How to Talk To Your Data with AI

Today, I wanted to share some observations from my recent work around AI for data analytics.

I’ve been knee-deep in this topic since 2022, and what’s clear is that the tools we have today are dramatically better than what we had back then. Today’s AI can write much better code, recover from its own mistakes and work for hours through complex analysis. (Seems like an eternity, but it’s crazy it’s only been a few years!)

Interestingly, many of the problems that made “chat with your data” unreliable back then are still the same problems we’re dealing with today.

What has changed, however, is the approach you can take to solve them.

I really think it comes down to three things you need to have in place – and picking the right starting point.

Let’s jump in!

A lot of the problems with AI for data analytics used to be model-related.

The AI was simply too “dumb” to reliably understand that:

They also regularly got lost halfway through an even mildly complex data analysis.

A lot of that has improved.

Upload a complex dataset to ChatGPT today (or connect it to a data warehouse), and it will generally be much better at interpreting what it sees, writing and checking code, and performing tedious analytical work for you.

All of that is really cool.

But it only solves one part of a much bigger problem.

Because a model can:

We have to stop wondering whether AI can analyze and interpret our data.

It can.

The more important question is whether we can give it the right context at the right time so it can find the answer that we are interested in and that matches our interpretation – or be candid when that answer might not be provided in the given dataset.

So it all boils down to data and context.

(Which is… drumroll… also data.)

And this is where, in my experience, relatively small changes can make a surprisingly big difference.

I call this data that wants to be chatted with.

Let’s take a look.

I found that there are three traits that make a particularly big difference when you run a “chat with your data” use case.

Your data should be (1) easy to query, (2) unambiguous, and (3) governed.

The first requirement is pretty simple:

The AI needs to be able to get to the right data without reverse-engineering everything else first.

In practice, data that is not easy to query is typically a nicely formatted Excel spreadsheet with multiple header rows, empty cells for “readability”, or sub-totals in random cells (ideally with wrong formulas).

What you want instead is a dataset that comes as a tidy, well-structured table – meaning boring things like:

Good news: if you’re querying data from a data warehouse, it’s typically already stored in a machine-friendly shape.

Even turning your Excel spreadsheet into a flat, “tidy” CSV first – under your supervision – makes a big difference.

For example, in my trainings I regularly upload an Excel spreadsheet that contains British Pound as a currency. In 8/10 cases, even Claude and ChatGPT with high thinking effort are not able to catch that info because it’s stored in the formatting and often gets ignored during import. 4/10 times they hallucinate a currency based on the user’s locale.

If you convert it to a flat table first, you could encode the currency directly in the column header, such as lifetime_revenue_gbp – or provide that additional context separately in the prompt.

Which brings me to part 2.

Being able to query data is one thing.

Giving the AI enough context to interpret what it’s looking at is another.

Imagine your warehouse contains columns called:

Which one should the AI use when someone asks:

“How much revenue did we make last month?”

A data analyst working on these kinds of requests every week knows the answer immediately.

“Of course it’s revenue – that’s why we put it there as a separate column!”

AI doesn’t know any of that.

This is where things like clear column names, descriptions, and schema definitions finally start getting useful. (All those things that got deprioritized for years by data teams because they looked more like a documentation exercise.)

At some point, encoding all relevant context in column headers hits a limit, of course.

That’s where things like Semantic Layers, data catalogs, data dictionaries, or knowledge graphs come in. They all solve slightly different problems, but the important part is that your AI has some place to look up what the different fields actually mean. And that those definitions don’t contradict each other.

Typically, building that context layer is the hardest part because it requires the organization to externalize knowledge that was previously sitting in the heads of a few people.

Very often, some of those people have already left.

But the earlier you get to a shared definition of meaning and context, the better in the long run.

You could write this off as another piece of context, but “Governed” in this case means something else. “Governed” clearly defines the guardrails of what the AI is allowed to do with this data and what not.

For example:

The more you look into this the more you will realize that this is less about telling the AI what is allowed, and more about defining the limitations of it.

A marketing data set might contain a “revenue” field, but it’s probably a good rule to say this revenue field must not be used for financial reporting. Because the number Marketing calls ‘revenue’ might not be one your CFO agreed to report.

These boundaries become especially important once you move beyond simple descriptive questions.

Calculating “revenue last month” is one thing, but explaining why revenue dropped is another. The correct answer might simply be:

“I don’t know.”

This is particularly important because in practice, people transition from descriptive analytics to diagnostic analytics pretty fast.

The first question is what, the second usually why.

And it might be a good hand-off to bring a professional data analyst in.

The key is that these rules shouldn’t live only inside a prompt – or worse, inside somebody’s head.

You can’t teach this at scale using training!

Some rules can live in the system instructions or Skills of your data analysis agents. Others should probably be encoded more deterministically in metric definitions, SQL, semantic layers, or code. The more you can define, the less the AI has to interpret (or hallucinate) during runtime.

Cheap Codification to the rescue, because historically the effort required to externalize and maintain all of these rules was often simply too high.

Which brings me to our practical steps.

Here are two things to help you get started:

Don’t start another data documentation project where people chase after other people to get their definitions updated. You want AI involved in creating and maintaining the context layer from day 1 – and automate more of that process over time.

Practically, if a new metric definition gets added, the system should be able to figure out:

Aim for building a context layer that can increasingly maintain itself – under human supervision.

This sounds harder than it is. But you have to start small.

To get to that AI-powered context layer, don’t start with:

“Let’s make our entire data warehouse chat-ready.”

You’ll overwhelm your organization before anybody gets value from it.

Instead, start with one useful outcome.

Choose a recurring, high-value question or analytical workflow.

Work backwards from there:

Make it exist first, then make it better.

Develop your roadmap from there.

If you want to chat with your data, build data that wants to be chatted with.

See you next Saturday,
Tobias