Analyzing Customer Feedback with AI

Recently, I spent a lot of time working with AI to analyze customer comments for different clients. But the goal was always the same: to find insights that drive business.

However, most comments are barely insightful. You get things like "good", "more sustainability," or just "F" Yes, that was the whole comment. Others read like high school essays. (Honestly, customer comments often say more about the person than the product, but that’s a story for another day.)

So how do you make sense of it all?

In the age of ChatGPT and nearly unlimited context windows, many teams just dump everything into an LLM and ask, "What do my customers want?" Of course, the model gives you a clever-sounding answer — but the results are often both wrong and random.

The problem isn’t the AI. It’s the approach. Done right, AI can actually help you extract what customers really want.

Let’s dive in.

📣 Workshop Invitation

The Customer Feedback Problem

In theory, customer feedback should be gold. But in practice, it’s often a vague mix of sentiment, complaints, and scattered feature requests.

Customers typically express symptoms, not root causes. When someone writes "The website is slow," they’re not asking for faster servers — they just want to complete a task faster.

So when you dump all your feedback into an LLM, you don’t get real insight. You get a bland summary:

• "Users want better usability"

• "They’re confused by onboarding"

Often, you didn’t need AI to tell you that.

But done right, AI can help. Not by summarizing — but by extracting the needs behind the words.

5-Step Framework for AI-Powered Customer Insight

Instead of the "dump and hope" approach, here's a systematic framework that's worked for me and many organizations I've consulted with:

1. Start with a problem hypothesis.
2. Build a taxonomy that maps to action.
3. Count what matters.
4. Analyze temporal trends.
5. Look for what’s missing.

Let's break these down.

Step 1: Start with a problem hypothesis

Don’t ask: "What are they saying?"
Ask: "Are they struggling with X?"

When you approach customer feedback like a scientist testing a theory, rather than an analyst fishing for random insights, you stay grounded in the business context.

For example, if churn is high after the first week, your hypothesis might be:
"Users aren’t finding value fast enough."

Now you can use AI to test that. You're no longer asking the model to "summarize feedback" — you're asking it to find signal related to that specific hypothesis. You’re guiding the model’s attention. And because LLMs are essentially pattern matchers, guidance is everything.

By anchoring analysis around a hypothesis, you're turning qualitative data into something falsifiable.

That’s where rigor begins.

Step 2: Build a taxonomy that maps to action

Next, you need structure. Specifically: a taxonomy that reflects business levers — not vague themes.

Too often I see taxonomies that are intellectually clean but practically useless. They categorize feedback into buckets like "usability," "performance," or "pricing". Nice in theory. But where do you go from there?

Instead, design categories that align with things you can actually change.

For example:

• "Signup friction (UI issues)"

• "Signup friction (copy confusion)"

• "Lack of onboarding guidance"

• "Missing core feature expectations"

• "Breakdowns in customer support loop"

Each label should suggest who needs to act and what they need to look at. If a tag doesn’t help someone make a decision or prioritize work, it doesn’t belong.

Yes, building this taxonomy takes time. But more importantly, it needs to be treated as a versioned asset – not buried in prompts or scattered across spreadsheets.

It will evolve as your business and users do. So version it like code. (And if you’re already versioning prompts, you can track category changes right alongside them.) Because when this mapping breaks or drifts, your insights degrade — even if your model improves.

Step 3: Count what matters

Still the most underrated skill in analytics: frequency by category.

Once you've labeled your feedback using a solid taxonomy, count. Not just overall volume.

You really want to count a lot:

• Count per category

• Count per per week

• Count per segment.

This isn’t sexy, but it’s where real insight starts to emerge.

Why? Because humans are bad at estimating what’s common vs. rare. You’ll read ten complaints about "speed" and think it’s a pattern — until you realize 90% of comments are actually about missing features. Frequency grounds you.

It also protects you from confirmation bias. It’s tempting to search for feedback that supports your narrative – and LLMs are great at mirroring your assumptions. But real counts push back. They tell you what’s actually showing up, not just what you’re hoping to see.

And don’t just look at absolute numbers — look at relative frequencies as well. A category showing up 30 times might seem minor. But if those 30 comments make up 90% of feedback from new users on mobile? That’s a signal. Frequency only becomes insight when it’s contextualized.

Here’s an example from a client project that shows how this plays out:

So the issue wasn’t the content. It was delivery failure. And counting — not just summarizing — is what made that visible.

That’s how frequency becomes leverage.

Talking about context…

Step 4: Analyze temporal trends

Feedback isn’t static — and treating it like a flat dataset is a common trap. What customers complain about today isn’t necessarily what they cared about three months ago. That shift is the insight.

AI can help you surface these changes, but only if you track feedback over time.

Think:

• What categories are rising?

• Which ones are fading?

• Are there seasonal patterns?

• Did a recent release trigger a spike in something you thought was fixed?

When people say something often matters more than what they say.

This is where a lot of teams get surprised — not because they didn’t have the data, but because they weren’t watching it change.

Set up your system to compare today’s feedback with last month’s, last quarter’s — whatever cadence matches your product velocity. Because if you're not analyzing change, you're just rereading history.

Step 5: Look for what’s missing

Not all signals are loud. Some are silent — and just as meaningful.

One of the most overlooked forms of insight in customer feedback is absence. No positive feedback in 100 reviews? That’s not "neutral". That’s a red flag – or at least a reason to dig deeper.

Same goes for missing topics. If a major feature launches and no one mentions it, that doesn’t mean it landed quietly — it might mean it didn’t land at all.

But — and this is key — you have to contextualize the absence. Sometimes the problem isn’t the experience itself, but the system around how and when you’re collecting feedback.

For example:

Absence is a signal. But without context, it’s easy to misread.

So don’t just ask: What are customers saying?
Also ask: What aren’t they saying — and why might that be?

Because in feedback, what’s missing is sometimes the most actionable thing of all.

Practical Tips

Start small — maybe just 50–100 comments from your most valuable customer segment. Apply the framework above, refine it for your context, and then scale up.

Remember:

1. Don’t just dump feedback into AI and expect magic — structure your approach.

2. Define categories that map to real business decisions.

3. Use AI to systematically tag feedback and surface latent needs.

4. Connect insights directly to product or service improvements.

5. Continuously loop back to validate and adjust.

This puts you on the right track.

The real work isn’t "put this into ChatGPT and see what it says". It’s designing the system: defining the right hypotheses, building taxonomies that guide decisions, tracking frequency, analyzing change, and spotting what’s missing.

Do that, and AI stops being a clever summarizer – and becomes a force multiplier for real insight.

Now go make your feedback pipeline smarter.

Want help getting started? I’m running a live workshop next week where we’ll turn your existing customer feedback into a validated offer idea — using AI, step by step.

See you next Friday,
Tobias