Confidence Calibration

At the end of one of my recent talks, someone in the audience asked me if I had any advice on effective AI upskilling within an organisation.

I gave my usual answer – teach the basics with on-demand courses, offer tool-specific training, and organize exchange around real use cases.

I still believe all of that, but I realized it's not the whole picture anymore. Because this explanation assumes organizations have one AI training problem.

When in fact, they have three.

Here’s what they are (and what to do about them).

Two years ago, I wrote that every knowledge worker – not just technical teams – needs AI training.

Back then I described AI learning as a pyramid inspired by Bloom's Taxonomy: build the foundations first, then gradually move toward application.

That made sense in a world where knowledge was relatively stable and learning followed a predictable sequence.

Modern AI doesn't work that way.

Trying AI costs almost nothing these days. Feedback arrives in seconds. You don't need weeks of preparation before experimenting – you can discover what you don't know simply by using the tools.

When trying becomes cheaper than studying, the learning order changes.

People learn what a context window is because a conversation starts forgetting earlier messages. They learn about hallucinations after seeing one. They learn prompt design because their first attempt produces mediocre results. Not because they saw it in a training video on slide 14.

So theory becomes just-in-time instead of just-in-case.

AI learning isn't a pyramid you climb once.

It's a pendulum that constantly swings between understanding and application. You try something, hit a limitation, learn why it happened, then apply that knowledge to the next round.

The problem is that not everyone gets stuck at the same point in that cycle.

Across organizations, I keep seeing three groups of people.

These people already use AI every day.

They summarize documents, prepare meetings, analyze spreadsheets, write code, draft proposals, or automate parts of their work. Many of them – estimates range from 50% to 80% – have never received formal AI training at all.

They're not waiting for permission – they're already building habits.

The question is whether those habits are any good.

The problem here isn’t that AI makes mistakes (it always will), but that people stop noticing them.

A well-known BCG field experiment with 758 consultants found that AI significantly improved performance on tasks within the limits of what the AI was capable of doing. But when participants tackled problems just outside that frontier, those using AI were substantially more likely to produce wrong answers.

We've already seen this in the real world. KPMG pulled back an AI report after AI-hallucinated case studies were exposed, and Deloitte Australia agreed to a refund after a government report turned out to contain AI slop.

A Microsoft and Carnegie Mellon study found the mechanism behind it: the more people trusted AI, the less critically they evaluated its output.

Today, this matters more than ever. Modern AI tools got extremely more powerful – evolving from simple chatbots to agentic systems that "stay with a project for hours if needed," and turn a goal into finished work – as you can see for example by the latest ChatGPT Work announcement from OpenAI.

These employees don't need another AI introductory course.

They need better judgment.

Teach them where AI performs reliably, where it struggles, and how to verify outputs before acting on them. For example, don't ask an LLM to "find every risk in this contract." Ask, "Is there a change-of-control clause? Quote the relevant section," because specific claims are much easier to verify than completeness.

The goal here is essentially to avoid the Dunning-Kruger effect – not by reducing their confidence when working with AI, but making sure their confidence matches reality.

In short, the best way to get practitioners to the next level is to make sure they don’t fall three levels back.

The second group has the opposite problem.

They know a surprising amount about AI.

They attend workshops, read newsletters, follow every product launch, and can explain hallucinations, context windows, and the basics of prompt engineering. But when you ask what they actually used AI for this week, the answer is often: not much.

Their pendulum is stuck on stuck on the Understand side. Mostly because they’re aware of all the problems that this technology brings. A little too aware.

The problem is that more understanding does not automatically create more confidence. More theory can make the problem worse, because it gives them a more sophisticated reason to wait.

These employees don't need more "inputs".

They need opportunities to experiment safely.

Give them learning labs with real tasks from their daily work – not generic exercises.

Let them summarize customer interviews, draft proposals, analyze meeting notes, or prepare presentations. Then have them compare the AI output with reality. Provide guidance or coaching from Practitioners if the work context fits. Actually, these two groups can highly benefit from each other.

Confidence doesn't grow from hearing that AI works. It grows from seeing where it works – and where it doesn't.

When someone discovers that AI produces an excellent first draft but still needs human judgment on pricing, positioning, or risk, they've learned something much more valuable than another list of prompting tips.

Their pendulum needs to swing toward application.

The third group simply hasn't engaged.

I don’t have a quotable figure, but in practice the number of people here appears to be still quite large.

They don't sign up for AI training. They’re not interested in AI developments. They may have tried ChatGPT once or twice but never integrated it into their work.

Their pendulum simply does not move.

And that is an entirely different training problem. Because if someone’s not interested in something it’s very hard to “convince” them using classical training paths or by telling them that AI is IMPORTANT! (They’ve already heard that a thousand times.)

What this group needs is a practical impulse.

What usually works is seeing someone they trust solve a real problem they can relate to.

An internal use case demonstrated by a colleague is often more persuasive than the most polished vendor demo. The use case itself doesn't even have to be spectacular. It just has to feel relevant.

Once it clicks, people can start experimenting or watch training materials. Either path is fine.

The important part is that their pendulum got moving.

Before launching another organization-wide AI upskilling initiative, ask 3 questions.

Who is already using AI every day?
Give them better judgment, verification habits, and a deeper understanding of AI's strengths and limitations.

Who keeps learning without applying?
Offer structured opportunities to safely experiment on real work in sandboxed environments.

Who hasn't meaningfully started?
Show relatable examples from colleagues doing work they recognize.

Don’t assume these groups need the same intervention. They don't.

AI upskilling isn't about moving everyone through Bloom's Taxonomy anymore.

The goal is to create good judgment.

You want a workforce that is motivated to use AI where it makes sense, knows when to trust its output, and recognizes when verification is needed.

In other words: confident enough to use AI, but skeptical enough to check.

That's confidence calibration.

And the next time someone asks me what modern AI upskilling should look like, that's the answer I'll give.

See you next Saturday,
Tobias