MODULE 02 — HOW AI THINKS · 08 AI Concepts

Why AI hallucinates

Not because it gets confused or unsure. Because nowhere in the machine is there a step that compares what it is about to say against the world.

your question the next-word machine picks the most plausible continuation an answer check it against a source nothing is wired to this box
The empty box is the whole explanation. Fluency and truth are produced by two different mechanisms, and only one of them is present.
Asked

What is Northwind Traders’ return policy?

Answered

Choosing the number in “returns within ___ days”

The myth this kills

It does not sound unsure, because it is not unsure

A FACT IT REALLY HASCapital of France?Paris97%Lyon1%Nice1% A FACT IT NEVER HADNorthwind’s return window?30 days61%14 days12%60 days9% peaked · correct peaked · invented Same shape. Confidence carries no information about truth.

“30 days” is the most common return window on the internet, reported with the confidence that frequency earns. It answered the question it can answer — what does a return policy usually say — and you asked a different one.

How it happens

Three ordinary steps, one missing one

plausible true not on the target
Plausible is the targetA convincing invention scores exactly as well as a fact.
no data typical no “I don’t know” token
Gaps get filled, not flaggedThe filler arrives in the same voice as everything else.
machine you check a source not wired
Nothing checks afterwardsNo second pass, and no flag saying this part was reconstructed.

Where to be careful

The shapes that invite it

Anything private and namedyour contract, a small vendor [1]Citations and case namespatterned text reproduces well 4.7Precise figures and datesas easy to produce as a real one Anything after the cutoffthe locked room, narrated well Obscure people and placesthin data, same delivery ?“Are you sure?”from habit, not by rechecking

So what

Do not ask it to be more careful. Put the source on the desk — which is exactly what the next page is about.