How AI reads language

It never sees your sentence. It sees a list of chunks — and the chunks are not the words you would have picked.

Type anything

Characters0 Tokens0

TRY

Real output from o200k, the tokenizer behind today’s GPT models — these chunks are what it calls tokens. Grey dots mark a leading space — it belongs to the token.

The problem it explains

Why it can’t count the r’s in strawberry

The same word is chopped differently depending on the space in front of it. Ask about letters and the model is guessing about pieces it never sees apart.

Same word, two different chops. This is also why models fumble rhymes, spelling and word games.

How it works

Chop, number, place

ChopCut from a fixed list of about 200,000 pieces. Common words survive whole.

NumberEach chunk becomes its ID. The model never handles letters again.

PlaceEvery ID has coordinates. Things used alike land near each other.

See it in action

The meaning map

Nobody labelled these groups — they fall out of which words keep showing up in the same company. Tap a word to see its nearest neighbours.

Positions here are hand-drawn to show the idea. A real map has hundreds of dimensions squashed down to two — but the grouping is real, and it is learned, not typed.

The payoff

Direction carries meaning

Move the same distance in the same direction and you make the same change in meaning.

Because meaning became geometry, “find me something similar” turns into “find the nearest points”.

That single move is what makes search, recommendations and — later in this series — a model looking things up before it answers all possible.

So what

Before any thinking happens, your sentence has already become chunks, then numbers, then coordinates.