MODULE 02 — HOW AI THINKS · 06 AI Concepts

Why it acts like an assistant

A next-word machine has no reason to answer you. Left as it came out of training, it would continue your question rather than reply to it.

Zoom into the training bar from the last page

Trillions of words. Months of compute. No idea what a question is.
A sliver: curated good answers, plus humans rating which of two replies is better.

Same model, two moments — before vs. after that sliver above. Pick a prompt:

Base model — no post-training yet, just continuing a document

These transcripts are written to illustrate how a pretrained model behaves before post-training — they are not captures from a live model. The behaviour they show is well documented.

The problem it explains

It answers with more questions because that is what pages look like

A raw model has one skill: continue this text plausibly. On the open web, a line ending in a question mark is most often followed by more questions — a quiz, a worksheet, an FAQ index.

WHAT THE TRAINING TEXT LOOKED LIKE GEOGRAPHY QUIZ — SHEET 3 1. What is the capital of France? 2. What is the capital of Spain? 3. What is the capital of Peru? 4. Name the longest river in Asia. Answers overleaf. so when you type line 1… “2. What is the capital of Spain?” Not a failure. A correct continuation. It knew Paris the whole time. Nobody had yet taught it that a question is an invitation to reply.
The knowledge was already there after pretraining. What was missing was the convention of answering.

How it works

Three stages, wildly different sizes

trillions of words continue no idea what a question is
Pretraining · enormousWhere essentially all the knowledge comes from.
request good reply request good reply a few tens of thousands — no new facts
Instruction tuning · tinyIt learns the shape of a conversation, not new facts.
reply A reply B a person picks nudged toward the winner
Rating · tinyThe model drifts toward whatever people keep choosing.

The side effect

Why it agrees with you too easily

ONE RATING ROUND, REPEATED THOUSANDS OF TIMES “Actually, you’re wrong.” “Great point — you’re right!” a person picks the nicer one nudge the model toward the winner round after round agreement becomes the default reply it folds when pushed — even when it was right

It is not weighing your argument. It is repeating a pattern that got rewarded.

So what

Post-training changed its manners, not its knowledge — worth remembering when someone offers to teach it new facts by retraining it (page 10 covers that trade-off).