Module 2.3

Logits, then one word

Stack the same block N times, then do one more thing nobody's shown you yet: turn the last block's output into an actual next word. That last step is the whole point of the other 999 numbers.

HOW MANY BLOCKS DEEP?

Every block is the identical function from Module 2.2 — same shape in, same shape out — just called again on the previous block's output.

THE OUTPUT HEAD — HIDDEN STATE → LOGITS → NEXT WORD

FINAL HIDDEN STATE

The math

One linear layer turns a vector into a vocabulary

Output projection

logits = h · W_head

h is 4-dim here; a real model's h might be 4096-dim

logits ∈ ℝ^(vocab size)

Next-token distribution

P(token) = softmax(logits)

argmax → "quota"

P(quota) = 0.61

Vocabulary size, for real

W_head: d_model × |V|

real |V| is 30,000–150,000+ tokens

6 candidates here, illustratively

This one matrix, W_head, is why vocabulary size drives so much of a model's parameter count — it's d_model × vocabulary_size numbers, applied at every single position.

The code

The last mile

block_to_model.py

def gpt_forward(tokens, embed, blocks, W_head, vocab):
    X = embed[tokens]                        # (seq_len, d_model)
    for block in blocks:                     # same function, called N times
        X = transformer_block(X, *block)

    h = X[-1]                            # hidden state of the LAST position only
    logits = h @ W_head                       # (vocab_size,)
    probs  = softmax(logits)
    next_token = vocab[np.argmax(probs)]      # greedy decoding
    return next_token, probs

This is the entire forward pass of a GPT-style model. Everything else — training, sampling strategies, the chat template — decides what data flows through this function, never changes what the function does.

Go deeper (optional)

Watch the whole thing get built, live

This is a full course-length build, not a quick clip (1h 56m) — Andrej Karpathy writes this exact forward pass from scratch, character by character, and trains it on real text. Nothing here required to move on; it's the deepest possible follow-up for this module.

Andrej Karpathy — "Let's build GPT: from scratch, in code, spelled out." Supplementary and optional — Modules 3–8 build out everything this video only has time to sketch (training, fine-tuning, retrieval, agents, evaluation).

The Nimbus thread

Where "quota" actually comes from

After enough blocks, the model's hidden state for the last position of "Why did my Nimbus export fail?" ends up pointing toward the region of vector space W_head maps to the word "quota" — not because it looked anything up, but because attention (Module 2.1) pulled "export" and "fail" together, and the feed-forward sublayers (Module 2.2) refined that into a direction the output head recognizes. Module 5 covers the alternative: actually retrieving the real Nimbus doc instead of hoping the weights remember it.

So what

Every one of a model's "billions of parameters" ends at one linear layer and a softmax — everything before that is just building a better hidden state to feed it.