Module 1.3 · Deepens page 31

From a wall of numbers to one word

One forward pass produces one raw score per vocabulary entry — 49,152 numbers for a single next token. Softmax turns them into probabilities; temperature decides how boldly you sample from them. All three, run for real on the same prompt.

PROMPT: "My favorite season is" — TOP-5 REAL PROBABILITIES FOR THE NEXT TOKEN

The script

Logits, softmax, and three temperatures

bc-03-logits-to-word.py

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
inputs = tokenizer("My favorite season is", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits[0, -1]     # raw scores, one per vocab entry

probs = torch.softmax(logits, dim=-1)          # turns scores into a real probability distribution

for temp in [0.2, 1.0, 1.5]:
    out = model.generate(**inputs, max_new_tokens=15, do_sample=True, temperature=temp)
    print(temp, tokenizer.decode(out[0]))

Temperature reshapes the same distribution before sampling — low temperature sharpens it toward the top choice, high temperature flattens it toward "anything goes." Download bc-03-logits-to-word.py ⬇

What you should see

terminal

python bc-03-logits-to-word.py
logits shape: (49152,)
top-5 raw logits:
   18.625   ' spring'
   18.250   ' the'
   17.625   ' summer'
   17.250   ' winter'
   17.250   ' always'
top-5 probabilities (after softmax):
  0.1777   ' spring'
  0.1221   ' the'
  0.0654   ' summer'
  0.0449   ' winter'
  0.0449   ' always'
temperature=0.2: ' spring?<|im_end|>'
temperature=1.0: ' spring?<|im_end|>'
temperature=1.5: ' warmer instead ofhogs vinyl (say Dec ative copypress posting vs Moscow'

At 0.2 and 1.0 the sample happened to land on the same top token both times — at 1.5 the distribution is flat enough that a much less likely token wins, and the result stops reading as English at all. That's not a bug; it's the softmax tail getting sampled.

Try this

Add torch.manual_seed(0) right before the temperature loop in bc-03-logits-to-word.py ⬇, then run it twice. Predict first: same output both times, or different? Sampling is random by default — a seed is the only thing that makes it repeatable.

So what

Every generated word starts as 49,152 competing scores — temperature only changes how much you trust the top of that list.