Module 1.2 · Deepens page 2

What tokenize() really returns

The same sentence page 2's diagram walked through — now printed for real, from the model's own tokenizer. Step through it and watch one word split into two pieces, live.

SENTENCE: "Tokenization splits unfamiliar words into pieces."

Token 1 of 8

Token id	Decodes to

The script

Eight lines to see the real ids

bc-02-tokenize.py

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

sentence = "Tokenization splits unfamiliar words into pieces."
ids = tokenizer(sentence)["input_ids"]

print("token ids:", ids)
for i in ids:
    print(f"  {i:6d} -> {repr(tokenizer.decode([i]))}")

tokenizer(sentence) never sees a "word" — only the vocabulary of ~49,152 pieces it was trained on, same idea as page bc-00's vocab.json. Download bc-02-tokenize.py ⬇

What you should see

terminal

python bc-02-tokenize.py
sentence: Tokenization splits unfamiliar words into pieces.
token ids: [20855, 1237, 30855, 14160, 1924, 618, 4443, 30]
count: 8
   20855 -> 'Token'
    1237 -> 'ization'
   30855 -> ' splits'
   14160 -> ' unfamiliar'
    1924 -> ' words'
     618 -> ' into'
    4443 -> ' pieces'
      30 -> '.'

"Tokenization" itself doesn't have its own id — it's Token (20855) + ization (1237), two pieces. Every other word here happened to be common enough to get one id each.

Try this

Change sentence to your own name, or a made-up word like "flibbertigibbet". Predict first: one token, or several? Uncommon words and names are exactly where splitting shows up.

So what

The model never reads "words" — only a fixed vocabulary of pieces, and unfamiliar ones get cut, not rejected.