Module 2.2

One block, start to finish

Attention is one sublayer, not the whole block. Follow "export"'s representation through the full pipeline — positional encoding, two attention heads concatenated, a residual add, layernorm, a feed-forward sublayer, and another residual add.

TOKEN: "export" (position 1 of 4) — STEP THROUGH THE BLOCK

1. Embedding

The math

Three formulas the block leans on

Positional encoding

PE(pos,2i)=sin(pos/10000^(2i/d))

PE(pos,2i+1)=cos(pos/10000^(2i/d))

PE(1) = [0.84, 0.54, 0.01, 1.00]

Add & norm

LN(x + Sublayer(x))

residual keeps the original signal from vanishing

mean ≈ 0, std ≈ 1 after LN

Feed-forward

FFN(x) = W₂ · ReLU(W₁x + b₁) + b₂

expands, squashes negatives, contracts back

d_model → d_ff → d_model

Every one of these is applied per token, independently and identically — the only place tokens talk to each other at all is inside the attention sublayer.

The code

The whole block, assembled

transformer_block.py

def layer_norm(x, eps=1e-5):
    mu, var = x.mean(), x.var()
    return (x - mu) / np.sqrt(var + eps)

def multi_head_attention(X, heads):
    outs = [attention_head(X, h) for h in heads]
    return np.concatenate(outs, axis=-1)   # back to d_model

def transformer_block(X, heads, W1, b1, W2, b2):
    X = X + positional_encoding(X.shape[0], X.shape[1])
    attn = multi_head_attention(X, heads)
    X = layer_norm(X + attn)                       # residual #1
    ffn = np.maximum(0, X @ W1 + b1) @ W2 + b2
    X = layer_norm(X + ffn)                        # residual #2
    return X   # same shape as the input — stackable

Same shape in, same shape out — that's what makes stacking N of these (Module 2.3) as simple as calling this function N times.

The Nimbus thread

One block deep, on the real query

This exact pipeline runs on every token of "Why did my Nimbus export fail?" simultaneously, not just "export" — the interactive above shows one token's journey so the numbers stay readable. A real model repeats this whole block dozens of times, each one refining every token's representation using what every other token now knows.

So what

A transformer block is attention, then a per-token feed-forward pass, each wrapped in a residual add and a normalize — repeat N times.