Module 8.2

Longer isn't free

Rotary position embeddings let a model reason about "how far apart" two tokens are, no matter how far into the sequence they sit. Rotate the same query/key pair at three different absolute positions and see what stays constant.

QUERY AT POSITION m, KEY AT POSITION n — SAME q=[1.0, 0.0], k=[0.6, 0.8] EVERY TIME

m = , rotation angle = rad n = , rotation angle = rad relative offset (n − m) =

rotated q → rotated k →

dot(rotated q, rotated k) — depends only on the offset above

The math

Three separate ideas, one page

RoPE

R(v,θ)=[v₀cosθ − v₁sinθ, v₀sinθ + v₁cosθ]

θ = position × base frequency — a rotation, not an addition

dot(R(q,m),R(k,n)) depends only on n−m

Sliding-window attention

token i attends to [i−W+1, i]

a fixed-width causal window, not the whole prefix

compute drops from O(n²) to O(n·W)

KV-cache memory

mem = 2 · layers · heads · d_head · n · bytes

grows linearly in n — n = context length, in tokens

at n=1M: hundreds of GB

RoPE is why the model still makes sense of position past its training length; the window and the cache are why doing that at 1M tokens still costs real memory and real compute.

Sliding-window attention, computed

What token 6 can actually see, with a window of 3

Each row would repeat this pattern shifted by one — every token sees at most 3 tokens back, however long the sequence gets. Full causal attention would light up the entire row up to the diagonal.

What "1M tokens of context" actually costs

Context length	KV-cache memory	Full attention FLOPs (relative)	Windowed (W=4096) FLOPs (relative)

The code

The rotation and the mask

long_context.py

def rope(v, pos, theta_base=0.1):
    theta = pos * theta_base
    return [v[0]*np.cos(theta) - v[1]*np.sin(theta),
            v[0]*np.sin(theta) + v[1]*np.cos(theta)]

# dot(rope(q, m), rope(k, n)) depends only on (n - m) — try any m, n with the same gap

def sliding_window_mask(seq_len, window):
    mask = np.full((seq_len, seq_len), -np.inf)
    for i in range(seq_len):
        lo = max(0, i - window + 1)
        mask[i, lo:i+1] = 0   # 0 = visible, -inf = masked out before softmax
    return mask

Most long-context production models mix strategies: RoPE (or a scaled variant) for position, a window or sparse pattern for most layers, and a handful of full-attention layers to still allow occasional long-range lookups.

The Nimbus thread

"Just raise the context window" isn't a free knob

If Nimbus-bot wanted to read a customer's entire 500-ticket history at once, the KV-cache table above is the real bill for that — not a vague "it'll be slower," but a specific number of gigabytes of GPU memory per concurrent conversation. This is exactly why Module 5's RAG pipeline (retrieve only the relevant tickets) usually beats "just paste everything into the prompt" once a customer's history gets long, even on a model that technically supports a huge window.

So what

A bigger context window is a real memory and compute bill, paid on every single request — not a setting you flip for free.