Module 4.2

The error you can actually compute

Same frozen weight matrix from Module 4.1. This time it gets quantized to int8 or int4 — and instead of saying "a little accuracy is lost," we compute exactly how much, per weight.

QUANTIZE THIS 4×4 WEIGHT MATRIX TO...

Max error: — Mean abs error: — Levels available: —

32-bit — full size

The math

Rounding, with a scale factor

Quantize

q = round(w / scale)

scale = max|W| / levels

scale = 0.0071

Dequantize + error

ŵ = q·scale, error = |w − ŵ|

error is a real, computable number per weight

worst case ≈ half a scale step

Memory

size = n_weights × bits / 8

for this 16-weight matrix

64 bytes (32-bit)

Every quantization scheme, no matter how exotic (NF4, GPTQ, AWQ...), is still fundamentally this: pick a scale, round to the nearest representable step, and accept the resulting error.

The code

Quantize, dequantize, measure

quantize.py

def quantize(W, bits):
    levels = 2**(bits - 1) - 1          # signed, symmetric range
    scale = np.abs(W).max() / levels
    q = np.round(W / scale).astype(np.int8)
    return q, scale

def dequantize(q, scale):
    return q.astype(np.float32) * scale

q, scale = quantize(W, bits=4)
W_hat = dequantize(q, scale)
error = np.abs(W - W_hat)
print(error.max(), error.mean())      # a real number, not a guess

This is the entire technique. QLoRA runs exactly this on every frozen weight in the model, then dequantizes on the fly during the forward pass so the math still happens in higher precision.

The Nimbus thread

Why QLoRA quantizes the frozen half, never the adapter

Module 4.1's W is exactly what QLoRA targets: quantize it to 4-bit (real QLoRA uses a scheme called NF4, tuned for how pretrained weights are actually distributed) and its memory footprint drops roughly 8× versus 32-bit. The LoRA adapter — B and A — stays in full precision the entire time, because it's already tiny (Module 4.1: a few thousand numbers, not millions) and it's the part actually being trained; quantizing something you're actively computing gradients through adds error you don't need to accept.

So what

Quantization error isn't a hand-wave — it's max|W − ŵ|, a number you can print, for every weight, at every bit-width.