Module 4.1

Freeze the big one, train two small ones

Full fine-tuning updates every entry of a weight matrix W. LoRA freezes W completely and learns a low-rank update ΔW = B·A instead — two skinny matrices standing in for one square one. Change the rank and watch what that costs.

RANK r — HOW SKINNY ARE B AND A?

Full fine-tune: 16 trainable params LoRA (this r): 8 trainable params

The math

The saving only shows up at real scale

Low-rank update

W' = W + (α/r)·B·A

W frozen; B is d×r; A is r×d; α is a fixed scale

only B and A ever get a gradient

Trainable parameters

params = 2·d·r vs d²

at toy scale (d=4) these are comparable

at d=4096, r=8: 65,536 vs 16,777,216

Initialization

B ← 0, A ← small random

so ΔW = 0 on the very first step

training starts from the untouched pretrained model

2·d·r grows linearly with d; d² grows quadratically. That gap is the entire reason LoRA works — and why it barely matters at d=4, but matters enormously at d=4096.

The code

A LoRA layer, end to end

lora_layer.py

class LoRALayer:
    def __init__(self, d, r, alpha=16):
        self.W = load_pretrained_weight(d, d)   # frozen, never updated
        self.B = np.zeros((d, r))                     # starts at zero
        self.A = np.random.randn(r, d) * 0.01
        self.scale = alpha / r

    def forward(self, x):
        frozen_path = x @ self.W                       # same as the base model always did
        adapter_path = (x @ self.B) @ self.A * self.scale
        return frozen_path + adapter_path

    def trainable_parameters(self):
        return [self.B, self.A]              # self.W is never in this list

The optimizer never even sees W — it isn't frozen by a flag that could be forgotten, it's structurally absent from the list of things being trained.

The Nimbus thread

Adapting Nimbus-bot's tone without touching the base model

Want Nimbus-bot to sound more like your support team's house style, without a full fine-tune? Freeze every weight from Module 3.3's instruction-tuned checkpoint, attach a LoRA adapter (rank 8 or 16 is typical) to the attention and feed-forward matrices, and train only B and A on a few hundred example replies in the right tone. The adapter alone might be a few megabytes; the frozen base model stays exactly as it was, shareable across as many different adapters — one per team, per tone, per client — as you want.

So what

LoRA doesn't change what gets computed — W+BA is still just a matrix — it changes how many numbers you have to learn and store to get there.