Module 3.3

Four datasets, one loop

Pretraining, fine-tuning, and instruction-tuning are the exact same next-token loop from Module 3.2 — only the data changes. RLHF/DPO is the one stage that changes the objective itself. Step through how Nimbus-bot's base model becomes a support assistant.

FOUR STAGES, SAME STARTING MODEL

Training data looks like

Objective

Nimbus-bot, asked "Why did my Nimbus export fail?"

The math

Three stages share a loss; one doesn't

Stages 1–3

L = −log P(next token | context)

pretrain, fine-tune, instruction-tune

same formula as Module 3.2 — different data only

Preference signal

P(y_w ≻ y_l) = σ(r(y_w) − r(y_l))

Bradley-Terry model of "which reply is better"

y_w = preferred reply, y_l = rejected reply

DPO loss

L = −log σ(β·Δlog π(y_w) − β·Δlog π(y_l))

Δ is relative to a frozen reference copy of the model

no separate reward model needed

RLHF trains a separate reward model first, then optimizes the policy against it with reinforcement learning (PPO). DPO proved you can skip the reward model and optimize this loss directly on preference pairs — same goal, one fewer moving part.

The code

Same trainer, four datasets

training_stages.py

# Stages 1-3: identical training loop, only `dataset` changes
for stage_dataset in [web_text, nimbus_support_qa, diverse_instructions]:
    for context, next_token in stage_dataset:
        loss = cross_entropy(model(context), next_token)
        model.step(loss)                     # same update rule every time

# Stage 4: DPO — needs (prompt, chosen, rejected) triples instead
def dpo_loss(model, ref_model, prompt, chosen, rejected, beta=0.1):
    logp_w = log_prob(model, prompt, chosen) - log_prob(ref_model, prompt, chosen)
    logp_l = log_prob(model, prompt, rejected) - log_prob(ref_model, prompt, rejected)
    return -np.log(sigmoid(beta * (logp_w - logp_l)))

The reference model (ref_model) is a frozen copy from before DPO started — it keeps the policy from drifting arbitrarily far just to win preference comparisons.

The Nimbus thread

Which stage actually taught "quota"?

Module 3.2's single gradient step was almost certainly a fine-tuning or instruction-tuning update — that's where Nimbus-specific support content enters training at all. Pretraining might have seen "quota" and "export" near each other on some indexed web page, by luck. RLHF/DPO wouldn't teach the fact at all — it only shapes which of two already-plausible answers reads better.

So what

Three of these four stages are the same math on different data — only the last one asks a genuinely different question: not "what comes next," but "which reply is better."