Module 7.2

A computable number, not an accusation

"The benchmark leaked into training" isn't a vibe — it's a literal string-overlap check. Compare the same benchmark question against two training snippets and watch the actual mechanism run.

BENCHMARK QUESTION: "what year was the eiffel tower completed and how tall is it in meters"

0%

The math

Overlap between two sets of word sequences

n-gram

gᵢ = (wᵢ, wᵢ₊₁, …, wᵢ₊ₙ₋₁)

every consecutive run of n words in the text

n = 5 here — real systems often use 13–50

Contamination score

score = |G_q ∩ G_c| / |G_q|

G_q = question's n-grams, G_c = corpus's n-grams

the fraction found verbatim in training text

Why n matters

n too small → false positives

short phrases repeat naturally in any large corpus

n too large → misses paraphrased leaks

Real contamination reports (GPT-3, GPT-4, Llama) use exactly this kind of substring/n-gram matching at scale — across the entire training corpus, for every benchmark question, before the model is even trained.

The code

The actual contamination check

contamination_check.py

def ngrams(text, n=5):
    words = text.lower().split()
    return {tuple(words[i:i+n]) for i in range(len(words) - n + 1)}

def contamination_score(benchmark_q, training_text, n=5):
    q_grams = ngrams(benchmark_q, n)
    corpus_grams = ngrams(training_text, n)
    overlap = q_grams & corpus_grams
    return len(overlap) / len(q_grams)

# run this for every benchmark question against the whole training corpus
# before training even starts — that's the actual mitigation

This runs once, at data-preparation time, over every benchmark you plan to evaluate on — the fix is deciding what to do with a high-overlap match (usually: drop that training document, or drop that benchmark question).

The Nimbus thread

If Nimbus built its own eval set from support forums

Imagine Nimbus builds an internal benchmark by pulling real questions from their support forum — and later fine-tunes on scraped forum data that includes those same threads, answers included. Nimbus-bot's eval score would look excellent, for a reason that has nothing to do with how well it handles a genuinely new ticket. This exact n-gram check, run before fine-tuning, is what would catch it.

So what

Contamination isn't "the model cheated" in some vague sense — it's a measurable fraction of exact word sequences, shared between test and training.