Module 7.1

Fluent isn't the same as right

Two candidate replies to the same Nimbus ticket — one correct, one confidently wrong but perfectly fluent. Watch which metrics actually catch the difference, and which one gets fooled.

CANDIDATE REPLY TO: "Why did my Nimbus export fail?"

The math

Three ways to measure "good"

Perplexity

PP = exp(−(1/n)Σ log p(tokenᵢ))

Module 3.2's cross-entropy, exponentiated

measures fluency — how "expected" the text is

Embedding similarity

score = cos_sim(embed(reply), embed(reference))

Module 1.2's formula, applied to answers not documents

measures semantic closeness to a known-good answer

LLM-as-judge

score = judge_model(rubric, reply)

a separate model call, scored against explicit criteria

can check things embeddings can't: "does it suggest a fix?"

Perplexity alone would call both replies below "good" — it has no way to know one of them is factually wrong.

The code

Scoring one reply, three ways

evaluate_reply.py

def perplexity(token_probs):
    log_probs = [np.log(p) for p in token_probs]
    return np.exp(-np.mean(log_probs))

def semantic_score(reply, reference):
    return cosine_sim(embed(reply), embed(reference))     # Module 1.2

RUBRIC = """Rate 1-5 on:
- Accuracy: does it identify the real cause?
- Helpfulness: does it suggest a concrete fix?
- Tone: professional and clear?"""

def llm_judge(reply, question):
    return judge_model.score(RUBRIC, question, reply)   # a real model call

No single one of these is "the" eval metric — production systems typically log all three and treat disagreement between them as a signal worth a human look.

The Nimbus thread

Why relying on perplexity alone would be dangerous here

Both candidate replies below are grammatically fluent — a "server outage" excuse reads just as smoothly as the real quota explanation, so their perplexity scores land close together. If Nimbus only tracked perplexity as its quality signal, a systematically wrong root-cause explanation could ship and look fine on the dashboard. Embedding similarity and LLM-as-judge both catch it because they check what was said, not just how smoothly it reads.

So what

Perplexity measures how confident a model sounds — it has no way to check whether confidence is deserved.