Module 7.3

A number against a threshold

A guardrail isn't a list of banned phrases somebody typed by hand — it's a small classifier scoring P(unsafe) on every candidate reply before it reaches the user. Watch it score three real Nimbus-bot drafts.

CANDIDATE NIMBUS-BOT REPLY, BEFORE IT'S SENT

P(unsafe) = 0.00

ALLOW

The math

The classifier is one dot product and a sigmoid

Score

z = w · x + b

x = trigger-feature vector, w = learned weights

z = w·x + b

Probability

P(unsafe) = 1 / (1 + e⁻ᶻ)

the same sigmoid as any logistic classifier

P(unsafe) = 0.00

Decision

block if P(unsafe) > τ

τ = 0.5 here — tuned in practice for precision/recall

τ = 0.5

This runs on the output side (before a reply ships) and often on the input side too (before a user prompt even reaches the model) — the same classifier shape, different training data.

The code

Scoring a reply before it ships

guardrail.py

W = [2.5, 3.0, 0.8]   # override, sensitive-data, competitor-mention
B = -1.0
THRESHOLD = 0.5

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def guardrail_score(features):
    z = sum(w*x for w, x in zip(W, features)) + B
    return sigmoid(z)

p_unsafe = guardrail_score(features)
if p_unsafe > THRESHOLD:
    block_and_log(reply, p_unsafe)
else:
    send_to_user(reply)

The weights and features are trained the same way as any classifier — on a labeled dataset of past safe/unsafe examples, not hand-tuned by someone guessing.

Beyond the classifier

Two more pieces of the safety stack

Constitutional AI training loop

Instead of only filtering at inference time, the model critiques its own draft against a written set of principles ("the constitution"), then revises it — and that critique-and-revise pair becomes training data. Over many rounds, the model internalizes the principles instead of needing a classifier to catch every violation downstream.

Red-teaming methodology

A dedicated process of writing adversarial prompts specifically designed to slip past the classifier and the model's own training — paraphrasing, role-play framing, encoding tricks — then feeding every successful bypass back in as a new labeled training example, closing the loop that keeps τ and the weights calibrated.

The Nimbus thread

Hardening Nimbus-bot before it ships a reply

Every draft reply above passes through this exact scoring step before the user sees it. The "fluent but wrong" answer from Module 7.1 would sail through this guardrail untouched — a wrong root cause isn't the same signal as an unsafe one. That's the whole point of running perplexity, embedding similarity, LLM-judge, contamination checks, and this guardrail as separate, purpose-built checks rather than expecting any single one to catch everything.

So what

A guardrail is a second small model scoring a number against a threshold — not a list of banned words somebody typed once.