Evaluations and A/B tests for prompts

“It feels better” is how automations quietly get worse. An evaluation turns a prompt change into a number.

What you will build

A golden set, a scoring function with a ship/no-ship gate, and a comparison between two prompt versions.

Figure 1. Nothing ships unless it meets minimums on a set you labelled yourself.

Step 1: Build the golden set

Take 30 to 100 real, redacted examples and label them by hand. Labels are the answer you would be happy with.

id,segment,intent,next_action
1,small_business,high,send_demo_invite
2,enterprise,high,send_demo_invite
3,freelancer,low,no_action
4,agency,high,send_demo_invite

Rules for a useful set:

• Include the hard ones. Ambiguous messages, empty notes, other languages.
• Cover every label. If no_action never appears, you cannot see the model overusing “good lead”.
• Keep it separate. Do not paste golden examples into the prompt as few-shot examples.
• Label with a second person for a sample, to check you agree with yourself.

Step 2: Score it

# evals.py
"""Score a classifier against a hand-labelled golden set."""
from __future__ import annotations
import csv
from collections import defaultdict


def load(path: str) -> dict[str, dict]:
    with open(path, newline="", encoding="utf-8") as f:
        return {r["id"]: r for r in csv.DictReader(f)}


def score(golden: dict[str, dict], predicted: dict[str, dict], fields: list[str]) -> dict:
    """Per-field accuracy over ids present in both, plus coverage and misses."""
    hits = defaultdict(int)
    misses = []
    common = [i for i in golden if i in predicted]
    for i in common:
        for f in fields:
            if golden[i][f] == predicted[i][f]:
                hits[f] += 1
            else:
                misses.append({"id": i, "field": f, "expected": golden[i][f], "got": predicted[i][f]})
    n = len(common)
    return {
        "n": n,
        "coverage": round(n / len(golden), 3) if golden else 0.0,
        "accuracy": {f: round(hits[f] / n, 3) if n else 0.0 for f in fields},
        "misses": misses,
    }


def passes(report: dict, minimums: dict[str, float]) -> bool:
    """Gate: every field must meet its minimum accuracy, and coverage must be complete."""
    return report["coverage"] == 1.0 and all(report["accuracy"][f] >= m for f, m in minimums.items())


def compare(old: dict, new: dict, fields: list[str], min_gain: float = 0.0) -> dict:
    """Which fields got better or worse between two reports. Use before switching a prompt or model."""
    better = [f for f in fields if new["accuracy"][f] - old["accuracy"][f] > min_gain]
    worse = [f for f in fields if new["accuracy"][f] < old["accuracy"][f]]
    return {"better": better, "worse": worse, "safe_to_ship": not worse}

score returns per-field accuracy, coverage (did every golden id get a prediction?) and the exact misses. passes is the gate: coverage must be complete and each field must meet its minimum. compare tells you which fields moved when you changed something.

Step 3: Run a prompt A/B

from evals import load, score, compare, passes
import csv

def predictions(path):
    with open(path, newline="", encoding="utf-8") as f:
        return {r["id"]: r for r in csv.DictReader(f)}

golden = load("data/golden.csv")
fields = ["segment", "intent", "next_action"]

report_a = score(golden, predictions("results_prompt_v1.csv"), fields)
report_b = score(golden, predictions("results_prompt_v2.csv"), fields)

print("v1:", report_a["accuracy"])
print("v2:", report_b["accuracy"])
print(compare(report_a, report_b, fields))
print("ships:", passes(report_b, {"segment": 0.9, "intent": 0.85, "next_action": 0.9}))
for miss in report_b["misses"][:10]:
    print(miss)                    # read the misses: they tell you what to fix

Step 4: Read the misses, then change one thing

The score tells you whether; the misses tell you why. Group them: is intent wrong for short messages? Does the model call every agency “high”? Change one thing (a rule, an example, the model), rerun, compare. Changing three things at once means you cannot tell which helped.

Step 5: A/B testing in the real world

The golden set checks the classification. Whether an email performs is a different question and needs a live test:

• Split a list randomly, not by sign-up date or alphabet.
• Change one variable (subject line, offer, send time).
• Decide the success metric and the sample size before you start.
• Do not stop the test the moment one arm looks ahead; small lists produce noisy gaps.

If your list is small, a statistical test may never reach significance. In that case treat the result as a hint and keep a log of tests so patterns can emerge across several.

Test it

The scoring tests check: per-field accuracy and the listed miss for a deliberate error, that the gate fails when a minimum is not met, that partial coverage fails the gate even when accuracy on the covered rows is perfect, and that compare reports a regression in one field as unsafe to ship even when another improved.

Worked example

A team swapped to a cheaper model to cut costs. Overall accuracy looked “about the same”, but compare showed intent down from 0.88 to 0.71 while segment was unchanged. They kept the cheaper model for segment labelling and the larger one for intent, and the bill still fell.

Common mistakes

• Evaluating on the same examples you tuned the prompt on.
• A golden set of 8 easy rows.
• Looking at one overall average and missing a regression in one field.

Checklist

• 30+ labelled, redacted examples including hard ones
• Per-field minimums written down
• Every prompt or model change is scored before it ships
• Live A/B tests change one variable and fix the metric up front