"It looked fine when I tried it"
Change a prompt or a model and some answers get better while others quietly get worse. Evals are the test suite of AI engineering: a set of inputs with checks on the outputs, run on every change. Compare versions case by case, and refuse to ship anything that drops below target or breaks a case that used to pass.
Two versions of an assistant
const cases = [
{ id: "momo", input: "Where can I eat cheap momo?", expect: { contains: ["momo"], cites: true } },
{ id: "trek", input: "Will it rain on my Pokhara trek?", expect: { contains: ["rain"], cites: true, maxLength: 80 } },
{ id: "cricket", input: "Who won the cricket?", expect: { equals: "I don't know" } },
{ id: "bug", input: "My javascript has an error", expect: { contains: ["deploy"], notContains: ["momo"], cites: true } },
{ id: "dal", input: "Is dal bhat spicy?", expect: { contains: ["dal bhat"], cites: true } },
];
const ANSWERS_V1 = {
"Where can I eat cheap momo?": "Try the momo near the bus park — cheap and good [1].",
"Will it rain on my Pokhara trek?": "Yes, monsoon brings rain, so pack rain gear for the trek to Pokhara and book hotels early [1].",
"Who won the cricket?": "Nepal, probably!",
"My javascript has an error": "Fix the javascript error before you deploy [1].",
"Is dal bhat spicy?": "Dal bhat is never too spicy.",
};
const ANSWERS_V2 = {
"Where can I eat cheap momo?": "Momo by the bus park is cheap [1].",
"Will it rain on my Pokhara trek?": "Yes — it's monsoon, expect rain [1].",
"Who won the cricket?": "I don't know",
"Is dal bhat spicy?": "Dal bhat is never too spicy [1].",
};
const v1 = async (q) => ANSWERS_V1[q];
const v2 = async (q) => {
if (!(q in ANSWERS_V2)) throw new Error("timeout");
return ANSWERS_V2[q];
};