Eval Starter Kit — catch the regression you didn't see coming

You probably need this and don't know it

Most teams shipping LLM features today are not training models or fine-tuning. They're calling GPT-4o, Claude, or Gemini through an API and shipping a feature on top. The feature is a combination of four things — and three of them can quietly change without anyone touching the code.

That's three out of four moving without anyone realizing. Without evals, you find out it broke when support tickets stack up three weeks later. With evals, the PR fails CI the same day.

Who this is actually for

You don't need to be doing "ML work" to need an eval loop. If your product calls an LLM anywhere in its hot path, this kit is for you.

A concrete example — what this actually catches

Imagine you ship a customer-support thread summarizer using gpt-4o-mini. Your system prompt says "summarize concisely, friendly tone." You wire the kit in, run 30 example tickets through it, and your baseline rubric scores look like:

Two weeks later, a teammate adds "…and include action items" to the system prompt. They test on three example tickets, it looks fine, they open a PR.

Without the kit: PR ships. Support tickets quietly get worse. Three weeks later a customer-success lead notices the summaries now invent action items on tickets that don't have any. You spend two days bisecting prompt history.

With the kit: The PR triggers pytest. The eval re-runs on the full 30 examples. The report comes back:

CI fails. The diff points at the two axes that dropped. The reviewer immediately sees: format broke because the model appends "Action items:" even on tickets with none, and hallucination ticked up because it's inventing items to fill that section. Fix the prompt in the same PR, scores recover, ship.

That regression — silently shipped, found three weeks late — is the thing the kit prevents. The whole point.

What's in the box

Five files plus a notebook, intentionally tiny:

• golden_dataset.jsonl — 30 PM-flavored prompts paired with the ideal answer
• rubric.md — four scoring axes graded 1 – 5: factual accuracy, tone, format adherence, hallucination
• judge.py — LLM-as-judge that scores a single response on the rubric
• eval.py — the runner: model under test → judge → JSON report
• test_evals.py — pytest integration so evals run in CI on every PR
• eval_walkthrough.ipynb — notebook you can demo in a standup
• prompt_template.md — copy-paste mode, for when you don't have API keys

Two modes. API mode clones the repo and runs python eval.py end-to-end. Copy-paste mode is the rubric folded into a single prompt you paste into ChatGPT, Claude, or Gemini to score one example at a time. Same shape; no keys required.

The 4-step PM playbook

The repo encodes one loop. Run it on every LLM feature you own.

1. Define the golden dataset. 30 – 50 real user queries paired with ideal responses, weighted toward the common case, the expensive failure mode, and the awkward edge cases.

2. Set your metrics. The defaults — factual accuracy, tone, format adherence, hallucination rate — are PM-grade axes that travel well across products. Keep the count small (≤ 5) so the signal stays legible.

3. Choose an eval method. Deterministic checks (exact string, regex, JSON schema) where the answer is exact. A model-based judge — a stronger model like GPT-4o or Claude 3.5 Sonnet grading on the rubric — for everything semantic. Both are wired in.

4. Wire it into CI/CD. Run the eval on every prompt or model change. Fail the build when the score drops below threshold. Thirty examples through gpt-4o-mini plus a gpt-4o judge runs in pennies per PR — cheap enough to live on every commit.

The cost math

A single full eval run:

• 30 examples × gpt-4o-mini response: ~$0.005
• 30 examples × gpt-4o judge × 4 axes: ~$0.030
• Total: ~$0.035 per CI run — three-and-a-half cents

Run it 100 times in a month: $3.50. Catch one regression that would have taken two days to debug: saved $1,000+ in eng time. The ROI is not subtle.

How it compares to bigger frameworks

Four heavier open-source frameworks worth knowing once you outgrow a starter kit:

This kit covers the first 80% so you can decide which 20% you actually need.

Why I built it

Every team I've worked with that didn't have a written rubric ended up arguing about taste on a Slack thread for the third time that month. Every team that did — even thirty lines and a spreadsheet — moved faster and made better calls.

The kit isn't sophisticated. The discipline is.

→ Read the launch post · Star the repo