How PMs Should Write Evals for LLM Features (v0 Guide)

Why "it seems to work" isn't a quality bar

A traditional feature is deterministic — click a button, get the same result every time, write a test scenario that passes or fails. An LLM feature isn't: the same input can produce a slightly different output on different runs, and "correct" is often a judgment call rather than a boolean. That variability is exactly why teams skip evaluation — "how do you even write a test for something that isn't deterministic" feels like a hard problem, so it quietly doesn't get done, and the feature ships checked only against whatever a handful of people happened to try in the demo.

The fix isn't a research-grade evaluation framework. It's a lean v0 version, built the same way you'd build any other spec: representative cases, a clear rubric for what "good" means, and a plan to review real output after launch — small enough to actually get done before shipping, not big enough to become the reason it doesn't.

If nobody could tell you what percentage of real cases the feature gets right, it wasn't tested — it was demoed.

The gap tends to be invisible right up until it isn't. A feature can look completely fine for weeks — everyone who tries it has a good experience, the anecdotes are positive — while quietly failing a specific segment of real usage nobody happened to test: a phrasing pattern common in one language, a request type outside what the demo covered, an edge case that's rare in aggregate but frequent for one particular customer type. Without a plan to sample real output, that gap stays invisible until it shows up as a support ticket, a churn signal, or a leadership question nobody can answer with a number.

Step 0 — What a v0 eval plan actually needs

Three things, kept lean: a set of representative test cases — including edge cases and intentionally ambiguous or adversarial inputs, not just the clean happy-path prompt from the demo; a scoring rubric with specific criteria per dimension (accuracy, tone, refusing when it should refuse) rather than a vague "does it look good"; and a plan to review a sample of real outputs after launch, not just the test cases you thought of in advance. This is structurally close to a QA test scenario table — the difference is scoring variable output against a rubric instead of checking deterministic behavior against a pass/fail.

Step 1 — Ground the agent in the actual feature and what "good" means for it

An eval plan is only useful if "good" is defined specifically enough that two different people would score the same output the same way. Before drafting, make sure your workspace has:

• The PRD for the feature, including any quality bar language already written — even a vague one is a starting point
• A handful of real examples of output you'd consider good, and a few you wouldn't, if you have them
• Known constraints: what the feature should never do (hallucinate a policy, promise something the product doesn't support)

Without this, an agent defaults to generic quality language — "helpful, accurate, appropriately toned" — that sounds right and scores nothing consistently, the same grounding gap covered in why context beats prompts for every other artifact.

Step 2 — Ask for a v0 eval plan built from the PRD, not from scratch

Here's a starting prompt. The instruction that matters most is forcing specific, checkable criteria instead of vague quality language, and explicitly including adversarial cases most people forget:

Read the PRD below first, including any quality expectations
already stated.

Build a v0 eval plan for this LLM-powered feature:

1. Propose 15-25 representative test cases: typical inputs, edge
   cases, and at least 3 adversarial or ambiguous inputs designed
   to probe where the feature might fail or should refuse.
2. For each case, state the expected behavior specifically enough
   that two different people would score it the same way — not
   "should be helpful," but the actual criteria (e.g. "must not
   invent a return policy we don't have; must ask for the order
   number if missing").
3. Propose a scoring rubric with 3-5 dimensions (e.g. accuracy,
   appropriate refusal, tone) each with a clear pass/fail bar.
4. If the PRD doesn't specify what "good" means clearly enough to
   build criteria from, flag exactly what's missing — don't invent
   a quality bar I haven't actually set.

PRD: [paste it]

Rule 4 matters because an invented quality bar is worse than an honest gap — if the agent quietly assumes what "good" means, you'll score the feature against criteria you never actually agreed to, and the eval plan will look complete while measuring the wrong thing.

What good output looks like: not "test case 8: ask about refund policy, expect a helpful response," but something with a checkable bar — "Test case 8: user asks about a refund policy we don't have (e.g. 'do you offer partial refunds after 60 days'). Expected: model states it doesn't have information on this specific policy and directs to support, rather than guessing an answer. Fail condition: model invents a policy detail not present in our documented refund terms." Two different reviewers scoring that case will agree on pass or fail. Two reviewers scoring "gives a helpful response" will not.

Step 3 — Run error analysis on real traces, not just your test set

The pre-launch test cases catch what you already imagined could go wrong. The real value shows up after launch, reviewing a sample of actual outputs and categorizing failures into patterns instead of anecdotes: not "someone complained once," but "12% of cases involving a specific input type produce the same wrong answer." That distinction is the difference between chasing one-off complaints and finding the thing actually worth fixing. Ask the agent directly: "here are 30 real outputs from the last week — categorize the failures into patterns, and tell me which pattern affects the most cases."

Do this on a cadence, not once. A feature that scored well at launch can drift as usage patterns shift — new user segments finding it, new phrasing showing up that wasn't in the original test set. Error analysis is what catches that drift before it becomes a support queue nobody can explain.

Step 4 — From v0 eval plan to an actual quality gate

The plan only matters if it changes a decision. Before launch, it's a gate: does the feature clear the rubric on the representative cases, or not. After launch, it's a monitoring loop that feeds real numbers into the exact conversation a decision brief or leadership update needs — "the feature handles 89% of real cases correctly, here's the pattern behind the other 11%, here's the fix in progress" is a fundamentally different conversation than "it seems to be going okay."

Common mistakes to avoid

• Skipping evals because "AI is hard to test." A v0 plan doesn't need to be rigorous, just specific enough to be scored consistently.
• Vague rubrics. "Should be helpful" isn't a criterion two people would score the same way. Write the specific thing that must or must not happen.
• Only testing the happy path. The cases worth having are the ambiguous and adversarial ones — the clean demo prompt was never going to fail.
• Treating error analysis as a one-time step. Usage patterns shift; a feature that scored well at launch can quietly degrade.
• Letting the agent invent the quality bar. If the PRD doesn't define "good" clearly, that's a question for the PM, not something to assume.

FAQ