How do you turn expert feedback on AI outputs into a grading rubric?

Ask one domain expert to review real outputs and write a short critique for each. Group the critiques into recurring failure types. Turn each type into a clear pass/fail check with a definition and examples. Then test the rubric against fresh expert labels, and revise it where the two disagree.

Why not write the rubric first?

Researchers who studied how people grade LLM outputs found that criteria change during grading. They call this “criteria drift”: “users need criteria to grade outputs, but grading outputs helps users define criteria” (Shankar et al., arXiv 2404.12272, 2024-04).

In plain terms, experts do not know all their criteria until they see outputs. Some criteria only appear after the expert has read a bad answer. Hamel Husain makes the same point: you cannot write a good judge prompt before you see the data (Hamel Husain, 2024-10-29).

So the rubric should come from the expert’s reactions to real outputs, not from a template.

What does the process look like?

The steps below combine guidance from the sources at the end of this post.

1. Choose one expert. The evals FAQ calls a single domain expert, a “benevolent dictator”, the most effective approach for most teams. That person sets the quality standard and can take input from others. Larger or multi-domain organizations may need several annotators, and then agreement should be measured with a chance-corrected metric such as Cohen’s kappa (Husain and Shankar, Evals FAQ, 2026-09-18).
2. Pull real outputs. Use a varied pool of about 100 traces. The same FAQ suggests hand-reviewing a first batch of at least 30 and continuing while you are still learning.
3. Ask for a verdict and a critique. For each output, the expert records Pass or Fail and writes why.
4. Group the critiques. The FAQ advises that you “group similar notes to see which problems recur”. The groups become a list of failure modes.
5. Write one check per failure mode. Each check gets a name, a one-sentence definition, and examples of Pass and Fail.
6. Test the rubric. Run it against expert labels the rubric has not seen. Check where it disagrees.
7. Revise and relabel. When graders disagree, find the rubric line that caused the split, write a rule to settle it, and relabel the affected examples.

What makes a good critique?

A critique is the raw material for the rubric, so quality matters. Hamel Husain says it should be detailed enough to use as a few-shot example in a judge prompt. A new employee should be able to understand it (Hamel Husain, 2024-10-29). He names terse critiques as a common mistake.

Here is the difference. The examples are illustrations, not real client data.

Weak critique	Stronger critique
”Wrong tone."	"The reply promises a refund the policy does not allow. The customer asked about a late fee, and the reply never mentions the fee."
"Bad answer."	"The summary leaves out the contract end date. A reader would miss the renewal deadline."
"Not helpful."	"The answer names the right form but does not say where to send it.”

The stronger version tells the next reader what to look for. It also helps the expert. Husain notes that writing critiques helps experts clarify criteria they had not fully put into words.

Why use pass/fail checks instead of a 1 to 5 scale?

Several practitioners prefer binary checks. Husain writes that “a binary pass/fail forces evaluators to make a clear judgment”, and pairs it with a detailed critique to keep nuance (Hamel Husain, 2025-03-24). The evals FAQ gives reasons: the gap between adjacent scale points is subjective, detecting real differences on a scale needs larger samples, and annotators drift toward the middle. For gradual progress, it suggests separate binary checks on sub-components.

Eugene Yan takes a similar position: where possible, he has his evaluators return binary outputs, which makes classification metrics easier to use (Eugene Yan, 2024-08).

Not every source says to avoid scales. Anthropic’s documentation on building tests shows an LLM-graded 1 to 5 scale for tone, with the end points defined, and a yes/no grader for privacy leaks (Anthropic docs, accessed 2026-10-30). A scale can work when each point is defined and the output is constrained. The point of agreement is that vague scales do not work.

How do you turn a failure mode into a check?

Write each check so that two people reading it would give the same verdict. Anthropic’s guidance says good success criteria are specific and measurable (Anthropic docs, accessed 2026-10-30). Its privacy example defines what counts as protected data before asking for a yes/no judgment.

Use this template for each check:

1. Name: a short label, such as “Promises unauthorized refund”.
2. Question: one yes/no question, such as “Does the reply promise a refund the policy does not allow?”
3. Definition: what counts, and what does not.
4. Pass example: one real output from the review.
5. Fail example: one real output, with the expert’s critique.
6. Check type: code check where a rule can do it, LLM judge where it cannot. Anthropic’s engineering team describes three grader types: code-based, model-based and human. Code-based graders are fast and reproducible. Model-based graders handle nuance but are non-deterministic (Anthropic, 2026-01-09).

If a failure mode can be tested with a rule, use a rule. Save the LLM judge for criteria that need judgment.

How do you know the rubric works?

Test it. Split the labeled examples. The evals FAQ suggests roughly 10 to 20% for training examples that may appear in the prompt, and roughly 40 to 45% each for dev and test. Tune on dev. Reserve the test set for one final check. If test scores fall well below dev scores, the judge may be overfitting (Husain and Shankar, Evals FAQ, 2026-09-18).

Report the judge’s true positive rate and true negative rate against the expert, not raw agreement alone. The related guide on sample size is How many labeled examples do you need before you trust an LLM judge?

Also read the transcripts. Anthropic’s engineering team says reviewing full records shows whether a failure is a real agent mistake or a grader rejecting a valid answer (Anthropic, 2026-01-09).

FAQ

How many expert reviews do I need to build the first rubric?

The evals FAQ recommends reviewing at least 100 traces, with a first hand-reviewed batch of at least 30. Keep going while you are still finding new problems. Stop when new failure modes stop appearing.

What if two experts give different grades?

Measure agreement with a chance-corrected metric such as Cohen’s kappa. Then find the rubric line that caused the split and write a rule that settles it. Relabel the affected examples. For many teams, naming one expert as the final judge avoids repeated splits.

Should I let an LLM write the rubric for me?

An LLM can draft candidate checks. The research above shows why a human must still grade: the LLM’s checks inherit the problems of the model, and experts refine their criteria by grading outputs. Use the LLM to propose, and the expert to decide.

How often should I update the rubric?

Update it when new failure patterns appear and after material changes such as a model update. Husain advises re-running the validation after such changes.

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm. Clarity’s evals consulting turns expert feedback into rubrics, golden datasets and LLM-as-judge checks that are validated against expert labels, with regression gates. It can be done with your team or for your team. Engagements start at $100K+, and the proposal sets scope, deliverables and duration. Lynn Chyi, AI Experience Architect at Dayforce, said the work improved “our human expert alignment scores by nearly 60%” (case study). See /evals/ and how to evaluate an AI agent.

Sources

• Shankar et al. — Who Validates the Validators? (2024-04)
• Hamel Husain — Creating a LLM-as-a-Judge (2024-10-29)
• Hamel Husain — A Field Guide to Rapidly Improving AI Products (2025-03-24)
• Hamel Husain and Shreya Shankar — Frequently Asked Questions about AI Evals (2026-09-18)
• Eugene Yan — Evaluating the Effectiveness of LLM-Evaluators (2024-08)
• Anthropic — Define success criteria and build evaluations (accessed 2026-10-30)
• Anthropic — Demystifying evals for AI agents (2026-01-09)