LLM-as-Judge: Using AI to Grade AI

An LLM-as-judge is a language model that grades AI outputs against a rubric. To build one from expert reviews, have one domain expert mark real outputs pass or fail with a written critique. Put those critiques in the judge prompt, compare its labels to the expert’s, and revise until they agree on fresh examples.

TL;DR

• LLM-as-judge lets you scale evaluation from dozens of human-reviewed interactions to thousands of automated assessments, but only when the judge is calibrated against domain expert judgment first
• Hamel Husain’s critique shadowing approach builds agreement between LLM and human graders iteratively, starting from a shared rubric and converging through disagreement analysis
• The judge becomes truly powerful when it understands the user via self-models, because the same response can be excellent for one user and terrible for another

LLM-as-judge is a method for scaling AI evaluation from dozens of human-reviewed interactions to thousands of automated assessments by training a language model to grade like a domain expert. An uncalibrated LLM judge is hard to trust. Hamel Husain’s critique shadowing approach [1] checks the judge against a domain expert’s labels, and he reports one case that reached more than 90% agreement after three iterations. This post covers the three-round calibration process, how to build a reliable judge prompt, and why self-models unlock the fit dimension that standard judges miss.

How Does the Critique Shadowing Process Work?

Hamel Husain calls his method for building LLM judges “Critique Shadowing” (Hamel Husain, 2024). The LLM judge shadows the human expert, and you close the gap between their assessments step by step. His guide uses binary pass/fail labels with written critiques. The rounds below adapt that process to a rubric with several dimensions.

Round 1: Establish the Baseline. Have your domain expert grade a starting set of real interactions using your rubric. Hamel Husain starts with around 30. Mark each dimension, such as relevance, coherence, depth and fit, as pass or fail, and write a short critique. Then have the LLM judge grade the same interactions with the same rubric. Compare the labels.

On round 1, expect clear gaps. The LLM may be too generous on some dimensions, too harsh on others, and miss the point on a few. This is normal.

Round 2: Analyze Disagreements. For every interaction where the LLM and expert disagree, document why. Is the LLM ignoring context? Is it applying the wrong standard? Is it failing to distinguish between technically correct and actually helpful?

Use these disagreements to refine the judge’s prompt. Add specific examples of what a pass versus a fail looks like for each dimension. Include the expert’s reasoning for their scores.

Round 3: Reconverge. Run the refined judge on a new set of expert-graded interactions it has not seen. Agreement should rise. Hamel Husain reports one case that passed 90% after three iterations. Report true positive and true negative rates separately, because raw agreement can mislead when labels are imbalanced. For the remaining disagreements, make a judgment call: is the LLM wrong, or is the expert’s criteria ambiguous? Sometimes the calibration process reveals that the rubric needs clarification, not the judge.

How Do You Build the Judge Prompt?

The judge prompt is the most important artifact in your eval system. Here is a structure that gives the judge what it needs.

Section 1: Role and Context. Tell the LLM it is evaluating AI product outputs against a specific rubric. Provide product context: what the AI does, who uses it, what a successful interaction looks like.

Section 2: The Rubric. Include the full rubric with definitions for each score level on each dimension. Not just what a 5 looks like: what a 1, 2, 3, 4, and 5 look like. The more specific, the higher the agreement.

Section 3: Calibration Examples. Include 5-10 expert-graded examples with the expert’s reasoning. These anchor the judge’s understanding. The examples should span the score range: a clear 5, a clear 1, and several in the ambiguous middle where context matters.

Section 4: Output Format. Require structured output with a score for each dimension plus a reasoning explanation. The reasoning is essential: it lets you audit the judge’s logic and catch systematic errors.

1// Pseudocode: build a calibrated LLM judge← Illustrative, not a published API
2const judge = await createJudge({
3  rubric: qualityRubric,
4  calibrationExamples: expertGradedSamples,
5  dimensions: ['relevance', 'coherence', 'depth', 'fit'],
6});
7
8// Grade an interaction with user context← Fit scoring needs user context
9const evaluation = await judge.evaluate({
10  interaction: aiResponse,
11  userContext: userProfile,
12  explainReasoning: true,
13});
14
15// Returns: { relevance: 4, coherence: 5, depth: 3, fit: 4,← Auditable reasoning
16//   reasoning: 'Fit scored 4: adapted technical depth for senior engineer...' }

The Self-Model Advantage for LLM Judges

Here is where most LLM-as-judge implementations hit a wall they do not see coming.

A standard LLM judge evaluates whether the output is generally good. But a recommendation that is perfect for a beginner is condescending to an expert. An explanation that delights a technical user confuses a business stakeholder. The same output deserves different scores depending on who received it.

Without user context, the judge cannot evaluate fit. Fit is a dimension a generic judge cannot score. You can have perfect relevance, coherence, and depth, and still lose the user because the output was not tailored to them.

Self-models solve this by providing the judge with the user’s context: their expertise level, their goals, their communication preferences, their domain knowledge. The judge can then evaluate whether the output matched the person, not just whether it matched the question.

This transforms the judge from a generic quality checker into a personalization quality checker.

What Are the Common LLM Judge Failures and Fixes?

Researchers have documented several judge biases, including position, verbosity and self-enhancement bias (Zheng et al., 2023) [2].

Positivity bias. LLM judges tend to grade generously. Fix: include examples of clearly poor outputs that received low expert scores. Anchor the low end of the scale explicitly.

Length bias. Longer responses tend to get higher scores, even when brevity is more appropriate. Fix: include calibration examples where the expert preferred the concise response and explain why.

Style over substance. The judge rewards well-formatted, articulate responses even when they miss the point. Fix: include examples where a technically correct but unhelpful response received a low score, with explicit reasoning about the difference between sounding good and being good.

Context blindness. The judge evaluates the response in isolation, ignoring the conversation history or user context. Fix: always include the full interaction context, previous messages, user profile, session intent, in the judge’s input.

What Are the Trade-offs of an LLM Judge?

LLM judges cost money to run. Evaluating every interaction with a judge model adds inference costs. Budget for it as a real line item.

Judge quality degrades over time. As your product evolves, the judge’s calibration examples become stale. Plan for regular recalibration with fresh expert-graded examples.

Circular dependency risk. If the same model family judges itself, systematic blind spots pass undetected. Use a different model family for judging than for production. Or better yet, use self-model context to ground the evaluation in user reality rather than model opinion.

Expert disagreement is real. Expert disagreement is real. Measure how often your experts agree with each other first. If they often disagree, the rubric needs clearer criteria before you tune the judge.

What Should You Do Next?

1. Start with a small set of expert-graded examples. Have your best domain expert grade around 30 real interactions using a defined rubric. This dataset is the foundation of everything, your calibration set, your baseline agreement measurement, and your first audit trail of what good looks like.

2. Build and calibrate in rounds. Adapted from Hamel Husain’s critique shadowing [1]: Round 1: baseline agreement. Round 2: disagreement analysis and prompt refinement. Round 3: convergence check. Deploy the judge once agreement is good enough for your risk level.

3. Add user context to the judge. Once your judge agrees with your expert on generic quality, extend it with user context for fit evaluation. Clarity by Epistemic Me offers evals consulting, including LLM-as-judge checked against expert labels. Engagements start at $100K+.

An uncalibrated LLM judge is just automated vibes. A calibrated one scales your best expert to every interaction. Build the judge.

FAQ

What is an LLM-as-judge?

An LLM-as-judge is a language model that grades the outputs of another AI system against a rubric. It lets you score many more interactions than human reviewers can. It is only trustworthy when you check it against expert labels.

How many expert-graded examples do I need to start?

Hamel Husain starts with around 30 examples and keeps going until new failure modes stop appearing [1]. Start small, then add a fresh set to test the refined judge. If your labels are imbalanced, report true positive and true negative rates separately.

How do I know my LLM judge is good enough?

Compare the judge’s labels to your domain expert’s on examples the judge has not seen. Hamel Husain reports one case that passed 90% agreement after three iterations [1]. The right bar depends on how costly a wrong grade is for your product.

Why can an LLM judge be wrong even with a good rubric?

Judges can show position, verbosity and self-enhancement bias [2]. Calibration examples with expert reasoning help, and so does using a different model family for judging than for production. Keep checking the judge as your product changes.

References

1. Hamel Husain, “Using LLM-as-a-Judge For Evaluation: A Complete Guide” (2024-10-29)
2. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023)
3. Hamel Husain, “Your AI Product Needs Evals”

Related

• Minimum Viable Evals: Start Here
• The AI Product Maturity Assessment
• The AI Product Health Check