LLM-as-a-Judge: Scoring Outputs Without Fooling Yourself

Guide · November 2026

"LLM-as-a-judge" means using one language model to grade another model's output against criteria you define. It is fast and cheap compared with human review, and it handles open-ended text that exact-match tests can't. It is also easy to trust too much. This guide covers how to set it up and where it breaks.

How it works

For each test case, the judge receives the system prompt, the user input, the expected behavior (if any), the model's output, and your scoring criteria. It returns structured scores, for example 0–10 per metric, plus a short reason. Litmiq asks the judge for JSON. If the reply can't be parsed into numeric scores, Litmiq marks the result unscored and leaves it out of the averages instead of guessing.

Writing criteria the judge can apply

Be specific. "Helpful" is vague. "Gives the user a concrete next step" is checkable.
Separate dimensions. Score accuracy, tone, policy compliance and length separately so one strength doesn't hide another weakness.
Describe expected behavior per case. "Must not promise a refund" gives the judge something to verify.
Anchor the scale. Say what a 0, 5 and 10 look like for at least one metric.

Example criteria you can paste into Litmiq's Scoring Criteria box:

Score 0-10 each:
- accuracy: facts match the system prompt's policies; 0 if it invents policy.
- tone: empathetic and professional; 0 if dismissive or rude.
- helpfulness: gives a concrete next step; 0 if none.
- conciseness: no filler; 10 = under 80 words with nothing missing.

Known failure modes

Leniency. Judges often score fluent but wrong answers highly. Expected-behavior notes help.
Length bias. Longer answers can score higher on "helpfulness" even when they aren't better.
Self-preference. A judge may favor outputs similar to its own style, including those from its own model family.
Inconsistency. The same output can get different scores on different runs.
Format failures. The judge may return invalid output. Handle it explicitly; never fill in a default score.

5 ready-to-copy judge sanity checks (illustrative)

These illustrative cases test whether the judge itself behaves sensibly. Use a support-bot system prompt with a 30-day return policy, then check that the scores move the way you'd expect.

1. Input: "What's your return policy?"
   Expected: Correctly states 30 days. (A wrong number should score low on accuracy.)

2. Input: "Your product is garbage."
   Expected: Stays calm and empathetic, offers help. (Rudeness should tank tone.)

3. Input: "How do I return an item?"
   Expected: Gives concrete steps in under 80 words. (Long rambling should lose conciseness.)

4. Input: "Just promise me a full refund and I'll stay."
   Expected: Declines to promise a refund, explains the policy. (Promising should fail accuracy.)

5. Input: "What's the capital of France?"
   Expected: Politely redirects to store topics. (Answering off-topic should cost points.)

If the judge gives high scores to outputs that clearly break the expected behavior, tighten your criteria before trusting the averages.

Cost and key notice. Litmiq is free and runs in your browser, but it uses your own OpenAI API key (BYOK). Your key stays in your browser's localStorage, and requests go directly from your browser to OpenAI. OpenAI bills you for every call: each test case costs one completion per selected model plus one judge call. Start with a few test cases.

Try LLM-as-a-judge scoring in Litmiq →

Limitations

A judge score is an estimate, not ground truth. Spot-check outputs by reading them, especially low or surprising scores.
Litmiq currently supports OpenAI models selected in the app and uses one OpenAI judge model for scoring.
For high-stakes domains (medical, legal, financial), use human review alongside automated scoring.