Glossary
LLM-as-a-judge
On this page
The short answer
- LLM-as-a-judge is using one language model to score or compare another model’s outputs against criteria you write. It scales judgment calls such as tone and helpfulness that code cannot check, but it inherits model biases, so its scores are estimates.
- This is a glossary entry, not a product claim or a benchmark result.
- All terms · Templates · Guides.
Definition
LLM-as-a-judge is using one language model to score or compare another model’s outputs against criteria you write. It scales judgment calls such as tone and helpfulness that code cannot check, but it inherits model biases, so its scores are estimates.
In practice
Calibrate the judge on a sample you have labelled. Use code checks for anything exact (JSON, length, forbidden strings). Litmiq’s judge scores accuracy, tone, helpfulness and conciseness from 0 to 10, each with a reason.
Related terms
FAQ
Does Litmiq implement “LLM-as-a-judge” as a built-in metric?
Only where the definition itself says so. Many terms are industry vocabulary. Litmiq is a single-turn, BYOK prompt tester with an LLM judge, not a full evaluation platform.
Where is the rest of the glossary?
The hub at /glossary lists every term. This page is the standalone entry for LLM-as-a-judge.
See the term in a suite
Many glossary entries have a matching template you can run in the browser.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI