Glossary

LLM-as-a-judge

On this page

The short answer

  • LLM-as-a-judge is using one language model to score or compare another model’s outputs against criteria you write. It scales judgment calls such as tone and helpfulness that code cannot check, but it inherits model biases, so its scores are estimates.
  • This is a glossary entry, not a product claim or a benchmark result.
  • All terms · Templates · Guides.

Definition

LLM-as-a-judge is using one language model to score or compare another model’s outputs against criteria you write. It scales judgment calls such as tone and helpfulness that code cannot check, but it inherits model biases, so its scores are estimates.

In practice

Calibrate the judge on a sample you have labelled. Use code checks for anything exact (JSON, length, forbidden strings). Litmiq’s judge scores accuracy, tone, helpfulness and conciseness from 0 to 10, each with a reason.

FAQ

Does Litmiq implement “LLM-as-a-judge” as a built-in metric?

Only where the definition itself says so. Many terms are industry vocabulary. Litmiq is a single-turn, BYOK prompt tester with an LLM judge, not a full evaluation platform.

Where is the rest of the glossary?

The hub at /glossary lists every term. This page is the standalone entry for LLM-as-a-judge.

See the term in a suite

Many glossary entries have a matching template you can run in the browser.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn