Glossary

LLM evaluation (eval)

On this page

The short answer

  • An LLM evaluation, or eval, is a repeatable test that measures how well a model or LLM application performs on a defined set of inputs. An eval has three parts: the inputs, a way to produce outputs (a prompt and a model), and a way to score them. “Eval” can mean a single test suite or the whole practice of measuring quality, including monitoring production traffic.
  • This is a glossary entry, not a product claim or a benchmark result.
  • All terms · Templates · Guides.

Definition

An LLM evaluation, or eval, is a repeatable test that measures how well a model or LLM application performs on a defined set of inputs. An eval has three parts: the inputs, a way to produce outputs (a prompt and a model), and a way to score them. “Eval” can mean a single test suite or the whole practice of measuring quality, including monitoring production traffic.

In practice

Use a fixed suite so you can compare two prompts or two models on the same cases. Litmiq covers the offline, single-turn kind: you paste a system prompt, add cases and run them in your browser.

FAQ

Does Litmiq implement “LLM evaluation (eval)” as a built-in metric?

Only where the definition itself says so. Many terms are industry vocabulary. Litmiq is a single-turn, BYOK prompt tester with an LLM judge, not a full evaluation platform.

Where is the rest of the glossary?

The hub at /glossary lists every term. This page is the standalone entry for LLM evaluation (eval).

See the term in a suite

Many glossary entries have a matching template you can run in the browser.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn