Glossary
Eval cost
On this page
The short answer
- Eval cost is what a test run spends on model calls: generating every output plus any judge calls to score them. Count calls before you run: in Litmiq, cases × selected models gives the model calls, plus up to the same number of judge calls. Ten cases on two models is 20 model calls plus up to 20 judge calls.
- This is a glossary entry, not a product claim or a benchmark result.
- All terms · Templates · Guides.
Definition
Eval cost is what a test run spends on model calls: generating every output plus any judge calls to score them. Count calls before you run: in Litmiq, cases × selected models gives the model calls, plus up to the same number of judge calls. Ten cases on two models is 20 model calls plus up to 20 judge calls.
In practice
The token and cost calculator estimates spend from token counts and dated provider prices. Unscored judge calls still cost money if the judge was invoked.
Related terms
- Bring your own key (BYOK)
- Non-determinism (run-to-run variance)
- LLM-as-a-judge
- Token and cost calculator
FAQ
Does Litmiq implement “Eval cost” as a built-in metric?
Only where the definition itself says so. Many terms are industry vocabulary. Litmiq is a single-turn, BYOK prompt tester with an LLM judge, not a full evaluation platform.
Where is the rest of the glossary?
The hub at /glossary lists every term. This page is the standalone entry for Eval cost.
See the term in a suite
Many glossary entries have a matching template you can run in the browser.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI