Glossary
Agent evaluation
On this page
The short answer
- Agent evaluation measures a system that takes multiple steps — planning, calling tools, reading results and perhaps trying again — not just one reply. Typical checks include whether it chose the right tool, looped, or finished the task.
- This is a glossary entry, not a product claim or a benchmark result.
- All terms · Templates · Guides.
Definition
Agent evaluation measures a system that takes multiple steps — planning, calling tools, reading results and perhaps trying again — not just one reply. Typical checks include whether it chose the right tool, looped, or finished the task.
In practice
Litmiq runs single-turn tests. You can still test the first step (did the model emit the right tool call as JSON?) with the tool-calling template, but you cannot replay a full trajectory here.
Tracing platforms and agent-focused libraries (for example TruLens or Giskard) are built for multi-step runs. Use Litmiq to regress the prompts inside an agent, one turn at a time.
Related terms
FAQ
Does Litmiq implement “Agent evaluation” as a built-in metric?
Only where the definition itself says so. Many terms are industry vocabulary. Litmiq is a single-turn, BYOK prompt tester with an LLM judge, not a full evaluation platform.
Where is the rest of the glossary?
The hub at /glossary lists every term. This page is the standalone entry for Agent evaluation.
See the term in a suite
Many glossary entries have a matching template you can run in the browser.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI