Agent evaluation

On this page

LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.

The short answer

Agent evaluation measures a system that takes multiple steps — planning, calling tools, reading results and perhaps trying again — not just one reply. Typical checks include whether it chose the right tool, looped, finished the task, or stayed within a budget of steps and spend.

Litmiq cannot replay a full trajectory. It is a single-turn prompt tester. You can still regress the instructions and the first tool call in the browser, then use a tracing platform or an agent-focused library for the rest. A shorter definition is in the agent evaluation glossary entry.

What to measure

  • Task success. Did the agent complete the job the user asked for, by a written definition of done?
  • Tool use. Right tool, valid arguments, no invented tools. The tool-calling template is a single-turn slice of this.
  • Control. Did it stop, loop, or exceed a step or cost budget?
  • Policy and safety. Hard rules (no refunds, no leaked secrets) should be pass/fail, including on intermediate steps if you have traces.
  • Judge or human review on the final answer, after the hard checks, with the same caveats as any LLM judge.

There is no official, complete metric set. We do not invent one. Start from failures you have already seen in traces.

Where to run it

To score a multi-step run you need a trace: each step's inputs, tool calls and outputs. That is what platforms such as LangSmith, Langfuse and Braintrust are for (see Langfuse vs LangSmith vs Braintrust). Offline, you replay a dataset of traces or tasks; online, you sample live threads.

Litmiq's place is narrower: lock the system prompt and the first-step tool-call format before you wire the agent, and re-check those when a model shuts down or a prompt changes. Use the tool-calling cases, then graduate the same expectations into your platform or CI framework.

Limits of this guide

This is a practice guide, not a benchmark and not a claim that Litmiq evaluates agents end to end. Tool capabilities change; see the linked vendor pages and our dated comparison pages. We are not affiliated with those vendors.

FAQ

What is agent evaluation?

Agent evaluation measures a system that takes multiple steps — planning, calling tools, reading results and perhaps trying again — not just one reply. Typical checks include whether it chose the right tool, looped, finished the task, or stayed within a budget of steps.

Can Litmiq evaluate a full agent trajectory?

No. Litmiq runs single-turn tests. You can test the first step, for example whether the model emits the right tool call as JSON, with the tool-calling template, but you cannot replay a full multi-step trajectory here. Tracing platforms and agent-focused libraries are built for that.

What should I test in an agent eval?

Start with task success (did it finish the job?), tool choice and arguments, whether it stopped or looped, and any hard policy rules. Add trace-level judges only after those checks exist. Keep a dataset of failed production traces if you have a platform that can store them.

Which tools do full agent evaluation?

Hosted tracing and evaluation platforms such as LangSmith, Langfuse and Braintrust are built to score traces and threads. Some libraries (for example Giskard or TruLens, which we compare on their own pages) document multi-step or agent checks. We do not rank them here.

How is this different from prompt evaluation?

Prompt evaluation usually scores one prompt and one reply. Agent evaluation scores a sequence of steps. You still need good single-turn prompt tests for the instructions inside the agent; they just are not sufficient on their own.

Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.

Sources

Try it in Litmiq

Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn