LLM evaluation

On this page

LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.

The short answer

LLM evaluation (often “eval”) is a repeatable way to measure how a language model or an LLM application behaves on a defined set of inputs. It has three parts: the inputs (test cases), a way to produce outputs (a prompt, a model, and any tools or retrieval around them), and a way to score those outputs. “Eval” can mean a single suite or the whole practice, including watching live traffic.

This is a full guide to that practice — what it is, which metrics are actually useful, and what an AI eval framework does or does not give you. A shorter definition lives in the glossary entry for LLM evaluation. We make Litmiq, a browser prompt tester; it covers only part of this map, and the limits are spelled out below.

What is LLM evaluation?

Software tests ask “does this function still return the same result?” LLM evaluation asks “does this prompt, model or application still behave well enough on cases we care about?” Outputs are often open-ended, so a single exact-match assertion is usually not enough. A practical eval therefore mixes:

  • Deterministic checks where a rule is yes or no (valid JSON, no refund promise, required citation format).
  • An LLM-as-judge or human review where quality varies by degree (was the answer grounded, was the tone right).
  • A fixed suite, so two prompts or two models are compared on the same cases, not on vibes.

Offline evaluation runs that suite before you ship. Online evaluation scores production traces as they arrive. Hosted platforms such as LangSmith document both; a browser tool such as Litmiq does the offline, single-turn kind only. See the LangSmith evaluation section on the LangSmith comparison, and the offline and online glossary entries.

LLM evaluation metrics

A metric is just a named way to score a case. There is no official, complete list. Useful families:

  • Hard rules. Schema, regex, forbidden or required strings, tool-call shape. These should be pass/fail, not a 1–10 scale.
  • Task metrics. For RAG, whether the answer is faithful to retrieved context and whether the context was relevant (libraries such as Ragas document metrics in this family). For extraction, whether the JSON is valid and complete. For support bots, whether a policy was followed.
  • Judge scores. An LLM grades the output against written criteria (accuracy, tone, safety). These are useful for open-ended text and easy to fool if the criteria are vague. The LLM as a judge guide covers biases and calibration.
  • Human review. Slow, still the reference when a judge or a metric disagrees with what you would ship.

Pick metrics from the failures you have already seen, not from a catalogue. Keep the suite stable when you compare prompts or models. We do not publish a ranked “best metrics” list, and we do not invent benchmark numbers here.

What an AI eval framework is (and isn't)

An eval framework is the software that stores cases, calls models, applies scorers and shows diffs so you can run the same eval again tomorrow. Broad types:

  • CLI and libraries — Promptfoo (YAML, MIT, Community listed as Free Forever on its pricing page, 19 Nov 2026), DeepEval (Apache-2.0, tests as code), Ragas (Apache-2.0, RAG-oriented metrics). These belong in CI. See open-source LLM evaluation frameworks.
  • Hosted platforms — LangSmith, Langfuse, Braintrust: traces, datasets, experiments, often online evals. See Langfuse vs LangSmith vs Braintrust.
  • Narrow testers — Litmiq runs a prompt and cases in the browser with your own key. It does not trace production traffic or replay a multi-step agent.

A framework does not write your cases, and it does not make a bad metric good. If you only need to compare two wordings of a system prompt this afternoon, start in the browser; graduate to a CLI or a platform when you need CI, traces or a team dataset.

Try this in Litmiq

Paste the system prompt you actually ship, add 10–20 cases that encode hard rules and typical user phrasing, and run them. Read the diffs. Export the JSON if you want a baseline. When the suite is stable, move it into Promptfoo or DeepEval for CI, or keep re-running it here after model changes.

Limits of this guide

This is a map of the practice, not a product review and not a benchmark. Tool features and prices cited on linked pages were checked on the dates given there (Promptfoo and LangSmith prices, 19 November 2026). Litmiq is one option in the “narrow tester” row, not a full eval platform. We are not affiliated with the other tools named.

FAQ

What is LLM evaluation?

LLM evaluation is a repeatable test of how a language model or an LLM application behaves on a defined set of inputs. An eval has inputs, a way to produce outputs (a prompt, a model, and any surrounding code), and a way to score those outputs — code checks, an LLM judge, human review, or a mix. The term can mean one test suite or the whole practice, including production monitoring.

What are LLM evaluation metrics?

A metric is a specific score or pass/fail rule you apply to an output or a trace. Common kinds include exact or schema checks (valid JSON, a required phrase), task metrics (did it answer the question, was the citation grounded), and judge scores on written criteria (accuracy, tone, safety). There is no universal metric. Pick ones that match the failure you care about, and keep a fixed suite so two prompts or two models are comparable.

What is an AI eval framework?

An eval framework is the software that runs those tests repeatedly: storing cases, calling models, applying scorers and reporting diffs. Examples include CLI and library frameworks such as Promptfoo and DeepEval, hosted platforms such as LangSmith or Langfuse, and narrower tools such as Litmiq, which runs single-turn prompt tests in the browser. A framework is not a substitute for writing good cases and reading failures.

Is Litmiq an LLM evaluation framework?

Litmiq is a free, no-signup browser app for single-turn prompt tests with an optional LLM judge, using your own OpenAI or Anthropic key. It is not a tracing platform, not a multi-step agent runner, and not a full CI-first framework, though Litmiq Pro can export a GitHub workflow that re-runs saved suites.

How is LLM evaluation different from a leaderboard?

A leaderboard ranks models on someone else's tasks. An eval you maintain ranks your prompt, your rules and your cases. Use a public benchmark as background; use your own suite to decide whether to ship a change.

Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.

Sources

Try it in Litmiq

Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn