How to choose an AI eval platform
On this page
LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.
The short answer
Choose an AI eval platform by the job, not by a leaderboard. If you need production traces, team datasets or online evaluation, you want a platform such as LangSmith, Langfuse or Braintrust. If you need evals as code in CI, you want a framework such as Promptfoo or DeepEval. If you only need to check a system prompt against a handful of cases this afternoon, a no-signup browser tester such as Litmiq is enough — and is not a platform.
This page is a decision guide. Dated plan details live on the linked comparison and pricing pages (official vendor pages read 19 November 2026). We do not rank products or invent review scores.
Start with the job
- Are you scoring live traffic, or a fixed suite before you ship? Online vs offline is the first split. See LLM evaluation.
- Do you need a trace of tool calls and retrieval, or only one prompt and one reply?
- Must the suite live in git and run on every pull request?
- Do you have to self-host, or is a vendor cloud acceptable?
When you need a platform
Langfuse (MIT core, Cloud Hobby free, paid Cloud plans from $29/month, self-host free), LangSmith (Developer $0/seat for one seat, Plus $39/seat/month), and Braintrust (Starter $0/month with limits, Pro $249/month) are the three we document with official prices as of 19 November 2026. Pick among them on Langfuse vs LangSmith vs Braintrust — for ecosystem, self-hosting, and plan shape, not for a Litmiq score.
When you need a CI framework
Promptfoo (MIT, Community listed as Free Forever on its pricing page, 19 Nov 2026) is typically a YAML config and a CLI. DeepEval (Apache-2.0) is typically tests as code. Ragas is a Python library aimed at RAG metrics. See open-source LLM evaluation frameworks and Promptfoo vs DeepEval.
When a browser tester is enough
Litmiq runs a prompt and cases in your browser with your own key. Use it to draft the suite, compare wordings, and re-check after a model change (for example the 11 December 2026 GPT-5 shutdown — see GPT-5 shutdown). Then move the same cases into a framework or a platform. Litmiq is not a cheaper Langfuse, LangSmith or Braintrust.
Limits of this guide
We make Litmiq. Prices and plan names are from official pages read on 19 November 2026 and may have changed. This page does not cover every vendor, and it is not a buying recommendation for a specific team.
FAQ
How do I choose an AI eval platform?
Start with the job. If you need production traces or online evaluation, choose a platform such as LangSmith, Langfuse or Braintrust. If you need evals as code in CI, choose a framework such as Promptfoo or DeepEval. If you only need a quick prompt check, a browser tester such as Litmiq is enough.
Which eval platform is best?
We do not publish a best-platform ranking or review scores. Compare official plan names and the jobs each tool is built for, then pick from that, not from a leaderboard.
Is Litmiq an eval platform?
No. Litmiq is a free browser app for single-turn prompt tests with your own key. It is not a tracing platform and not a full CI-first framework, though Litmiq Pro can export a GitHub workflow.
Where are current prices?
Official list prices for Langfuse, LangSmith, Braintrust and Promptfoo, read 19 November 2026, are cited on Langfuse vs LangSmith vs Braintrust and on the Promptfoo comparison and pricing pages linked from this guide.
What if I already use one tool?
Stay if it covers the job you actually have. Switch or add a narrower tool only when the job changes — for example, adding CI around a suite you drafted in the browser.
Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.
Sources
- Litmiq, Langfuse vs LangSmith vs Braintrust (official prices dated 19 Nov 2026).
- Litmiq, open-source LLM evaluation frameworks.
- Promptfoo, Pricing (checked 19 Nov 2026).
Try it in Litmiq
Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI
Litmiq Pro
Saved suites, CI on a schedule and regression alerts for $19/month. The app stays free.
See pricing →Related guides
Compare & reference
- Prompt testing tools compared
- LangSmith alternative
- LangSmith pricing
- Litmiq vs Promptfoo
- Litmiq vs OpenAI Playground
Share this guide