Open-source LLM evaluation frameworks
On this page
LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.
The short answer
Open-source LLM evaluation frameworks let you keep test cases and scorers in your own repo and run them locally or in CI. This page compares three that we already document from their public materials — Promptfoo, DeepEval and Ragas — and says honestly where Litmiq, a free browser tester, does and does not belong on the same list.
When to choose a framework instead of a browser tester
- You want evals as code or as a config file, reviewed in pull requests.
- You need many model providers, red teaming, or RAG-specific metrics in Python.
- You want the same suite to run on every commit.
Promptfoo
Promptfoo is an MIT-licensed CLI and library for evaluation and red-teaming. You typically write a promptfooconfig.yaml, run promptfoo eval, and inspect the matrix in a local viewer. Its pricing page, checked 19 November 2026, lists Community as Free Forever (including all LLM evaluation features, all model providers, red teaming with up to 10,000 probes a month, and self-hosting) and Enterprise and On-Premise at custom pricing. See Litmiq vs Promptfoo and Promptfoo alternative.
DeepEval
DeepEval is open source under Apache-2.0. You write eval tests in Python or TypeScript and run them with deepeval test run, locally or in CI. That is the unit-test shape of LLM evals. Litmiq needs no code and is the step before you write those tests, not a replacement for them. See Litmiq vs DeepEval.
Ragas
Ragas is an open-source Python library (Apache License 2.0) for evaluating RAG and LLM applications. Its documentation describes experiments, datasets and metrics, including RAG-oriented measures such as faithfulness and context precision, and integrations with frameworks such as LangChain and LlamaIndex. You install it with pip and run it in your own environment. See Litmiq vs Ragas and RAG evaluation.
Where Litmiq fits
Litmiq is free in the browser and needs no install, but it is not an open-source CLI framework. Use it to draft cases and compare prompt wording quickly, then export (including a promptfooconfig.yaml) into Promptfoo, or rewrite the cases as DeepEval or Ragas tests. Litmiq Pro can export a GitHub workflow that re-runs saved suites; that is still not a full framework.
Limits of this comparison
We make Litmiq. Promptfoo, DeepEval and Ragas are described from their public docs and from our existing comparison pages; Promptfoo's plan names were rechecked on its pricing page on 19 November 2026. We don't publish rankings or benchmarks of these frameworks against each other. We are not affiliated with them. OpenAI announced an agreement to acquire Promptfoo on 9 March 2026; both companies said the project remains open source and MIT licensed (sources on the Promptfoo comparison).
FAQ
What are open-source LLM evaluation frameworks?
They are libraries or CLIs you can run yourself to store cases, call models and score outputs. This page compares Promptfoo, DeepEval and Ragas, which are open source, and Litmiq, which is a free browser app but not an open-source CLI framework.
Is Promptfoo free and open source?
Yes, the project is MIT-licensed. Promptfoo's pricing page, checked 19 November 2026, lists Community as Free Forever, including evaluation features and up to 10,000 red-team probes a month. Enterprise and On-Premise are custom-priced.
How do Promptfoo and DeepEval differ?
Both are open source and can run in CI. Promptfoo is typically a YAML config and a CLI (promptfoo eval). DeepEval is typically tests as code in Python or TypeScript (deepeval test run), Apache-2.0. Litmiq is the no-install browser step before either. See also Litmiq vs Promptfoo and Litmiq vs DeepEval.
Where does Ragas fit?
Ragas is an open-source Python library (Apache-2.0) aimed at evaluating RAG and LLM applications, with documented metrics such as faithfulness and context precision. It is not a general browser prompt tester and not a tracing platform.
Is Litmiq open source?
The Litmiq app runs for free in your browser with your own key; this page does not claim Litmiq is an open-source eval framework. If you need evals as code, use Promptfoo, DeepEval or Ragas. Litmiq can export a promptfooconfig.yaml to help you move cases.
Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.
Sources
- Promptfoo, Pricing (checked 19 Nov 2026); Intro and GitHub README (MIT, checked 11 Nov 2026) as cited on Litmiq vs Promptfoo.
- Litmiq, Litmiq vs DeepEval (DeepEval described as Apache-2.0,
deepeval test run). - Litmiq, Litmiq vs Ragas (Ragas described from its public docs as Apache-2.0, with faithfulness and context precision among its metrics).
Try it in Litmiq
Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI
Litmiq Pro
Saved suites, CI on a schedule and regression alerts for $19/month. The app stays free.
See pricing →Related guides
Compare & reference
- Prompt testing tools compared
- LangSmith alternative
- LangSmith pricing
- Litmiq vs Promptfoo
- Litmiq vs OpenAI Playground
Share this guide