Promptfoo vs DeepEval

On this page

LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.

The short answer

Promptfoo and DeepEval are both open-source tools for running LLM evals in your own repo and in CI. They are not the same shape. Promptfoo is typically a YAML config (promptfooconfig.yaml) and a CLI (promptfoo eval), MIT licensed, with Community listed as Free Forever on its pricing page (checked 19 November 2026). DeepEval is typically tests as code in Python or TypeScript, run with deepeval test run, Apache-2.0.

Litmiq is a third, narrower option: a no-install browser tester. It is the step before either framework, not a replacement for CI. See Litmiq vs Promptfoo and Litmiq vs DeepEval.

When to choose Promptfoo

  • You want a config file and a matrix of prompts, providers and assertions, rather than test classes.
  • You want the Community plan's published scope, including evaluation features and up to 10,000 red-team probes a month (Promptfoo pricing page, 19 Nov 2026).
  • You already think in YAML, or you will export a promptfooconfig.yaml from Litmiq.

When to choose DeepEval

  • You want evals to look like the rest of your unit tests, in Python or TypeScript.
  • Your team already reviews test files in pull requests and wants the same for prompts.

When neither is the first step

If you do not yet have a stable suite, draft the cases in the browser first. If you need production traces, use a platform instead of either CLI. OpenAI announced an agreement to acquire Promptfoo on 9 March 2026; both companies said the project remains open source and MIT licensed (sources on the Promptfoo comparison).

Limits of this comparison

We make Litmiq and we have not benchmarked Promptfoo against DeepEval. Promptfoo plan names were rechecked on 19 November 2026. DeepEval is described from our existing comparison page. We are not affiliated with either project. Check their current docs before you standardise on one.

FAQ

What is the difference between Promptfoo and DeepEval?

Both are open source and can run in CI. Promptfoo is typically a YAML config and a CLI (promptfoo eval), MIT licensed. DeepEval is typically tests as code in Python or TypeScript (deepeval test run), Apache-2.0.

Is Promptfoo free?

Yes, the project is MIT-licensed. Promptfoo's pricing page, checked 19 November 2026, lists Community as Free Forever, including evaluation features and up to 10,000 red-team probes a month. Enterprise and On-Premise are custom-priced.

Which should I use?

Choose Promptfoo if you want a config-file matrix and a CLI. Choose DeepEval if you want evals to look like the rest of your unit tests. We do not publish a benchmark of one against the other.

Where does Litmiq fit?

Litmiq is a no-install browser tester for drafting and comparing prompts. Use it first, then move a stable suite into Promptfoo or DeepEval for CI.

Is Promptfoo still open source after the OpenAI agreement?

OpenAI announced an agreement to acquire Promptfoo on 9 March 2026. Both companies said the project remains open source and MIT licensed. Sources are on the Litmiq vs Promptfoo page.

Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.

Sources

Try it in Litmiq

Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn