Litmiq vs Promptfoo: Browser Prompt Tests or CLI Evals?

On this page

Litmiq and Promptfoo both test prompts against a set of cases, but they are built for different moments. Litmiq is a free browser app: you paste a system prompt, add test cases, pick OpenAI models and get side-by-side outputs with LLM-judge scores, with nothing to install. Promptfoo is an open-source (MIT) command-line tool and library: you describe prompts, providers, test cases and assertions in a config file and run them locally or in CI. This page compares the two honestly, using Promptfoo's own documentation as checked on 11 November 2026.

The short answer

Use Litmiq when you want a quick, visual answer to "did my system prompt change make answers better or worse?" without installing anything or writing config. Use Promptfoo when tests need to live in your repository, run in CI on every pull request, cover providers beyond OpenAI and Anthropic, use code-based assertions, or include red-team scans. Many teams can use both: draft and explore cases in Litmiq, then move the cases that matter into Promptfoo as a merge gate.

Key numbers

  • OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
  • Only 2 of 12 open models (Gemma 4 26B-A4B, gpt-oss-120b) resisted all 8 prompt-injection cases; the weakest (GLM-4.7 Flash) resisted 1. (Litmiq LLM Reliability Benchmark, 12 Nov 2026)

What is Promptfoo?

Promptfoo describes itself as "an open-source CLI and library for evaluating and red-teaming LLM apps." You install it with npm, Homebrew or pip (or run it with npx), then define an evaluation in a declarative config file, usually promptfooconfig.yaml. The config lists prompts, providers (models), test cases and assertions. Running promptfoo eval produces a matrix of outputs, and promptfoo view opens a local web viewer to compare them.

Promptfoo's docs list support for OpenAI, Anthropic, Azure, Google, Hugging Face, open-source models such as Llama, and custom API providers. It adds caching, concurrency and live reload, and it has a dedicated red-teaming feature for scanning apps for vulnerabilities. It is MIT-licensed. On 9 March 2026 OpenAI announced it had agreed to acquire Promptfoo; both companies said the project stays open source, and the GitHub README now says "Promptfoo remains open source and MIT licensed."

What is Litmiq?

Litmiq is a free, browser-based prompt evaluation tool. There is no signup and no Litmiq backend for evals. You add your own OpenAI API key once; it is stored in your browser's localStorage and sent only to OpenAI's API. Each test case is a single user message with an optional expected behavior. Every case runs on every OpenAI model you select, the outputs line up side by side, and an LLM judge scores each output for accuracy, tone, helpfulness and conciseness from 0 to 10, with a short reason. If the judge's reply can't be read, the result is marked unscored rather than given a default number. Your setup autosaves in the browser, and you can export a run as JSON.

How do Litmiq and Promptfoo compare, feature by feature?

QuestionLitmiqPromptfoo
What is it?Browser appCLI and Node.js library, with a local web viewer
InstallNone; open litmiq.com/appnpm, Homebrew, pip or npx
How you define testsForms: system prompt, test cases, optional expected behavior, scoring criteriaConfig file (YAML) or code: prompts, providers, tests, assertions
ModelsOpenAI and Anthropic models listed in the appMany providers, including OpenAI, Anthropic, Azure, Google and custom APIs
ScoringLLM judge with four fixed 0–10 scores plus reasons, guided by your criteriaAssertions you choose: deterministic checks (equals, contains, regex, is-json), model-graded checks (llm-rubric, g-eval) and custom JavaScript or Python
CI / pull-request gatesNoYes, designed for it
Red teamingNo attack library; you write your own cases (see the prompt injection playbook)Yes, a dedicated red-team feature
Conversation shapeSingle-turn: one system prompt and one user message per caseConfigurable; its docs cover prompts, agents and RAG apps
Where it runsYour browser; calls go from the browser to OpenAIYour machine or CI runner; calls go from there to the providers
CostFree; OpenAI bills your keyOpen source and free to run; providers bill your keys
LicenseHosted free tool (not open source)MIT

Promptfoo facts above come from its intro docs and GitHub repository. Litmiq facts describe the app as it ships today.

When is Litmiq the better choice?

When is Promptfoo the better choice?

Can you use both together?

Yes, and the two fit a natural order. Start in Litmiq while the prompt is still changing every hour: write cases, look at real outputs and learn which behaviors matter. Once a case has caught a real problem, it has earned a place in your permanent suite. Move those cases into a Promptfoo config, turn the expected behaviors into assertions (exact checks where possible, model-graded checks where not) and run them in CI. Keep using the browser for exploration and the CLI for gates.

A Litmiq test case maps cleanly onto a Promptfoo test: the user message becomes a test variable, the system prompt becomes part of the prompt, and the expected behavior becomes an assertion. Expected behaviors written in the Must / Must not / Okay style from our expected behavior guide are the easiest to convert, because each line is already a checkable rule.

What does the same test look like in each tool?

Take one support-bot case: the user asks for a refund 90 days after purchase, and the policy allows 30 days. In Litmiq you would type the user message "I bought this 90 days ago, refund me." into a test case and write the expected behavior "Must say the 30-day window has passed. Must not promise a refund. Should offer the next step." Then you pick models and click Run.

In Promptfoo the same test lives in a config file: the prompt template holds the system prompt, the test sets a variable for the user message, and an assertion checks the behavior. An llm-rubric assertion can hold the same sentence as the expected behavior, and a deterministic assertion such as not-icontains can check that a phrase like "refund approved" never appears. The idea is identical; the difference is whether you click or commit.

What does each tool cost to run?

Neither tool charges for running evals itself. Litmiq is free and has no plans or quotas of its own; OpenAI bills your key for every model call and judge call. Promptfoo is open source and free to run; model providers bill your keys. In both, the real cost is model calls. In Litmiq, each case runs once per selected model, and each output gets up to one judge call, so 10 cases on 2 models is 20 model calls plus up to 20 judge calls.

Coming from OpenAI Evals? The migration guide generates a promptfooconfig.yaml from your eval export and maps each OpenAI grader to a Promptfoo assertion.

Have a promptfooconfig.yaml? Open it in your browser with no install, or add a Run in Litmiq badge to the repo. Going the other way, the app exports a promptfooconfig.yaml for CI.

Limits of this comparison

FAQ

Is Litmiq a Promptfoo alternative?

For one job, yes: quick side-by-side prompt checks with judge scores, in a browser, on OpenAI and Anthropic models. It is not a replacement for Promptfoo's CI gates, multi-provider support or red teaming.

Is Promptfoo still open source after the OpenAI acquisition?

Yes, according to both companies. OpenAI announced the agreement on 9 March 2026, and the Promptfoo README says it "remains open source and MIT licensed."

Do I need to write code to use Litmiq?

No. You fill in a system prompt, test cases and optional scoring criteria in the browser. Promptfoo needs a config file and a terminal.

Can I move my Litmiq test cases into Promptfoo?

Yes. Export a run as JSON from Litmiq; each case's input and expected behavior map to a Promptfoo test variable and an assertion. There is no automatic converter, so you write the config yourself.

Does either tool see my prompts?

Litmiq has no eval backend: requests go from your browser directly to OpenAI. Promptfoo says its evals run locally and talk directly to the model providers.

Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.

Sources

Get model deprecation alerts

We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.

We store only your email, through our form provider. See the privacy policy.

Try it in Litmiq

Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn