Litmiq vs Promptfoo: Browser Prompt Tests or CLI Evals?
On this page
Litmiq and Promptfoo both test prompts against a set of cases, but they are built for different moments. Litmiq is a free browser app: you paste a system prompt, add test cases, pick OpenAI models and get side-by-side outputs with LLM-judge scores, with nothing to install. Promptfoo is an open-source (MIT) command-line tool and library: you describe prompts, providers, test cases and assertions in a config file and run them locally or in CI. This page compares the two honestly, using Promptfoo's own documentation as checked on 11 November 2026.
The short answer
Use Litmiq when you want a quick, visual answer to "did my system prompt change make answers better or worse?" without installing anything or writing config. Use Promptfoo when tests need to live in your repository, run in CI on every pull request, cover providers beyond OpenAI and Anthropic, use code-based assertions, or include red-team scans. Many teams can use both: draft and explore cases in Litmiq, then move the cases that matter into Promptfoo as a merge gate.
Key numbers
- OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
- Only 2 of 12 open models (Gemma 4 26B-A4B, gpt-oss-120b) resisted all 8 prompt-injection cases; the weakest (GLM-4.7 Flash) resisted 1. (Litmiq LLM Reliability Benchmark, 12 Nov 2026)
What is Promptfoo?
Promptfoo describes itself as "an open-source CLI and library for evaluating and red-teaming LLM apps." You install it with npm, Homebrew or pip (or run it with npx), then define an evaluation in a declarative config file, usually promptfooconfig.yaml. The config lists prompts, providers (models), test cases and assertions. Running promptfoo eval produces a matrix of outputs, and promptfoo view opens a local web viewer to compare them.
Promptfoo's docs list support for OpenAI, Anthropic, Azure, Google, Hugging Face, open-source models such as Llama, and custom API providers. It adds caching, concurrency and live reload, and it has a dedicated red-teaming feature for scanning apps for vulnerabilities. It is MIT-licensed. On 9 March 2026 OpenAI announced it had agreed to acquire Promptfoo; both companies said the project stays open source, and the GitHub README now says "Promptfoo remains open source and MIT licensed."
What is Litmiq?
Litmiq is a free, browser-based prompt evaluation tool. There is no signup and no Litmiq backend for evals. You add your own OpenAI API key once; it is stored in your browser's localStorage and sent only to OpenAI's API. Each test case is a single user message with an optional expected behavior. Every case runs on every OpenAI model you select, the outputs line up side by side, and an LLM judge scores each output for accuracy, tone, helpfulness and conciseness from 0 to 10, with a short reason. If the judge's reply can't be read, the result is marked unscored rather than given a default number. Your setup autosaves in the browser, and you can export a run as JSON.
How do Litmiq and Promptfoo compare, feature by feature?
| Question | Litmiq | Promptfoo |
|---|---|---|
| What is it? | Browser app | CLI and Node.js library, with a local web viewer |
| Install | None; open litmiq.com/app | npm, Homebrew, pip or npx |
| How you define tests | Forms: system prompt, test cases, optional expected behavior, scoring criteria | Config file (YAML) or code: prompts, providers, tests, assertions |
| Models | OpenAI and Anthropic models listed in the app | Many providers, including OpenAI, Anthropic, Azure, Google and custom APIs |
| Scoring | LLM judge with four fixed 0–10 scores plus reasons, guided by your criteria | Assertions you choose: deterministic checks (equals, contains, regex, is-json), model-graded checks (llm-rubric, g-eval) and custom JavaScript or Python |
| CI / pull-request gates | No | Yes, designed for it |
| Red teaming | No attack library; you write your own cases (see the prompt injection playbook) | Yes, a dedicated red-team feature |
| Conversation shape | Single-turn: one system prompt and one user message per case | Configurable; its docs cover prompts, agents and RAG apps |
| Where it runs | Your browser; calls go from the browser to OpenAI | Your machine or CI runner; calls go from there to the providers |
| Cost | Free; OpenAI bills your key | Open source and free to run; providers bill your keys |
| License | Hosted free tool (not open source) | MIT |
Promptfoo facts above come from its intro docs and GitHub repository. Litmiq facts describe the app as it ships today.
When is Litmiq the better choice?
- You just edited a system prompt and want to check it in minutes. Paste the prompt, add five to ten cases, run them on two models and read the outputs next to each other.
- The person testing doesn't use a terminal. Product managers, support leads and prompt writers can run the same cases an engineer would, in a browser tab.
- You want judgment scores without writing assertions. Describe what good looks like in plain language and let the judge score against it. Read the LLM-as-a-judge guide for where judge scores go wrong.
- You want a ready-made starting suite. The templates page has illustrative suites that open in the app with the system prompt, scoring criteria and cases filled in.
- You don't want another service holding your prompts. Litmiq has no eval backend; the setup lives in your browser and requests go directly to OpenAI.
When is Promptfoo the better choice?
- Evals must block bad merges. If a prompt change should fail a pull request when a test fails, you need a CLI in CI. Litmiq doesn't do this.
- You test more than OpenAI. Comparing Claude, Gemini, open-weight models or your own API endpoint is Promptfoo territory.
- You need exact checks. Format, JSON schema, string and regex checks are better as code than as a judge's opinion. Our JSON extraction guide explains why.
- You need security testing. Promptfoo's red-team feature generates attacks; Litmiq only runs the cases you write.
- Your tests should be versioned with code. A config file in the repo gives you diffs, review and history for free.
Can you use both together?
Yes, and the two fit a natural order. Start in Litmiq while the prompt is still changing every hour: write cases, look at real outputs and learn which behaviors matter. Once a case has caught a real problem, it has earned a place in your permanent suite. Move those cases into a Promptfoo config, turn the expected behaviors into assertions (exact checks where possible, model-graded checks where not) and run them in CI. Keep using the browser for exploration and the CLI for gates.
A Litmiq test case maps cleanly onto a Promptfoo test: the user message becomes a test variable, the system prompt becomes part of the prompt, and the expected behavior becomes an assertion. Expected behaviors written in the Must / Must not / Okay style from our expected behavior guide are the easiest to convert, because each line is already a checkable rule.
What does the same test look like in each tool?
Take one support-bot case: the user asks for a refund 90 days after purchase, and the policy allows 30 days. In Litmiq you would type the user message "I bought this 90 days ago, refund me." into a test case and write the expected behavior "Must say the 30-day window has passed. Must not promise a refund. Should offer the next step." Then you pick models and click Run.
In Promptfoo the same test lives in a config file: the prompt template holds the system prompt, the test sets a variable for the user message, and an assertion checks the behavior. An llm-rubric assertion can hold the same sentence as the expected behavior, and a deterministic assertion such as not-icontains can check that a phrase like "refund approved" never appears. The idea is identical; the difference is whether you click or commit.
What does each tool cost to run?
Neither tool charges for running evals itself. Litmiq is free and has no plans or quotas of its own; OpenAI bills your key for every model call and judge call. Promptfoo is open source and free to run; model providers bill your keys. In both, the real cost is model calls. In Litmiq, each case runs once per selected model, and each output gets up to one judge call, so 10 cases on 2 models is 20 model calls plus up to 20 judge calls.
Coming from OpenAI Evals? The migration guide generates a promptfooconfig.yaml from your eval export and maps each OpenAI grader to a Promptfoo assertion.
Have a promptfooconfig.yaml? Open it in your browser with no install, or add a Run in Litmiq badge to the repo. Going the other way, the app exports a promptfooconfig.yaml for CI.
Limits of this comparison
- We make Litmiq. We've tried to describe Promptfoo by its own documentation, not by guesses, but read its docs before deciding.
- Tools change quickly. Promptfoo facts were checked on 11 November 2026; anything after that date may differ.
- We don't claim rankings, market share or review scores, and we don't publish benchmark numbers for either tool.
- Litmiq is intentionally narrow: OpenAI and Anthropic models only, single-turn, browser-only, no CI. If you need those, that is a reason to choose Promptfoo, not a gap we're hiding.
- We are not affiliated with Promptfoo or OpenAI.
FAQ
Is Litmiq a Promptfoo alternative?
For one job, yes: quick side-by-side prompt checks with judge scores, in a browser, on OpenAI and Anthropic models. It is not a replacement for Promptfoo's CI gates, multi-provider support or red teaming.
Is Promptfoo still open source after the OpenAI acquisition?
Yes, according to both companies. OpenAI announced the agreement on 9 March 2026, and the Promptfoo README says it "remains open source and MIT licensed."
Do I need to write code to use Litmiq?
No. You fill in a system prompt, test cases and optional scoring criteria in the browser. Promptfoo needs a config file and a terminal.
Can I move my Litmiq test cases into Promptfoo?
Yes. Export a run as JSON from Litmiq; each case's input and expected behavior map to a Promptfoo test variable and an assertion. There is no automatic converter, so you write the config yourself.
Does either tool see my prompts?
Litmiq has no eval backend: requests go from your browser directly to OpenAI. Promptfoo says its evals run locally and talk directly to the model providers.
Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.
Sources
- Promptfoo, Intro (checked 11 Nov 2026).
- Promptfoo, Assertions & metrics (checked 11 Nov 2026).
- Promptfoo, GitHub repository and README, MIT license (checked 11 Nov 2026).
- OpenAI, OpenAI to acquire Promptfoo, 9 Mar 2026.
- Promptfoo, Promptfoo is joining OpenAI, 9 Mar 2026.
Get model deprecation alerts
We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.
We store only your email, through our form provider. See the privacy policy.
Try it in Litmiq
Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI
Litmiq Pro
Saved suites, CI on a schedule and regression alerts for $19/month. The app stays free.
See pricing →Related guides
Compare & reference
Share this guide