Tool · free
Promptfoo without the CLI: run a promptfooconfig.yaml in your browser
On this page
The short answer
- Promptfoo is a CLI-first, MIT-licensed eval tool: you install it, run
promptfoo eval, then view results in a local web viewer. - Ownership: OpenAI announced an agreement to acquire Promptfoo on 9 March 2026 (Promptfoo blog); Promptfoo's README now says it is part of OpenAI and remains open source and MIT-licensed.
- To run a
promptfooconfig.yamlwith nothing installed, paste it below. Litmiq converts the prompt, tests and assertions in your browser and opens them in the app. - Deterministic assertions become plain-language rules for Litmiq's LLM judge. Code assertions are flagged and never run.
- Keep CI in Promptfoo; use this for a quick look, for reviewers without a terminal, or for a Run in Litmiq badge on your repo.
Key numbers
- OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
- Re-running the 11 security cases gave the same pass/fail on 130 of 132 model–case pairs at temperature 0; 2 flipped. Run important cases more than once. (Litmiq LLM Reliability Benchmark, 12 Nov 2026)
Import your promptfooconfig.yaml
Config on GitHub? Skip the paste: litmiq.com/app?suite=<raw URL> loads it directly. Open Promptfoo's getting-started example that way, or make a badge on Run in Litmiq.
What converts
- Prompts. The first inline prompt. A chat prompt (a JSON list of messages) keeps its system message as the system prompt and its last user message as each case's input;
{{ var | dump }}is handled. A plain-text prompt with variables becomes the system prompt, and the variable values are sent as the user message. - Tests. Each entry in
tests, withdefaultTestvars and assertions merged in. List-valued vars expand into one case per value, as in Promptfoo. Up to 60 cases. - Providers.
openai:providers that match a model in the app are preselected. - Assertions, as in this table:
| Promptfoo assertion | What it checks | In Litmiq |
|---|---|---|
equals | Output is exactly the value | Rule: “Output is exactly …”, scored by the judge |
contains / icontains | Substring, case-sensitive / any case | Rule: “Output contains …” |
contains-any, contains-all (and i variants) | Any / all of a list | Rule listing the values |
starts-with, regex | Prefix, regular expression | Rule quoting the prefix or pattern |
is-json, contains-json | Valid JSON, optional JSON schema | Rule: “valid JSON and nothing else”, schema quoted |
is-xml, is-sql, is-html (and contains-) | Format validity | Rule naming the format |
llm-rubric, model-graded-closedqa, g-eval | Model-graded rubric | The rubric text, as written |
factuality, similar, levenshtein, rouge-n, bleu… | Closeness to a reference | Rule quoting the reference answer |
not-… on any of the above | Negation | Rule prefixed with “NOT:” |
assert-set | Group of assertions | Rules joined with AND |
javascript, python, ruby, webhook | Your code returns a score | Not converted: Litmiq never runs code from a suite |
cost, latency, perplexity, tool-call checks | Metrics on the API response | Not converted (no equivalent) |
Assertion types as documented in Promptfoo's assertions reference, read 13 Nov 2026.
What doesn't convert
- Anything from a file:
file://prompts, tests and values, and globs. A browser can't read your disk, so paste them inline. - Code:
javascript,python,rubyandwebhookassertions, transforms and custom providers. - Several prompts at once: Litmiq tests one system prompt per run, so the first is used. Re-import with another one first to compare.
- Red teaming and
scenarios. For injection tests, start from the prompt injection test cases. - Other providers. Litmiq runs the OpenAI and Anthropic models in the app.
Is there a Promptfoo GUI?
For results, yes. Promptfoo's docs describe a web viewer you open with npx promptfoo@latest view after running an eval. It has filters for failures and differences, ratings and comments, side-by-side comparison of evals, an eval creator to edit and re-run, exports, and share links hosted at promptfoo.app or self-hosted (Promptfoo web viewer). It still starts from the CLI on your machine.
Litmiq is a different tool, not a Promptfoo front end: no install, an LLM judge instead of assertions, OpenAI and Anthropic models only. It's useful when the person checking a prompt doesn't have Node.js, or when you want a reviewer to try a config from a link. For a full comparison, see Litmiq vs Promptfoo.
Use both: browser for drafts, Promptfoo for gates
- Draft cases in Litmiq and read the outputs side by side.
- Export with the app's Export Promptfoo YAML button: every case becomes a test with an
llm-rubricassertion. - Tighten it by replacing rubrics with
equals,containsoris-jsonwhere the rule is exact. - Gate pull requests with the workflow on prompt testing in CI/CD, and add a Run in Litmiq badge so anyone can open the same suite in a browser.
Limits
- Converted assertions are judged by an LLM, not checked exactly. Expect close, not identical, results.
- The converter reads the config structure documented by Promptfoo as of 13 Nov 2026. Unusual configs may need hand edits; every conversion note is shown above the preview.
- Litmiq is not affiliated with Promptfoo or OpenAI.
FAQ
Can I use Promptfoo without the command line?
Promptfoo itself is installed and run from a terminal (npx, npm, Homebrew or pip), and its web viewer opens after you run an eval with the CLI. To run a promptfooconfig.yaml with no install, import it into Litmiq: it converts the prompts, tests and common assertions in your browser and runs them on OpenAI and Anthropic models with your own key.
Is there a Promptfoo GUI?
Yes, for results: promptfoo view opens a local web viewer where you can filter, rate, compare and share evals, and re-run from its eval creator. It is started from the CLI. Shared evals can be hosted at promptfoo.app or self-hosted. Litmiq is a separate, unaffiliated browser app.
Does Litmiq run my javascript or python assertions?
No. Litmiq never executes code from a suite. Code assertions are listed as not converted, so you can describe what they check in the expected behavior or keep them in Promptfoo.
Will the scores match Promptfoo's?
Not exactly. Promptfoo checks equals, contains, regex and is-json deterministically. Litmiq turns them into rules that an LLM judge scores, which is close but not deterministic. Use Litmiq for a quick browser check and Promptfoo where you need exact pass/fail.
Is my config uploaded?
No. The importer runs in your browser. Open in Litmiq saves the suite in your browser's localStorage; a share link carries it after the # sign, which browsers don't send to servers.
Which providers are supported?
Litmiq runs the OpenAI models offered in the app. openai: providers in your config that match those models are preselected; other providers are ignored.
Sources (read 13 Nov 2026)
- Promptfoo: Using the web viewer:
promptfoo view, cell actions, eval creator, sharing. - Promptfoo: Assertions and metrics: deterministic, model-graded and code assertion types.
- Promptfoo getting-started example and repository (MIT): the sample config loaded by the first button.
Get model deprecation alerts
We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.
We store only your email, through our form provider. See the privacy policy.
Run your Promptfoo tests in a browser
Paste a config above, or open the getting-started example directly in the app.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI