Tool · free

Promptfoo without the CLI: run a promptfooconfig.yaml in your browser

On this page

The short answer

  • Promptfoo is a CLI-first, MIT-licensed eval tool: you install it, run promptfoo eval, then view results in a local web viewer.
  • Ownership: OpenAI announced an agreement to acquire Promptfoo on 9 March 2026 (Promptfoo blog); Promptfoo's README now says it is part of OpenAI and remains open source and MIT-licensed.
  • To run a promptfooconfig.yaml with nothing installed, paste it below. Litmiq converts the prompt, tests and assertions in your browser and opens them in the app.
  • Deterministic assertions become plain-language rules for Litmiq's LLM judge. Code assertions are flagged and never run.
  • Keep CI in Promptfoo; use this for a quick look, for reviewers without a terminal, or for a Run in Litmiq badge on your repo.

Key numbers

  • OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
  • Re-running the 11 security cases gave the same pass/fail on 130 of 132 model–case pairs at temperature 0; 2 flipped. Run important cases more than once. (Litmiq LLM Reliability Benchmark, 12 Nov 2026)

Import your promptfooconfig.yaml

Paste a promptfooconfig.yaml (or drop the file) to start. Nothing leaves this page.

Config on GitHub? Skip the paste: litmiq.com/app?suite=<raw URL> loads it directly. Open Promptfoo's getting-started example that way, or make a badge on Run in Litmiq.

What converts

Promptfoo assertions and what the Litmiq importer does with them
Promptfoo assertionWhat it checksIn Litmiq
equalsOutput is exactly the valueRule: “Output is exactly …”, scored by the judge
contains / icontainsSubstring, case-sensitive / any caseRule: “Output contains …”
contains-any, contains-all (and i variants)Any / all of a listRule listing the values
starts-with, regexPrefix, regular expressionRule quoting the prefix or pattern
is-json, contains-jsonValid JSON, optional JSON schemaRule: “valid JSON and nothing else”, schema quoted
is-xml, is-sql, is-html (and contains-)Format validityRule naming the format
llm-rubric, model-graded-closedqa, g-evalModel-graded rubricThe rubric text, as written
factuality, similar, levenshtein, rouge-n, bleu…Closeness to a referenceRule quoting the reference answer
not-… on any of the aboveNegationRule prefixed with “NOT:”
assert-setGroup of assertionsRules joined with AND
javascript, python, ruby, webhookYour code returns a scoreNot converted: Litmiq never runs code from a suite
cost, latency, perplexity, tool-call checksMetrics on the API responseNot converted (no equivalent)

Assertion types as documented in Promptfoo's assertions reference, read 13 Nov 2026.

What doesn't convert

Is there a Promptfoo GUI?

For results, yes. Promptfoo's docs describe a web viewer you open with npx promptfoo@latest view after running an eval. It has filters for failures and differences, ratings and comments, side-by-side comparison of evals, an eval creator to edit and re-run, exports, and share links hosted at promptfoo.app or self-hosted (Promptfoo web viewer). It still starts from the CLI on your machine.

Litmiq is a different tool, not a Promptfoo front end: no install, an LLM judge instead of assertions, OpenAI and Anthropic models only. It's useful when the person checking a prompt doesn't have Node.js, or when you want a reviewer to try a config from a link. For a full comparison, see Litmiq vs Promptfoo.

Use both: browser for drafts, Promptfoo for gates

  1. Draft cases in Litmiq and read the outputs side by side.
  2. Export with the app's Export Promptfoo YAML button: every case becomes a test with an llm-rubric assertion.
  3. Tighten it by replacing rubrics with equals, contains or is-json where the rule is exact.
  4. Gate pull requests with the workflow on prompt testing in CI/CD, and add a Run in Litmiq badge so anyone can open the same suite in a browser.

Limits

FAQ

Can I use Promptfoo without the command line?

Promptfoo itself is installed and run from a terminal (npx, npm, Homebrew or pip), and its web viewer opens after you run an eval with the CLI. To run a promptfooconfig.yaml with no install, import it into Litmiq: it converts the prompts, tests and common assertions in your browser and runs them on OpenAI and Anthropic models with your own key.

Is there a Promptfoo GUI?

Yes, for results: promptfoo view opens a local web viewer where you can filter, rate, compare and share evals, and re-run from its eval creator. It is started from the CLI. Shared evals can be hosted at promptfoo.app or self-hosted. Litmiq is a separate, unaffiliated browser app.

Does Litmiq run my javascript or python assertions?

No. Litmiq never executes code from a suite. Code assertions are listed as not converted, so you can describe what they check in the expected behavior or keep them in Promptfoo.

Will the scores match Promptfoo's?

Not exactly. Promptfoo checks equals, contains, regex and is-json deterministically. Litmiq turns them into rules that an LLM judge scores, which is close but not deterministic. Use Litmiq for a quick browser check and Promptfoo where you need exact pass/fail.

Is my config uploaded?

No. The importer runs in your browser. Open in Litmiq saves the suite in your browser's localStorage; a share link carries it after the # sign, which browsers don't send to servers.

Which providers are supported?

Litmiq runs the OpenAI models offered in the app. openai: providers in your config that match those models are preselected; other providers are ignored.

Sources (read 13 Nov 2026)

Get model deprecation alerts

We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.

We store only your email, through our form provider. See the privacy policy.

Run your Promptfoo tests in a browser

Paste a config above, or open the getting-started example directly in the app.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn