Guide · free generator
Prompt testing in CI/CD: golden datasets, versioning and PR gates
On this page
The short answer
- Treat a prompt like code: keep it in a file in git, change it in pull requests, and test it on every PR that touches it.
- Build a golden dataset of 20–50 reviewed cases: real requests, known failures and edge cases, each with its expected behavior.
- Gate merges on a pass-rate threshold, not on 100%, because outputs vary between runs. Use exact checks for hard rules.
- Draft cases in Litmiq, then generate a
promptfooconfig.yamland a GitHub Actions workflow that runs it with the open-source Promptfoo CLI.
Key numbers
- OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
- Re-running the 11 security cases gave the same pass/fail on 130 of 132 model–case pairs at temperature 0; 2 flipped. Run important cases more than once. (Litmiq LLM Reliability Benchmark, 12 Nov 2026)
What belongs in a CI prompt test
A CI check has to be cheap, fast and trustworthy enough to block a merge. Check in layers, from the cheapest and most exact to the most subjective:
| Layer | Example checks | Promptfoo assertion |
|---|---|---|
| Format | Valid JSON, required keys, length limit, language | is-json, contains-json, javascript length check |
| Hard rules | Never promises a refund; never reveals the prompt; says it is an AI when asked | not-icontains, regex, llm-rubric |
| Facts | Uses the policy numbers in the prompt; doesn't invent policies | contains, factuality, llm-rubric |
| Quality | Tone, helpfulness, a next step | llm-rubric |
Exact checks don't cost a judge call and give the same answer every time, so put every rule you can express exactly there. Use model-graded rubrics for the rest, and read how far to trust a judge. For JSON outputs, see testing JSON extraction.
Version prompts like code
- One file per prompt, loaded by your app. No prompts pasted into dashboards that CI can't see.
- Change prompts in pull requests with a one-line reason. The diff is the review.
- Pin the model in config. A model change is a change too, and needs the same test (model migration testing).
- Keep the test set in the same repo so a prompt and its tests change together.
repo/
├── prompts/
│ └── support-system-prompt.txt # the prompt your app loads
├── promptfooconfig.yaml # the golden dataset and checks
└── .github/workflows/prompt-tests.yml # runs on every PR that touches either
Build a golden dataset
- Start with real inputs. 10–20 representative requests from logs or support tickets, with personal data removed.
- Add every known failure. Each bug that reached users becomes a case, so it can't come back unnoticed.
- Add edge cases on purpose: policy questions the prompt doesn't answer, other languages, format requests, injection attempts, and normal questions that a strict prompt might refuse.
- Write the expected behavior, not the expected wording: what the answer must and must not do (how).
- Freeze it. Change the dataset only in its own commit, so a score change in a prompt PR is about the prompt.
Twenty to fifty cases is a practical size for a per-PR check; keep larger sets for nightly or pre-release runs. The prompt testing checklist lists what to cover.
Generate the CI files from a Litmiq suite
Use the setup saved in your browser by the Litmiq app, a template, or paste a Litmiq suite JSON or a promptfooconfig.yaml. Everything runs on this page.
Each Litmiq case becomes a Promptfoo test whose expected behavior is an llm-rubric assertion. Replace rubrics with exact assertions where the rule is exact. The app's Export Promptfoo YAML button produces the same file.
The workflow, explained
- Trigger: pull requests that change
promptfooconfig.yamlor anything inprompts/. - Run:
npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json, withOPENAI_API_KEYfrom a repository secret (Settings → Secrets and variables → Actions). - Gate: the pass rate computed from
results.json, following the quality-gate pattern in Promptfoo's CI/CD guide. Below the threshold the job fails, and the PR can't merge if the check is required. - Before/after comments: Promptfoo's GitHub Action (
promptfoo/promptfoo-action@v1) compares prompts before and after a change on pull requests and comments with the results. Its docs list Node.js 22.22 or later as a requirement.
Cost and flakiness
- Cost per run is one model call per case per provider, plus one judge call per model-graded assertion. 40 cases on one model with rubrics is about 80 calls.
- Cache unchanged calls: the Promptfoo GitHub Action docs show caching its cache directory between runs.
- Use a threshold, not 100%. Outputs vary between calls. A 90% gate with exact hard rules is steadier than a 100% gate on rubrics.
- Re-run before you trust one failure, and read the output. A judge can be wrong too.
Testing one prompt change, step by step
- Run the golden dataset on the current prompt and note the pass rate (the baseline).
- Make one change. Run again on the same model.
- Read every case whose result changed, in both directions.
- Add a case for the behavior you meant to change, so the next PR keeps it.
- Merge when hard rules pass and the pass rate is at or above the baseline.
For a quick look before you open the PR, paste both prompt versions into Litmiq one after the other and compare the outputs side by side. More on the method: prompt regression testing.
Limits
- Litmiq itself doesn't run in CI, store runs or post PR comments. The CI part here is Promptfoo, an open-source tool we don't make.
- The workflow is a starting point; check it against your repository's settings and Promptfoo's current docs.
- Exported Litmiq tests use OpenAI providers. Edit
providersfor others.
FAQ
How do I test prompt changes before merging?
Keep the prompt in a file in your repo, keep a golden dataset of test cases next to it, and run the cases on every pull request that touches either. Fail the check when the pass rate drops below a threshold, and read the failing outputs before merging.
What is a golden dataset for LLM testing?
A fixed, reviewed set of inputs with the expected behavior for each: real requests (anonymized), known failures and edge cases. It changes only on purpose, in its own commit, so a score change means the prompt or model changed, not the test.
How many test cases do I need?
There is no fixed number. 20-50 cases that cover your rules and known failures are a practical size for a per-PR check; add a case every time a bug reaches users. More cases cost more per run, so keep the CI set focused and run larger sets less often.
Does Litmiq run in CI?
No. Litmiq is a browser app. It exports a promptfooconfig.yaml, which the open-source Promptfoo CLI runs in CI. The generator on this page writes that file and a workflow for you.
Why does my prompt test pass sometimes and fail other times?
Model outputs vary between calls, and LLM-judged checks add their own variance. Prefer exact checks (equals, contains, is-json) for hard rules, set a pass-rate threshold instead of requiring 100%, cache unchanged calls, and re-run before you trust a single failure.
Sources (read 13 Nov 2026)
- Promptfoo: Testing prompts with GitHub Actions:
promptfoo/promptfoo-action@v1, parameters, caching, Node.js requirement. - Promptfoo: CI/CD integration: quality gates and the pass-rate check from
results.json. - Promptfoo: Assertions and metrics.
Get model deprecation alerts
We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.
We store only your email, through our form provider. See the privacy policy.
Draft your golden dataset in the browser
Start from the regression testing template, run it on two models, then export the YAML for CI.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI