How to test prompts
How to test prompts in GitHub Actions (drafted in Litmiq)
On this page
The short answer
- There is no official GitHub Actions integration and no Litmiq package to install.
- You test the prompt your app already uses: copy it, add cases, run them in the browser with your own key.
- For CI, export promptfooconfig.yaml and run it in your own workflow. Litmiq does not receive that workflow or its secrets.
Litmiq is a browser app. It does not run inside GitHub Actions, is not a GitHub App, and does not receive your repository secrets. There is no official Litmiq Action to install.
The workflow that does work is: draft and review cases in Litmiq, export them, and run the export in CI with a tool that already has a runner, such as Promptfoo.
1. Draft the suite in the browser
Use a template or your own cases. Run it once yourself so you trust the expected behavior. Export promptfooconfig.yaml (or JSON) from the app.
2. Store the file in the repo
Commit the exported config next to the prompt it tests. On pull requests, a Run in Litmiq badge can point reviewers at /app?suite= plus the raw file URL, so they can try it in a browser without installing anything. The CI job is separate and does not depend on that badge.
3. Run the export in Actions
The prompt testing in CI/CD guide shows how to generate a workflow that runs the YAML with a CI runner, using a secret you already manage in GitHub (your provider API key). That key never goes to Litmiq. Fail the job on assertion failures or a threshold you choose.
Limits
A browser LLM judge is not a substitute for deterministic CI assertions. Prefer code checks for JSON, length and forbidden strings, and keep the judge for tone and helpfulness if you use one in CI at all.
FAQ
Is there an official GitHub Actions integration?
No. This page describes a manual workflow using Litmiq features that already exist: the app, suite links and YAML export.
Will Litmiq call my app?
No. You copy a prompt and cases out of the app, or you point /app?suite= at a file you host.
Where do my keys go?
In Litmiq, they stay in your browser and are sent only to OpenAI or Anthropic. In CI, they stay in your own secrets store.
Open a starter suite
Load an illustrative pack, then replace the fictional facts with your prompt.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI