How to test prompts

How to test prompts in GitHub Actions (drafted in Litmiq)

On this page

The short answer

  • There is no official GitHub Actions integration and no Litmiq package to install.
  • You test the prompt your app already uses: copy it, add cases, run them in the browser with your own key.
  • For CI, export promptfooconfig.yaml and run it in your own workflow. Litmiq does not receive that workflow or its secrets.

Litmiq is a browser app. It does not run inside GitHub Actions, is not a GitHub App, and does not receive your repository secrets. There is no official Litmiq Action to install.

The workflow that does work is: draft and review cases in Litmiq, export them, and run the export in CI with a tool that already has a runner, such as Promptfoo.

1. Draft the suite in the browser

Use a template or your own cases. Run it once yourself so you trust the expected behavior. Export promptfooconfig.yaml (or JSON) from the app.

2. Store the file in the repo

Commit the exported config next to the prompt it tests. On pull requests, a Run in Litmiq badge can point reviewers at /app?suite= plus the raw file URL, so they can try it in a browser without installing anything. The CI job is separate and does not depend on that badge.

3. Run the export in Actions

The prompt testing in CI/CD guide shows how to generate a workflow that runs the YAML with a CI runner, using a secret you already manage in GitHub (your provider API key). That key never goes to Litmiq. Fail the job on assertion failures or a threshold you choose.

Limits

A browser LLM judge is not a substitute for deterministic CI assertions. Prefer code checks for JSON, length and forbidden strings, and keep the judge for tone and helpfulness if you use one in CI at all.

FAQ

Is there an official GitHub Actions integration?

No. This page describes a manual workflow using Litmiq features that already exist: the app, suite links and YAML export.

Will Litmiq call my app?

No. You copy a prompt and cases out of the app, or you point /app?suite= at a file you host.

Where do my keys go?

In Litmiq, they stay in your browser and are sent only to OpenAI or Anthropic. In CI, they stay in your own secrets store.

Open a starter suite

Load an illustrative pack, then replace the fictional facts with your prompt.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn