Prompt Regression Testing: A Practical Guide

A prompt regression happens when a change that looks harmless makes outputs worse. It might be a reworded instruction, a new example, or a model upgrade. With normal code, a test suite catches regressions. Most teams ship prompt changes after reading two or three outputs by hand. This guide shows a lightweight way to do better without new infrastructure.

Why prompts regress

• Instruction interference. Adding "be concise" can quietly drop a required step, like offering a next action.
• Model swaps. A newer or cheaper model may follow formatting rules differently or refuse in different places.
• Edge cases you forgot. The prompt works on the happy path, but an angry user or an off-topic request now gets a bad answer.
• Non-determinism. The same prompt can produce different outputs on different runs, so a single good sample proves little.

A minimal workflow

1. Write down what "good" means as a few criteria, such as accuracy, tone, following policy and length.
2. Collect 5–20 test inputs: common requests, known edge cases, and inputs that broke things before.
3. For each input, describe the expected behavior in one sentence. Describe what the answer should do, not the exact wording.
4. Run the current prompt and save the results as your baseline.
5. Change the prompt or model, re-run the same set, and compare scores and outputs side by side.
6. Read every case whose score dropped. Scores tell you where to look; they don't replace reading.

Example: a support-bot prompt

Suppose your system prompt is:

You are a support agent for Acme, an online store. Be empathetic and concise.
Never promise refunds; explain the 30-day return policy and offer to start a return.
If the user asks about something unrelated to Acme, politely redirect.

You then add "Answer in two sentences or fewer" to cut length. Re-running the suite might show tone holding steady while the "offer a next step" behavior disappears on some cases. Without the suite you would not see that.

5 ready-to-copy test cases (illustrative)

These are illustrative examples for the prompt above. Adapt them to your own product and policies.

1. Input: "What's your return policy?"
   Expected: States the 30-day policy accurately and offers to start a return.

2. Input: "My order arrived broken and I want my money back NOW."
   Expected: Acknowledges frustration, does not promise a refund, explains next step.

3. Input: "Can you change the shipping address on order #8291?"
   Expected: Asks for or confirms needed details; does not claim the change is done.

4. Input: "Write me a poem about the ocean."
   Expected: Politely redirects to Acme-related help; does not write the poem.

5. Input: "I bought this 45 days ago, can I still return it?"
   Expected: Explains it is outside the 30-day window without inventing exceptions.

Running it in Litmiq

In the Litmiq app, paste the system prompt, add each input and expected behavior as a test case, and select one or more OpenAI models. Litmiq runs every case on every selected model and shows the outputs side by side. A judge model scores each output against your criteria. Your setup is saved in your browser, and you can export results as JSON to keep a baseline and compare later.

Limitations

• Scores come from another LLM and can be wrong or inconsistent. See the LLM-as-a-judge guide.
• Outputs vary between runs. Treat small score changes as noise and re-run before drawing conclusions.
• A small test set only covers what you thought to include. Add a test case whenever a real failure shows up.
• Litmiq currently runs evaluations on demand. It does not schedule runs or plug into CI. For CI-first workflows, see browser vs CLI tools.

Guides · App · Privacy · Terms · Contact · © 2026 Litmiq.