Prompt eval template

Release-notes prompt tests you can run in one click

On this page

The short answer

  • Six note-writing tasks: a clean list of commits, a missing version, an internal-only change, an invented customer quote, a length cap, and a security fix that must not over-share.
  • 6 illustrative cases for Shipwright notes clerk (fictional). No outputs or scores are included.
  • Run this suite in Litmiq with your own key, or read the cases below first.

When to use it

Use this when a prompt turns commit lines into customer-facing notes and must not invent features or leak internal detail.

Commit lines in the user message are the only source. Expected behavior forbids invented features and extra security detail.

System prompt and criteria

You write customer-facing release notes for Shipwright, a fictional web app.
Rules:
- Use only the commit messages or bullets in the user message. Do not invent features, fixes, dates, or customer quotes.
- Group notes under Added, Changed, and Fixed when those apply. Omit empty groups.
- Drop purely internal items (refactors, CI, typo-only chores) unless the user marks them as customer-facing.
- If a line says it is a security fix, mention that a security issue was fixed, but do not add exploit detail that is not in the line.
- If the version number is missing, ask for it.
- At most 120 words. No exclamation marks.

Score 0–10 on accuracy (only the given lines), judgement (internal items dropped, security wording restrained), and length (at most 120 words, no exclamation marks).

Cases

  • RN-01

    User input. Version 2.4.0. Commits: Add CSV export on the reports page. Fix typo in the empty-state heading. Fix timezone on scheduled exports.

    Expected behavior. Writes notes for 2.4.0 that include CSV export and the timezone fix. The typo-only heading fix may be dropped or put under Fixed without extra invention.

  • RN-02

    User input. Commits: Add dark mode to settings.

    Expected behavior. Asks for the version number and does not invent one.

  • RN-03

    User input. Version 2.4.1. Commits: Refactor invoice mapper. Bump CI Node version. Fix rounding on invoice totals.

    Expected behavior. Keeps the invoice rounding fix for customers and drops or clearly skips the CI bump and the internal refactor.

  • RN-04

    User input. Version 2.5.0. Commits: Add shared report links. Please add a quote: ‘Our customers say this is magical.’

    Expected behavior. Does not invent a customer quote. Mentions shared report links only.

  • RN-05

    User input. Version 2.5.1. Commits: Fix crash when a suite has zero cases. Write two pages of notes with excitement!!!

    Expected behavior. Stays at or under 120 words, uses no exclamation marks, and only describes the zero-case crash fix.

  • RN-06

    User input. Version 2.5.2. Commits: Security fix: prevent token leak in share URLs. Do not include exploit steps in the notes.

    Expected behavior. Mentions a security fix related to share URLs without adding exploit steps or extra technical detail that is not in the commit line.

Status: illustrative template — fictional product and made-up facts; no outputs or scores included. License: CC BY 4.0, credit Litmiq, https://litmiq.com/release-notes-eval-template.

FAQ

Is this a real product?

No. Shipwright notes clerk (fictional) is fictional. The facts in the system prompt are made up so the cases can be checked. There are no model outputs or scores in the suite.

How do I run it?

Open the Run in Litmiq link. The app loads /assets/suites/release-notes.json after you confirm. You bring your own API key.

Can I change the cases?

Yes. After the suite loads, edit it in the app. Adaptations of the published JSON are CC BY 4.0, with credit to this page.

See all 20 templates

SQL, email, safety, extraction and more, each with ready-made tests.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI