How to test prompts
How to test prompts from a LlamaIndex app in Litmiq
On this page
The short answer
- There is no official LlamaIndex integration and no Litmiq package to install.
- You test the prompt your app already uses: copy it, add cases, run them in the browser with your own key.
- For CI, export promptfooconfig.yaml and run it in your own workflow. Litmiq does not receive that workflow or its secrets.
LlamaIndex is a framework for connecting LLM applications to data, often through indexes and query engines. Litmiq is not an official LlamaIndex integration and does not run your index or retriever.
The useful split is: LlamaIndex retrieves; Litmiq tests the prompt that reads the retrieved text.
1. Snapshot the prompt and the context
Copy the query-engine or response-synthesis prompt you actually send, and paste a realistic block of retrieved nodes underneath it (the same pattern as the RAG eval template). Include at least one noisy or conflicting chunk so you can see whether the prompt stays faithful.
2. Write groundedness cases
Ask questions the passages support, questions they do not, and questions where two chunks disagree. Expected behavior should say which passage may be used and that unknown questions must not be invented. See faithfulness and groundedness.
3. Run in Litmiq, measure retrieval elsewhere
Litmiq will score the written answer given the context you pasted. It cannot compute context precision or recall. For those numeric RAG metrics in Python, a library such as Ragas is a better fit. The two are complements, not substitutes, and neither is an official LlamaIndex plugin from Litmiq.
FAQ
Is there an official LlamaIndex integration?
No. This page describes a manual workflow using Litmiq features that already exist: the app, suite links and YAML export.
Will Litmiq call my app?
No. You copy a prompt and cases out of the app, or you point /app?suite= at a file you host.
Where do my keys go?
In Litmiq, they stay in your browser and are sent only to OpenAI or Anthropic. In CI, they stay in your own secrets store.
Open a starter suite
Load an illustrative pack, then replace the fictional facts with your prompt.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI