RAG evaluation
On this page
LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.
The short answer
RAG evaluation tests a retrieval-augmented system on two layers: did we retrieve useful context, and did the model stay faithful to that context when it answered — including when the honest answer is “I don’t know”?
Litmiq cannot see your index or your retriever. It can still test the prompt and the generator against a fixed context you paste into a case. For a ready-made suite of that kind, use the RAG prompt-tests template (grounded answers, citations and I don’t know). For retriever-plus-generator metrics in Python, a library such as Ragas is in a different category.
RAG evaluation metrics
- Faithfulness / grounding. Claims in the answer should be supported by the retrieved (or provided) context, not by the model’s prior.
- Context quality. Names such as context precision or context relevance ask whether the retrieved passages were the right ones. You need the actual retrieved set to score this. Ragas documents metrics in this family in its own docs; we do not re-implement them here.
- Answer adequacy. Did it actually answer the question, once the context is sufficient?
- Abstention. When the context is missing or contradictory, does the system say it does not know instead of inventing a source?
- Citation form. If you require a quote or a document id, that is a hard pass/fail, not a 1–10 score.
There is no official complete list, and we do not invent one. Start with the failure you have already seen: hallucinations, dropped citations, or retrieval misses.
How to test in practice
- Build cases as (question, context, expected behavior), including at least one empty or irrelevant context.
- Score hard rules first (citation present, abstention, no invented URLs).
- Use a judge only for grounding and adequacy, with criteria that mention the provided context.
- When you can log production retrieval, add a retriever metric or a platform trace; do not pretend a pasted context is your live index.
You can run the generator-and-prompt slice in Litmiq with your own key, starting from the RAG template. Move the same expectations into CI or a platform when the suite is stable.
Limits of this guide
This page does not report a Litmiq RAG benchmark. Ragas and platform features are described from our existing dated comparison pages and from those projects' public materials; see the sources. Litmiq remains a single-turn prompt tester.
FAQ
What is RAG evaluation?
RAG evaluation tests a retrieval-augmented generation system: not only the final wording, but whether the retrieved context was relevant and whether the answer stayed faithful to that context, including when the right answer is to say it does not know.
What metrics are used for RAG?
Common published metric families include faithfulness or grounding (the answer is supported by the retrieved context), context relevance or context precision (the retrieved passages were the right ones), and answer quality on the question. Libraries such as Ragas document specific metrics in this family. There is no single official set. Also keep hard checks: citations present, I-don't-know when the context is empty, no invented sources.
Can Litmiq evaluate a full RAG pipeline?
Not the retriever. Litmiq tests a prompt and a model on cases you paste, in the browser. You can still write cases that include a fixed context snippet and expected behavior (grounded answer, citation, or I don't know). That checks the generator and the prompt, not whether your index retrieved the right passage in production. The RAG prompt-tests template is a starting suite.
When should I use Ragas or a platform instead?
Use a RAG-oriented library such as Ragas, or a tracing platform that can score retrieval and generation on real traces, when you need to measure the retriever and the generator together, in code or in production. See Litmiq vs Ragas.
What should every RAG eval include?
At least: a case the context supports, a case the context does not support (the model should refuse or say it does not know), a citation or quote check if you require one, and a hard ban on invented sources. Add retriever metrics only when you can log what was actually retrieved.
Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.
Sources
- Litmiq, RAG prompt tests: grounded answers, citations and I don’t know.
- Litmiq, Litmiq vs Ragas, which describes Ragas from its public docs as an open-source (Apache-2.0) Python library with RAG-oriented metrics such as faithfulness and context precision.
Try it in Litmiq
Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI
Litmiq Pro
Saved suites, CI on a schedule and regression alerts for $19/month. The app stays free.
See pricing →Related guides
Compare & reference
- Prompt testing tools compared
- LangSmith alternative
- LangSmith pricing
- Litmiq vs Promptfoo
- Litmiq vs OpenAI Playground
Share this guide