AI Prompt to Build a Golden Test Set for Evaluating RAG Retrieval Accuracy

0 views Updated Oct 16, 2026

This RAG test set prompt helps you generate a golden evaluation set for a retrieval-augmented generation pipeline, built for developers and ML engineers who need to measure retrieval quality instead of guessing from a handful of manual spot checks. You feed the model a batch of source documents, and it produces question-answer pairs tied to specific passages, along with the exact source text each answer should come from.

The prompt works by asking the model to read each document chunk and invent realistic user questions that a retrieval system should be able to answer using only that chunk, plus a few adversarial questions designed to have no good answer in the corpus. This mix lets you test both retrieval precision (does the right chunk get retrieved) and the system's ability to say "I don't know" when nothing relevant exists. Running this periodically as your knowledge base grows catches retrieval drift before users notice it.

If you're assembling this test set from a large batch of PDFs, contracts, or support docs, pulling only the relevant passages first with Context Extractor makes it easier to generate accurate, source-grounded question-answer pairs instead of working from whole documents at once.

Prompt template

prompt-template

248 tokens

You are a test-set generator for evaluating a retrieval-augmented generation (RAG) system. CONTEXT: Source document: [DOCUMENT TEXT OR CHUNK] Domain: [DOMAIN, e.g. internal HR policy, product documentation, legal contracts] Number of questions needed: [NUMBER] TASK: 1. Read the source document carefully. 2. Generate [NUMBER] question-answer pairs that can be fully answered using only this document. 3. For each pair, include the exact sentence or passage the answer is drawn from. 4. Additionally generate [NUMBER OF ADVERSARIAL QUESTIONS] plausible-sounding questions that CANNOT be answered from this document, and label them as such. 5. Vary question phrasing: include direct factual questions, comparison questions, and questions phrased the way a real user would type them (casual, incomplete, or ambiguous). CONSTRAINTS: - Do not invent facts not present in the source document. - Do not answer questions using outside knowledge. - Flag any question where the answer requires combining multiple parts of the document. OUTPUT FORMAT: Return a numbered list. For each item include: - Question: - Expected Answer: - Source Passage: - Answerable: [Yes/No]

Want it sharper? Optimize this prompt with Prompt Optimizer, check it with the Prompt Debugger or shorten it with the Token Optimizer.

Example input

example-input

104 tokens

Source document: "Employees are entitled to 15 paid vacation days per calendar year, accrued monthly at a rate of 1.25 days. Unused vacation days may be carried over up to a maximum of 5 days into the following year. Carried-over days must be used by March 31st or they are forfeited. Sick leave is tracked separately and does not count toward the vacation balance." Domain: internal HR policy Number of questions needed: 3 Number of adversarial questions: 1

When to use it

You're launching a new RAG pipeline and have no baseline for retrieval accuracy
Your knowledge base has grown and you suspect older queries no longer retrieve the right chunks
You need a repeatable benchmark to compare two chunking or embedding strategies
You're debugging user complaints about wrong or missing answers from a RAG assistant

Best practices

Generate questions from a representative sample of documents, not just your longest or most detailed ones
Include a deliberate share of unanswerable questions to test refusal behavior, not just correct retrieval
Have a human spot-check a sample of generated Q&A pairs before trusting the set as ground truth
Re-run the test set after any change to chunk size, embedding model, or reranking logic

Common mistakes

Only testing easy, single-fact questions that don't reflect how real users phrase queries
Generating the test set from the same documents used to tune the retrieval system, which hides overfitting
Treating a one-time test set as permanent instead of refreshing it as content changes
Skipping unanswerable questions, so the system's hallucination rate never gets measured

FAQs

What is a golden test set in RAG?

A golden test set is a curated collection of questions paired with known-correct answers and source passages, used as a benchmark to measure whether a retrieval-augmented generation system retrieves and answers accurately over time.

How many test questions do I need to evaluate a RAG pipeline?

There's no fixed number, but most teams start with 50-100 questions covering a representative mix of document types, question styles, and at least 10-15% unanswerable questions to test refusal behavior.

Can ChatGPT or Claude generate realistic test questions from my documents?

Yes, both can generate question-answer pairs grounded in a provided document when given clear instructions and the source text, though a human should still review a sample before treating the output as ground truth.

Which Cuelara tool can help me prepare documents before building this test set?

Context Extractor — it pulls only the relevant passages from large PDFs or CSVs first, which makes it easier to generate accurate, source-grounded test questions from big document sets.

Found this prompt useful? Share it.

More in RAG & Knowledge Retrieval

RAG & Knowledge Retrieval

AI Prompt to Evaluate RAG Answers for Hallucinations Against Retrieved Context

This RAG hallucination detection prompt checks a generated answer against the exact passages a retrieval augmented generation system pulled…

ROLE: You are a strict fact-checking evaluator for a retrieval-augmented generation (RAG) system. You do not answer questions or improve the answer; you only judge it.

CONTEXT:
Retrieved passages:
[PASTE THE EXACT RETRIEVED CONTEXT CHUNKS HERE, NUMBERED]

Generated answer to evaluate:
[PASTE THE RAG SYSTEM'S GENERATED ANSWER HERE]

Original user question:
[PASTE THE ORIGINAL QUESTION HERE]

TASK:
1. Break the generated answer into individual factual claims.
2. For each claim, decide whether it is:
- SUPPORTED: directly backed by a specific passage
- PARTIALLY SUPPORTED: related to a passage but adds unsupported detail, numbers, or certainty
- UNSUPPORTED: no passage backs this claim, even if it sounds plausible
3. For every SUPPORTED or PARTIALLY SUPPORTED claim, cite the passage number it relies on.
4. Do not use outside knowledge to judge truth; judge only whether the provided context supports the claim.

CONSTRAINTS:
- Evaluate every claim individually; do not summarize the whole answer as one verdict
- Be skeptical of specific numbers, dates, or named entities in the answer that are not explicitly in the passages
- If a claim restates the question without adding information, mark it as NOT APPLICABLE rather than forcing a verdict

OUTPUT FORMAT:
Return a numbered list, one entry per claim, each with:
Claim: [the exact claim text]
Verdict: [SUPPORTED / PARTIALLY SUPPORTED / UNSUPPORTED / NOT APPLICABLE]
Evidence: [passage number and quote, or "none found"]

End with:
Overall hallucination rate: [X of Y claims unsupported or partially supported]
Recommendation: [APPROVE / REVISE / REJECT the answer]

RAG & Knowledge Retrieval

AI Prompt to Retrieve and Summarize Airtable Records Using MCP

This prompt is for teams who want an AI assistant to pull specific records from an Airtable base through an MCP server connection and summar…

Role: You are a data assistant with access to an Airtable MCP connection.

Context:
- Base: [BASE_NAME]
- Table: [TABLE_NAME]
- View or filter: [VIEW_NAME_OR_FILTER_CRITERIA]
- Question to answer: [SPECIFIC_QUESTION_ABOUT_THE_RECORDS]

Task:
1. Query the specified table and view through the Airtable MCP connection, applying the stated filter.
2. Identify the records that match the question.
3. Summarize the findings in plain language, including the total count of matching records.

Constraints:
- Base the answer only on records actually retrieved from the base, not assumptions about typical values.
- List the primary field value (e.g. record name or ID) for each record referenced in the summary.
- If no records match, say so explicitly rather than approximating an answer.
- Flag any records with missing or inconsistent data relevant to the question.

Output format:
- Answer: 1-2 sentence direct answer to the question
- Matching records: bulleted list with primary field value and relevant field values
- Total count: number of matching records
- Data issues: bulleted list of any missing/inconsistent fields noticed, or "None found"

RAG & Knowledge Retrieval

AI Prompt to Make a RAG Assistant Say 'I Don't Know' When Context Is Missing

This is a RAG prompt for stopping a retrieval augmented assistant from guessing when the retrieved documents don't actually contain the answ…

ROLE: You are a question-answering assistant that must only use the provided context to answer questions. You must never use outside knowledge, even if you know the answer.

CONTEXT:
[PASTE RETRIEVED DOCUMENT CHUNKS HERE]

QUESTION: [USER'S QUESTION]

INSTRUCTIONS:
1. Check whether the context above contains enough information to fully answer the question
2. If it does, answer using only that information, and quote or reference the specific part of the context you used
3. If the context only partially answers the question, answer the part you can and explicitly state which part is not covered
4. If the context does not contain relevant information at all, respond exactly with: "[INSUFFICIENT CONTEXT RESPONSE, e.g. I don't have enough information in the provided documents to answer this.]"
5. Do not guess, infer beyond what the context states, or use any knowledge from outside the provided context

CONSTRAINTS:
- [ANY ADDITIONAL CONSTRAINT, e.g. keep answers under 150 words, cite chunk numbers if provided]

OUTPUT FORMAT: A direct answer to the question, or the insufficient-context response, followed by a short reference to which part of the context (if any) was used.