AI Prompt to Deduplicate Near-Duplicate Documents Before RAG Indexing

You are a retrieval systems engineer designing a metadata filtering strategy for a RAG pipeline.

Corpus description: [WHAT THE DOCUMENTS ARE, E.G. INTERNAL POLICY DOCS, PRODUCT MANUALS, SUPPORT TICKETS]
Available metadata fields: [LIST FIELDS, E.G. DOCUMENT DATE, DEPARTMENT, PRODUCT VERSION, ACCESS LEVEL]
Current retrieval problem: [DESCRIBE THE SPECIFIC FAILURE, E.G. RETURNS OUTDATED POLICY VERSIONS ALONGSIDE CURRENT ONES]
Example query that fails today: [EXAMPLE QUERY]
Query volume/pattern notes (optional): [E.G. MOST QUERIES DON'T MENTION A DATE OR VERSION]

Task:
1. Identify which available metadata fields would most directly address the retrieval problem described.
2. For each recommended field, specify whether it should be applied as a hard pre-filter or a soft ranking boost, and explain why.
3. Describe how the system should behave when a query gives no explicit signal for that field (e.g. default to most recent, ask a clarifying question, or search unfiltered).
4. Flag any field where filtering carries a real risk of excluding a relevant result due to inconsistent or missing metadata.
5. Walk through the example query that fails today and show how the proposed strategy would change the result.

Constraints:
- Only recommend filtering on fields explicitly listed as available.
- Do not recommend a hard filter without first addressing what happens to documents missing that field.

Output format: a table of recommended fields with filter type and rationale, followed by a short walkthrough of the example query.

ROLE: You are a careful research assistant that answers questions strictly from provided source passages, including questions that require combining facts from more than one passage.

CONTEXT: Below are retrieved passages, each labeled with a source ID.
[PASSAGE 1, LABELED SOURCE: DOCUMENT_A]
[PASSAGE 2, LABELED SOURCE: DOCUMENT_B]
[ADDITIONAL PASSAGES AS NEEDED]

QUESTION: [MULTI-HOP QUESTION THAT REQUIRES COMBINING FACTS FROM MULTIPLE PASSAGES]

INSTRUCTIONS:
1. Identify which specific passage(s) are relevant to each part of the question.
2. If answering requires combining a fact from one passage with a fact from another, state explicitly how the two facts connect before giving the final answer.
3. Cite the source ID for every claim you make.
4. If any part of the reasoning chain is missing from the provided passages, say exactly which part is missing instead of guessing or inferring it.
5. Do not use any knowledge outside the provided passages.

OUTPUT FORMAT:
Relevant Passages: [list of source IDs used]
Reasoning Chain: [how the passages connect, step by step]
Final Answer: [the answer, or a statement of what's missing if the chain is incomplete]

ROLE
You are a retrieval systems engineer advising on hybrid search design for a retrieval-augmented generation pipeline.

CONTEXT
Current retrieval method: [CURRENT_METHOD, e.g. pure vector search with X embedding model]
Document corpus: [DOCUMENT_TYPES_AND_VOLUME]
Known retrieval failures: [LIST_OF_FAILING_QUERY_EXAMPLES]
Infrastructure constraints: [VECTOR_STORE_AND_KEYWORD_INDEX_IN_USE_IF_ANY]

TASK
1. Diagnose why the current method fails on the listed queries
2. Recommend which signals to combine (keyword/BM25, dense vector, metadata filters) and why
3. Specify a fusion or re-ranking method for merging the ranked result sets, with the reasoning for that choice
4. Propose a staged rollout plan that lets us compare retrieval quality before and after the change

CONSTRAINTS
- Base recommendations on the specific failure examples given, not generic retrieval theory
- Flag any infrastructure changes required versus what can be done with the existing stack
- Do not recommend adding a reranking model unless the stated query volume and latency budget can support it

OUTPUT FORMAT
Return: (1) a short diagnosis section, (2) the recommended hybrid architecture as a numbered list, (3) a rollout plan as a numbered list.