A retrieval-augmented generation system searches external information and supplies selected results to a language model. In a typical application, that means finding relevant passages and including them in the prompt. The model can still draw on its training or ignore the evidence, so retrieving a document does not by itself make an answer accurate.
The original RAG paper combined a neural retriever with a trained sequence-to-sequence generator. Current applications use the term more broadly, including systems that assemble prompts around an existing model API.
When an answer is wrong, I want a trace that shows the source documents, retrieved candidates, final context and generated answer. That makes it possible to locate the failure before changing the system.
Start with one failed answer
Take a real query and inspect the data at each stage. Keep the original wording; a simplified version may hide the problem.
| Observation | Check next |
|---|---|
| The source is absent from the index | Ingestion status, parsing and source synchronisation |
| The source is indexed but absent from candidates | Search query, filters, embeddings and candidate count |
| The useful passage is a candidate but absent from the prompt | Ranking, deduplication and token-budget handling |
| The prompt contains sufficient evidence but the answer is wrong | Instructions, evidence conflicts and generation |
| The answer cites a real document that does not support the claim | Claim-to-passage verification |
| The answer exposes another user’s information | Authorisation, retrieval filters, caches and logs |
For a controlled diagnostic, place a known sufficient passage directly into the prompt. If the answer improves, investigate how the normal pipeline selected its context. If it remains wrong, examine how the model is using the evidence. This separates retrieval failures from generation failures without rebuilding either component.
Check what was indexed
A successful ingestion job can still produce unusable content. Read extracted samples from the formats your users rely on. Tables may lose column relationships, scanned pages may need OCR, and headings can become detached from their sections.
Track a stable source identifier, version or content hash, ingestion status and the resulting chunk IDs. A changed document should replace or supersede its indexed version. Deleting a document should remove its searchable chunks and invalidate affected caches.
Choose chunks around the document’s structure where possible. A section may work well for a manual; a row with its column names may work better for a table. Short passages can improve search specificity while requiring a larger surrounding section for the final answer. Parent-document retrieval is one way to separate those two needs.
There is no fixed chunk size that works across corpora. Compare a few plausible sizes and inspect the failures. Excessive overlap can fill a result set with near-duplicates; large chunks can combine unrelated subjects and consume the context budget.
Metadata and current permissions
Keep enough metadata to identify the source and apply the filters the application needs. A policy may need an effective date and a superseded flag. A support ticket may need a customer, status and product version. “Latest” requires a defined ordering: publication time, last modification and effective date can mean different things.
Permissions need an authoritative source and enforcement at request time. One approach stores permitted user or group identifiers with indexed documents and filters against the authenticated caller. Microsoft’s security-filter pattern illustrates this design. The application must derive allowed identities from trusted authentication, rather than accepting arbitrary group IDs from the requester.
Other systems use separate tenant indexes or an authorisation service. In every design, revoked access must propagate, and unauthorised text must be excluded before it reaches the model or a reranking service. Cache keys and trace access must respect the same boundaries. Asking the model to hide restricted content is not an access-control mechanism.
Compare retrieval methods
Keyword search is useful for identifiers and exact terms. Dense retrieval can match related meanings even when wording differs. Hybrid retrieval combines both result sets, with a fusion or scoring method to produce an ordering.
Azure AI Search provides one concrete implementation: text and vector searches run together, and reciprocal rank fusion combines their ranked results. Its behaviour is an implementation choice, not a definition that every hybrid system follows.
Test keyword, vector and hybrid search on the same labelled queries. Include product codes, paraphrases, date-sensitive requests and questions requiring evidence from multiple documents. Inspect which query groups improve and which regress.
A reranker scores an initial candidate set again. Cross-encoders are one option; model-based ranking is another. Reranking can promote a useful passage that initial retrieval ranked poorly, but cannot recover a passage missing from its candidates. Measure the final context quality and added latency.
Query rewriting can resolve references or extract filters. It can also remove a product code or change the question. Keep the original query in the trace and compare retrieval with and without the rewrite. Extra searches need a stopping rule and a time budget.
Build the context the answer needs
Remove duplicate passages, retain source labels and include enough surrounding material to interpret the evidence. A table value without its heading or unit is easy to misread. When documents disagree, preserve their dates and provenance so the answer can explain the disagreement.
A large context window does not imply equally reliable use of every position. Lost in the Middle found that information placement affected performance in its retrieval and question-answering experiments. Test your own model and prompts with evidence in different positions, especially when increasing the number of passages.
Citations require two checks. First, the cited source must exist in the authorised material supplied to the model. Second, its passage must support the associated claim. A real URL beside an invented statement fails the second check.
Give the model a clear response format for insufficient evidence. Evaluate that behaviour with questions the corpus cannot answer. Also treat retrieved documents as untrusted input: instructions found inside them must not grant tool permissions or override the application’s rules.
Measure retrieval and answers separately
A useful evaluation set contains realistic questions, relevant source IDs and expected answer criteria. Include unanswerable questions, outdated sources and access-control cases. Synthetic questions can help exercise an index, but questions generated from chunks often resemble those chunks more closely than user queries do.
For labelled retrieval results, common metrics include:
- Recall@k: the fraction of labelled relevant items present in the first
kresults. - Precision@k: the fraction of the first
kresults labelled relevant. - Hit rate@k: the fraction of queries with at least one relevant result in the first
k.
Define relevance at the document or passage level and keep it consistent. A query that needs three supporting passages may pass a hit-rate check while lacking enough evidence to answer.
Answer evaluation asks different questions: is the response correct, does it address the request, and are its claims supported by the supplied evidence? Ragas proposes automated metrics for these aspects of a RAG system. Model-based scores are useful for triage and comparison; check their agreement with human review on a sample of your actual tasks.
There is no universal recall percentage that establishes readiness. Set acceptance criteria from the intended use and the cost of errors. Keep a test set separate from the examples used to tune prompts and retrieval settings.
Record enough to investigate production failures
For a sampled request, record the model and prompt versions, search settings, candidate IDs, final context IDs, timing and outcome. Store text only where needed and permitted, with restricted access and a retention policy. A shared chat channel is usually a poor destination for raw customer prompts and documents.
Monitor ordinary service failures as well as answer quality. Retrieval APIs can time out, parsers can fail, and model calls can return errors. Plausible but unsupported answers require separate evaluation because they may return a successful HTTP status.
Measure end-to-end latency, including retries, and report a distribution rather than just an average. Track ingestion, embeddings, reranking, generation and storage costs. Repeated agent searches can change both latency and cost substantially.
Review a sample of interactions and group failures by cause. Add representative cases to the evaluation set before making a change. A revised prompt, new embedding model or altered chunker should face the same regression cases.
When a simpler tool fits
A billing date held in a database can be returned through an authorised lookup. A numeric aggregate is usually better computed by a query or program. A small document that fits comfortably in context may not need an index. Compare those approaches before adding retrieval infrastructure.
Code assistants can combine file reading, exact search, symbol navigation and semantic retrieval. Reading files or running a search tool is still retrieval in the broader sense. The useful comparison is between particular search and context strategies, rather than a universal claim that coding agents should abandon RAG.
Additional agents introduce handoffs and coordination work. They can help with independent tasks, but that benefit needs to appear in task success, elapsed time or cost. The number of agents is not an answer-quality metric.
A repeatable debugging sequence
- Reproduce one failure and inspect its trace.
- Verify that the source exists, is current and is authorised for the caller.
- Check whether sufficient evidence survives retrieval, ranking and context assembly.
- Test generation with known sufficient evidence and inspect each citation.
- Change the identified stage, then rerun the held-out cases and measure cost and latency.
Keep the failed query and the fix together in the engineering record. It should be possible to see what changed and whether that particular failure returns.