A RAG system can return a fluent answer even when its search results never contained the passage needed to answer the question. Retrieval-augmented generation gives a model material to work from. The retrieval layer decides which material gets that chance.
That distinction gives you somewhere concrete to start debugging. A passage missing from the search results needs a different investigation from a passage that was found, ranked too low and left out of the prompt. A cross-encoder reranker can reorder candidates. It can't recover a passage none of your searches returned.
Lexical search, vector search and graph traversal give you different ways to find candidates. The useful combination depends on the questions people ask and the information your corpus contains. Exact identifiers, paraphrased explanations and relationships between entities create different retrieval problems. Buying another database before identifying the problem adds another component to operate.
RAG Knowledge Engines covers the broader architecture, including citations and the decision to use RAG at all. Useful passages have to reach a bounded result set before the answering model can assess them.
Chunk boundaries can separate a passage from its explanation
My Archon setup needed document-specific processing
My local Archon retrieval system had to handle my own Markdown writing, PDF textbooks, and YouTube transcripts and comments. Those sources don't organize information the same way. A heading in an essay, a code block in a textbook and a reply in a comment thread give a chunker different clues about where an idea begins and ends.
I started with Archon because it already worked end to end: ingest documents, connect an agent through a local Model Context Protocol server, and query the material. I then changed the chunking pipelines for the document types I was using. The work included cluster-semantic chunking and task-specific embeddings. That gave my agents access to the writing and reference material I wanted them to use.
I also described Archon as a crude, home-developed RAG system. My account supplies no controlled benchmark of the refinements' effect. Those changes need evaluation on another corpus before anyone treats them as a general prescription.
A relevant match can contain an incomplete instruction
A returned chunk has to contain a usable idea. A passage can mention the right function and omit the condition that explains when to call it. A search engine may rank that passage highly while the agent receives an incomplete instruction. Adding a reranker leaves the missing condition missing.
Inspect the source and the returned chunk together before changing the search stack. Check where the document was split, what context the split removed, and whether the system can recover the surrounding section. Chunk size is a parameter to evaluate against actual questions. Small pieces can isolate a fact. They can also separate it from its qualification. Large pieces preserve more surroundings and consume more of the prompt. Evaluate chunk boundaries and prompt cost together.
Different questions need different ways to find evidence
A contract identifier and a request for an explanation put different demands on search. The identifier needs its exact characters preserved. The explanation may use words that never appear in the source. A question about who approved a change may require following a recorded relationship. Each demand suggests a retrieval method, and each method has limits.
Lexical search preserves the importance of exact words
Lexical search matches terms in a query against indexed text. OpenSearch uses BM25 by default, scoring matches with term frequency, inverse document frequency and document-length normalization. A distinctive term can carry more information than a word found throughout the corpus.
Names, product codes and clause references are reasons to test lexical retrieval. Phrase matching, field weights and filters still need deliberate configuration. A request for a particular identifier shouldn't quietly become a search for something with a similar name.
Typesense is another option for lexical search. Its documented ranking differs from OpenSearch's BM25 default. Typesense documents its own text-match ranking, including token overlap, edit distance and proximity. Choose the matching behavior the task needs, then inspect the results. The category name alone doesn't specify the ranking algorithm.
Vector search can find passages that use different wording
Dense retrieval represents a query and document chunks as embeddings, then searches for nearby vectors. Similarity can connect a question with a passage expressing the same idea in different language. A nearby vector still needs checking against the question. Similarity doesn't establish that a passage contains a required fact or applies to the right customer, document version or date.
Qdrant is a dedicated vector-search option. pgvector adds vector search to Postgres, including exact search and approximate indexes such as HNSW. An existing Postgres installation is a reason to evaluate pgvector before adding a separate service. Corpus size alone doesn't settle that choice. Filtering, memory, update patterns and the recall required by the application affect the workload.
Embedding selection belongs in the same evaluation. Check the model's retrieval task, language coverage and supported vector representation against the stored corpus. A named encoder and a dimensionality figure can't substitute for relevant results.
Graph retrieval follows relationships recorded in the data
Graph retrieval starts from entities and follows typed relationships. A document mentioning two people can be semantically relevant without establishing that one approved the other's work. A recorded approval relationship gives the system something more specific to query.
Neo4j is an option for graph storage and traversal. Graphiti adds a temporal-knowledge-graph approach when changing facts are part of the problem. A simpler edge table can be sufficient for a narrower relationship query. The graph is useful only to the extent that its entities and relationships are supported and maintained.
My Research Layer combines full-text, vector and graph retrieval. A narrower application may need fewer retrieval methods. Add relationship retrieval when the questions require relationships that text matching alone doesn't establish.
Bound the candidates before selecting prompt evidence
Running several searches can improve coverage and also produce duplicate or irrelevant passages. Passing the entire union to the answering model spends prompt space on whatever each search happened to return. A separate selection stage lets you control that trade-off.
Joint scoring needs a bounded set of candidates
A cross-encoder reads the query and a candidate together and scores their relevance as a pair. A bi-encoder embeds them separately. Joint scoring gives the reranker a different opportunity to assess the match, at the cost of processing each candidate pair. Apply that work to a bounded set. Cohere offers a hosted reranking service. The BAAI bge-reranker family offers open-weight models to run yourself. Evaluate them against the same questions and operating constraints.
Candidate budgets don't predict the size of the merged set
The following candidate budgets illustrate the sequence. They aren't a measured recommendation:
- Retrieve up to 30 lexical candidates and 30 dense candidates. Retrieve up to 10 graph candidates when the question warrants that search.
- Merge the results and deduplicate repeated chunks by stable identity. Keep the source and version metadata needed to inspect each result.
- Score the remaining query-candidate pairs with the reranker.
- Select up to 10 results, subject to relevance and the available prompt-token budget. Keep fewer when the evidence doesn't support filling the set.
Deduplication and reranking are separate operations. A relevance model doesn't remove repeated records merely by assigning them scores. The size of the merged set also depends on overlap between searches. The budgets don't predict how many unique chunks will remain.
Qdrant's hybrid-search tutorial separates retrieval from reranking, using late interaction for its reranking stage. Cross-encoders are one way to perform that stage. The design decision is how much candidate coverage to retain and how much ranking work the application can afford.
Measure retrieval and reranking latency separately, then measure the complete request. Cache state, candidate length, model hosting and network calls affect the result. A latency target needs a benchmark with those conditions recorded.
Resolve the query without inventing its missing facts
A rewrite must preserve known references
A conversation can contain enough information to understand a question that looks useless in isolation. A user asks, "What's the cap?" The previous turn may identify a contract and an indemnification clause. Sending only the latest sentence to an index discards that context before retrieval begins.
Query rewriting makes the request explicit for search. Suppose the conversation already identified MSA-2024-014 and schedule 2 as the relevant contract material. A rewrite such as "indemnification cap MSA-2024-014 schedule 2" preserves those known references and adds the topic supplied by the conversation. The contract number is an illustrative identifier, not a customer record. A rewriter that introduces the same details without that context has guessed which document the user meant.
Preserve exact names, numbers and restrictions when rewriting. Keep the original request available alongside the rewritten query so a trace can show what changed. Ambiguity that affects which contract applies may require a clarification from the user. Search expansion can't establish a missing fact.
Multiple query variants can widen the candidate pool when different phrasings express the same request. They also add retrieval work and repeated results. Merge and deduplicate the candidates before reranking, and evaluate whether expansion recovers useful evidence the original query missed.
A router can exclude the right collection
Semantic routing makes a different decision: which collection or retrieval method should handle the request? A clause lookup can emphasize a contracts collection and lexical matching. A synthesis question can call for semantic search. A question about recorded relationships can justify a graph query.
A small classifier or a structured-output model call can emit a route and its configuration. That extra decision can also exclude the right evidence. Compare routed retrieval with a broader search on the same questions, and keep uncertain routes visible. Corpus permissions must constrain every route. A model's classification doesn't grant access to another collection.
Evaluate candidate coverage separately from ranking
A final answer can fail because the required evidence was absent, because selection excluded it, or because the answering model mishandled evidence already in its prompt. An answer-only score combines those failures. Keep a record of the intermediate results so the next change addresses the stage that lost the evidence.
Candidate recall checks whether useful evidence was found
Start with held-out questions and source passages judged relevant to each question. Include the actual question shapes the system must handle: exact references, paraphrases, follow-up questions and relationship queries where applicable. Include questions the corpus can't answer. An evaluation set containing only answerable questions won't test whether the system recognizes an evidence gap.
Recall@K measures the fraction of known relevant passages present in the first K retrieved results. A test with one labelled target passage reduces to checking whether that target was found. A question needing several passages requires a broader relevance set; finding one doesn't establish that the evidence is complete. The Stanford information-retrieval text distinguishes set coverage from ranked-result evaluation.
Define which candidate list the cutoff applies to. Per-search recall@30 and recall on a merged candidate pool answer different questions. Keep the evaluation corpus and relevance labels versioned with the result.
Reciprocal rank checks where useful evidence was placed
Mean reciprocal rank averages the reciprocal of the position of the first relevant result across queries. TorchMetrics' reciprocal-rank implementation applies a top-K cutoff. For MRR@10, a query scores zero when no relevant result appears within the first 10. The metric rewards placing a relevant passage near the top. It doesn't measure whether every passage needed for a multi-part answer survived selection.
A drop in candidate recall directs the investigation upstream: ingestion, chunk boundaries, query rewriting, routing, filters and search configuration can all affect what enters the set. Stable recall with worse reciprocal rank points toward ranking or a changed candidate mix. Compare the actual passages before assigning the cause. A metric identifies a symptom more reliably than a single broken component.
Run these comparisons when a retrieval configuration changes, including in CI where that's part of the delivery process. Evaluate the final answer separately for support, citation correctness and behavior when evidence is missing.
Production traces connect the scores to real failures
A useful retrieval trace records the request, rewritten query, chosen route, per-search candidate identities and scores, deduplication, reranked order, selected prompt passages and stage timings. LogFire can hold those spans. Protect sensitive query and source text through access controls, redaction and retention limits. An observability system becomes another place the data must be governed.
Low reranker scores can identify cases to inspect, but a score isn't automatically a calibrated probability of correctness. Disagreement between retrieval methods, weak results and negative user feedback are investigation signals. Inspect them alongside the corresponding answer and cited passages. A thumbs-down alone doesn't tell you which stage failed.
Ask a retrieval supplier to show where the evidence went
A retrieval service can support a search interface, a RAG assistant and a content system looking for source material. Those consumers can share evidence retrieval while keeping their own requirements for selection, citations and answers. The value of the shared service depends on whether each consumer receives usable, traceable context.
A vendor demonstration should make that path inspectable. Bring questions tied to source material you're allowed to use and ask the supplier to show:
- The source passage that should answer each question, including its conditions and document version.
- The candidates each search returned, and the evidence that justified adding a retrieval method.
- The merged set, duplicate removal, reranked order and passages selected for the prompt.
- The rewrite and route for an ambiguous follow-up, including what happens when context is missing.
- The held-out evaluation results, stage timings and behavior when no supporting passage exists.
A lexical-only or vector-only design may fit a bounded task. A multi-method design may earn its extra operating cost. The decision needs examples of what the simpler design misses and what the added component recovers.
Evals & Observability develops the broader evaluation discipline. The retrieval decision remains concrete: did the system find the passage, keep it through selection and give the answering model enough evidence? A fluent answer leaves those questions open. A trace back to the source lets you inspect them.