# Retrieval Tooling

Canonical: https://andydataguy.com/wiki/ai-and-technical-development/default/rag-retrieval-tools

Author: Anand Houston (AndyDataGuy)

Lexical, semantic and structural retrieval each find a different kind of relevance, and the reranker can only order what reached it. Buy vector search alone and two of the three retrieval lanes in the figure stay dark while the stack on the right keeps looking full.

## The thesis

Retrieval is engineering, not magic. A vendor that sells you "vector search" sells you one third of a system. A team that builds against that one third and wonders why its RAG hallucinates is rediscovering what the search-engineering community has known for two decades: lexical, semantic and structural retrieval each catch a different kind of relevance, and a production retrieval stack composes all three. Tools exist that make the three composable, and patterns exist that make them cheap. None of it is exotic, and all of it is what separates a search demo from a search system.

This article is the practitioner companion to the broader [RAG Knowledge Engines](/wiki/ai-and-technical-development/default/rag-systems) piece. That piece covers the full architecture (chunker, index, citations, evals, when not to RAG). This one zooms in on the retrieval layer: which tools to use, how to compose them, how to handle queries before they hit the index, and how to keep the whole thing fast enough to be interactive.

## The three retrieval stacks

Lexical search ranks with BM25, the Okapi-derived function that scores documents on term frequency, inverse document frequency, and document length. It's forty years old and still the right default for exact-phrase and named-entity queries. Typesense and OpenSearch are the production-grade implementations I reach for. Both have native typo tolerance, faceting, and per-field weighting, and both run on commodity hardware at scale. The configuration cost is hours, not weeks.

Dense vector search (cosine similarity over learned embeddings) is the modern technique that gets all the press. The query and every document chunk are embedded into a high-dimensional vector space (Gemini Embedding 2 at 3072 dimensions is my default; OpenAI's text-embedding-3-large is the comparable alternative). Similarity in that space approximates semantic similarity. Qdrant is the production-grade vector store I prefer, and pgvector is the pick if the team is already on Postgres and the corpus is under a million chunks. Both ship HNSW indexes, filter-aware search, and reasonable defaults.

Graph retrieval traverses typed relationships between entities to find structurally relevant context the other two can't see. Neo4j with Graphiti is my default for the temporal-knowledge-graph case (institutional memory across time, with edge-level invalidation when facts change). For lighter cases, a simpler graph store (or even a Convex table modeling edges directly) is enough. Graph retrieval is for the queries where structure matters more than text similarity, not for every retrieval. The [Research Layer](/case-studies/research-layer-metagraph-compiler) case study is the production proof for composing full-text, vector, and graph retrieval behind one system.

## How the three compose

The naive composition is "run all three, union the results." That version works for small corpora and breaks at scale because the union has too many candidates and the model gets noisy context. The grown-up composition is "run all three with bounded K per source, union with deduplication, rerank the union with a cross-encoder, take top N for the prompt."

The cross-encoder reranker is the load-bearing piece most teams skip. It takes each (query, candidate) pair and produces a relevance score in a single forward pass. Cohere Rerank is the easiest hosted option. The bge-reranker family from BAAI is the strongest open-weights option; running it on a GPU is simple, and the latency is manageable for a top-50-into-top-10 reduction. Cross-encoders see the query and the candidate together, so they catch contextual relevance that bi-encoder embeddings (which see them separately) can't.

In numbers, BM25 returns the top 30, dense returns the top 30, and graph returns the top 10 (graph queries are inherently more selective). The deduplicated union comes to roughly forty candidates, the reranker scores all forty, and the top eight or ten go into the prompt. The latency budget for the full retrieval-and-rerank pipeline runs about 200-400ms on a warm cache, which is fast enough to feel interactive in a chat UI.

Forty noisy candidates go into the reranker, and ten ordered, scored, prompt-ready chunks come out.

## Query rewriting and semantic routing

A user's query is rarely the right query for the index. Users underspecify, use pronouns that refer to prior turns, mix several questions into one, and use brand-new terminology the index doesn't contain. The retrieval layer has to handle that translation gap, and the translation is itself a model task.

Query rewriting runs each user query through a small model that emits one or more rewritten queries optimized for retrieval. The simplest version expands "what's the cap?" to "indemnification cap MSA-2024-014 schedule 2." A more capable version emits multiple variant queries to run in parallel, broadening recall. Each rewritten query gets retrieved against, the union goes to the reranker, and the reranker handles the deduplication. The cost is one extra small-model call per user query, and it's well worth it for the recall lift on the queries production users ask.

Semantic routing is the upstream sibling: the system decides which retrieval stack (or which subset of the corpus) to use based on the query shape. A query for a specific clause routes to the lexical stack with high weight on the contracts collection. A query asking for a synthesis routes to the dense stack across the full corpus. A query about relationships between entities routes to the graph. The router is a small classifier (a fine-tuned model or a structured-output prompt) that takes the query and emits a route plus configuration. The overhead is one classification call, and the savings come from not running all three stacks against the full corpus when only one is relevant.

Rewriting expands recall and routing picks the stack, and that one small-model call upstream pays for itself many times over.

## Retrieval-augmented evals

Eval discipline for retrieval is the same shape as eval discipline anywhere: a held-out set of queries with known-correct chunks, a runner that executes the stack and checks recall@K and rerank position of the correct chunk, a CI integration that runs on every change. The retrieval-specific twist is that the eval has to test both the retrieval (did the right chunk get into the candidate set) and the rerank (did the right chunk end up in the top N).

Two metrics matter most. Recall@30 measures whether the candidate set included the right chunk before reranking, which catches retrieval-stack failures. Mean reciprocal rank (MRR) of the correct chunk in the reranked top-10 measures whether the reranker placed the right chunk near the top, which catches reranker failures. When recall@30 drops, the chunker or the index broke. When MRR drops but recall@30 holds, the reranker broke or the prompt changed in a way that affected reranking. These are different fixes for different failures, and the metrics tell you which is which.

The other gauge is production. Log every retrieval as a LogFire span with the query, the per-stack candidate sets with scores, the reranked output, the chunks that landed in the prompt, and any user feedback signals. The dashboard surfaces queries with low rerank scores (the system wasn't confident), queries with high reranker disagreement (the candidate set was diverse), and queries with thumbs-down feedback (the user disagreed with the answer). Those three views answer "where is retrieval failing in production" without requiring the operator to read raw logs.

## What this gets you

A retrieval layer built this way is the substrate every other AI feature in the system depends on. The RAG agent gets better answers. The semantic search UI returns more relevant results. The content compiler, my pipeline that turns sources into articles, finds the right citations when it resolves each claim's evidence. The agent's tool calls land on the right context. The investment compounds across every downstream feature because retrieval is a horizontal capability, not a vertical one.

If you're evaluating a retrieval team or vendor, the questions are about which stacks they compose, whether they ship a reranker, whether they handle query rewriting and semantic routing, whether they have a recall-and-MRR eval harness, and whether their production observability surfaces low-confidence retrievals. If they sell you "vector search" and stop there, what they have is one third. If they walk you through composition and rerank and rewriting and evals, what they have is the system.

// RELATED

For the full RAG architecture this layer plugs into, read [RAG Knowledge Engines](/wiki/ai-and-technical-development/default/rag-systems). For the eval discipline that catches retrieval failures, read [Evals & Observability](/wiki/ai-and-technical-development/default/evals-observability). For the architectural pattern these tools live inside, read [Unified Architecture](/wiki/ai-and-technical-development/default/unified-architecture).

Source: https://andydataguy.com/wiki/ai-and-technical-development/default/rag-retrieval-tools