The thesis
A retrieval-augmented generation (RAG) system is an answer machine bolted to a search engine, and the interesting failure mode is the search, not the model. When a RAG demo collapses on a real corpus, the model is almost never the cause. The retrieval layer pulled the wrong chunks, the chunks were the wrong shape, the chunks lost the document structure they came from, or the answer was rendered without any auditable link back to where it came from. Fix the retrieval and ninety percent of the hallucination story disappears with it.
The frame I work from is simple: embeddings are necessary and not sufficient. Semantic search finds nearby meaning and misses structurally relevant context, and graphs do the reverse. Once the corpus is real, the hybrid of the two stops being optional. And the answer that gets returned to the user has to carry its sources with it as first-class objects, not as a post-hoc footnote. If the citation is a string concatenation, the system isn't auditable. If the citation is a typed reference back to a chunk that knows its parent document and its parent section, the system can be debugged the way you debug any other production data path.
Why vector-only fails on a real corpus
The vector demo wins because the demo corpus is small and the questions are flattering: twenty PDFs and a handful of curated queries. Cosine similarity finds plausible chunks and the model writes something coherent. It looks like a working system, and it isn't.
Three things break the moment the corpus crosses a few thousand documents. The first is recall. Dense embeddings cluster meaning, not exact phrases. When a user asks for a clause that contains a specific defined term ("Material Adverse Event," "Permitted Transferee," "Net Revenue Retention") the embedding doesn't know that term is load-bearing. It finds chunks that talk around the topic and misses the chunk where the term is actually defined. BM25 keyword search catches this trivially because lexical match is what BM25 does. Hybrid stacks exist because each half compensates for the other half's blind spot.
The second is structure loss. A large corpus is a hierarchy, not a pile of independent essays, and contracts are the clearest example: the master agreement points to schedules, the schedules reference exhibits, and the exhibits reference defined terms in the master. A naive chunker reading those documents into 800-token windows shreds the hierarchy and the embedding has no way to reconstruct it. The user asks "what is the indemnification cap" and the system returns a chunk from the schedule that quotes the cap, missing the master clause that says the schedule's cap is overridden by the side letter. The answer is wrong by structural omission, not by retrieval distance.
The third is freshness. Embedding indexes go stale as new documents land and old ones get superseded. The naive build re-embeds the full corpus on every change. The grown-up build maintains an incremental index keyed on document version with a backfill path for re-embedding when the underlying model changes. An incremental index isn't exotic, and it's the difference between a system that runs and a system that quietly drifts until the day a user catches it citing a contract that was rescinded six months ago.
The hybrid architecture in practice
The shape that works is three retrieval stacks composed behind a single retrieval-and-rerank API. Lexical search (BM25 via Typesense or Elasticsearch) catches exact phrases and named entities. Dense vector search (Qdrant, pgvector, or whatever you're running) catches semantic neighborhoods. Graph traversal (Neo4j with Graphiti, or a lighter-weight graph store) catches structural relationships the other two can't see. Each returns a candidate set with scores. A reranker, usually a cross-encoder (Cohere Rerank, or a fine-tuned bge-reranker), takes the union, evaluates each candidate against the query in pairwise fashion, and produces a final ordered list. The model never sees the raw retrievals, only the top N reranked chunks plus their metadata.
The metadata is where the system earns its keep. Every chunk carries a document ID, a section path (an array of section IDs from root to leaf), a span (start and end character offsets in the source), a version, and a checksum. The model is instructed to answer the question and to attach a citation array where each citation is a typed reference to one or more of those chunks. The rendering layer turns the citation array into footnotes the user can click. If the user clicks a citation, the rendering layer fetches the chunk by ID, fetches the parent document, and shows the user the exact span highlighted in context. If the chunk version no longer matches the live document version, the rendering layer warns the user and offers to re-resolve.
It sounds like a lot, and it is, but every piece is load-bearing. Without lexical, the named-entity question fails. Without dense, the conceptual question fails. Without graph, the structural question fails. Without rerank, the union is too noisy for the model. Without typed citations, the system is unauditable. Skipping any of the five components saves a week of build time and costs you the production deployment.
Citations as first-class objects
A first-class citation is a typed Pydantic model (a Python class with validated fields) that lives in the database, has a UUID, references a chunk and a document and a version, and is queryable. A footnote in a string is a presentational artifact: it's the last step of rendering, not the data shape.
The reason typed citations matter is debugging. When a user reports a wrong answer, the question isn't "why did the model say that." The question is "which chunks did the retrieval return, what was the rerank score, what citations did the model attach, and which of those citations actually supported the claim the user is disputing." If the citations are strings, the answer is reconstructive guesswork. If the citations are typed objects with foreign keys, the answer is one query: pull the citation, pull the chunk, compare to the model's claim, identify whether the failure was retrieval, rerank, or generation. That single query saves entire days of debugging on a production system.
Typed citations also pay off for the user, because users trust answers that show their work. The citation pill that opens an inline panel showing the source paragraph with the relevant span highlighted converts a doubtful user into an evaluating user. The doubtful user closes the tab. The evaluating user comes back with a follow-up question. Only a system that supports evaluation grows the relationship.
How you know retrieval is working
Every RAG system needs an eval harness, and most don't have one. The harness is a set of held-out questions with known-correct chunks. For each question, you run the retrieval stack, you check whether the top K results contain the known-correct chunk, and you log the recall@K, the rerank position of the correct chunk, and the latency. You do this on every deploy. You do this on every embedding model upgrade. You do this when you change the chunker.
The eval set is an unglamorous CSV. The first time you build one, you sit with a domain expert for half a day and write thirty questions. Thirty is enough to catch most regressions. Three hundred is enough to claim statistical significance on small differences between retrieval configurations. The number you pick is determined by how often you intend to change the retrieval stack and how much you care about the difference between configurations. Most teams don't need three hundred. They need thirty and the discipline to run them on every change.
The other gauge is production. Log every retrieval. Log the query, the candidate set with scores, the reranked top N, the chunks that ended up in the prompt, the citations the model produced, and the user's downstream behavior (did they click a citation, did they ask a follow-up, did they give a thumbs-down). Logged as spans in LogFire, an observability platform, this data answers the question "what is going wrong, where, and how often" in a single dashboard. Without those spans you're running blind on a system whose failures stay silent until they're catastrophic.
When RAG is the wrong tool
RAG is the right answer when the corpus is large, the queries are open-ended, and the user benefits from being shown the source. RAG is the wrong answer when any of those three conditions is missing. If the corpus is small (under a few hundred pages of substance), put the whole thing in the context window and stop pretending you need retrieval. If the queries are narrow (always one of five forms), build a structured pipeline with explicit lookups and let the model do final synthesis on a known set of fields. If the user doesn't benefit from sources (a chat interface for casual questions), citations are noise and a smaller, faster, cheaper architecture wins.
The pattern I see most often in client work is RAG used because it's the visible solution, when the actual problem is a structured-data problem dressed up in document language. In those cases the fix is correctly identifying that the system needs schemas and mappings and a SQL or Convex database query, with the model on top doing language framing rather than language reasoning. Better retrieval isn't the fix. RAG is a hammer, and not every problem is a nail.
The metagraph
The graph part of the hybrid stack hints at something larger. Treat the relationships between concepts as first-class and retrieval changes. It stops searching for the nearest chunks and starts navigating how the corpus actually hangs together. I think about a body of knowledge the way a bat thinks about a cave. You get the room by sending signals into the whole space and rebuilding it from what comes back, not by shining a flashlight at one wall. Map the concepts, map how they connect, go several layers deep, and what you end up with is a world model you can query.
A metagraph is that world model: knowledge held as structure rather than as a pile of passages that happen to sit near each other. Two paragraphs can be neighbors in embedding space and mean opposite things, because one was written by an authority and one by a stranger, one is current and one was superseded last quarter, one states a claim and one refutes it. Flat retrieval averages all of that into a single fuzzy neighborhood and loses the distinctions that decide the answer. A metagraph keeps them. Who said it, when it was true, what it contradicts, and where it came from each become a property on the graph itself. Retrieval then walks that structure, following a claim to its source, a term to its definition, a decision to the three decisions that depended on it. The multi-hop questions flat search can't reach turn into a traversal.
The same frame works on a market. A customer lives inside an economy, and that economy has its own customers, its own pressures, its own history. Model that as a graph and you can navigate the relationships between concepts in ways a keyword match or a cosine score never surfaces. The full treatment lives in the WikiDesignCo library. From RAG to Metagraph picks up exactly where this article stops, on the architecture. Echolocation is the long read on the sensing method itself, reading a market by what pings back instead of by its demographics.
What this gets you
A RAG system built this way does the unglamorous thing of staying right. My retrieval engine indexes a 5-million-word personal corpus (hundreds of Markdown files, PDF textbooks, and YouTube transcripts) on the open-source Archon platform, with document-type-specific chunkers and task-typed Gemini embeddings. Its citation accuracy climbed past baseline RAG because of the retrieval architecture, not the model. The base model stayed the same off-the-shelf model, and what changed was the chunker (structure-aware, per document type), the index (hybrid), the rerank (cross-encoder), and the citation surface (typed, queryable, clickable). Every one of those changes was boring, and compounding all of them is what made retrieval trustworthy.
If you're evaluating a RAG vendor or an internal team, ask about the chunker, the index topology, the rerank strategy, the citation data shape, the eval harness, and the production observability, not about the model. If they can't answer those questions with specifics, what they've built is a demo. If they can, what they've built is a system.