# Retrieval Engine: A Business Finds What It Already Knows

Canonical: https://andydataguy.com/wiki/ai-and-technical-development/default/retrieval-engine

Published: 2026-11-21

Author: Anand Houston (AndyDataGuy)

> I build retrieval for an Alaska equipment dealer that owns more manuals, films and pictures than one person could read. The same engine fits any business with a pile.

> Figure: Andy holds up a lighthouse lens inside a dark dealership warehouse full of manuals, film reels and photographs, aiming one beam at a single open page while a bulldozer waits under the aurora outside.

## If retrieval works for heavy equipment in Alaska it probably works for your niche

### I build a data platform for Alaska's only Topcon dealer

The company I'm building for right now sells the gear that tells a bulldozer exactly where its blade is. It owns 4,685 manufacturer documents that run to 52,363 pages, plus 26,026 pictures and 666 videos. No one person could read all of it, and nobody there should have to. They have machines to sell and customers to keep running. Still, most of the answers a customer asks for are somewhere in that pile, and so is the raw material for every post, film and product page the company could publish. Finding the right page when somebody asks takes machinery. The trade calls it retrieval, and I think it's the engine under any business that owns a pile of material and has plenty to say about it.

> Figure: Four thousand manuals, six hundred films and a lot of photographs. Nobody reads that pile by hand.

I'm a senior data engineer. My specialty is agent infrastructure, which is the plumbing that lets AI agents do real work inside a company. Right now I'm the forward-deployed engineer for GPS Alaska, which means I work inside one company and build for that company. My other background, performance advertising, comes back in chapter four, because it's the reason any of this makes money.

This picks up where my article on data folding left off. That one was about the infinite content system, which is how a small company keeps making posts, films and pages without running dry. This one goes back underneath it, into [hyper-personalization](https://andydataguy.com/wiki/business-intelligence/default/hyperrelevance-cartography), which is content that fits one particular reader. It grounds that in [vector engineering](https://andydataguy.com/wiki/ai-and-technical-development/default/vector-engineering), which is the craft of storing meaning as numbers a computer can compare. You don't need to have read it first, because this piece explains everything it relies on as it goes.

The idea I'm carrying over is the one that article ended on. You build one world model, which is my name for a single organized record of what a company knows: its products, its customers, its stories. Every piece of content is that one model seen from one side, so one change to the model shows up in every piece. Douglas Hofstadter carved a block of redwood for the cover of *Gödel, Escher, Bach* whose shadow is a G from one direction, an E from another and [a B from the third](https://mathworld.wolfram.com/Trip-Let.html). He wrote that the three men in his title "were only shadows cast in different directions by some central solid essence." An article, a reel and an email about the same machine are shadows of that kind.

> Figure: Turn the lamp to throw an article, a reel and an email from one block, then add a fact and watch all three change.

**GPS Alaska is the live example, and you can go and look at it.** It's [the only authorized Topcon dealer in the state](https://www.gpsalaska.com/), an Anchorage company that sells machine control and survey equipment to contractors, surveyors and engineers. Topcon makes positioning equipment. Machine control means that sensors, satellite positioning and a 3D model of the site guide the blade of a dozer or a grader, so the operator cuts to the plan without a crew hammering stakes into the ground first. The state it serves has more than twice the land of Texas, and [about 82 percent of its communities aren't on the road system](https://dot.alaska.gov/stwdplng/freightplan). The industry it serves came late to software. When the McKinsey Global Institute [ranked American industries by how digital they'd become](https://www.mckinsey.com/mgi/our-research/reinventing-construction-through-a-productivity-revolution), construction came second to last.

> Figure: Satellites, a site plan and a sensor tell the blade where to cut. That's the business I build for.

I made [a short presentation of the platform](https://gps-alaska-briefing.vercel.app/briefing) for the dealer, with one plain card for each thing it can now do. This article is the engineering behind those cards: what each one means and how it works.

This isn't really an article about one dealer. It's for anyone whose business runs on many kinds of material and who needs a steady stream of content about it. If the method works for something as narrow as machine control on heavy equipment, it will very likely work in your niche too.

So what does a pile like that look like from the inside?

### Most of a dealer's knowledge sits in manuals and films and one person's head

**A manual looks searchable and mostly isn't.** From the inside, those 4,685 documents came to 52,363 pages once a machine had read every one. A manual is pages of diagrams, tables and photographs with captions, and a plain text search sees none of the pictures. On [a library of manufacturer PDFs I've written up](https://andydataguy.com/case-studies/technical-pdf-library-55k-pages), an early pass found the outlines of 38,233 pictures and saved none of them. Every picture has to be cut out, indexed and correlated with the page it came from. [Detail that gets dropped at capture can never be searched later](https://andydataguy.com/wiki/business-intelligence/default/shape-of-data-ceiling).

Film is harder, because it has to come apart before anyone can search it. The 666 videos became 10,255 clips, 27.3 hours of footage and 972 soundbites. A person searching wants the ten seconds where the blade drops, and a full-length video file can't hand them that. Some of the footage comes from the manufacturers' own YouTube channels, with their blessing. Vendors and dealers, we're on the same side of the table. Add the pictures and the platform now holds 39,630 clips, soundbites and pictures on one map.

> Figure: Pour one dealer's pile into the stream, then swap the niche and watch the same stream run.

**Most companies are sitting on a pile like this.** IDC, a research firm, [estimated that 90 percent of the data organizations generated in 2022 was unstructured](https://www.datapressepremium.com/rmdiff/2012956/IDC_Unstructured_Data_White_Paper.pdf). Unstructured is the trade's word for anything that doesn't sit in tidy rows and columns: a PDF, a photo, a recording of a call. By the same estimate only about half of it ever gets analysed.

> Figure: Tidy rows are the tip. Most of what a company knows sits below the waterline.

Meanwhile the demand for content keeps climbing. Adobe sells content tools, so weigh this accordingly, but when it [surveyed more than 1,600 marketers](https://business.adobe.com/blog/71-percent-of-marketers-say-content-demand-to-increase-5x) in 2025, 62 percent said demand for content had grown fivefold or more in two years. For a dealer, that demand means volume: many short pieces, one for each situation a crew meets on a job site. Not all of them are about the technology. The ones that travel are true stories, funny and familiar, that an operator forwards to a friend. Nobody can write that many true stories from memory. They have to come out of the pile.

**The best part of the pile was never written down.** At a dealership I've [written about before](https://andydataguy.com/case-studies/contentfactory-positioning-dealership), decades of specifics about what fits which fleet and what breaks in the field "lived in one head." The philosopher Michael Polanyi had the sentence for this in 1966: "[we can know more than we can tell](https://press.uchicago.edu/ucp/books/book/chicago/T/bo6035368.html)." His example was a face. You can pick it out of a million and you can't describe it. An owner who can get a stranded machine moving from a phone call knows more than any manual holds. Recordings are how that kind of knowledge becomes data, and [a call archive](https://andydataguy.com/case-studies/call-archive-21-case-studies) is searchable in a way a memory isn't.

> Figure: The best knowledge lives in one person's head until a recording turns it into data.

So that's the pile: pages, film and memory, in forms a search box can't read. The tempting move is to skip the engineering. Why can't you hand the whole pile to an AI model and ask it?

### I hand an AI model a few right pages instead of the whole pile

**A model that's given too much to read loses the middle of it.** The whole pile won't fit in one request to begin with. Suppose it did. Nelson Liu and his colleagues [tested this](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00638/119630/Lost-in-the-Middle-How-Language-Models-Use-Long) and found that models answer best when the fact they need sits at the beginning or the end of what they were handed, and a good deal worse when it sits in the middle. Chroma, a company that makes a database for this kind of search, [ran 18 models](https://www.trychroma.com/research/context-rot) and found that performance grew less reliable as the input grew. A 2025 paper [found the drop even when the right passage was guaranteed to be in there](https://aclanthology.org/2025.findings-emnlp.1264): somewhere between 13.9 and 85 percent, from length alone. So the model gets a handful of pages that answer the question, and finding that handful is the job. That finding step is retrieval.

> Figure: Models read the start and the end. The page that mattered was in the middle.

**Lighthouses wasted most of their light until 1823.** An open flame throws its light in every direction, and by the National Park Service's account it [loses nearly 97 percent of it](https://www.nps.gov/caha/learn/historyculture/fresnellens.htm). In July 1823 Augustin-Jean Fresnel lit a lens at Cordouan, on the French coast, that bent one lamp's light into a beam ships could see more than 20 miles out. The lamp stayed the same size and the light got aimed. Pasting a whole pile into a model is the open flame, and retrieval is the lens.

> Figure: Flood the archive with light, then narrow it to a beam and count the shelves that answer.

**Patrick Lewis named retrieval-augmented generation and later apologized for the name.** In 2020 he and eleven co-authors [published a paper](https://arxiv.org/abs/2005.11401) that had a language model look things up before it answered. They shortened the name to RAG. Lewis [said later](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation) that they "definitely would have put more thought into the name" had they known how far the work would travel, and that "no one had a better idea." They gave the field its engine and its least flattering acronym. One of his co-authors, Douwe Kiela, has said that "what people call RAG now is almost a baseline in the original RAG paper."

The search half is much older. In 1972 Karen Spärck Jones, at Cambridge, [worked out](https://www.staff.city.ac.uk/~sbrp622/idf.html) that a rare word in a question should count for more than a common one. In "the hydraulic valve on a dozer," the word "the" tells a search nothing and "hydraulic" tells it plenty. Stephen Robertson and his team built that into a formula called BM25 in the 1990s. It is still the ground floor of every search I build.

> Figure: A rare word in a question should count for more than a common one, which is the idea under keyword search.

RAG is the default now. When Menlo Ventures [surveyed 600 IT decision-makers](https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise) in the fall of 2024, RAG was the design behind 51 percent of enterprise AI systems, up from 31 percent.

**Question-and-answer RAG is the basic version.** You ask, it answers, and the answer comes from somewhere reasonable. That alone rarely changes a company's numbers. The money is in running the same retrieval all the way into content: RAG all the way through to a content generation pipeline, where one retrieval step answers the question and also feeds the post, the film and the product page. That kind of tool is rooted in making a better experience for the person running the business, and in this case we're helping somebody better serve their customers. So I look at all of this application-first. When I ask why we're learning a technique, the example on the table is the dealer's own manual, not an arbitrary one from a research paper. Once a style of content works there, we should be able to apply it to quite literally anything, from heavy construction equipment automation to forward-deployed AI engineering.

> Figure: One lookup can fill the post, the film and the product page at once.

If the engine is this well understood, why do so many companies still fail to get one running?

### A forward-deployed engineer builds for one customer from inside the business

**Many company AI projects die after the demo.** In July 2024 Gartner [predicted](https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025) that at least 30 percent of generative AI projects would be abandoned after proof of concept, which is the demo stage, by the end of 2025. It named four causes: poor data quality, inadequate risk controls, escalating costs or unclear business value. In January 2026 it [wrote](https://www.gartner.com/en/articles/genai-project-failure) that at least half had been. All four causes are problems of fit between the engine and one particular business, with its own data, its own risks, its own costs and its own reasons.

> Figure: Many company AI projects are abandoned after the demo stage.

The software company Palantir built a job around that fit. In the early 2010s it started putting engineers inside its customers, and it called them Deltas. Its own blog [draws the line](https://blog.palantir.com/dev-versus-delta-demystifying-engineering-roles-at-palantir-ad44c2a6e87) this way: an ordinary developer's focus is "one capability, many customers," and a Delta's is "one customer, many capabilities." The Delta has a twin called the Echo, who finds the real problem before anyone builds. The rest of the industry now calls the job forward-deployed engineer. Lightcast, which tracks job postings, [counted](https://lightcast.io/resources/blog/forward-deployed-engineer-ai-job) about 200 of them in the United States in 2024, about 1,200 in 2025 and more than 5,200 in the first seven months of 2026. Gergely Orosz has written [the fullest history of the role](https://newsletter.pragmaticengineer.com/p/forward-deployed-engineers).

> Figure: One customer, many capabilities. I build from inside the shop.

When you hire someone to build this, the useful question is less what they have built and more what they know about your kind of problem. On a small job like this one, one person does both halves of Palantir's pairing: finding the real problem, and building the retrieval that solves it. My finding half comes from performance advertising and agency operations, where the job is to make a business earn more online. Under that sits an operational background of building CRMs (the customer databases a sales team lives in), company automations, voice systems, sales training and sales team systems. [I built CRMs for teams of 20 to 250 people and ran my own agency](https://andydataguy.com/about) before I spent two years teaching myself data engineering.

Several of my earlier pieces cover parts of this in the abstract, and this article is where they get applied. [An essay on agent handoffs](https://andydataguy.com/wiki/ai-and-technical-development/default/the-ai-agent-industry-has-automated-the-handoff-back-to-you) covers the buyer's side of the fit problem: the agents finish their parts, and the buyer still has to make the result usable. I work [people first, then the product, then the process](https://andydataguy.com/wiki/operational-systems/default/people-product-process). Part 7 covers [Agent Redwood](https://andydataguy.com/case-studies/agent-redwood-blueprint), my twelve-component framework for deciding how deep each part of a build should go. The data folding article covered the questions that come before it. They are the people phase of PPP, my people, product, process framework, which asks about the problem, the future the person wants and the transformation between them. That article also made the case for having a senior operator behind a build, whatever you're trying to optimize for, so I won't make it twice.

For years my designs rested on what was technically possible and what I thought would be impressive. Without a real customer, that was theory. Now we have real field experience, and we've collected so much data along the way that agents can dig around in our own stuff and find millions of words that help. [My own library is over five million words](https://andydataguy.com/case-studies/rag-knowledge-engine). And if that's not helpful enough, there are tens of millions of words' worth of textbooks to reference.

My own trade is advertising and data. So where does the knowing come from, for a subject I haven't lived in for twenty years?

### I start every new subject from its textbooks

**I learned to read many books at once at a library table.** When I was in school I used to go to the library, pick up pretty much as many books as I could carry, go to a table, spread them all out and bounce between them. I could spend hours just digging through books. The system I build does the same thing across hundreds of books at once. Mortimer Adler and Charles Van Doren gave the habit a name in the 1972 edition of *How to Read a Book*. They called it syntopical reading: you read many books on one subject side by side and build "an analysis of the subject that may not be in any of the books."

> Figure: Same habit as school. Carry as many books as I can and bounce between them.

**Textbooks hold other people's stories, and most of them overlap with mine.** That's why we start from the textbooks. A textbook is the nearest thing to a room of practitioners who have already met your problem. Most of what a new subject throws at me turns out to be a version of something I've lived through in another field, and the book shows how this field handles it. Many parts of any plan I make are already in those books, and the skills I need next are the ones I'd be learning from them.

> Figure: Pull a book off the shelf and see how much of its story I'd already lived.

**The retrieval pipeline I'm describing started from one book.** Keith Bourne wrote it and called it "[Unlocking Data with Generative AI and RAG](https://www.packtpub.com/en-us/product/unlocking-data-with-generative-ai-and-rag-9781806381654)." It's in a 606-page second edition, and it lays out a sound starting design for a retrieval pipeline. I kept his stages and built them on my own tools, code and data model. A published textbook has already been through reviewers and editors, so its stages were tested by people other than me. Part 3 walks through what we took from it.

The subject gets the same treatment as the method. One book on the shelf for this client is a construction handbook of more than a thousand pages. Most operators know it exists, and few have time to read it, so we read it for them. The owner of the dealership is an expert who already knows everything in the textbooks and more. We read them so we know what to visualize and create, and then the owner guides us. Every single time we get feedback, we learn more and we get even better.

> Figure: Operators know the thousand-page handbook exists and never have time to read it, so we read it for them.

What goes into the system is the raw information, bundled and packaged in a way that abstracts the textbook part away. That's the idea behind embeddings, which turn a passage into numbers that sit near other passages with a similar meaning. Part 2 explains them. The result works like a Mad Lib with typed blanks, each filled by the passage that fits this reader best.

**Books are one layer of three.** The second is research agents that read the live web, so the work reflects what is true this month. The third searches our own notes. The order matters: the textbook first, then current research, then what practitioners say in public, and my own view last, tested against all three.

> Figure: Books on the bottom, live research in the middle, our own notes on top.

Start with the fundamentals. Different people want different things from the same book, so I read each textbook for the fifth of it that serves the job in hand, and the system keeps that fifth from every book it has read. The longer it runs, the more it holds.

None of this starts fancy. The first pipeline is the simple version a textbook student would build, on our own tools, and each layer after it has to earn its place by adding value. I took that posture from Warren Powell's work at Princeton.

That gets you a library you can search. It isn't yet a library you can trust, and [checking an answer against its source is half the job](https://andydataguy.com/wiki/ai-and-technical-development/default/rag-systems). The first version of [a sales chatbot built on a library like this one](https://andydataguy.com/case-studies/equipment-chatbot-sourced-answers) was asked about a Cat 320 excavator and came back with Volvo passages. I keep [a public ledger](https://andydataguy.com/corpus) of how my own retrieval measures. So where does plain retrieval go wrong?

## An AI is only as right as the pages it gets handed

> Figure: Most of the damage happens far upstream of the answer.

A library you can search isn't yet a library you can trust. When an AI that reads a company's files gives a wrong answer, my experience is that the damage was done before the model typed a word. It was done when the files were cut into pieces, when the pieces were turned into numbers, or when a table or a picture got thrown away on the way in. I'd fix all three before I paid for a smarter model.

I'm in the middle of that job right now. The dealer's platform already takes in more than 50,000 pages of brochures and manuals, and this month it gets rebuilt. It's inspired by a textbook, and it includes a method called RAPTOR and a couple of my own layers. Those come in later parts. The failures come first, because every one of those layers sits on top of them.

> Figure: Fifty thousand pages of brochures and manuals, and this month they get rebuilt.

### A language model writes well and has never read my files

**A language model is a fluent new hire who has never opened my filing cabinet.** It's the kind of AI that sits behind a chatbot. It learned to write from a huge pile of public text, and it writes well about nearly anything. It has never seen my price list, my service manuals or the email where I promised a customer a delivery date. Ask it about those and it answers anyway, in the same sure voice it uses for everything else.

> Figure: Fluent, confident and has never seen your files.

The fix the field settled on is simple to say. Before the model answers, a search step goes into the company's own files, pulls out the few passages that look most useful, and hands them over with the question, and the model writes from those. It's a small trick with an ugly name, which is retrieval-augmented generation, or RAG. Retrieval is the looking up. Generation is the writing.

**The name was a placeholder that nobody replaced.** Patrick Lewis and eleven co-authors at Facebook AI Research, University College London and New York University [named the idea in a 2020 paper](https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf). They took a copy of English Wikipedia from December 2018, split it into about 21 million passages of 100 words each, and let a model look things up in that pile before it answered. Lewis later [told NVIDIA's blog](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation), "We always planned to have a nicer sounding name, but when it came time to write the paper, no one had a better idea." A whole category of software is named after a sticky note.

> Figure: A whole category of software, named after a sticky note.

**An answer can't be better than the pages the model was handed.** Hand it the right page and a decent model does fine. Hand it the wrong page, or half of the right one, and you get a well-written wrong answer. That's why [the sales chatbot we built for a heavy-equipment automation company](https://andydataguy.com/case-studies/equipment-chatbot-sourced-answers) shows the manual pages behind each answer, so a person can check them. If that works on manuals for machine control on bulldozers, it'll work on a price list.

So what happens when the wrong pages come back?

### The wrong pages still produce an answer that sounds sure

Nothing warns you. The model gets the wrong pages and writes from them in the same confident voice, and sometimes the first person to notice is a customer.

**Five researchers in Australia wrote down seven ways a lookup system fails.** In 2024 Scott Barnett and four colleagues at Deakin University [studied three working systems](https://arxiv.org/html/2401.05856). One answered questions about research papers, one tutored students from course material, and one answered biomedical questions from about four thousand PDFs. They came away with seven named failure points. The first three happen before the model reads anything. What was asked for isn't in the files at all. Or it's there and the search ranked it too low. Or the search found it and it got dropped when the pages were packed up for the model. The other four happen afterward, when the model has the right page and still misses the detail, puts it in the wrong format, answers too broadly or leaves part of it out. The paper doesn't count how often each one happens, so I won't either. Its conclusion matches what I've lived: a system like this can only be validated while it's running, and its robustness "evolves rather than designed in at the start." I've written before about [telling a missing source apart from a failed search and from evidence the model ignored](https://andydataguy.com/wiki/business-intelligence/default/space-crystals). Their list is the best map of that territory I've found.

> Figure: Seven places the pipe can leak. Three of them come before the model reads a word.

**A chatbot told a grieving customer one thing and linked to a page that said another.** In November 2022 Jake Moffatt's grandparent died, and he asked the chatbot on Air Canada's website about bereavement fares. The chatbot told him he could apply for the reduced rate within 90 days of the ticket being issued. In the same answer it linked to the airline's bereavement page, which said the policy doesn't apply once travel is finished. He believed the chatbot, paid $1,630.36 for two flights and didn't get the reduced rate. In February 2024 a British Columbia tribunal [ordered Air Canada to pay him $812.02](https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html). The tribunal pointed out that the airline never explained why its webpage deserved more trust than its own chatbot. I don't know how that chatbot was built, and the decision says the airline didn't describe it. I do know that the right page existed, sat one click away, and lost. It's the reason I want [every answer checked against the page and the version it came from](https://andydataguy.com/wiki/ai-and-technical-development/default/rag-systems).

> Figure: The right page was pinned one step away. He believed the kiosk.

**Legal tools built on lookup were still wrong between 17 and 33 percent of the time.** Lawyers have been sold research tools that look things up in case law before they answer. Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher Manning and Daniel Ho at Stanford [tested the leading ones](https://onlinelibrary.wiley.com/doi/10.1111/jels.12413). Each tool hallucinated, which is the trade's word for a confident answer that's false or unsupported, between 17 and 33 percent of the time. The authors say many of those failures "stem from poor retrieval". That's a public test of products lawyers pay for, and it's worth reading before you buy one.

**Three parts of my own platform passed their tests with no data in them.** In [my own platform](https://andydataguy.com/corpus), three components passed their tests and had never touched real data, which is the most misleading green light a system can give. One workspace held 545 stored pieces that turned out to be 318 distinct texts. Each of the 227 duplicates could push a better passage out of the top results. [A request can succeed and still be wrong](https://andydataguy.com/wiki/ai-and-technical-development/default/observability-manifesto). By November, patching that would have cost more than starting again from a tested design, so I started again.

> Figure: Duplicate copies push the better passage out of the top results.

So much of this starts upstream of the model. Which step do I check first?

### A bad cut separates a fact from the words that find it

I check the cut. It's the same habit as [tracing a wrong number back through the pipeline that loaded it](https://andydataguy.com/wiki/business-intelligence/default/data-pipelines): start at the earliest step that could have broken. A manual is too long to hand over whole, so every one of these systems begins by cutting files into pieces. The trade calls a piece a chunk and the cutting chunking. Chunks are measured in tokens, and a token is a word or a piece of one. If the cut is wrong, everything after it is wrong too.

**Text has perforations and a count ignores them.** Picture a strip of raffle tickets. The paper tells you where to tear. Writing does the same thing. A thought ends, the subject turns, and there's a natural place to separate. The lazy way to chunk ignores all of that and cuts every so many tokens. That's like running scissors across the ticket strip every three inches. A few cuts land on a perforation by luck. Most go through the middle of a ticket.

> Figure: Change the count and watch the blade cut through thoughts the perforations keep whole.

**A piece cut loose from its page no longer says what it's about.** Anthropic's engineers gave the example I reach for most. A chunk reads, "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? The sentence was fine on its page. Cut loose, it can't be found by anyone asking about ACME's second quarter of 2023. Their fix was to write a line or two of context onto each chunk before storing it. [In their own tests](https://www.anthropic.com/engineering/contextual-retrieval) failed lookups fell by 35 percent, by 49 percent when the keyword index got the same treatment, and by 67 percent with a second ranking pass on top. Those are a vendor's numbers on a vendor's data, so I hold them loosely. I hold the example tightly. I've argued elsewhere that ["smaller chunks" is an incomplete prescription](https://andydataguy.com/wiki/business-intelligence/default/shape-of-data-ceiling), because [a boundary can separate a passage from its explanation](https://andydataguy.com/wiki/ai-and-technical-development/default/rag-retrieval-tools).

> Figure: A piece cut loose from its page no longer says what it's about.

**Two people at Chroma measured what the cut does.** Chroma makes a database for these pieces. In July 2024 Brandon Smith and Anton Troynikov [published a comparison of cutting methods](https://research.trychroma.com/evaluating-chunking), and two of their measures matter here. Recall asks whether the right words came back at all. Precision asks how much of what came back was what you asked for. A common default, 800 tokens a piece with half of each piece repeated in the next, scored 87.9 percent on recall and 1.4 percent on precision. Nearly everything handed to the model was beside the point. Their own method scored about the same recall, 87.3 percent, and 8.0 percent precision, the best in the table. It starts from crumbs of 50 tokens or fewer and groups the crumbs that mean alike, up to a size limit. That limit was 200 tokens in the row I'm quoting, and the pieces averaged about 103. They built on Greg Kamradt, who made cutting where the meaning shifts popular in 2023. Their cluster semantic chunker is the one I use.

**One study argues against cutting by meaning and I cite it anyway.** Renyi Qu and Forrest Bao at Vectara, with Ruixuan Tu at the University of Wisconsin-Madison, [asked whether cutting by meaning is worth the computing it costs](https://aclanthology.org/2025.findings-naacl.114.pdf). On the real document sets they tried, plain fixed-size pieces won at finding the right document. Cuts by meaning won clearly only on documents stitched together from mixed topics. Their paper says, "the computational costs associated with semantic chunking are not justified by consistent performance gains." I think they're right about their documents. I read an equipment manual as closer to their stitched case, because one page can hold a warning, a torque table and a wiring note. That's a reading. I can only settle it with [a comparison on my own questions](https://andydataguy.com/wiki/ai-and-technical-development/default/vector-engineering) against [a baseline I keep](https://andydataguy.com/wiki/ai-and-technical-development/default/evals-observability).

I aim for pieces of 50 to 150 tokens, cut with that cluster semantic chunker, and I treat that as a target range rather than a wall. The size has a reason. Pieces have to be small enough to cluster cleanly. A piece that holds one thought lands in one neighbourhood. A piece that holds four lands somewhere between them.

A target is not a wall. What matters is that the average stays near the band, with no giant balloon pieces and no tiny useless ones. That holds for every number in a system like this: each one needs a reason behind it.

> Figure: A target band, not a pen. A few strays are fine.

On one handbook, 94 percent of 7,991 chunks landed inside the band after one change. Any chunk under 50 tokens was joined to its nearest neighbour by meaning, inside its own section. The remaining 6 percent were complete short sections. That was good enough, and the useful question was what the figure had been before the change, because a number with no history tells you nothing. On 32,565 passages of my own dictation, the middle piece is 70 tokens long and 79.5 percent sit inside the band. I'm fine with both.

Once a piece is cut well, how does the machine know what it means, and where does that knowing fail?

### A search by meaning grabs lookalikes and misses exact part numbers

It knows by address. In the article before this one I explained that each piece is turned into a long list of numbers called an embedding, and that the list works like a location. Pieces that mean similar things land close together. What lands there can be anything. A point can stand for anything: a passage, a picture, a clip or a page. I use one model for all of them, Google's Gemini Embedding 2, because it handles every kind of media in the same space. Every point has 3,072 numbers, and every text piece is 50 to 150 tokens. Part 6 shows one way to look at the result.

> Figure: An embedding gives every piece an address in a space of meaning, whatever the piece is.

**A question and its answer are different kinds of writing.** The address is where the last article stopped, and one step further in there's a catch. A question is short and it asks. Its answering passage is long and it states. A key looks far more like the other keys on the rack than like the door it opens. So on my platform a stored passage and an incoming question are embedded under two different instructions. One says this is a document to be found. The other says this is a question looking for a document. My corpus page gives the reason: "A question and the passage that answers it are different kinds of writing, and asking one model to place both in the same space with the same instruction costs recall." Google's newest embedding model words the instruction differently, and I haven't confirmed which wording our rebuilt pipeline sends. The idea is the same in both.

> Figure: A question looks like other questions, so I tell the model which one it's holding.

**A closeness score is a ranking.** Google's own documentation says the score between two embeddings ["isn't a calibrated probability"](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/embeddings/get-multimodal-embeddings), and it tells builders to rank results and avoid a fixed cut-off. A high score doesn't mean the system is sure. It means this passage beat the others. Close only means the two are about the same subject, and two passages on one subject can disagree.

**A legal tool answered a question about "moral wrong" from a passage about "moral turpitude".** The Stanford team from two chapters back caught the cleanest case I know. Asked to define the "moral wrong doctrine", one tool leaned on a source that defines moral turpitude. That phrase is, as the authors wrote, "a legal term of art with a seemingly similar but actually unrelated meaning." The two phrases share a word and a courtroom. A search by meaning treated them as neighbours and came back with the wrong one.

> Figure: The claw grabs the capsule that looks right, and I find out what's inside at the chute.

It gets worse with codes. Google's search documentation says a search by meaning can only find what the model can make sense of, and it lists ["arbitrary product numbers or SKUs"](https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/vector-search/about-hybrid-search) among the things that don't work. An equipment catalogue lives on model numbers and part numbers, and a customer who types one expects that exact part. There's a harder limit underneath. In 2025 Orion Weller, Michael Boratko, Iftekhar Naim and Jinhyuk Lee at Google DeepMind [showed that the number of different result sets a search like this can return is capped by the length of the address](https://arxiv.org/abs/2508.21038). They built a deliberately simple test, and "even state-of-the-art models fail" it. I haven't seen their per-model scores, so I'll leave it at their own sentence. I read it as the reason my pipeline never trusts one address per piece on its own.

> Figure: Lookalikes are everywhere in a parts drawer. The customer wants one exact bolt.

The fix is older than embeddings. You run a plain keyword search beside the search by meaning and merge the two lists by rank. BM25 is the classic formula for scoring a keyword match. It's the ground floor, and the other layers can't all go on at once. The next part builds them in order.

That handles sentences. What about the parts of a manual that aren't sentences at all?

### A table torn in half loses its answer

A book is more than its sentences. Breaking one down properly means separating it into its parts: running text, lists, tables, images and formulas. Each becomes its own piece.

**One model takes every kind of piece and charges each a different fare.** Text, images, video frames, audio and whole PDF pages all go through the same door. Google's page for Gemini Embedding 2 lists the fares. An image costs 258 tokens. A PDF page costs 258, because the model looks at it as a picture. Each frame of video costs 66, and each second of audio costs about 25. All of it shares one budget of 8,192 tokens, and anything past that gets "silently truncated". Nothing fails and nothing warns you. The tail of a long recording is gone, which is one more reason I cut first.

> Figure: One model takes every kind of media through the same door, charges each a different fare, and silently drops what runs past the limit.

The payoff is that all the textbooks get chunked into the same atomic-level soup of vector embeddings. So instead of retrieving from an individual book, we're retrieving from the entire library holistically. Pictures ride along. Each image stays attached to its page and its caption, so it comes back whenever that page does. Because the model reads images directly, a relevant picture can also turn up on its own. In my library the image vectors "come from the pixels themselves, not a caption."

> Figure: Lift the glass walls and one question reaches the picture, the clip and the page at once.

**A 50-token cut scored 30 percent on table questions.** LangChain [ran that test in December 2023](https://www.langchain.com/blog/benchmarking-rag-on-tables) and showed why: the cutter broke the tables apart. A text cutter reads a table the way it reads a paragraph, left to right and row by row, and it cuts when the count runs out. The header row ends up in the first piece. Every later piece is a row of numbers with no column names. In 2026 a paper in the journal AI [took the opposite approach on product manuals and engineering documents](https://www.mdpi.com/2673-2688/7/5/161). It kept each clause, table, figure, caption and procedure step connected to its neighbours, and the share of answers that pulled the exact right evidence went from 0.510 to 0.942. Reliable answers on technical documents depend, in their words, "less on retrieving more passages than on preserving the relations that make evidence interpretable."

**My first pass found the outlines of 38,233 pictures and saved none of them.** That's in [the case study on 55,001 pages of technical PDFs](https://andydataguy.com/case-studies/technical-pdf-library-55k-pages). The rebuilt library stores 71,030 picture placements, each with its page and its position on the page, and 30,490 usable images are tagged and embedded. Cutting the words well is only half of it. For every image, check four things: that it was captured at all, that it is tied to the right page and section, that its caption came with it, and that any model or part number beside it is kept.

> Figure: Pull the page apart and the line from the bolt to its part number stops at the torn edge.

A builder who can answer those questions has pieces worth searching. Good pieces are half the job, and I didn't invent the other half. A mentor of mine put it plainly: if a barbecue restaurant across the street is making money, open another barbecue restaurant. Copy what works, and skip what doesn't. So I followed the process in a published book, with its code, instead of inventing my own.

> Figure: My mentor's rule. If it makes money, build another one across the street.

The book's process is what turns good pieces into something you can ask. How does a library of good pieces get read the way a person reads a textbook?

## I borrowed my retrieval pipeline from a textbook on purpose

> Figure: The line starts on the page of a book somebody already wrote.

The last part ended on a library of good pieces and a question about how it gets read. My answer is a pipeline. I didn't invent it. A pipeline is the line of steps a document passes through between the day you upload it and the day it helps answer a question. Mine came out of a published textbook. I'd make that choice again for any business with a room full of files.

### I stopped patching and started over from a published book

**I had a retrieval system and it had turned into a patchwork of fixes.** Starting again from a tested design was cheaper than patching it further. I trust a textbook as scaffolding for a plain reason. Someone else has already checked it. Two technical reviewers and a publisher's editors went through the one I picked before I did.

> Figure: A patchwork of fixes costs more to keep than starting fresh from a proven design.

**My way into any new subject has always been a table covered in books.** These days the table is digital: a library of a few hundred core textbooks, chosen from more than 3,000, that [my agents search](/case-studies/rag-knowledge-engine) alongside my own writing and video. Going into a subject blind, when someone has already written it up, is hubris. Hold anyone who builds a system for you to the same rule.

**One book in that library became the map for the whole pipeline.** Keith Bourne wrote it and called it "[Unlocking Data with Generative AI and RAG](https://www.packtpub.com/en-us/product/unlocking-data-with-generative-ai-and-rag-9781806381647)". Packt published its second edition at the end of 2025. RAG is the practice from Part 2, where a model looks things up in your files before it answers. I kept the book's stages and built them on my own tools, my own code and my own data model. One setting shows the split. His first exercise cuts text into blocks of about a thousand characters. I cut by meaning, in pieces of 50 to 150 tokens, for the reasons in Part 2. I kept his stages and changed the settings.

Bourne's book earned its place in three ways. His book carries one running example from a first small pipeline in chapter 2 to an agent that learns in chapter 19. That's the order I wanted to build in, with the simple version first and the layers later. It sorts the whole job into three stages. That sorting cleared my head. Indexing files the documents, retrieval finds the right pages, and generation writes an answer from them. It ends in territory I already work in, with agents, knowledge graphs and memory. Its copyright page says he wrote it himself, with AI used only for ideas, phrasing and diagram drafts, and that he and Packt's editors checked the technical content. If you're hiring someone to build with AI, that's a line worth asking them for: what the machine did, and who checked it. If I'm going to build on somebody's work, I want to know whose it is.

> Figure: Library rule: learn from the book, give it back, thank whoever wrote it.

**Three more books sit beside his on the table and an older one sits under them.** Ofer Mendelevitch and Forrest Sheng Bao wrote [*Hands-On RAG for Production*](https://ofermend.github.io/hands-on-rag), which covers the distance between a demo and a system people depend on. Marlon Vidal Ferreira's *Spec-Driven Development with BMAD* gave us the working method, which is to write down what you're building before you write the code. Chad Sanderson, Mark Freeman and B. E. Schmidt wrote *Data Contracts*, the book behind my rule that every store of data has a written contract for what goes into it. The older one is [*Introduction to Information Retrieval*](https://nlp.stanford.edu/IR-book/information-retrieval-book.html). Christopher Manning, Prabhakar Raghavan and Hinrich Schütze published it in 2008 and put it online for free. Its first chapter builds a search index from scratch. Retrieval was a library science long before it was a chatbot feature.

Bourne's book is on that table twice. The book went through the same pipeline as every other book, so the map now sits in the library it mapped. That leaves a gap between his book and my build. He gives three stages. Why does my line have nine stations?

### A mistake at one station spoils every station after it

**Three stages turn into nine stations once somebody has to build them.** Indexing alone is five jobs. A document gets ingested, which means taken in and logged. It gets parsed, which means a program works out what's on each page. It gets chunked, which means cut into pieces. Each piece gets embedded, turned into the list of numbers from Part 2 that stands for its meaning. Then the pieces get indexed, which means filed where a search can reach them. Retrieval is the search and, where it's needed, a second read that reorders what the search found. Generation is the model writing an answer from those pages. The ninth station is measure. It's there because [a run can succeed at every station and still deliver a wrong number](/wiki/business-intelligence/default/data-pipelines). What comes off the end of the line is one search tool that every agent calls, the way a program calls a basic command. [One tool serving every caller](/wiki/ai-and-technical-development/default/systems-design) means one place to fix.

> Figure: Nine stations on one line. Every agent calls the same one.

**Damage on this line travels one way and every later station inherits it.** A bad cut ruins everything after it. A page that's cut in the wrong place gets embedded wrong, filed wrong, found wrong and quoted wrong. Every station after the bad cut does its job perfectly on garbage. Scott Barnett and his colleagues at Deakin University named [seven failure points](https://dl.acm.org/doi/10.1145/3644815.3644945) in systems like this one. That's the list from Part 2. Three of the seven happen before the model has read a word. I sort every bad answer the same way: the source was missing, the search failed to find it, or the model had the evidence and ignored it. Each cause has a different fix. That's also why every document goes through the same line. If half the library went down an older line, I can't tell a bad answer from a bad batch.

> Figure: Break one station and watch what arrives at the far end of the line.

**Parsing reads the anatomy of a page before anything else touches it.** A PDF looks like a document to you and like a scatter of ink to a computer. The parser I use is Docling, which came out of [IBM Research](https://arxiv.org/html/2408.09869v5). It looks at every page and hands back the page's anatomy: these lines are a heading, this block is a table, this picture has a caption, the reading order runs this way, and here's the page number for each of them. I treat this as the easy station, and the rule is simple: give the parser the PDF and let it work. It's only easy because a team at IBM did the hard part. The anatomy matters because [detail dropped at capture can never be asked for later](/wiki/business-intelligence/default/shape-of-data-ceiling). Antonio Jimeno Yepes and his colleagues tested this on financial reports. They cut along the document's own elements, so a title starts a new piece and a table stays whole. A system built that way answered [53.19 percent of their test questions correctly against 48.23 percent](https://arxiv.org/html/2402.05131) for fixed pieces of 512 tokens. The size of a piece follows the anatomy as well, and that's why the size target is a range. I've already run these first stations at size, with [2,452 PDFs and 55,001 pages](/case-studies/technical-pdf-library-55k-pages) made searchable in a heavy-equipment niche.

> Figure: Before anything else, the parser works out what's on the page and in what order.

Now the pieces are clean. How does the line find the right one among thousands?

### A word search and a meaning search find different pages

**The oldest part of my search is a word-matching formula from 1994.** It is the ground floor, and nobody can build every layer above it at once. So the layers go in one at a time, from the bottom. BM25 scores how well a page matches the words in a question. The BM stands for Best Match. Stephen Robertson, Steve Walker and their team at City University London described it in [a 1994 paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/okapi_trec3.pdf) for a search system called Okapi. It gives a page more credit for a rare word than for a common one. It also stops rewarding a page for repeating itself. In 2021 Nandan Thakur, Nils Reimers and their colleagues ran ten search systems across [18 different collections](https://arxiv.org/html/2104.08663). The old formula held up as a strong baseline. Several newer systems did worse than it on subjects they hadn't been trained on.

> Figure: Keyword search is the ground floor. Everything else goes on top, one floor at a time.

**A meaning search finds the paraphrase and loses the part number.** In the data folding article I turned every page, clip and photo into numbers so that similar things sit near each other, and Part 2 showed what that buys. A search can match on meaning. Ask it about a blade that drifts and it can find a page that says the cutting edge wanders. The two share no words. Its blind spot is the rare exact string. Researchers at Meta showed [a well-known meaning-based retriever failing](https://aclanthology.org/2022.findings-emnlp.19.pdf) on a name as distinctive as "Thoros of Myr", which a word search finds at once. In heavy equipment, Thoros of Myr is a part number. [Each representation answers different questions](/wiki/business-intelligence/default/shape-of-data-shapes), so I run both searches on every question.

> Figure: Switch on both lamps and see which pages only one of them finds.

**Two ranked lists become one list when each page is scored by its rank.** A word search and a meaning search score on different scales, a 12.4 here and a 0.87 there, so their scores can't be added. Their rankings can. Gordon Cormack and Charles Clarke at the University of Waterloo and Stefan Büttcher at Google published the method in [2009](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) and called it Reciprocal Rank Fusion. On each list, a page earns 1 over 60 plus its rank. A page that's first on one list and third on the other earns 1/61 plus 1/63, while a page that's first on one list and missing from the other earns 1/61, which is about half as much. The page both searches found wins. In their tests the fused list beat the best single system by 4 to 5 percent on average. They reported that the exact value of 60 mattered little. Bourne's hybrid search exercise fuses the same two searches the same way. [My research tools already work like this](/case-studies/research-layer-metagraph-compiler): exact matches come back beside related passages, with a badge on each one that says which path found it.

> Figure: Merge two ranked lists by rank, and the page both searches found wins.

**Fusion can bury a good page on a question it was supposed to help.** An [IEEE study from 2026](http://ieeexplore.ieee.org/document/11738423) found that fusion beat meaning search overall and still pushed relevant pages down on 13 to 48 percent of individual questions. That's why I don't take the gain on faith. I [hold the model and the collection still, change one retrieval path, and compare the evidence and the cost](/wiki/ai-and-technical-development/default/vector-engineering).

Both searches together hand me a shortlist, and it's still too long to give the model whole. Who cuts it down?

### A slower second read decides which few pages reach the model

**The first searches are fast because neither one reads your question and a page together.** They prepared every page long before you asked anything. At question time they only compare what they prepared. A reranker is a second model that does the slow thing. It reads your question and one candidate page side by side, start to finish, and scores how well that page answers that question. Then it does the same for the next candidate. It's the difference between sorting a ring of keys by how they look and trying each key in the lock. That's far too slow for a whole library and fine for a shortlist. Rodrigo Nogueira and Kyunghyun Cho showed what it buys in 2019. On a standard test collection called MS MARCO, a second read by a model named BERT took a ranking score from 0.184 for BM25 alone to 0.372. Omar Khattab and Matei Zaharia at Stanford then found [a middle path called ColBERT](https://dl.acm.org/doi/10.1145/3397271.3401075), which compares the question and the page word by word. It ran about a hundred times faster than the full second read, with competitive results.

> Figure: Try each key in the lock and watch the rack change its order.

**A model can lose a page that sits in the middle of a long stack.** Nelson Liu and his colleagues [measured this](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00638/119630/Lost-in-the-Middle-How-Language-Models-Use-Long) and named it "lost in the middle". When the one useful page sat in the middle of a long stack, a popular model did worse than it did with no pages at all. A later attempt to reproduce the effect on two open models found accuracy [roughly flat across positions](https://arxiv.org/html/2605.27105). So I say a model can lose the middle. I still put the best page first and keep the list short, because that design is safe either way. Anthropic's engineers [published their own numbers](https://www.anthropic.com/engineering/contextual-retrieval) for the second read. They took the top 150 candidates down to 20 with a reranker, and failed retrievals fell from 2.9 percent to 1.9 percent. Those are their tests on their data.

**In my build the second read has to earn its station on the line.** The reranker is in the textbook's design and in my build order. It goes in only if it proves its worth. It costs time on every single question. A [paper with Khattab and Zaharia among its authors](https://arxiv.org/html/2411.11767) also warned that rerankers can get worse when they're handed too many pages after a strong first search. The station goes in where a measurement shows the shortlist needs it. Debugging this stretch of the line means [comparing the candidates, the reranked list and what finally reached the prompt](/wiki/ai-and-technical-development/default/rag-retrieval-tools), because [a request can succeed and still be wrong](/wiki/ai-and-technical-development/default/observability-manifesto).

A shortlist only works if every piece on it still knows what it is. How does a piece keep its place in the book?

### Every piece keeps the address of the page it came from

**A book already has a filing system and my pipeline keeps it.** Part, chapter, section, page: somebody already did that sorting work. A cutter that works by character count throws it away at the first cut. My line holds on to it. I ran a heavy construction handbook of more than 1,200 pages through it as a local test. The handbook came out the other side as 54 chapters holding 9,026 pieces. Every record in the system carries its owner and its history as well. Each one records where it came from and what was done to it, so any answer can be traced back to its page. Part 5 explains the labelling scheme behind that.

**An address lets a small matched piece borrow the section around it.** A piece of 50 to 150 tokens is good for matching and thin for reading. "Tighten to the value in table 4" is no use to anyone until it remembers whose table 4. Part 2 covered Anthropic's fix, a short context line written onto each piece before it's filed, and the numbers they reported for it. The address gives me a second fix. I match on the small piece and then hand the model the section it lives in. LangChain and LlamaIndex, two popular toolkits, both ship this as a standard pattern. In LlamaIndex's own example the gain was small, so I treat it as a pattern to test, not a proven win.

> Figure: Pull one card from the pile and one from the cabinet and see what comes with each.

**Each station gets its own grade so I know which one to fix.** One score for the whole line can't tell me that. Shahul Es, Jithin James, Luis Espinosa-Anke and Steven Schockaert built [a toolkit called Ragas](https://aclanthology.org/2024.eacl-demo.16/) that scores the stations apart. Did the search bring back what an answer needed? Did the model stick to what the search brought back? Bourne's evaluation chapter uses it too. My own rule sits on top, and it's to [keep a baseline and a reason for every accepted difference](/wiki/ai-and-technical-development/default/evals-observability). The payoff looks like [a heavy-equipment sales chatbot](/case-studies/equipment-chatbot-sourced-answers) that answers from the manuals, shows its sources with every answer.

**I keep the borrowed book and my own book report on the same shelf.** The system stores each book and a short report on each book, and it searches both. The book gives the raw material. The report gives the parts that matter for this client. Underneath both are embeddings, so a search reaches the ideas and not only the titles.

> Figure: The book and my report on it sit side by side, and both get searched.

**The borrowed map ends where RAPTOR and my own layers begin.** My one-line description of this pipeline is that it's inspired by that textbook and includes RAPTOR and a couple of my own layers. Bourne's book never mentions RAPTOR. I added it. Part 4 is about nothing else. On top of it sit a couple of layers of my own. Under all of it, the method hasn't moved from the library table. Every layer starts from what the textbooks already know. Part 6 follows that habit as far as it goes. One name runs through both ends of this part. Christopher Manning, whose name is on that free 2008 textbook, is also an author of the RAPTOR paper.

A table of contents is a tree that an author wrote down. What if the system grew a tree of its own, one summary at a time?

## I summarize a library before anyone asks and I group customers the same way

> Figure: Search finds a tree. Somebody asked about the forest.

The last part ended with a pipeline that can find a passage. You ask a question, the search pulls the pieces of text that sit closest to it, and a second pass puts the best ones on top. That works, and I'd defend every stage of it. It also has a hole in it, and the hole only shows up when somebody asks an easy question.

### A pile of small pieces can't tell you what a manual is about

**A search can find one bolt and still have no idea what chapter 7 is for.** Picture a 300-page equipment manual, cut into pieces of about a paragraph each. In this trade a piece is called a chunk. Somebody asks, "What's the torque setting for the M8 bolt on the mounting bracket?" The search goes straight to the torque table and hands it back with its source attached. Now somebody asks, "What does chapter 7 cover, in plain words?" The search does the same thing it did before. It hands back the five chunks that sit closest to the question, and they're probably a torque table, a warning box, a part list and two sentences about calibration. All five came out of chapter 7. None of them says what chapter 7 is for.

The RAPTOR researchers made the same point with a fairy tale. Ask a search over small pieces how Cinderella found her happy ending and it can collect the scenes. Then ask it, in their words, "What is the central theme of the story?" No single scene can answer that. The first question is about one tree and the other is about the forest. I call that a forest question, and [tens of thousands of pages of technical PDFs is exactly where it bites](https://andydataguy.com/case-studies/technical-pdf-library-55k-pages). A pile of chunks is [one shape of data, and it answers one kind of question](https://andydataguy.com/wiki/business-intelligence/default/shape-of-data-shapes).

> Figure: The theme of a story is written in no single scene.

**The theme of a group is written nowhere inside the group.** In the data folding article I showed that every piece has neighbours, and that the villages show up without anyone tagging them. A village can be every passage about one type of mistake a crew makes. I can see it on the map. I can't search for it, because the sentence "this village is about calibration mistakes" isn't in any of the passages. Nobody wrote it down.

I first came at that gap visually. Put the pieces in a 3D cloud and a person can see the clusters and sense what each holds. A sense is a start. The next step is a written summary inside every cluster, made by a model the moment someone opens it, and saved for the next person.

> Figure: A 3D cloud of images lets a person sense the clusters, and a summary written inside each cluster is the next step.

That version writes the summary when I click. What if the forest were written down before anyone asked about it?

### Six people at Stanford built a tree that summarizes a library layer by layer

**RAPTOR came out of Stanford in early 2024 and six people wrote it.** Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie and Christopher Manning are the six, and [their paper appeared at ICLR 2024](https://arxiv.org/abs/2401.18059), one of the main research conferences in machine learning. The name stands for Recursive Abstractive Processing for Tree-Organized Retrieval. Somebody clearly started with the word RAPTOR and worked backwards to the words. I respect a team that knows the brand comes before the science. By his own site's account, Sarthi led the work as a graduate researcher in Manning's group.

> Figure: Six people at Stanford, and a tree that summarizes upward.

RAG is the practice of handing a language model the right passages before it answers, and this tree is the part of it I was most stubborn about. It answers two kinds of question from one index: the broad one that runs across a whole chapter, and the narrow one that goes straight down to a detail. Horizontal is the forest question, the one that runs across a whole chapter. Vertical is the bolt, straight down into one detail. I want one index that serves both.

**The build is one short loop that runs until a single summary is left.** The text gets cut into leaves of about 100 tokens (a token is a word or a piece of one), and a sentence that would cross the limit moves whole to the next leaf. Each leaf becomes a list of numbers, which is the embedding from Part 2. Leaves that mean similar things get grouped. A language model writes a short summary of each group. Then the summaries get treated as new pieces. They're embedded, grouped and summarized in turn, and the loop repeats until one summary sits above everything. The paper measured the trees it built. On average each summary sat over about 6.7 pieces, ran about 131 tokens, and came to about 28 percent of the length of the text underneath it.

The grouping step stands on older work. It shrinks the map with UMAP, the tool [Leland McInnes, John Healy and James Melville published in 2018](https://arxiv.org/abs/1802.03426). It fits soft-edged groups with a Gaussian mixture, using a method that Dempster, Laird and Rubin published in 1977. And it picks how many groups to keep with a score from [a four-page paper Gideon Schwarz wrote in 1978](https://projecteuclid.org/journals/annals-of-statistics/volume-6/issue-2/Estimating-the-dimension-of-a-model/10.1214/aos/1176344136.full). The score rewards a good fit and charges for every extra group, which stops the model from inventing one group per leaf. The leaves themselves were embedded with SBERT, from Nils Reimers and Iryna Gurevych.

I can show you the shape on a real one. On one internal run for the GPS Alaska platform, a single handbook came apart into 9,026 leaves. The first pass wrote 903 summaries. Then came 91, then 10, and the fourth pass wrote one. Ours came out near ten pieces under each summary, a little wider than the paper's 6.7.

> Figure: Press next layer and 9,026 pieces get 903 summaries above them, then 91, then 10, then one.

**One piece is allowed to feed two summaries.** The paper says its groups are soft. Nodes, in its words, "can belong to multiple clusters without requiring a fixed number of clusters." The [reference code](https://github.com/parthsarthi03/raptor) gives every piece a share in every group and joins it to each group where that share is above 0.1. A passage about a survey's method and its results can feed both summaries. This is the single most realistic feature in all of corporate AI. Somebody finally modelled the employee who's on two projects and behind on both. It also means a piece can have two parents, so strictly the thing is a graph and the name should change, except that "Recursive Abstractive Processing for Directed-Acyclic-Graph-Organized Retrieval" doesn't spell anything.

> Figure: One passage, two parents. Every office has this employee.

Hold on to the two-parents detail. It comes back when I get to customers.

The base of this system came from [Bourne's textbook](https://www.amazon.com/dp/1806381656), built on my own tools, and RAPTOR sits on top of it.

So the design is settled. It starts from the textbook, RAPTOR stays in, and two layers of my own sit on top. The same tree now [runs over my own writing, textbooks and video](https://andydataguy.com/case-studies/rag-knowledge-engine), so I can test it on material I know well.

Once the tree exists, how does a question find the right branch?

### I search every layer at once and broad questions land on the summaries

**The paper tried two ways to search the tree and the simpler one won.** The first way walks the tree. It starts at the top summary, keeps the best few matches, steps down a layer and repeats. The other way ignores the layers. It puts every node, the leaves and the summaries of every level, into one pool and pulls the closest matches until a budget is full. The authors call that the collapsed tree. On 20 stories from one of their test sets it "consistently performs better", and the budget that worked best was 2,000 tokens, which is about the top 20 nodes.

Nobody tells that search which layer to use. A broad question is worded like a summary, so it lands near summaries. A narrow one is worded like a detail, so it lands near a leaf. Across the paper's test sets, between 18.5 and 57 percent of what the search pulled were summaries. I've written before about how [retrieval has optics](https://andydataguy.com/wiki/business-intelligence/default/space-crystals), and this is the same idea with a ladder in it: one library looks different from each height.

> Figure: Slide from one bolt to the whole manual and watch the lit pieces climb from the bottom layer to the top.

Back in the manual, the chapter 7 question now pulls a summary of the chapter, two summaries of its main sections and maybe one leaf with a key specification. The M8 bolt question pulls the torque table, exactly as before, because [a narrow question still lands on a leaf, which is why the answers can cite the page](https://andydataguy.com/case-studies/equipment-chatbot-sourced-answers). On that bolt question the tree adds cost and no value. It stays out of the way.

**The headline number is big and the controlled number is small.** Paired with GPT-4, RAPTOR scored 82.6 percent on QuALITY, a test of questions about long passages, against 62.3 for the best earlier system. On QASPER, a test of questions about research papers, it scored 55.7 against 53.9. Those runs changed the reader model as well as the index, so the paper also ran a controlled test with one small reader held constant. There the tree alone added between 0.5 and 2.2 points. I trust a paper more when it prints the small number next to the big one.

The authors printed a second caution too, and it's about invention. A summary is written by a model, and a model can make things up. They checked 150 of their summaries by hand and found a minor invented detail in six, which is 4 percent, and none of the six carried up into a higher layer. I keep every summary tied to the leaves it was written from anyway, because [a summary is only trustworthy if you can walk it back to its sources](https://andydataguy.com/wiki/ai-and-technical-development/default/rag-systems).

**The criticism measures where the tree earns its keep.** An EMNLP 2025 paper gave the reader model a much larger budget of text and found that [simpler pipelines matched or beat the tree](https://aclanthology.org/2025.emnlp-main.1656.pdf). At a 20,000-token budget on QuALITY, RAPTOR scored 84.4 percent against 91.4 for a pipeline that hands over the original passages. That result is real, and it has limits, because other studies measured different conditions. A clinical study at RANLP 2025 found RAPTOR with smaller local models beating GPT-4o on complex questions about clinical trials, and the same gain didn't show up on QuALITY. An ICLR 2026 paper by Xiang and colleagues indexed one book of about 56,000 tokens with several structured methods: [RAPTOR took 135 seconds and 115,541 tokens, and Microsoft's GraphRAG took 292 seconds and 654,673](https://proceedings.iclr.cc/paper_files/paper/2026/file/6c9e01d6cefbbf4cdd265032550e767f-Paper-Conference.pdf). So the tree is cheap to build, and it helps most when the model reading the results is small or the data can't leave the building.

If a passage can belong to two groups at once, can a person?

### A person stands in several neighbourhoods at once

**Netflix worked this out about people years before six researchers worked it out about passages.** In 2016 Todd Yellin of Netflix told Wired that the company [assigns each subscriber three to five clusters, weighted by how well each one matches their taste](https://www.wired.com/2016/03/netflixs-grand-maybe-crazy-plan-conquer-world). By 2017 and 2018 reporters were counting about 2,000 of those "taste communities", and Yellin had a name for the strangers who share one with you: taste doppelgangers. Pinterest's engineers published the same lesson in 2020. Aditya Pal and his co-authors built a system called PinnerSage on the finding that [one embedding per user falls short](https://medium.com/pinterest-engineering/pinnersage-multi-modal-user-embedding-framework-for-recommendations-at-pinterest-bfd116b49475). A light user gets three to five interest clusters and a heavy user gets 75 to 100. They also showed what goes wrong when you average a person. Three unrelated pins, averaged, landed on a point best described as "energy boosting breakfast". Average a person and the result is an address they've never visited.

> Figure: Switch one customer from a single address to three and watch which buildings light up.

**I let people show me their shape the same way I let passages.** The readers of the content go into the same space as the passages, placed by [the language they use](https://andydataguy.com/wiki/business-intelligence/default/market-research). I don't force a structure on that space. I look at the clusters it forms, color them, and watch which pieces pull together and which push apart. That's [vector engineering](https://andydataguy.com/wiki/ai-and-technical-development/default/vector-engineering) in one breath: deciding what sits near what. And [I measured the shape of my own corpus before trusting any picture of it](https://andydataguy.com/corpus).

> Figure: I don't force a shape onto data. I push it around and watch what moves.

Content lands in that space too. A clip of a tilt rotator lands near a photo of one (a tilt rotator is the attachment that gives an excavator's bucket a wrist), so a single search for "tilt rotator" brings back the clips, the photos and the manual pages together. Now put a customer in the same space and run the search from where they stand. That's what hyper-personalization comes to once it's grounded in vectors. It's a search where the question is a person.

> Figure: Clip, photo and manual page about the same attachment end up as neighbours.

**Membership is a share.** This is where the two-parents detail comes back. RAPTOR asks how much of a passage belongs to each group, and it counts the passage wherever that share clears one tenth. Netflix does the same thing to a viewer with its three to five weighted communities. A hard sort would've forced each of them into one box and thrown the rest away.

> Figure: Drag one passage between three ponds and it stays counted in every pond that gets more than a tenth of it.

**Relevance shows up as money.** McKinsey's 2021 report on the field found that personalization [most often drives a 10 to 15 percent revenue lift](https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying). Netflix published its own test in 2026. Against the recommender it runs today, [random recommendations cut engagement by 16 percent](https://netflixtechblog.com/measuring-the-impact-of-personalized-recommendations-4c26be3a4d96). My favourite proof comes from an ad. For Diwali in 2021, Cadbury and its agency Ogilvy India, working with Wavemaker and Rephrase.ai, recreated the actor Shah Rukh Khan's face and voice so that each viewer's ad named local shops near them. They generated about 130,000 unique videos, and [sales grew 35 percent](https://apaceffie.com/docs/default-source/resource-library/apac-effie-2022-winner_not-just-a-cadbury-ad-2-0). Sukesh Nayak of Ogilvy called it "personalization at scale and more importantly, generosity at scale." That campaign earned the click by knowing which shops were near each viewer. In my own work [personalization at scale is a testing problem](https://andydataguy.com/case-studies/programmatic-split-testing), [relevance shows up as conversion](https://andydataguy.com/wiki/sales-and-growth/default/cro), and generic content gets you [an audience that agrees with you and still doesn't move](https://andydataguy.com/wiki/attention-engineering/default/the-mirror-ocean).

> Figure: One ad, rebuilt for every neighbourhood's own shops. Sales went up 35 percent.

If a person stands in five neighbourhoods and keeps moving, what does a one-page persona miss?

### A one-page persona describes someone who has already changed

**Alan Cooper gave software teams the persona and it was a gift.** [By his own account](https://urbanmobilitycourses.eu/wp-content/uploads/2020/08/cooper.com-The-origin-of-personas.pdf), his first one was Kathy, in 1983. She was based on a woman who did "traffic" at an advertising agency, and he used her to design a project-management program. The method reached the rest of the industry through his book *The Inmates Are Running the Asylum*. It got engineers to design for one particular person.

> Figure: The persona started as one useful card about one real person.

Cooper himself complained when the idea got diluted. In 2021 he wrote that his own firm "tightly restricted the number of personas", while other teams had "hundreds of personas, one for each feature".

**A persona goes stale in two directions.** One is detail. In 2006 Christopher Chapman and Russell Milham, both at Microsoft, published [the sharpest early critique](https://journals.sagepub.com/doi/10.1177/154193120605000503). They wrote, "Personas cannot be adequately verified or falsified and therefore have no demonstrable validity." Two years later Chapman and his co-authors showed that the share of real people who match a persona "decreases rapidly as attributes are added". Every detail that makes the page more vivid makes it true of fewer customers. The other is time. Bernard Jansen, Joni Salminen, Soon-gyo Jung and their colleagues [regenerated 15 data-driven personas from a publisher's YouTube channel every month for 32 months](https://www.sciencedirect.com/science/article/abs/pii/S0957417421010083). The personas changed by 40 percent on average, and 78 percent of them changed more than they stayed the same in what they were interested in.

> Figure: Step the months forward and watch the town outgrow the paper skyline that was cut from it.

**I model the person in more than two dimensions.** A true persona model follows a person as they change. The flat persona that marketers usually write captures one moment and then sits on a wall. A model built from the data can hold many dimensions at once and be updated as the buyer moves.

My own profiles run long. I use a checklist of more than 120 points, covering personality and mindset: demographics, beliefs, backstory, perspectives and more. Few customers need every point filled, and the job decides how deep to go. A 120-point page about an invented average customer would fail Chapman's test. The points earn their place because they hang on neighbourhoods the data found, and because [a buyer model has to be revised as the buyer keeps moving](https://andydataguy.com/wiki/business-intelligence/default/hyperrelevance-cartography).

The detail I put into the model is the depth I get back out, which was the argument of the data folding article. People come first in every framework I use. My [people, product, process](https://andydataguy.com/wiki/operational-systems/default/people-product-process) framework, or PPP, has nine stages in three phases. Its people phase maps the person's problem, the future they want and the transformation between them before any product or process gets designed. So the model of the person has to be the deepest part of the system. Part 6 has a picture for a person's evolution, and it's a willow tree that hangs upside down. Part 7 is about how I set the level of depth for a given job.

Trees and neighbourhoods find what's similar. What finds what's connected?

## Being close to something is different from being connected to it

> Figure: Seven bridges. Euler kept the crossings and threw away the distances.

### A map of connections throws the distances away

**Part 4 put every piece of the library in a neighbourhood, and a neighbourhood only knows who lives nearby.** Two manuals for the same machine sit close together because they use the same words. The neighbourhood can't tell me that one of them replaced the other, and it can't tell me which receiver fits which machine. Those are connections. A connection is its own form of knowledge, separate from a likeness, and I've written before that [graphs, vectors and tables each answer a different kind of question](https://andydataguy.com/wiki/business-intelligence/default/shape-of-data-shapes).

**People have been throwing away distance to keep connection since 1735.** The city of Königsberg had seven bridges, and the locals argued about whether a person could cross each one exactly once on a single walk. Leonhard Euler told a correspondent the puzzle had little to do with mathematics, and then [he solved it](https://scholarlycommons.pacific.edu/euler-works/53). He ignored the streets, the bends in the river and every distance in the city. He kept two things, the pieces of land and the bridges between them. That's usually called the first paper in graph theory, and a graph in that sense is just dots with lines between them.

Almost two centuries later a man with no job did the same thing for commuters. Harry Beck was an engineering draughtsman who'd been laid off by the London Underground. In 1931, in his spare time, he [redrew the Tube map](https://www.ltmuseum.co.uk/collections/stories/design/transforming-tube-map-harry-becks-iconic-design) the way he'd draw an electrical circuit. He kept the order of the stations and every connection. He straightened each line to horizontal, vertical or 45 degrees, and he dropped the geography. His bosses turned it down as too revolutionary and accepted it a year later. The first pocket edition came out in January 1933 with a print run of 750,000, when a normal run was 10,000 to 20,000. Beck was paid 10 guineas, by Transport for London's account. An out-of-work draughtsman beat the professionals with a diagram.

> Figure: Drag the map from likeness to connection, and every tangled line straightens without losing a stop.

**I only need two words to work with a graph.** An entity is a thing with an identity, like a machine, a manual, a person or a town. A relation is a named line between two entities, like "fits", "replaced" or "written by". Collect enough of both and you have what Aidan Hogan and his colleagues defined in 2021 as a knowledge graph, "a graph of data intended to accumulate and convey knowledge of the real world". In the data folding article I borrowed Google's phrase "things, not strings" for this, and I admitted that my own graph is spread across a handful of tools, each holding its own piece.

If the neighbourhoods already find everything that looks alike, when do I need the lines?

### Some questions can only be answered by walking from one fact to the next

**A similarity search answers "what looks like this", and it's good at that.** The neighbourhoods come from [vector engineering](https://andydataguy.com/wiki/ai-and-technical-development/default/vector-engineering), which turns a passage into a point in a space and finds the points nearest your question. Now take a question from heavy equipment. I made this one up to show the shape. Which receiver works with this machine on this software version? No single page holds that answer. One page says which receivers fit the machine, and another says which software each receiver runs. The answer sits where those two facts meet. [Plain retrieval](https://andydataguy.com/wiki/ai-and-technical-development/default/rag-systems) checks an answer against the passage it came from, and nobody built it to walk from one fact to the next. The [equipment chatbot I built](https://andydataguy.com/case-studies/equipment-chatbot-sourced-answers) shows its sources, and that question asks it for two sources at once.

**A walk spreads out from the things the question names.** In 2024 Bernal Jiménez Gutiérrez and a team at Ohio State published [HippoRAG](https://proceedings.neurips.cc/paper_files/paper/2024/file/6ddc001d07ca4f319af96a3024f6dbd1-Paper-Conference.pdf), which they modelled on how the hippocampus indexes memory. It finds the entities in your question and drops a scent on each one. The scent spreads along the lines of a graph and pools where the paths meet. Their example question was "In which district was Alhandra born?", and the passage that answers it never mentions Alhandra. On one benchmark of multi-step questions, the share of questions where every needed passage was found rose from 37.1 percent to 75.7.

> Figure: No brain, and it still finds the shortest route.

The picture I keep in my head for this is a slime mould. In 2000 Toshiyuki Nakagaki put one in a small maze with oat flakes at two exits. It spread into every corridor and then pulled back from the dead ends. In 17 of 19 trials it finished as [one strand along the shortest route](https://www.science.org/content/article/slimy-not-stupid). It has no brain. Nakagaki went on to win two Ig Nobel Prizes for this line of work, and I think that's the correct number.

> Figure: Run one question as a match and then as a walk, then switch to a simple question and watch the match win.

**Graphs lose on the easy questions, and someone measured it.** Zhishang Xiang and his colleagues built [a benchmark called GraphRAG-Bench](https://graphrag-bench.github.io/) to find out when a graph helps. On simple fact lookups, plain retrieval with a reranker (the second pass from Part 3 that re-sorts the shortlist) scored 60.92 and Microsoft's GraphRAG scored 49.29. On questions that need several steps of reasoning, the graph method HippoRAG 2 scored 53.38 and plain retrieval scored 42.93. A 2026 master's thesis at Aalto University ran a similar test on heavy-machinery manuals. It built a graph where every node was labelled as a kind of thing, like a part number or a machine, and every fact traced back to its source passage. That graph gave more faithful answers on one company's manuals and made no significant difference on a second company's. A graph bolted on without the grounding made answers worse, and the thesis concludes that "ungrounded, uncalibrated structure actively harms retrieval quality".

**I'm a sceptic about graphs on their own.** A graph database stores triplets, which means one thing, one relation and one other thing. That's a real limit, because a lot of what a business knows doesn't fit one-thing-one-relation-one-thing. Embeddings don't have that limit. Text, pictures, audio and video all become points in one space, and any of them can be a thing in the graph too. So I don't want to trade the dots for the lines. I want both views of the same things.

What kind of structure can hold the match, the walk and the source of every fact at the same time?

### A metagraph records where each fact came from and when it was true

**A metagraph is a graph where you can point at a line the way you point at a thing.** It takes three steps to get there. A plain graph joins two things with a line. Let one line join several things at once and you have a hypergraph. The third step gives the line itself an address, so I can attach facts to it and draw another line to it, and that's a metagraph. On my own site [I use the word](https://andydataguy.com/wiki/business-intelligence/default/shape-of-data-provenance) for "a richer structure in which relationships are addressable and can themselves participate in relationships."

> Figure: In a metagraph the bridge has its own address.

**The word has two families behind it.** Amit Basu and Robert Blanning named [the metagraph](https://link.springer.com/book/10.1007/978-0-387-37234-1) in 1995. Their metagraph maps sets of things to sets of things. The one I mean comes from Doctor Ben Goertzel. His [2020 paper](https://arxiv.org/abs/2012.01759) describes "nodes and links, where links can point to nodes, links or whole subgraphs". The nesting part is older than both. David Harel drew blobs inside blobs in 1988, and in 1994 Alexandra Poulovassilis and Mark Levene described "a graph whose nodes can themselves be graphs".

> Figure: Zoom in on a line between two things and it turns into a thing, with a source and dates of its own.

**Two books on one subject show why a line needs facts of its own.** Take two books on text analysis. Their covers and tables of contents look alike, but one is about engineering decisions and the other about foundational concepts. In a plain graph, both books point at the same topic with the same line. In a metagraph each line says which book, which year and how deep.

The rest of the field is heading the same way. In 2025 Preston Rasmussen and the team at Zep published [Graphiti](https://arxiv.org/html/2501.13956), which stores every fact with the date it became true and the date it stopped, and keeps the raw episode the fact came from. Remember Celina from the data folding article? My model of that town went stale because nothing in it carried a date.

**I've been trying to build a metagraph for a while.** Earlier attempts used Neo4j, Graphiti and my own data models. The goal is a full metagraph, which needs a whole layer of infrastructure under it, the way agents turned out to need one. What runs today is a pseudo-metagraph. [My agents' memory](https://andydataguy.com/case-studies/metagraph-agent-memory) is a temporal graph on Neo4j built on Graphiti, which I extended and host myself. Temporal means every fact carries its dates. Its home is a project called [WikiDesignCo](https://andydataguy.com/looikos/infra/wikidesignco).

> Figure: Circles inside circles, drawn one more time. It's a whole stack.

That covers the lines. If a node can hold a whole graph, though, what's inside a node?

### A relationship is one more part you snap onto a thing

**The inside of a node is the same ECS I described last time.** In that article I said I look at everything as entities, components and systems, and I told the history from *Dungeon Siege* to Adam Martin's bicycle. Through all of it a component only ever described one entity. In 2022 Sander Mertens, who writes an ECS library called Flecs, [showed relationships as pairs](https://ajmmertens.medium.com/building-games-in-ecs-with-entity-relationships-657275ba2c6c) you add like any other component. His examples add the pair (Likes, Dogs) to an entity and the pair (ChildOf, Sun) to Earth. The Flecs manual says relationships "make it possible to describe entity graphs natively in ECS". Bevy, an open-source game engine, built the same feature and credited Flecs for it.

Game engines show what this structure can do. A simulated city or factory is a state space: every situation that world could be in. A company's data can be modelled the same way, and content is what you get when you show one situation in it to one reader.

> Figure: A game world is a state space. So is a company's content.

**A few kinds of part make an absurd number of things.** In 2004 a mathematician in Copenhagen named Søren Eilers doubted LEGO's arithmetic. The company said six of its eight-stud bricks could be combined 102,981,500 ways, and that figure only counted towers six bricks high. Eilers counted every arrangement and [got 915,103,765](https://web.math.ku.dk/~eilers/lego.html). A high-school student named Mikkel Abrahamsen wrote his own program and confirmed it. Components multiply the same way, so a short list of them can describe a huge world.

> Figure: Snap a part onto a plate and bridge two plates with a long one, and the machine only stops where it finds its colour.

**Data science already had its own word for a component, and the word is feature.** A lot of what we're doing with this ontology is very similar to a feature store. An ontology is the agreed list of what kinds of things exist and how they relate. Mike Del Balso and Jeremy Hermann introduced one at Uber in 2017, and it held about 10,000 shared features. A feature store keeps one definition of each fact so that every model reads the same one. Every record follows the same entity-and-component model, and each record type has a [typed contract](https://andydataguy.com/case-studies/python-systems-stack) that says which fields it holds. That pays off fast. Define one world well, and most of the next one is already built. My rough, unmeasured estimate is about 80 percent.

**The walk and the match meet inside a node.** In 2014 a method called DeepWalk treated random walks across a graph as sentences and learned a vector for every node. So a node in the graph can also be a dot in the space, and one platform can [run full-text, vector and graph retrieval](https://andydataguy.com/case-studies/research-layer-metagraph-compiler) over the same entities. A cluster of those dots makes a neighbourhood, a cloud, and that cloud itself has something of a shape. The local database I test on is Convex, which is my main database, plus Neo4j and Graphiti. The test setup is built to let a person see and explore everything in it.

With that much structure in place, how do I find the route from what hurts to what somebody wants?

### A problem, a wanted future and a transformation make a path

**The form from the last article becomes a route inside a graph.** The data folding article used a short form from the first phase of my [people, product, process](https://andydataguy.com/wiki/operational-systems/default/people-product-process) framework, or PPP. PPP has nine stages in three phases, and the people phase comes first. Its three stages are the problem, the future the person wants and the transformation between them, and the form turns them into 15 to 45 questions. Every answer is an entity or a relation. A problem connects to the solutions that address it. Each solution connects to its trade-offs, and one of them connects to the transformation. That chain is a path through the graph. PST is a different framework with a different job. It stands for problem, story, transformation, and it models the person rather than the situation. Its problem is the loop the person keeps repeating. Its story is the set of beliefs that keeps the loop running. Its transformation is the change that could get them out. In the graph, PST adds a layer to every piece of content: where it sits in that loop and which way it pulls the reader.

> Figure: The person's problem is a loop they keep repeating, and the transformation is the way out of it.

Performance advertisers have walked this path for more than forty years. In 1982 Jonathan Gutman linked a product's attributes to their consequences and then to a person's values. In 1988 he and Thomas Reynolds described an interview method called laddering, built on one question, "Why is that important to you?" Their example ladder climbs from a flavored chip to a strong taste, to eating less, to not getting fat, to a better figure, to self-esteem. They added the ladders together into what they called a hierarchical value map, which is a graph. They said it "typically serves as the basis for the development of advertising strategies". Anybody who's written an ad that converts has climbed that ladder. Finding the cheapest way up a graph is old work too. Edsger Dijkstra designed his shortest-path method in 1956, in about twenty minutes, without pencil and paper.

> Figure: Pick what hurts and close off a fix that won't work for you, and the lit rungs find another way up.

**Neo4j is a database built for that walk, and in this first build it's tested locally.** Emil Eifrem, Johan Svensson and Peter Neubauer came up with its model in 2000. About half of their engineering team was spending its time, in Eifrem's words, "fighting with a relational database". Eifrem says "It was a broad team effort", and the team included master's students at IIT Bombay. Andrés Taylor invented its query language, Cypher, in 2011. Since 2023 Neo4j has also had a vector index, so both kinds of search can run in one database on a laptop.

> Figure: Half their team was fighting a relational database. So they built a different one.

So far I've run Neo4j only on my own machine, to measure it before trusting it. The best-known test looks one-sided. In the book *Neo4j in Action*, a friends-of-friends query over 1,000 users took a relational database 92.613 seconds at five steps deep, and it took Neo4j 0.07. Then a 2026 study timed Neo4j against PostgreSQL over 3,200 runs and concluded "[No paradigm dominates](https://www.mdpi.com/2079-9292/15/19/4399)". It found that how you write the query mattered more than which kind of database you chose.

One handbook on my own machine shows what a graph adds. Neo4j now holds it. The handbook's 54 chapters became 9,026 chunks, 17,206 entities and 38,829 links between a chunk and an entity. One manual turned into tens of thousands of named things and links between them, which is the structure a keyword search never sees. Those are counts from a local test run, and Neo4j isn't deployed anywhere. The production-grade build is planned for December.

> Figure: One handbook, tested on my own machine first.

A node can hold a graph, and a whole graph can become one dot. Part 6 builds a tree out of blocks, and each block is itself an embedding. First there's a gap to close. The graph tells me what's true and what connects, and it can't tell me which of two headlines is better. How does each piece I send out get better than the last one?

## An upside-down willow can show every tournament I run

> Figure: Every contest we run, hanging from one tree.

Part 5 ended with a graph that knows what's true and what connects, and with a question it couldn't answer. A graph can't tell me which of two headlines is better. This part is about how each piece I send out gets better than the last one. Every piece I ship gets decided by many small contests. I wanted one picture that shows all of them at once, and I found it in a tree I used to sit under as a kid.

### Every layer of an article gets its own tournament

**Something as simple as an article is a stack of layers.** It has headlines and subheads, sections and chapters, figures, and bold and italic text inside the sentences. Each is a layer, and the same world model has to show up in every one.

> Figure: Even an article is a stack of layers, and each one can be dialled in.

**The world model has to be able to project onto every single layer.** That reaches down to how a button settles after it moves, across to the colours and when not to use them, up to the voice of the copy, and finally to the person reading, which matters most.

Jiggle physics has a proper name, by the way. In 1981 two Disney animators, Frank Thomas and Ollie Johnston, set out twelve principles of animation in a book called *The Illusion of Life*. One of the twelve, follow-through and overlapping action, covers the loose parts of a thing that keep moving and then settle after the main body stops. John Lasseter carried the principles into 3D in [a 1987 paper](https://research.pixar.com/docs/1987.SiggraphPapers.L.pdf). So the jiggle on a button is a decision with more than forty years of craft behind it. The voice layer is the same kind of decision, and [mine has rules learned from copy I approved](https://andydataguy.com/wiki/ai-and-technical-development/default/voice-compiler).

> Figure: Pick a layer and run its contest, then watch what the winner does to the page.

**When you're dialing something in, each one of those layers has its own scenarios to run.** So each one gets an adversarial tournament. A tournament is the plain idea from the last article. Agents make a lot of versions, every version is scored against a rubric that was written before anyone saw it, and the survivors go another round. Headlines get one, and so do styles, the hook of a joke and the line that creates tension in an article. Anything important gets one.

Agencies already work this way. A copy team writes hundreds of headlines, narrows them to fifty, and hands them to a director, who rejects nearly all of them and sends back two or three that might work. Agents can run the same funnel, with each one arguing from a different angle and five times more ideas on the table than anyone would write by hand.

> Figure: Hundreds of headlines get narrowed to fifty, and the director keeps two or three.

Upworthy did this in public. Its editors wrote 25 headlines for every story, and the archive they left behind holds [32,487 experiments](https://www.nature.com/articles/s41597-021-00934-7) run between January 2013 and April 2015. In paid advertising it's older news still. [I designed automated ad testing](https://andydataguy.com/case-studies/programmatic-split-testing) for a large-budget team that was aiming for more than 100 tests a week.

**You can open the results of two of these tournaments yourself.** [The GPS Alaska briefing](https://gps-alaska-briefing.vercel.app/briefing) has a card that says 700 headlines were written and 76 cleared the bar. Another card holds a bank of 105 jokes from the trades, each one rated, with write-ups on why 44 of them landed. A round narrows a hundred entries to twenty-five, then to ten, and a person makes the final call.

> Figure: Seven hundred headlines in. Seventy-six cleared the bar.

A tournament is only as good as whoever judges each round. So who sits at the table?

### Pixar's Braintrust runs on rules that agents can follow

**Pixar puts a whole film up for criticism about eight times before release.** That testing, by a group called the Braintrust, is why the studio can trust a film before it ships. Mary Coleman, who ran story development at Pixar, says [a film goes up on reels an average of eight times](https://gointothestory.blcklst.com/go-into-the-story-interview-mary-coleman-d01937ae903a). A reel is a rough version of the whole film made from storyboard drawings. In *Creativity, Inc.*, Ed Catmull and Amy Wallace wrote that one 90-minute reel takes about twelve thousand drawings. Story teams commonly draw ten times that number. Coleman says Andrew Stanton likes to say "Be wrong fast."

> Figure: Pixar puts a whole film up for criticism about eight times before release.

**The Braintrust is a short list of rules, so it can be copied.** The last article quoted Catmull on how the first versions of all their films suck. This time I want the rules. Catmull calls the Braintrust Pixar's [primary delivery system for straight talk](https://www.fastcompany.com/3027135/inside-the-pixar-braintrust). It has no authority. The director doesn't have to take a single note, and after the meeting the director decides what to do about the notes. People with a lot of power in the room are supposed to stay quiet for the first ten to fifteen minutes. Catmull's reason is that "If a powerful person talks, they're setting the tone in the room." During *Toy Story 2* the group was five people with nine months to deliver the film, and Catmull credits Stanton with the name.

**So we're creating our own digital brain trust.** A team of agents can follow every one of those rules. Each agent can be given its own background and point of view, so the room argues from real differences instead of agreeing with itself.

> Figure: Seat someone new at the table or let the boss speak first, and watch the notes land somewhere else.

In 2023 a Stanford and Google team published [Generative Agents: Interactive Simulacra of Human Behavior](https://doi.org/10.1145/3586183.3606763). They gave twenty-five agents personalities and goals, put them in a small virtual town and let them plan together.

The researchers gave one agent the wish to throw a Valentine's Day party and gave nobody anything else. Two game days later, 13 of the 25 agents knew about the party, and five of the twelve who were invited showed up. Thank you to all six authors. Bernstein is also an author of Flash Organizations, the paper my imaginary mastermind studied last time.

> Figure: One agent wanted a party. Two days later, five guests showed up.

**The follow-up measured what a deep persona is worth.** In 2024 Park and his colleagues [built agents from two-hour interviews with 1,052 real people](https://arxiv.org/pdf/2411.10109v1). Each agent matched its person's survey answers 85 percent as well as the person matched their own answers two weeks later. Agents built from demographics alone scored 14 to 15 percentage points lower.

The limits are measured too. Three researchers at Harvard Business School tested language models as stand-ins for market research and called them "a supplement to, not a substitute for, human studies". A 2024 paper in *Political Analysis* found that personas built from real survey respondents drifted when the prompt changed slightly. So a simulated panel never gets the last word. The scoring rules are written before the work, [a separate reviewer checks the result](https://andydataguy.com/case-studies/ai-agent-team-checked-work), and a person decides.

A tournament works the same way as that town: many agents, each with a point of view, competing to produce the best headline, joke or design. One round is only the start, though. A single round can only find the best of what was in the room. So how do the rounds build on each other?

### I build each round from what lost the one before

Each round builds on the last, using three old ideas: boosting, which fixes what the last round got wrong; tree search, which decides where to look next; and evolution, which keeps what worked. All three apply to copy as well as code.

**XGBoost builds a strong answer out of small corrections.** It's a method for making predictions from tables of numbers. It starts from a simple guess, measures what the guess still gets wrong, and trains one small decision tree on those errors alone. Then it adds the tree and repeats. Each new tree is scaled down on purpose, which in its authors' words "leaves space for future trees to improve the model". Tianqi Chen and Carlos Guestrin reported that [17 of the 29 winning solutions](https://arxiv.org/pdf/1603.02754) published on Kaggle's blog in 2015 used it. Kaggle is a competition site for data scientists. I aim a round of copy the same way, at what the last round got wrong.

Boosting fixes what's already on the table, and the second idea decides where to look next. The published name is Monte Carlo tree search, and Rémi Coulom [coined it in 2006](https://www.remi-coulom.fr/CG2006/CG2006.pdf) for a program that played Go. The search picks a promising branch, adds a new option to it and plays that option out to see how it does. Then it sends the result back up, so the next pick is better informed. In the same year Levente Kocsis and Csaba Szepesvári worked out how to balance options that look good against options that have barely been tried. A tournament needs that balance, or the strange idea never gets a second look.

> Figure: Tree search plays out a promising branch and sends the result back up, so the next pick is better informed.

**Evolution is the oldest version, and it explains the hill.** In 1932 the biologist Sewall Wright drew fitness as a landscape, with peaks of good designs and valleys between them. He left the axes unlabelled, because the real space has too many dimensions to draw. A climber who only ever steps uphill stops on the first peak he reaches, even when a taller one stands a valley away. A computer science lecture at James Madison University describes it as "Like climbing Everest in thick fog with amnesia." John Holland's 1975 book on genetic algorithms gave the fix its modern form. You send many climbers, keep what the best ones found, and start the next round from there. Google's AI co-scientist [runs a tournament of scientific hypotheses](https://www.nature.com/articles/s41586-026-10644-y) on that plan. Every new hypothesis starts at a rating of 1,200 and moves up or down as it wins and loses debates. Its authors note that the rating is the system grading itself, which is the caution I ended the last chapter on.

> Figure: Drop one climber and watch him stall, then send a crowd and let it learn from the ones that lost.

**Climbing the hill means doing better than what we just did previously.** Copy that got approved was acceptable, which isn't the same as good. A round moves toward good only when it beats the one before it. [Using each result to guide the next attempt](https://andydataguy.com/wiki/ai-and-technical-development/default/mapping-agent-work) is the whole method.

> Figure: Approved isn't the same as good. Each round has to climb higher.

Every round is recorded in a database, with a couple dozen [typed fields](https://andydataguy.com/case-studies/python-systems-stack) for each entry, so the reasons for each score are kept along with the score. It's the same database the last article described, seen one layer at a time.

There is still a winner. The ranking just lets us sort the entries. Every entry is scored from 1 to 5 on five criteria: the idea, its truth, the craft, its fit to the reader and its range. What a 5 and a 1 look like is written down before the round starts, so anyone can see what was tested and why the winner won. [Keeping the losers and their reasons](https://andydataguy.com/wiki/ai-and-technical-development/default/evals-observability) is what makes the next round smarter.

Every layer starts from what is already known, in the same order Part 1 described, and the system gets better each time it runs.

These contests run on every layer of every piece. How would anyone see all of them at once?

### I draw a long tournament as a long branch and a quick one as a short branch

**Everything we break a piece into has to fit in one picture.** I draw it as a tree of blocks hung upside down, like a weeping willow, with its root at the top. It belongs here because it's a RAG visualization. It shows what retrieval handed us and everything we made from it.

**The upside-down weeping willow is a shape we can apply for all these tournaments.** It's a weeping willow because some tournaments are very long, and they create a longer branch. Then there are the tournaments where we come to a conclusion relatively quickly, where we find a local maxima of improvement fast. That's the nearest peak from the last chapter, and those ones get a shorter branch. And then those branches expand. A willow might hold a handful of branches or tens of thousands, so the root can be thick or a single strand. There are all sorts of creative ways we can play with colors and density. You can zoom in and out, and clicking each block would be, in itself, an entire world model.

> Figure: Turn the willow, change how many tournaments hang from it, and open any block to find the world inside.

Every block on that willow is one scored entry. Each cluster of blocks down a branch is one round, and the clusters get thinner as entries are knocked out. The bright block at the tip is the pick. Losers stay on the branch, dimmer, because they're information. I made up most of these branches to show the range. One branch carries numbers, and that one is my real funnel: 100 entries, then 25, then ten, then my feedback.

**Others have drawn parts of this.** James Rosindell and Luke Harmon built [OneZoom](https://www.onezoom.org/), which puts the tree of life on one zoomable page with more than two million species on it. Maps of embeddings exist as well, and they come with a warning from inside their own field. Martin Wattenberg, Fernanda Viégas and Ian Johnson showed that in one popular kind of map [the sizes of clusters mean nothing](http://distill.pub/2016/misread-tsne). The distances between clusters might not mean anything either. On the willow, the length of a branch is a count of rounds, so it means what it looks like it means. I haven't found a picture that shows how long a contest ran, keeps its losers in view and opens every entry into a space of its own.

That covers how to read it. Why a willow, though, of all the trees?

### I sat under weeping willows in Frisco when it was still a cow town

**I grew up in a neighborhood near a place called Plantation, and there were lots of weeping willow trees.** They're beautiful. I liked to sit by them, and it's very peaceful the way that they sway in the wind.

> Figure: A creek, a willow and a lot of time to think.

**When you look at the weeping willows, it's dark if you think about it.** Growing up in Texas, I'd sit under one by the creek a few blocks from Smith Elementary in Frisco and think about the suffering the South had seen, and about the freedom I had just to sit there. The tree has carried that weight for a long time. A willow bent over an urn took hold as the mourning picture on New England gravestones in the 1780s. Virginia's Department of Historic Resources calls the willow ["a symbol of both mourning and rebirth"](https://www.dhr.virginia.gov/blog-posts/the-evolution-of-funerary-symbolism-or-whats-with-all-the-willow-trees/). A willow earns the rebirth half because it regrows so easily.

**I sat there thinking about how this cow town was going to turn into something completely different.** You understand that you don't know where it's going to go. Looking back, though, you can see the connection of time. I had the cow town part right. Frisco began in 1902 as a water stop on a railroad, a few miles from a trail that longhorn cattle were driven north on. [The census counted 6,138 people there in 1990 and 200,509 in 2020](https://www.tshaonline.org/handbook/entries/frisco-tx). Last time I said Frisco isn't the same as Lewisville, and that a town is a world model. This is the other half of that thought. The world model changes while you're sitting in it.

> Figure: A water stop on the railroad grew into a city of two hundred thousand.

**The same way that a city expands and evolves and grows, so does a mind.** So does a perception. So does a person evolve. A buyer model has to change with the person, or it describes someone who no longer exists. The interview-built agents from two chapters back put a number on it, when a person's own history beat their demographics by 14 to 15 points. That's why [I keep revising a buyer model](https://andydataguy.com/wiki/business-intelligence/default/hyperrelevance-cartography) for as long as the buyer keeps moving.

**What makes it multidimensional is the parallel nature of everything that we do.** A flat persona is one list. The willow is every contest we ran for a person who keeps changing, and all of those contests ran side by side.

The willows also gave me a taste for dark questions, and that taste now runs through Depths of the Void, a science-fiction universe I'm building across games and novels, where one city's war becomes one family's story and every story changes the world the next one inherits. I wrote it up in full [here](https://andydataguy.com/wiki/ai-and-technical-development/default/depths-of-the-void).

> Figure: The willows gave me a taste for dark questions that now runs through a science-fiction universe of my own.

On the willow, clicking a block opens a world model. So what's a block when you stand far enough back?

### The blocks on the willow are embeddings too

Inside each block are the embeddings of the piece it holds, and each block is itself an embedding. Part 2 called an embedding a long list of numbers that works like a location, and ours run to 3,072 numbers. A block has a location too, which means we're able to even turn the blocks into neighborhoods and clusters. Say a losing headline about price lands beside a winning joke about price. They hung from different branches, and they're neighbors anyway.

I hold a neighborhood to a standard before I believe it. [My corpus page reports a shape result measured against a shuffled copy of the same data](https://andydataguy.com/corpus). Nearness on a map like this stays a hint until it passes a check like that.

> Figure: Fold the willow into neighborhoods of its own blocks, then tap one block to see where its branch-mates landed.

**Dr. Ben Goertzel was folding these structures before I was.** Part 5 defined the metagraph and thanked him for it. A group of blocks that becomes one block is that same structure at work. Dr. Goertzel has written that in 2020, with travel stopped by the pandemic, he convinced himself that ["basically all the algorithms needed for AGI"](https://arxiv.org/html/2310.18318) in his approach could be represented as "folding and unfolding, enacted over metagraphs". AGI means artificial general intelligence, and he more or less launched the term himself, with Cassio Pennachin, in a book they edited in the mid-2000s. The folding in the data folding article starts there.

> Figure: A group of blocks folding into one block is the metagraph at work.

He's also the reason I keep physics close. A game engine makes a world feel alive by giving it physics. On a map only direction means something. Add physics and things also have speed, mass and energy, and a space where things move under forces starts to evolve.

**The three parts I take from Dr. Goertzel are quantum mechanics, fluid dynamics and reinforcement learning.** That's my reading of his work, and he hasn't listed them as three anywhere I've found. Quantum mechanics describes each layer's many possible states. Fluid dynamics describes how things flow and settle. Reinforcement learning gives the whole system a direction, through a reward it tries to earn. In [a March 2026 post](https://bengoertzel.substack.com/p/rethinking-classical-quantum-brain) Goertzel works through known links between the equations behind all three. I lean on him to keep the physics honest.

**Reinforcement learning is trial and error with a score.** Richard Sutton and Andrew Barto define it as learning what to do by trying things, so as to collect as much of a numerical reward as possible. They [received the 2024 Turing Award](https://awards.acm.org/about/2024-turing) for it, and Barto can say it in five words: "Trial and error is search." A tournament is a reward written down. The rubric says what earns points, and the willow keeps every trial we ran against it.

You can build tournaments, willows and metagraphs as deep as you like, and every level costs money. Most jobs don't need all of them. So how deep should I build any of this for one client's job?

## I score twelve components before I build anything for a client

> Figure: Plenty of machinery on the table. Nobody should build all of it.

I build only as deep as the job pays for. By now there's plenty of machinery on the table: a chunker, an index, a tree of summaries, a graph and a willow full of tournaments. Nobody should build all of it. I decide which parts get built deep with a framework I call Agent Redwood.

### Agent Redwood scores twelve components from 1 to 5

**I start every build by asking how it makes money.** Before I design anything, I break down how an automation could help to make money, save money or mitigate risk. I like to start from there. Ideally it does all three, but you brainstorm and then narrow it down based on value. I ask the same thing at every layer of the stack. A button's animation has to earn its place for the same reason a whole campaign does.

> Figure: Make money, save money or cut a risk. That question comes first.

After the money question I start simple, and I owe that habit to two authors. Warren Powell's framework for decisions shows why. Powell sorts every way of making a decision into [four classes](https://warrenpowell.org/policies), from a plain rule, which he calls a policy function approximation or PFA, up to methods that simulate the future before every move. Most people try to start off with the classes that estimate or simulate the future, when the fact is the vast majority of the time all people need is a PFA. Powell reports the same from practice, where the simplest classes are "by far the most widely used". Start simple, because a simple rule tells you quickly whether you're solving the right problem. Add complexity only when the value justifies it.

John Gall wrote down why that order works. He was a pediatrician in Ann Arbor, about 30 publishers rejected his book before he printed it himself, and it holds that [a complex system that works](https://en.wikipedia.org/wiki/John_Gall_%28author%29) "is invariably found to have evolved from a simple system that worked." Erik Schluntz and Barry Zhang gave agent builders the same advice at Anthropic in 2024. Their guide, ["Building effective agents"](https://www.anthropic.com/research/building-effective-agents), recommends "finding the simplest solution possible" and adds that "This might mean not building agentic systems at all."

> Figure: Every complex system that works started as a simple one that worked.

**Agent Redwood is where simple gets specific.** It's the design framework I run before I build any agent. It names twelve components, and it scores each one from 1 to 5 for the build in front of me. The twelve come in no particular order, because the order isn't the ranking. The scores do the ranking. A score says how much that component matters to what this agent is being built to do. It isn't a grade of how good the component is. An agent whose whole job is a single scheduled lookup doesn't need an elaborate reward model or a deep memory system. An agent that runs a long autonomous mission needs memory, evaluation and awareness scored high, because those keep it coherent over a long stretch of work.

_Agent Redwood's twelve components, in the order the framework lists them. The order isn't a ranking, because the scores do the ranking._
| Component | What it is | What the score weighs |
| --- | --- | --- |
| **Personality** | It is how the agent sounds and behaves. | The score weighs how much tone and register matter to the people it serves. |
| **Planning** | It is how the agent breaks a goal into steps and orders them. | The score weighs how many steps sit between a request and a result. |
| **Mission** | It is the job the agent exists to do. | The score weighs how precisely success and scope have to be written down. |
| **Constraints** | They are the things the agent must never do. | The score weighs how much harm a wrong action could cause. |
| **Memory** | It is what the agent keeps between sessions. | The score weighs how much the work depends on what came before. |
| **Evaluation** | It is how the agent's output gets judged. | The score weighs how costly it is when a bad answer looks like a good one. |
| **Tools** | They are the outside capabilities the agent can call. | The score weighs how much the agent has to do beyond writing text. |
| **Awareness** | It is the agent's read on its own state and surroundings. | The score weighs how much its behavior should change with conditions. |
| **Reward model** | It is what the agent optimizes for. | The score weighs how many objectives compete and how sharply. |
| **Metadata** | It is the structured record of what the agent is. | The score weighs how many agents and owners have to be tracked. |
| **Strategy** | It is how the agent handles cases no rule covers. | The score weighs how often the work turns ambiguous. |
| **Integrations** | They are how the agent connects to other agents and systems. | The score weighs how many other parts it has to work with. |

**The scores decide where the budget goes.** A high-scored component gets real depth and real engineering. A low-scored one gets the minimum that works. The result is an agent exactly as complex as its job requires, which is the whole point of starting simple. Each of the twelve deserves a proper look, because a score only means something once you know what it's measuring.

> Figure: Pick a job and the redwood grows each limb only as far as its score.

Here are the twelve in the framework's order. Each one gets what it is, what a high score buys, what goes wrong when it's missing, and a made-up moment from an ordinary business that shows the difference.

### Personality decides whether people trust the agent

**Personality is how the agent communicates.** It covers the tone it takes, the register it speaks in and the voice it presents. Scoring it means deciding how much of that has to be designed on purpose. A high score buys a written voice profile with tone, vocabulary, formality and the phrases the agent must never use. It also buys a check that the voice holds steady across a long conversation and matches the brand's own guidelines. The same agent may need to speak one way to a chief executive and another way to a technician.

A mismatch here never shows up in an error log. People just stop finishing conversations and stop coming back. Picture a family law firm that puts an intake agent on its website. A parent writes in at 11 at night about a custody hearing. An agent with no designed voice answers in the breezy tone of a shoe store chat, and the parent closes the tab. An agent with a voice profile slows down, uses plain words, says what happens next and who will call in the morning. The facts are the same in both replies, and only the second one books the consultation.

> Figure: Personality decides whether people trust the agent; the same facts in the wrong voice lose the client.

### Planning keeps a multi-step job from wandering

**Planning is how the agent breaks a goal into steps and puts them in order.** A high score buys a way to turn one objective into ordered sub-tasks, to tell which steps can run side by side, and to track what's done and what remains. It also buys the ability to replan when a step fails, plus a limit on how deep the breakdown goes and how much time each step may take.

Without planning, an agent given a broad task wanders. It produces output, but the output doesn't add up to an answer. Picture a restaurant group whose agent gets this request: a rehearsal dinner for 60 on Saturday, two vegan guests, about $4,000 all in. An agent without planning answers the menu question first and never checks the room, so the quote goes out for a night the private room is already booked. An agent with planning checks the room, then the menu, then the price, and sends one quote that can actually be signed.

> Figure: Planning checks the room, then the menu, then the price, so the quote can actually be signed.

### Mission is the one sentence every other score is judged against

**Mission is what the agent exists to accomplish and what success looks like.** A high score buys a written, versioned mission statement with measurable success criteria. It names what's in scope and what's out, and it gives the agent a way to check whether a request belongs to its mission at all.

Most agents in production have no explicit mission. They have a system prompt that says something like "you are a helpful assistant" and a set of tools. An implied mission drifts, and an agent with email, calendar and web search will try to do anything it's asked, including things it should refuse. Picture a dental clinic that gives its front-desk agent those tools and no written mission. A patient asks whether their insurance covers a crown, and the agent guesses yes. The clinic now owns a promise it never made. With a one-sentence mission, book and move appointments and answer office questions, the same agent hands the insurance question to the billing team.

> Figure: A written mission keeps an agent from promising what it was never meant to answer.

### Constraints are the lines the agent never crosses

**Constraints are what the agent must not do, whatever the task.** They aren't the reverse of the mission. A copywriting agent's mission is marketing copy, and a constraint says it never invents a statistic. The mission doesn't imply that rule, so it has to be written down. A high score buys a register of constraints sorted by kind: safety, legal, brand, operational and technical. Some are absolute and some can be overridden with a recorded reason. Each one is checked before a tool runs, before an answer goes out and before anything gets changed, and every check is logged.

Without constraints an agent drifts and overreaches, and the harm lands on the business. Picture an online store whose support agent is trying to calm an angry customer about a late package. Nothing in its mission says it can't, so it offers 40 percent off and free express shipping on the next order, and it does the same thing all week. One written constraint would have stopped it: never offer a discount above 10 percent without a person approving it.

> Figure: Without a written constraint an agent overreaches, and the harm lands on the business.

### Memory turns a function call into something that learns

**Memory is what the agent keeps between sessions, what it forgets and how the past shapes what it does next.** A high score buys several kinds of memory. Short-term memory holds the current conversation. Long-term memory holds preferences, facts and past decisions. Episodic memory recalls a specific earlier exchange. Semantic memory holds what many exchanges taught. It also buys rules for what to keep, what to drop and what to do when two memories disagree, and a way to search all of it.

Without memory every conversation starts from zero, so the customer explains the same thing twice and the agent never gets better at the job. Picture a roofing contractor's agent. In March a homeowner tells it the leak is over the back bedroom and the dog gets out if the side gate is left open. In April a different crew shows up knowing none of it, and the dog is three streets away by lunch. With memory, the April booking carries the March notes, and the crew closes the gate.

> Figure: Without memory every visit starts from zero, and the dog is three streets away by lunch.

### Evaluation is how anyone knows a bad answer from a good one

**Evaluation is how the agent's output gets judged.** A high score buys quality scoring against stated criteria such as accuracy, completeness, tone and grounding. It buys a self-check before the answer leaves, a way for human reviewers to score answers and feed the scores back, and written rubrics for each kind of output. It also buys a history of scores, so a drop shows up, and routing that publishes strong answers and sends weak ones to a person.

Without evaluation, an agent producing bad answers looks exactly like one producing good answers, and the business finds out from a customer. Picture a real estate brokerage that lets an agent write its listing descriptions. One listing says four bedrooms when the house has three. Nobody checks, the listing goes live, and the brokerage hears about it from an annoyed buyer standing in the third bedroom. With evaluation, every description is checked against the listing data before it posts, and the mismatch goes to a person instead of the public.

> Figure: Without evaluation a confident wrong answer looks exactly like a right one until a customer finds it.

### Tools set the ceiling on what the agent can do

**Tools are the outside capabilities the agent can call.** They include APIs, databases, files, other agents and MCP servers, which are a standard way to plug a tool into a model. A high score buys a catalog of tools with their inputs, outputs and access rules, and rules about which agent may use which tool. It buys isolation so a bad call can't crash the agent, a check that every tool result has the expected shape, and a record of each tool's use, failures and speed.

Without tools an agent can only write text. Picture a boutique hotel's agent. Without a booking tool it can tell a guest that rooms are usually open in May, which helps nobody. With the tool it can check the real dates and hold the room while the guest decides. With the tool and no controls it could also cancel someone else's reservation by mistake. So the tool comes with a rule: the agent may check, hold and book, and only a person may cancel.

> Figure: A tool lets an agent act, and the tool comes with a rule about who may do what.

### Awareness lets the agent change course when conditions change

**Awareness is the agent's read on its own state and its surroundings.** A high score buys self-monitoring of the tokens, time and context it has used. It buys checks on the services it depends on, knowledge of what other agents are doing, and a sense of the user's situation, such as how long they've been waiting. It also buys behavior that changes with those signals. The agent can switch to a faster model when it's slow, or hand the conversation to a person when its confidence drops.

Without awareness the agent is a black box that answers the wrong question well. Picture a veterinary clinic's agent halfway through a routine answer about vaccine schedules when the owner adds that the dog just ate a bar of dark chocolate. An agent without awareness finishes the vaccine answer. An aware agent notices the urgency, stops, gives the emergency line and flags the front desk.

> Figure: Awareness lets the agent drop the routine answer the moment the situation turns urgent.

### The reward model decides what the agent optimizes

**The reward model is what the agent optimizes for and how it balances goals that compete.** A high score buys a written list of objectives with weights, the signals that show whether they're being met and the math that turns those signals into one score. It buys a way for that score to change the agent's behavior. It also buys explicit trade-offs between speed and accuracy, helpfulness and safety, and thoroughness and cost.

Every agent optimizes something. Without an explicit reward model it optimizes whatever its training rewarded, which is usually being agreeable and wordy. Picture a gym that rewards its membership agent for one thing, keeping members. The agent learns to stall, so people who want to cancel spend twenty minutes in a chat loop and then write a one-star review about it. A reward model that also counts a clean, quick cancellation when the member insists keeps more goodwill than it loses. That's the same trap as the Klarna case further down, where cost became the main measure and quality slipped.

> Figure: An agent optimizes whatever it is rewarded for, so a reward for keeping members becomes a trap.

### Metadata makes the agent and its answers traceable

**Metadata is the structured record of what the agent is and what it did.** A high score buys an identity, a version history and a machine-readable list of what the agent can do. It buys a named owner and a change-approval process, a record of where the agent runs and what it depends on, and a searchable catalog of every agent in the system.

Without metadata, managing agents is manual and error-prone, and nobody can prove where an answer came from. Picture a medical billing company that runs three agents and updates one of them on a Friday. On Monday a batch of claims comes back rejected. Nobody can say which agent wrote them or which version was running, so the team checks everything by hand. With metadata, every claim carries the agent's name, its version and the sources it used, and the bad batch is traced in minutes.

> Figure: Metadata makes every answer traceable to the agent and version that wrote it.

### Strategy is what the agent does when no rule applies

**Strategy is how the agent handles a situation its designer didn't foresee.** A high score buys a way to notice that a case falls outside the rules and a library of fallback moves for common kinds of uncertainty. Those moves include asking a clarifying question, applying a conservative default, splitting the problem or passing it to a person. It also buys rules for choosing between those moves, an order to try them in and a record of which ones worked.

Without strategy, an agent facing an unclear case either freezes or guesses, and a confident guess is the expensive outcome. Picture a florist's agent that gets one message that is half complaint and half new order: the centerpieces wilted before the reception, and the customer wants flowers for a second event next month. An agent without strategy either ignores the complaint or refuses the order. An agent with strategy splits the message, sends the complaint and the photos to the owner, and takes the new order.

> Figure: Strategy splits an unclear case instead of freezing or guessing.

### Integrations decide whether the agent fits the business

**Integrations are how the agent connects to everything else.** That means other agents, the company's systems and the data platforms it plugs into. A high score buys structured messages between agents with a sender, a recipient and a priority, plus shared records that several agents can update without overwriting each other. It buys a standard way to hand work to another agent and get the result back, a way to settle disagreements and a full log of what passed between them.

Without integrations, a group of agents is just a set of programs that happen to share a database, and a person ends up carrying the work between them. Picture a heating and cooling company whose agent books repair calls but can't reach the dispatch system. Someone retypes every booking, and now and then a technician drives to the wrong address. With integrations, the booking lands in dispatch with the address, the unit's model and the customer's notes already filled in, and the technician sees it before leaving the shop.

> Figure: Integrations carry the work between systems, so the technician sees the job before leaving the shop.

### The scores come out of a questionnaire

Those scores come out of a questionnaire. We go through it together, and you can dial in how intensive you want it to be, based on either how many questions or how much time you're willing to spend. There's a small, a medium and a large, and ideally you want to do the large. You want to let the system drill the hell out of you and extract a bunch of insights, because the more context you can give the system, the better it's going to make something for you. With only an hour, start with reward model and constraints. The reward model is how the agent knows it did well, so this is where you define what "good" looks like versus what's forbidden. Before touching code, you evaluate what matters for your specific use case, rank by priority and decide implementation depth.

Readers of the data folding article have met a questionnaire like this one. That article's form came from the people phase of PPP, and it asked about the problem, the future the person wants and the transformation between them. That form says what a build has to do for a person. Redwood picks up where it stops and says how deep each component has to go to do it.

**The mission gets named first, and components get cut against it.** In a working session the mission gets named first, then each component is decided against it: whether the first version needs it, and how far to go. In [one session](/case-studies/agent-redwood-blueprint) a team's agent had one job, a daily shortlist. The TradingView integration they'd asked for did nothing for that job, so it was cut before anything was built. The AI projects I've seen fail started as scoping failures.

> Figure: Name the mission first, then cut every component that doesn't serve it.

Other people have drawn maps of an agent, and I use them. Lilian Weng's 2023 post puts [planning, memory and tool use](https://lilianweng.github.io/posts/2023-06-23-agent) around the model. The CoALA paper by Theodore Sumers, Shunyu Yao, Karthik Narasimhan and Thomas Griffiths organizes an agent around [memory, an action space and a decision procedure](https://arxiv.org/html/2309.02427). By my reading those maps name three of my twelve, and the other nine are what a client adds: a brand, a budget, rules and other software the agent has to live with. So what do the scores look like when the thing I'm building is retrieval?

### A parts counter shows where the scores land

**Here's what scoring looks like when the job is retrieval.** Picture a plumbing supply house whose counter staff answer the same contractor questions all day: does this valve fit that line, is it in stock, and what does it cost. The agent's mission is one sentence: answer fit, stock and price questions before the contractor drives over. Constraints score a 5, because a made-up price or a wrong part costs a sale and a wasted trip. Evaluation scores a 5, because a confident wrong answer looks just like a right one until someone checks. Metadata scores a 4, because the contractor wants to see which spec sheet the answer came from. Integrations score a 4, because a ready order should land at the counter with the part and the job already filled in. Personality and strategy score a 2, and memory scores a 1, because each question stands on its own. The depth goes where a mistake costs money.

> Figure: A made-up price costs a sale and a wasted trip.

**I can't build every layer of retrieval at once, and neither can you.** Keyword scoring is the ground floor. In my order, keyword search comes first and vectors, which search by meaning, come next. Then comes a reranker, which takes a second look at the shortlist, then the RAPTOR summary tree from Part 4, and a graph last. Researchers have their own ladder. Yunfan Gao and colleagues [named its rungs](https://arxiv.org/html/2312.10997): naive RAG, which indexes, retrieves and generates; advanced RAG, which adds work before and after the retrieval; and modular RAG, with parts you can swap. They wrote that the upper rungs were "a response to these specific shortcomings" in the first one, which is Gall's law showing up in a survey paper.

I keep a third ladder for scoping talks. When I [audited one AI company's 29 recorded sales calls](/case-studies/ai-company-sales-call-audit), three kinds of build covered about 90 percent of what buyers asked for: content, research and a tool people could talk to. I sorted every possible build into five tiers by effort, from an assistant loaded with the company's own material, a couple of hours' work, up to systems that learn from their own mistakes. With the tiers on the table, a buyer and a team could agree on the size of a problem before anyone quoted it.

> Figure: Add one layer, prove it pays, and only then reach for the next.

**Evaluation gets a high score from the first day.** Scott Barnett and colleagues studied retrieval systems after they shipped and found [seven separate points where they fail](https://ieeexplore.ieee.org/document/10556182). They concluded that "validation of a RAG system is only feasible during operation", and that its robustness "evolves rather than designed in at the start." That's why the test comes early. I've written up [what a retrieval engine should check before it answers](/wiki/ai-and-technical-development/default/rag-systems), and a fixed set of questions with known answers, run before every change, is the smallest version of it.

Klarna is the cautionary case here, and its record is contested. In February 2024 the company said its AI assistant had handled 2.3 million conversations in its first month, two thirds of its customer service chats. In May 2025 its chief executive, Sebastian Siemiatkowski, told Bloomberg, [as CX Dive reported](https://www.customerexperiencedive.com/news/klarna-reinvests-human-talent-customer-service-AI-chatbot/747586), "As cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality." The company began recruiting human agents. That July, in correspondence with the Securities and Exchange Commission, Klarna said the remark was about its outsourced human service, not the AI, and had been misreported, and that the assistant still handled about two thirds of chats. I can't settle from here which reading is right. Both readings accept the part that matters for scoring, which is that measuring mostly on cost hid where quality was failing. In Redwood terms that's a reward model with one number in it.

> Figure: Measuring mostly on cost hides where quality is failing.

**Sometimes every score comes back low, and I say so.** I once made [55,001 pages of technical PDFs](/case-studies/technical-pdf-library-55k-pages) searchable, and that library needed the depth. If your library is a few hundred pages your team already knows, keyword search will do and I'll tell you so. If your product questions fit on one FAQ page, build the FAQ page. I ask two things of any design. Does it fit the depth I need? Is it going to extrapolate to everything else? Either way we're going to start relatively small. So why does a data engineer score everything by money?

### Ad platforms put a price on relevance in every auction

**I score by money because I used to be paid in it.** In performance advertising most of my income came from profit, so I couldn't blame the creative, the site or the copy when something didn't sell. The advertiser owns the whole funnel, which means understanding the whole business.

A real funnel has five to eight dozen steps. The slide version of a funnel is TOFU, MOFU, BOFU, which is top of funnel, middle of funnel and bottom of funnel. A real one has sixty to a hundred steps. By steps I mean something like how a great hotelier puts together touchpoints in a property to create a certain type of experience for guests. In advertising, the unit of measurement is the impression, the atomic measure of what we think is happening. An impression is one showing of one ad to one person.

> Figure: When you're paid on the profits, you own every step of the funnel.

Take the middle of that range, six dozen steps. If a person gets through each step 95 times in 100, about 2 or 3 people in 100 reach the end. Lift every step to 98 in 100 and about 23 do. I made those odds up to show the shape, and no account of mine produced them. The shape is why I never treat a better fit at one step as a small thing.

> Figure: Tighten all 72 joints a little and watch what reaches the far end.

**The ad platforms are retrieval systems, and they charge for a bad fit.** Meta's engineers wrote in 2024 that their ads retrieval engine, Andromeda, has the job of ["selecting ads from tens of millions of ad candidates into a few thousand relevant ad candidates"](https://engineering.fb.com/2024/12/02/production-engineering/meta-Andromeda-advantage-automation-next-gen-personalized-ads-retrieval-engine) before any ranking happens. They reported that its launch brought a 6% gain in recall, which is the share of the right ads it finds, and an 8% gain in ads quality on the segments they measured. Those are Meta's own figures. That's the retrieve-then-rank pipeline from Part 3 with a bill attached. Google prices the result openly. Its [Ad Rank](https://support.google.com/google-ads/answer/1752122) weighs the quality of an ad and its landing page beside the bid, and Google's help page says that "even if your competition has higher bids than yours, you can still win a higher position at a lower price by using highly relevant keywords and ads."

> Figure: Tens of millions in, a few thousand out, before any ranking happens.

**Relevance is worth revenue you can count.** McKinsey [collected the figures](https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-personalization): 71% of consumers expect personalized interactions, and personalization can lift revenues by 5 to 15 percent. Treat conversion like a trading strategy: test every idea in the live market, manage the risk, and keep what the market rewards.

Retrieval pays wherever a business makes money online, and that reaches well past content. So where else in a company does one retrieval core pay?

### One retrieval core plugs into a CRM and a sales floor and a phone line

**A company already runs on a couple dozen tools.** A CRM, payroll, a calendar, email, a scheduler, a phone line: no owner wants another subscription on that list.

So I don't build retrieval as one more subscription. I build it as one tool that every agent and every system calls. Old machine shops ran on the same idea. One engine turned a long shaft under the ceiling, and every machine on the floor took its power from that shaft through its own belt, at its own speed.

> Figure: Stop the one wheel and count how many machines stop with it.

**The CRM gets it first.** A CRM has to be built for the reps who live in it every day, not for management's reporting wish list. At [a 13-person software company](/case-studies/crm-handover-13-person-company), leads and deals lived in hundreds of chat threads. My estimate was that seeing those chats in one place would have put the company three to five deals further ahead over the two months before. The rule I took from that job is to get the reasons behind each deal into the CRM before the person who knows them leaves. James Oldroyd, Kristina McElheran and David Elkington studied 1.25 million sales leads for Harvard Business Review in 2011, and firms that tried to reach a lead within an hour were [nearly seven times as likely to qualify it](https://hbr.org/2011/03/the-short-life-of-online-sales-leads) as firms that tried an hour later. Elkington ran a company that sells lead-response software, so weigh that as you like. A retrieval core puts a deal's whole history in front of whoever answers first.

Automation is the level up underneath it. Tools like n8n, Make.com and Zapier connect one app to another without much code. They do for workflows what a spreadsheet does for data: someone without an engineering background can get most of the way, at a smaller scale. That improves margins, and it makes the business less dependent on any one person. I built an automated [round-robin system](/case-studies/solar-round-robin-media-buying) for an eight-figure agency managing 50+ solar accounts, and it dealt each task to whichever media buyer was free. That agency is also where I learned that systems which work at $200 a day blow up at $1,000 a day. Depth has to follow the load.

> Figure: The system dealt each task to whoever was free.

**A sales floor learns from the same core.** Erik Brynjolfsson, Danielle Li and Lindsey Raymond measured a reply assistant across 5,172 customer support agents. [Issues resolved per hour rose 15%](https://academic.oup.com/qje/article/140/2/889/7990658) on average and about 30% for the less experienced agents, and people two months into the job with the tool performed like people six months in without it. The assistant had been trained on the firm's own chats with the best agents weighted up, so it spread their habits. My older version was a written method. At two companies that grew [from 4 to more than 20 people and from 3 to more than 30](/case-studies/remote-teams-4-to-20-3-to-30), every media buyer I hired went through the same training, step by step. Now [the recorded calls and playbook rules](/wiki/operational-systems/default/enablement) can sit in the core a new rep practises against.

Sales teams that sell to advertisers already speak in conversion rates, click-through rates and split tests. So [measure your own selling the way you'd measure an ad account](/case-studies/agency-sales-pipeline-forecast), and put the value in financial terms first.

**A phone line scores the twelve differently.** I won't walk through a voice build here. I'll only show why the scores move. Speakers leave about 200 milliseconds between turns when they talk, and Tanya Stivers and her coauthors [timed that gap across ten languages](https://www.pnas.org/doi/10.1073/pnas.0903616106). A practitioner's budget on AWS's builder site gives a voice agent [roughly 700 milliseconds end to end](https://builder.aws.com/content/3JDEQi9AVukeX7JbxagjF0eROCf/five-rules-that-keep-a-voice-agent-inside-its-latency-budget) in a good case, and it warns that one unbudgeted retrieval lookup adds 300 to 800. Nobody has peer-reviewed those practitioner figures. On a phone line, speed earns a score it would never get in an article pipeline. The core is the same one, and the belt runs at a different speed.

> Figure: On a phone line every lookup has to fit inside the fraction of a second people leave between turns.

The team around the machines is a design problem too. Melissa Valentine and her colleagues at Stanford [showed how](https://dl.acm.org/doi/10.1145/3025453.3025811). They hired experts on demand through a freelance marketplace and organised them with roles, teams and a hierarchy, the way a film crew works, and they filled open roles in about fourteen minutes. The same structure works for a team of agents. With one core serving all of that, how would an operator in some other niche start?

### The tallest tree on Earth has roots about a dozen feet deep

**A coast redwood stays up by reaching sideways.** The tallest known redwood stands about 380 feet. The National Park Service says a redwood's roots [go only 10 to 13 feet down](https://www.nps.gov/articles/000/coast-redwood.htm). Then they spread 60 to 80 feet and intertwine with the roots of the redwoods around them. I want a first build to have that shape. It goes a few feet deep, it spreads wide, and it's tied into the tools you already run. In Redwood terms that's a low score almost everywhere and a high score on integrations.

> Figure: Shallow and wide, tied into the neighbours. That's a good first build.

**I've told you more about process than about any one client, and I did it on purpose.** That's deliberate. Specific client stories belong to the clients, and the method travels better than any one story. The better question is how I'd teach it to you in a way that makes sense and is exciting. By exciting, in the marketing sense, I mean: how does it make money? How does it make an impact?

**That brings back the operator from Part 1, who has lots of kinds of data and a pile of content to make.** My one live example has been GPS Alaska, where I work as a [forward-deployed engineer](/looikos/infra/wikidesignco), which means I build inside the client's business instead of handing software over a wall. If it works for machine control on heavy equipment, it will work in your niche too. When you choose someone to build it, ask what they know about your kind of problem, not only what they have built before. That's why every build here starts from the textbooks: they hold other people's experience of the same problem.

> Figure: It worked for heavy equipment in Alaska. Fill in your own niche.

I'd start in your seat with one sentence that names the mission. Score the twelve components against it. Build the smallest thing that makes money, saves money or cuts a risk, and let it prove itself before you add a layer. What's your mission, and which of the twelve would you score a 5?

Source: https://andydataguy.com/wiki/ai-and-technical-development/default/retrieval-engine

