Skip to content
andydataguy
AI & SYSTEMS

I corrected AI memory that kept returning outdated answers

Ten stale facts received end dates and three valid facts were restored. Superseded facts remain available as dated history.

My agents share one memory whose facts carry their own dates, so a newer fact closes the old one and the old one stays on the record.

My AI agents share one memory whose facts carry their dates. When three written corrections failed to retire a stale answer, the fix closed it by its ID, kept it on the record and restored the correct facts; 1,253 of the memory's 3,154 facts have stopped being true and stay on the record.
3,154
facts in my agents' memory
1,253
marked no longer true
380
entries written to the memory
73 of 73
saves read back from the graph
Case Studies

Context

My business runs on a team of AI agents that plan, build and check work in parallel. I built and ran remote teams across Europe, Asia and US time zones before that. My agents' memory, which I call the Metagraph, is a temporal graph on Neo4j built on the open-source Graphiti project, which I extended and host myself.

Problem

The more capable my agents get, the more their work depends on what they're grounded in. A search that finds matching text can't tell an old answer from a current one. Remote teams have the same problem: a fact gets written down once and goes stale while everyone keeps reading it. On a client platform I run, 204 product records still showed a billing error months after the billing was fixed, and for weeks agents kept reporting those records as a live billing emergency.

Approach

I built the memory so each fact carries its own history. Every fact records when the memory learned it, and most also record when they became true. A newer fact closes the old one with an end date and leaves it in place, so an agent can ask what's true now and what was true last month. Saving was the weak point. Graphiti's own save tool reported success the moment it queued an entry, and entries were lost when the embedding step failed. So I built a save path any agent can run from a command line: it writes the entry to disk first, writes it to the graph, reads it back, and fails loudly if any step breaks. Turning a document into facts costs money per document, so I choose what goes into memory and log the real token cost of each document, which checks the estimate made beforehand. The next layer is in the data model of WikiDesignCo, my knowledge platform: a confidence score on each extracted entity, and a record of which processing run produced it, with its model, version and cost, so two runs over the same material can be compared and the better one kept. An agent that learns something the others need saves it before its session ends. My standing rule for any stored error or fact is to date it and check it live before anyone raises an alarm over it.

Each entry is written to disk first and read back, and all 73 saves since July were confirmed.

My agents' old save path lost entries. Now each entry is written to disk first, then to the memory, and read back to confirm it: all 73 saves since July were confirmed.

Tools used

  • Graphiti on Neo4j, extended and self-hosted
  • Facts dated by when they were true and when they were recorded
  • A save path that writes to disk, then reads back from the graph
  • Real per-document extraction cost from token counts
  • Gemini extraction and 3072-dimension Gemini embeddings
  • WikiDesignCo data model with confidence scores and run records

Result

10
stale facts closed in the recorded correction

An agent asked the memory which model one of my coding agents runs on, and it got an answer that had been retired the day before. Three written corrections didn't retire it. The step that turns entries into facts attached them to the old fact and marked the correct new facts as invalid. The fix closed the 10 stale facts by their IDs, each with the date the change took effect and the date the memory learned it, and restored 3 correct facts that same step had closed. Nothing was deleted, so every change can be reversed. At last count the memory held 3,154 facts from 380 entries, and 1,253 of them were marked no longer true.

Impact

Agents from two different AI providers save to the same memory through one path, and all 73 saves through it were confirmed by reading them back from the graph. A retired fact stays on the record with its dates, so the history behind a current answer is still there to check.

Lessons

A stored fact is a claim with a date on it. Here that meant closing an old fact with the date it changed, because a correction written beside it left the old fact looking current.

Who this is for

For teams building AI agents or knowledge systems that have to stay correct while the facts underneath them change, and for companies whose people and agents work from the same records. Plain search is the better first build for an assistant that answers from a few documents that rarely change, and I'd start there.

Get a quote
On this page

The method behind this result