The Shape of Data III: Time and Provenance. When a fact was true, where it came from, and what contradicts it are shape properties, and none of them can be retrofitted.

The three things you cannot add later [summarize]

Three questions almost no schema can answer

Every system I've walked into can tell me what's true right now. That's the easy part, and it's most of what a database gets asked for on an average day. Ask it three other things and it goes quiet.

What did we believe last March, and when did we find out we were wrong? Where did this number come from, and what happens to it if one of its sources gets retracted? What else in this system disagrees with this number, and how much do we trust either version? None of those are exotic. A CFO restating a quarter needs the first one and needs it under audit. Anyone reconciling two systems that report different revenue needs the third, weekly, forever. The second shows up the first time a regulator, an acquirer, or a customer asks a company to show its work.

The usual response is that the system could answer those questions with some additional engineering. Sometimes that's true. Usually it isn't, and the reason it isn't has nothing to do with engineering effort.

Adding the column is cheap. Recovering the history isn't possible.

You can add a valid_from column to a production table this afternoon. It's one ALTER TABLE, it's backward compatible, and it'll be correct for every row written after you ship it. It'll also be empty, or worse, backfilled with the deploy date, for every row that already exists. Nothing you do recovers what those rows said in March, because the March values were overwritten by an ordinary UPDATE that ran months ago and left no trace by design.

That asymmetry is what makes these three properties different from everything else in a data stack. Almost every technical decision in a system is reversible at some price. You can swap Postgres for Snowflake. You can add an index, rewrite the API, change the vendor, move to a new cloud, and refactor a schema that got ugly. All of that costs money and calendar and nobody enjoys it, and all of it is a migration. These three are the exception, and they're the exception for one reason: the data that would populate them stopped existing at the moment of a write that already happened.

Part I of this series called that the ceiling, and made the case that unanswerable is a different category from hard. A hard question costs compute. An unanswerable question has no correct answer available at any budget, because the fact that would answer it was never recorded or was destroyed. Time, provenance, and contradiction are where that distinction bites hardest, because all three feel like features you can schedule for next quarter and none of them are.

All three failures share one mechanism

A single write pattern destroys all three properties at once, and it's the default write pattern in essentially every system anybody ships. An UPDATE replaces a value in place. The new value is correct. The old value is gone, along with the fact that it was ever believed, the interval during which it was believed, whatever produced it, and any disagreement it had with something else in the system.

Databases are extremely good at this. Replacing in place is fast, it keeps the table small, it makes reads simple, and every ORM in existence generates it without being asked. Nobody chose to destroy their history. They called save() on an object, several thousand times a day, for four years.

The three properties and what kills each one line up cleanly enough to put in a table, and this table is the map for the rest of the article:

Each row of that table is a shape decision rather than a feature decision. A uniqueness constraint isn't a policy somebody can relax later; it's a statement that the model has room for exactly one answer, and the code above it has been written against that assumption for years. Relaxing it means every consumer now has to decide what to do with two rows where it expected one.

What the rest of this covers

Six sections follow, and then two that put the pieces together. Bitemporal modeling handles the first property precisely, separating when a fact was true in the world from when the database believed it. Slowly changing dimensions are where most warehouses lose the fight, and Type 1 is the canonical history murder, so it gets its own section including the specific report that can never be reproduced afterward. Event sourcing is the strongest available answer and also the most expensive, so that section states its real costs and names when it's the wrong choice.

Then provenance, which the database literature has thought about far more rigorously than the average data team realizes, and contradiction, which almost nobody models at all. Those five threads converge on the medallion ladder, where an artifact's value is derived from its lineage rather than assigned by hand, and then on the metagraph, which is the shape that carries all three properties natively. The last section is what to do tomorrow morning, including who should read all of this and then do none of it.

What did we believe on Tuesday [summarize]

Two clocks, and why one of them isn't enough

Every fact in a business system has two independent timelines attached to it, and most schemas record at most one.

Valid time is when a fact was true in the world. A price took effect on the first of the month. An employee's title changed on their promotion date. A deal closed on March 3. That clock belongs to reality and it doesn't care what any database thinks.

Transaction time is when the system believed it. The price change got entered four days after it took effect because somebody was on vacation. The promotion hit HR's system the following Monday. The corrected deal amount landed in May. That clock belongs to the database, and it's the only clock an audit can hold anyone to, because it records what the organization actually knew at the moment it acted.

These two clocks disagree constantly, and the gap between them is where money goes missing. Data arrives late. Corrections arrive later. Somebody backdates an entry so the reporting looks right, which fixes valid time and quietly destroys transaction time. A schema with a single updated_at column has to pick one of the two clocks, and whichever it picks, it has thrown the other one away for good.

A commission that was right and wrong at the same time

Take a case built out of a shape I've run into in a few different businesses. A deal closes on March 3 and gets recorded at $80,000. Commission runs on the fifteenth, the rep gets paid against that number, and the month closes. In May, somebody in finance finds that a contracted discount was never applied, so the deal was always worth $72,000. It was worth $72,000 on March 3. Nobody knew that until May 12.

Now the business has two entirely reasonable questions and they have different answers.

The first question is what that deal was worth on March 3. The answer is $72,000, and it always was. Any restated revenue report, any commission recalculation, any board deck covering Q1 needs that number, because that's what was true.

The second question is why the rep got paid against $80,000. The answer is that on March 15, the system believed the deal was worth $80,000, and the payment was correct given what was known. Reproducing that payment run, which is exactly what a payroll auditor or a disputing rep will ask for, requires the database to return the value it held in March rather than the value it holds now. Both answers are correct, they contradict each other, and a table with one row per deal can only ever store one of them.

What usually happens next is that somebody runs the UPDATE. The deal row now says $72,000. The March commission report, if anyone regenerates it, comes back showing $72,000 and a commission figure that doesn't match the payment that actually went out. The report is now internally consistent and historically false, which is the worst of the available outcomes, because it looks fine. Nobody flags a report that reconciles. They flag the payment, and then spend two weeks in spreadsheets reconstructing what the system used to think.

A bitemporal table stores both. The deal has a valid-time period saying it was worth $72,000 from March 3 onward, and it also has a transaction-time period recording that from March 3 until May 12 the database asserted $80,000, and from May 12 onward it asserts $72,000. Nothing was deleted. The correction was appended as a new assertion and the old assertion was closed rather than removed.

Four question classes, and the one that only bitemporal answers

Once both clocks are present, four distinct queries become available, and the difference between them is the difference between a clean audit and a bad quarter:

Rows one and two are what a request for history usually means, and a plain append-only audit table gets you partway to row two. Row three is where single-timestamp designs fall apart, because reproducing a past belief requires knowing when each belief started and ended, which is a period rather than an instant. Row four is unanswerable without both clocks, and it's the row that shows up in a courtroom.

Row four also shows up in something far more common than litigation, which is debugging a model that made a bad automated decision. A pricing engine, a credit rule, or a bid algorithm acted on data as it stood at 09:14 on a Tuesday. Explaining that decision six weeks later means reconstructing the exact inputs the model saw, including the ones that have since been corrected. A team without transaction time will reconstruct the decision using today's data, find that the model's output looks reasonable, and close the ticket. The model still has the bug.

What the standard gives you, and what your database actually implements

This got standardized. SQL:2011 added temporal features to the language, documented by Krishna Kulkarni and Jan-Eike Michels in Temporal features in SQL:2011 (ACM SIGMOD Record 41(3), pp. 34-43, 2012). The standard defines two mechanisms that map exactly onto the two clocks. Application-time period tables let the application declare valid-time periods, and system-versioned tables have the DBMS maintain transaction-time history automatically. A table can be both, which is what "bitemporal" means in standards language. The standard also adds period predicates, so you can ask whether two periods overlap, contain, or precede one another without writing the interval arithmetic by hand every time.

Richard Snodgrass had been working this ground for years before that, and Developing Time-Oriented Database Applications in SQL (Morgan Kaufmann, 1999) is still the book to read if you want the full treatment rather than the standard's summary.

Vendor support is uneven, and this is where teams get surprised, because the feature is in the standard and people assume the standard means their database has it:

• IBM Db2 and Microsoft SQL Server (since 2016) implement system-versioned temporal tables natively.
• MariaDB implements them natively.
• Oracle is partial: SQL:2011-compliant syntax arrived in 12c, alongside the older proprietary Flashback mechanism it already had.
• PostgreSQL core does not implement system-versioned tables. An unmaintained third-party extension exists, and the major managed-Postgres platforms don't offer it.

That last one matters more than the other four combined, because Postgres is what a very large share of businesses are running. If you're on Postgres and you want transaction time, you build it. The workable pattern is an append-only history table written by a trigger on every change, with the current table kept as a projection for the queries that only need the present. Postgres range types plus exclusion constraints will enforce that validity periods for a given key don't overlap, which is the check that keeps a hand-rolled temporal table from silently accumulating garbage.

The cost nobody warns you about

Bitemporal tables are harder to query, and the difficulty is the dangerous kind, because forgetting a predicate returns rows instead of an error. Every single query against a bitemporal table needs constraints on both periods. Miss the transaction-time predicate and you get every historical assertion the table ever held, so your revenue number is inflated by every correction ever made. The query runs. It returns a number. The number is garbage and it looks like every other number on the page.

Teams handle this by never exposing raw bitemporal tables to analysts. The temporal table is the write model, and a set of views or materialized projections carry the current-state and corrected-history versions that most consumers actually want, with the as-of-then queries reserved for the small number of people who know they need them. That's the same one-source-many-projections discipline Part II lands on, applied along the time axis rather than the shape axis.

The second cost is volume. A system-versioned table grows with every update rather than with every entity, and a chatty table that gets touched by a nightly sync will grow fast. That's a partitioning and retention problem with known solutions, and it's worth knowing before you turn versioning on across an entire schema rather than after.

The overwrite that erases a year [summarize]

What a slowly changing dimension is, in plain terms

Warehouse data splits into facts and dimensions. Facts are the things that happened: an order, a click, a payment, a support ticket. Dimensions are the descriptive context you group those facts by: which customer, which product, which sales rep, which campaign, which region. Facts are append-only in practice, because an order that happened on March 3 stays happened. Dimensions change, slowly, which is why Ralph Kimball and Margy Ross call them slowly changing dimensions in The Data Warehouse Toolkit (3rd edition, Wiley, 2013).

A customer moves from one segment to another. A product gets recategorized. A rep changes territory. Each of those is a small, boring administrative event, and each one forces a decision that almost nobody makes deliberately: what happens to the old value? Kimball's original three answers are still the ones that matter.

Higher-numbered types get quoted around this subject a lot, so worth being precise about what they are. Types 5, 6 and 7 are formalizations of techniques the practitioner community had already been using, and they were type-numbered retroactively in the 2013 third edition rather than shipping alongside the original three. Type 6 is credited by the Kimball Group itself to an outside engineer at HP in 2000. They're useful hybrids and they're worth knowing. They aren't a separate lineage of original Kimball doctrine, and anyone presenting them that way hasn't read the source.

The Q1 report that changes every time you run it

Type 1 is where the year gets erased, and the mechanism is worth walking slowly because it's silent.

Say a company classifies accounts as SMB, Mid-Market, or Enterprise, and that classification lives as a column on the customer dimension. In April, sales ops runs a cleanup and reclassifies a few hundred accounts that had outgrown their original tier. Reasonable work. Somebody's job. It executes as an UPDATE against the customer table, which is Type 1 by default, because Type 1 is what an UPDATE is.

Nothing happened to the facts. Every order row is exactly where it was, with the same amount, the same date, the same customer key. The fact table is untouched and correct.

Now rerun the Q1 revenue-by-segment report. Same SQL. Same fact rows. Different answer. Every Q1 order placed by a reclassified account now joins to the current segment value, so revenue that was SMB in the Q1 board deck is Mid-Market today. The Q1 deck said one thing; the query that produced it now says another; both were generated by the same code against the same facts.

The specific artifact that's now unreproducible is the segment mix in that Q1 board deck. The total is fine, because a sum doesn't depend on the dimension. Every cut that groups by a Type 1 attribute is affected, and all of them changed the moment the UPDATE committed:

• Revenue by segment, once any account is reclassified
• Pipeline by territory, once any rep changes patch
• Retention by plan tier, once any plan is renamed or regrouped
• Margin by product category, once any SKU is recategorized

None of them error out and all of them quietly restate the past.

This breaks an assumption almost everyone holds without examining it, which is that a report is reproducible from its query plus its data. It isn't, when the dimension is Type 1. The query is deterministic and the fact table is immutable and the output still moves, because a third input nobody counted as an input, the current state of the dimension, is being read at query time.

The damage compounds in a specific way that makes it hard to catch. Trend analysis across a Type 1 dimension is measuring two things at once: real change in the business, and accumulated reclassification. A chart showing Mid-Market revenue growing 40 percent year over year might be showing growth, or migration of accounts into that bucket, or both in unknown proportion. Nobody can decompose it afterward, because the reclassification events left no record.

Backups are disaster recovery. History is a different product.

The standard response, and I've had this conversation more times than any other one on this subject, is that the old values are in the backups. They usually aren't, and where they are, having them isn't the same as having history. Five specific reasons, and each one is enough on its own.

Retrieval cost makes it not a query. Answering "what segment was this account in on March 3" from a backup means provisioning an instance, restoring a full database image to it, running one SELECT, and tearing the instance down. That's an operation a DBA performs a handful of times a year under duress. It is not something an analyst does while building a report, which means in practice the answer is unavailable even when it technically exists.

Retention windows are short. Thirty days is common. Ninety is generous. Point-in-time recovery windows on managed database services are usually measured in days rather than years. A value that changed fourteen months ago is not in any backup the company still holds, and the retention policy that deleted it was written by someone thinking about ransomware and storage cost, with no idea it was also a records-retention decision.

Snapshot granularity loses intermediate states. Backups run on a schedule. Any attribute that changed twice between two snapshots has a middle value that was never captured by anything, so even a complete backup archive has holes at exactly the resolution that matters for a fast-moving attribute.

There's no index across time. A backup archive can't answer "when did this value change" or "how many accounts were reclassified in April" without restoring every snapshot in the range and diffing them pairwise. The question that history makes trivial is the question backups make combinatorial.

And the two things solve different problems. A backup exists so the business can get its system back after a disaster. History exists so the business can say what its system asserted on a given date. Those are separate requirements that happen to touch the same bytes, and satisfying one has never satisfied the other. A company with excellent backups and Type 1 dimensions has full disaster recovery and no memory.

When Type 1 is the right call

Type 1 has a legitimate use and it's worth naming clearly, because a rule that says "never overwrite" is one nobody can follow and everybody ignores. Type 1 is correct when the old value was never true, and it's wrong the moment anybody acted on the old value.

Somebody typed a customer's name as "Micheal" and it's being fixed to "Michael." The old value was a typo. It was never a fact about the world, nobody made a decision because of it, and no report meaningfully grouped by it. Overwrite it and move on. Storing the history of a misspelling is ceremony, and it makes the table worse for everyone downstream.

Compare that with the segment reclassification. That old value was true. It was true for the whole period during which the account was actually smaller, commission plans and territory assignments and quarterly targets were built on top of it, and a board deck reported it. Overwriting it destroys the context for every decision made under it.

The practical test I run is a single question: did anyone or anything read this attribute and then do something? If a compensation plan, a routing rule, a pricing tier, a report, or a model consumed it, the value is load-bearing and needs Type 2. If the attribute exists purely for display and nothing has ever branched on it, Type 1 is fine and cheaper. Most teams have never asked that question per attribute, which is how entire dimensions end up Type 1 by default, and which is also why the fix is usually cheap: it's a handful of columns that matter, not the whole table.

State as a fold over events [summarize]

The log is the record, and state is something you compute

Every design so far in this article has tried to preserve history alongside a mutable current value. Event sourcing inverts which one is real. The sequence of things that happened is the system of record, and current state is a value you derive by replaying that sequence. The account balance is computed from every deposit and withdrawal ever recorded.

A fold is the operation that does the deriving. It walks a list from the beginning, carrying one accumulated value, and applies a function at each step. Start at zero, add each deposit, subtract each withdrawal, and the value you're holding at the end is the balance. Nothing about that is exotic; it's what a running total is. The shape claim is that the list is the durable thing and the total is disposable, which is the exact reverse of how essentially every application stores data.

That inversion buys three questions that a mutable table cannot answer at any price:

• The sequence. What order did things happen in, and what else was happening around them? A current-state table has no order. It has a value.
• State at any instant. Replay the log up to a timestamp and stop. That gives you the exact state the system held then, without having planned in advance to snapshot that particular moment.
• The first event that broke an invariant. Not that the balance is wrong now, which you already knew, but which specific event first made it wrong, and what the system looked like immediately before it.

Venetian merchants shipped this in the fifteenth century

Accountants solved this problem roughly five hundred years before software had the problem, and they solved it in the same shape. A ledger is append-only. An entry that turns out to be wrong doesn't get erased or edited; it gets corrected by a new reversing entry, so the mistake and its correction both stay on the record permanently. The balance is derived by folding the entries. Luca Pacioli documented the method in his Summa de arithmetica in 1494, describing what Venetian merchants had already been doing for generations rather than inventing it himself.

The reason that discipline exists is worth sitting with, because it's the same reason it belongs in software. Double-entry bookkeeping is append-only because somebody might be lying. A ledger that can be edited proves nothing, since any state it shows could have been produced by editing rather than by transactions. The append-only constraint is what converts a record from a claim into evidence. Erasability and provability are the tradeoff being made, and merchants who were being audited by people with an incentive to catch them picked provability.

Software defaulted the other way, and it defaulted that way for a reason that made sense at the time: storage was expensive and keeping only the current value was the cheap option. That constraint stopped binding decades ago. The default didn't move, because defaults rarely do, and most systems today are storing less history than a fifteenth-century merchant while paying for object storage that costs fractions of a cent per gigabyte.

The inventory that went negative

A warehouse system shows a stock quantity of negative three for a SKU. It's a physical product. There are not negative three of it in the building.

With a mutable quantity column, that's where the investigation begins and usually where it ends. The column says -3. It doesn't say how it got there. Somebody checks the recent orders, finds nothing obviously wrong, adjusts the count back to zero, and closes the ticket. Two weeks later a different SKU goes negative. Nobody connects them, because there's nothing to connect: each incident is a single number that was wrong once.

With an event log, the same investigation is a replay. Fold the events for that SKU and watch where the running total first goes below zero. The answer might be that a return got recorded twice, or that a shipment event arrived out of order from a warehouse integration, or that a cancellation released stock that had never been reserved. The log doesn't just show that the count is wrong; it identifies the event class that produced it, which is the difference between fixing a number and fixing a bug. And once you know the event class, you can replay every SKU against the same test and find the other seventy that are quietly wrong by amounts nobody noticed.

That last part is the compounding benefit and it's the one that justifies the cost. An invariant violation found in the log is a query you can run across the entire history, forever. An invariant violation found in a mutable table is an anecdote.

What it actually costs

Mechanism first, then limits, with no romance about it. Event sourcing is expensive in four specific ways, and teams that adopt it on enthusiasm rather than on a named requirement tend to discover all four at once about eight months in.

Persisted event schemas become a contract you can't migrate. A table schema can be altered, because there's one current row per entity and you can rewrite them all. Events are immutable by definition, so an event written three years ago has to stay readable by today's code, forever. Changing an event's shape means versioning it and writing upcasting logic that translates old versions forward, and that logic accumulates permanently. Teams underestimate this because the first year is fine.

Replay gets expensive, which forces snapshots. Rebuilding state by folding ten years of events takes as long as it takes. The standard fix is periodic snapshots, so replay starts from a recent checkpoint rather than from the beginning. That works, and it also reintroduces a stored current state that can drift from the log, which is precisely the thing event sourcing existed to eliminate. Snapshots need their own validation.

Projections go stale. Read models built off the log usually update asynchronously, which means a user who just performed an action can refresh and not see it yet. That's a genuine product problem, and it lands on the people least equipped to reason about it, which is support. Handling it well takes deliberate design rather than an apology in the docs.

Event granularity is a modeling decision you can't take back. Choosing to record "order updated" instead of "shipping address corrected" throws away the distinction permanently, for every event already written. This is the ceiling argument from Part I applied inside the log itself: an event log records exactly the resolution you chose when you designed it, and coarse events are as unrecoverable as coarse table grain.

When it's the wrong choice

Most applications should not be event sourced, and saying so plainly matters more than the enthusiasm does.

If nobody in the business has ever asked a historical question, and the domain genuinely has no audit, dispute, or reconstruction requirement, then the current state is the whole truth and storing a log is engineering cost with no buyer. A content management system, an internal admin tool, most catalog and configuration data, and the large majority of CRUD applications fall here. They're better served by a Type 2 dimension on the two or three attributes that matter and nothing else.

Team capability is the second disqualifier and it's the one people skip. Event sourcing changes how every developer on the team reasons about reads, writes, consistency, and debugging. A team of three that's never run it will spend its first year building the infrastructure rather than the product, and the failure mode is a half-migrated system with both a log and a mutable table where neither one is authoritative and they disagree.

The requirement that justifies it is specific: somebody outside the engineering team needs to reconstruct what the system did, and needs to do it more than once. Finance, compliance, disputes, safety, anything with a regulator, and any domain where an automated decision has to be explained afterward. Those requirements are worth the four costs above. General good hygiene is not.

Where a number came from [summarize]

A claim's standing should be a query

Ask most organizations where a number came from and the retrieval mechanism is a person. Somebody who was there remembers which spreadsheet fed the model, roughly, and which extract fed the spreadsheet, approximately, and whether the January figures were the restated ones. That person is usually competent and usually right. They're also the only index, they leave, and nothing they know is queryable.

Believability should be a computed property of the support structure. A claim's standing becomes a query against its sources rather than a judgment about whoever asserted it. That's a shape requirement before it's a policy requirement, because you can't query a derivation graph you never built.

The business version arrives as an ordinary request that turns into a two-week project. An acquirer asks how a cohort retention figure was calculated. A regulator asks which inputs fed a pricing decision. A client asks where a claim in a deliverable came from. Each of those is a lineage query, each is answerable in seconds by a system that stored derivations, and each costs two weeks of spreadsheet archaeology in a system that stored results.

The correction that was wrong in the more dangerous direction

My own worst version of this is worth putting on the record, because it runs opposite to the direction people expect.

Working through some of my own material, I flagged an attribution as a probable fabrication: a confident-sounding technical label pinned onto a well-known researcher, which I judged to be the kind of thing a generator invents and nobody checks. I wrote the correction. Checked properly against primary sources afterward, the label was the researcher's own term, used by him in his own published writing, and the original attribution had been substantially right. I introduced the defect by failing to check before I corrected. Part II covers that specific attribution in its section on where the metagraph term actually comes from.

Being wrong while skeptical looks like diligence, which is why that class of error survives review. An over-trusting error gets caught eventually, because somebody asks for the source. An over-skeptical error ships as a correction, carrying the institutional signal of a person doing their job carefully.

Both failures skipped exactly one operation, and it's the same operation: checking the primary source before ruling. Two opposite reflexes, one missing step, and no amount of care in either direction substitutes for it.

The reason the error was possible at all is a shape fact. The material lived in documents, which is to say in prose, outside any structure where a claim could be adjudicated against its evidence. Nothing could make the error visible from either side, because there were no provenance edges to query. Promote the same claims into a structure where each carries its sources and its standing, and both the over-trusting and the over-skeptical version of the mistake become mechanically visible instead of depending on who happened to read carefully that day.

A much older institution runs the same discipline. Evidence law worked out centuries ago that a physical artifact with a broken chain of custody is inadmissible, regardless of what it is or how obviously relevant it looks. The evidence is unprovable rather than wrong, because nobody can establish what happened to it between the scene and the courtroom. An artifact's evidentiary value comes from its custody record rather than from its contents, which is the claim this article makes about data, arrived at several hundred years earlier and with better enforcement.

Three provenance questions the literature already separated

Database researchers split this with more precision than most data teams realize, and the three-way split is operationally useful rather than academic. Each question fails differently and each gets asked by a different person.

The gap between why and how matters more than it sounds. Why-provenance tells you a result depended on records A, B and C. How-provenance tells you whether it needed all three together or whether any one of them alone would have produced it. Those imply completely different actions when C turns out to be garbage. Under one derivation structure, retracting C invalidates the result. Under another, the result stands on A and B and nothing changes. A lineage system that records only the set of inputs can't tell you which situation you're in.

Todd J. Green, Grigoris Karvounarakis and Val Tannen formalized this in Provenance Semirings (ACM PODS, pp. 31-40, 2007). Their framework unifies incomplete databases, probabilistic databases, bag semantics, and why-provenance as instances of evaluation in a commutative semiring. The mechanically important part for anyone building systems: when inputs carry semiring annotations, provenance propagates compositionally through positive relational algebra, with addition capturing alternative derivations and multiplication capturing joint use.

That compositionality is the whole argument for calling provenance a shape property rather than a logging feature. If inputs carry annotations, every derived result carries a correct derived annotation automatically, because the operators do the propagation. If inputs don't carry annotations, no amount of downstream effort reconstructs what they would have been, since the structure that would have propagated was never there. A system storing only collapsed current facts has nothing left to propagate through.

Entity, Activity, Agent, and the one everybody skips

There's a standard for describing all of this, and it's short enough to actually read. The W3C published PROV-DM: The PROV Data Model as a Recommendation on 30 April 2013. It models provenance with three core types, defined in the specification as follows.

Entity is "a physical, digital, conceptual, or other kind of thing with some fixed aspects; entities may be real or imaginary." Activity is "something that occurs over a period of time and acts upon or with entities; it may include consuming, processing, transforming, modifying, relocating, using, or generating entities." Agent is "something that bears some form of responsibility for an activity taking place, for the existence of an entity, or for another agent's activity."

Most systems record entities well, record activities badly or not at all, and record agents almost never. You have the source file and the output file, and nothing durable about the transformation between them beyond a job name in a scheduler that's been rewritten twice since. Agent is the type everybody skips and the one compliance actually asks about, because responsibility is a question about people and services, and it stays relevant long after anyone remembers what the pipeline was called.

The practical read on PROV is that it's a vocabulary rather than a product. You don't have to adopt the RDF serialization or any of the tooling to get the value. Recording that this artifact was generated by this activity, which used these inputs, and was associated with this agent, in whatever store you already run, gets you the queries. The standard's contribution is settling what the three things are, so that separate systems mean the same thing by them.

Record the altitude, not a true-or-false stamp

Provenance systems fail in a specific way when they try to be verdict machines. The instinct is to mark each claim true or false and move on, which collapses a rich structure into one bit and discards the part carrying the value.

The better move records a claim at its actual altitude, with its caveat attached. An established fact resting on citable sources sits at one altitude. An attributed synthesis, where somebody real combined established pieces into a claim that's plausible and unproven, sits at another. A live research program with no settled result sits at a third, and the correct annotation says so explicitly rather than rounding it toward either true or false.

Translate that to an operating business and it stops sounding philosophical. A revenue figure pulled from the billing system, a revenue figure from a rep's forecast, and a revenue figure from a propensity model are three different altitudes wearing the same currency symbol in the same column of the same dashboard. All three are legitimately useful. Flattening them into one number is the defect, and the schema commits it, because a decimal column has nowhere to record which kind of number it's holding.

Once altitude travels with a value, a whole class of question becomes askable. Show me this forecast with modeled inputs excluded. Show me which board-deck figures rest on a source that's since been superseded. Show me every downstream artifact that inherited an unproven claim. None of those are answerable against a schema storing a number and a name, which brings the argument to the property that makes altitude possible at all: the system has to be able to hold two things that disagree.

Holding two facts that disagree [summarize]

The resolution happens at write time, in a job nobody remembers

Nearly every schema in production enforces one value per key. A customer has an employee count. A product has a price. An account has an owner. That constraint feels like a description of reality, since a company does have some specific number of employees, and it's actually a statement that the system will accept exactly one answer and reject or overwrite the rest.

Reality supplies more than one answer constantly. A sales rep typed 45 into the CRM after a discovery call. An enrichment vendor returns 120. The customer's own signup form said 60. Three sources, three numbers, all captured, all plausible, and a schema with room for one.

So something resolves it, and that something is usually a transformation step written by a contractor three years ago. It might be last-write-wins, meaning whichever job ran most recently is the truth. It might be a source priority list, meaning the enrichment vendor beats the rep because somebody decided that once in a meeting. It might be a COALESCE that takes the first non-null value in an arbitrary column order. Each of those is a substantive business decision about whose information to trust, encoded as a line in a pipeline, invisible to everyone consuming the output.

Downstream, that number renders on a dashboard as a fact. It arrives with the same visual weight as the order total, which came from an actual transaction and has exactly one correct value. Nothing on the page distinguishes a measured quantity from an arbitrated one.

Disagreement is evidence, and collapsing it destroys the evidence

The information lost in that collapse is more valuable than the number that survives.

Consider two accounts. For the first, all three sources say 120 employees. For the second, the rep says 45, the vendor says 120, and the form says 60. After resolution both accounts show 120, and the two rows are now indistinguishable. Full agreement across independent sources and a three-way conflict resolved by policy have been rendered identical.

That difference is exactly what a person would want to know. The first account's headcount is reliable enough to set a pricing tier on. The second one's is a guess, and the spread between 45 and 120 straddles most segment boundaries a business would care about. Any team using that field to route accounts, set quotas, or size a deal is treating both as equally solid, because the schema told them to.

Disagreement also predicts things. Accounts where sources conflict are often accounts undergoing change, which is to say the interesting ones: rapid hiring, an acquisition, a rebrand, or a data-entry process that's broken for a whole segment. A system that resolves conflicts silently throws away a leading indicator and reports a clean number instead.

There's a second signal in the time dimension, and it's only computable if the earlier sections' machinery is in place. How long an assertion stood before something replaced it is itself information. A value corrected within a day was probably a typo. A value that held for two years and then changed probably reflects something real that happened in the world. Same column, same correction, completely different meaning, and the only shape that can tell them apart is one that recorded when each assertion started and stopped.

Refusing to resolve at write time

The structural move is to stop choosing at write time and make resolution a parameter of the query instead. Store all three assertions with their sources, and let each consumer resolve according to what it's doing.

That sounds like a dodge until you notice different consumers legitimately want different answers. Finance wants the conservative figure, because understating a tier costs less than overstating one. Sales wants the optimistic figure, because it sets the ceiling on the conversation. A model wants all three as separate features plus their variance, because the disagreement carries predictive weight the resolved number destroyed. One value per key forces those three consumers to share an answer that's wrong for at least two of them.

Every strategy for handling this destroys something, and the useful thing is knowing exactly what each one costs before picking:

Rows three and four are the same architecture at two levels of ambition, and both start from the same primitive. A fact becomes an assertion rather than a value. Each assertion carries what was claimed, who claimed it, when, and how much standing it has. Nothing gets deleted when a new assertion arrives; the old one gets superseded, which is a relationship rather than an erasure, and the earlier bitemporal machinery is exactly what records when the supersession took effect.

Confidence as something the structure computes

Once assertions are addressable, they can point at each other. One assertion supports another. One attacks another. Those edges make credibility a computed property instead of a field somebody maintains by hand, which is the thing that keeps a confidence score from decaying into a number nobody trusts.

The computation is a propagation. Priors flow along support and attack edges until the whole structure settles, and the equilibrium it reaches is the standing of every claim given every other claim. A source that's been contradicted repeatedly by better-grounded sources loses weight everywhere it appears, automatically, without anyone auditing it. A claim sitting inside a cycle of mutual support from well-grounded sources gains weight the same way.

This is where the whole article converges, and it's worth naming the dependency explicitly. Contradiction handling needs provenance, because an attack edge is worthless if you can't say what's doing the attacking. Provenance needs time, because a source that was authoritative in 2023 and superseded in 2025 has to carry both facts. The three properties aren't three features; they're one structure, and a system that has two of them is usually about to discover it needs the third.

When one value per key is the right answer

Plenty of data has exactly one correct value and should be constrained to one. An order total is what was charged. An invoice number is assigned once. A timestamp on a payment is whatever the payment processor recorded. Modelling contradiction for those is ceremony that makes every downstream query harder and buys nothing, because there's no second source with standing to disagree.

The test is whether more than one independent source can legitimately produce a value for the field. Measured quantities from a single system of record get a uniqueness constraint. Anything assembled from multiple sources, inferred by a model, entered by a human, or bought from a vendor should be an assertion, because all four of those produce values that can be wrong in ways the system can detect only by comparison.

Most schemas would need this treatment on a small minority of their columns. Firmographics, segments, attribution, model outputs, enrichment fields, anything with the word "estimated" in its description. The rest can stay simple, and keeping them simple is what makes the complicated ones affordable.

The medallion ladder [summarize]

Five tiers, and what actually separates them

Everything so far has argued for properties. This section is what those properties buy, and it's the part that turns a data-modelling discipline into something a business can price.

The ladder runs bronze, silver, gold, platinum, diamond, and the separations are structural rather than qualitative. No one grades an artifact and awards it a tier.

Bronze never changes, and everything else derives from it. That's the immutability rule from the event-sourcing section applied to whole artifacts instead of individual events. An original document is never edited in place; a processed version becomes a separate silver artifact and the original stays exactly as ingested. The upload event is the birth record.

Silver is broad on purpose. Any changed copy is silver, which covers chunks, embeddings, enrichments, research artifacts, templates, drafts, and rubric evaluations. It's the construction dust of production: enormous in volume, necessary, and by itself no evidence of anything. Silver is where teams generate the most output and where they most often mistake volume for progress.

Gold is defined by events, which is the load-bearing choice

Gold means events are being tracked against the artifact. Something happened in reality involving it, that occurrence got captured as data, and the captured stream is what makes it gold.

Defining it that way rather than by quality is the decision the entire ladder rests on. A quality judgment is somebody's opinion, it's assigned at a moment, and it never updates. An event is a thing that occurred, it's recorded, and it accumulates. Gold is proof because events are proof.

Shipping something client-facing is one way to start producing events, and it isn't the definition. An internal prompt whose pipeline runs are tracked is gold, because runs are events. A knowledge base an agent retrieves from is gold, because retrievals are events. A pull request a maintainer reacted to is gold. The common property is a stream accruing against the artifact, and whether a customer ever saw it is beside the point.

It's also the answer to a question every content and knowledge operation eventually faces, which is how to tell a valuable asset from an expensive one. Word count doesn't answer it. Effort doesn't answer it. Somebody's confidence in the asset doesn't answer it. An event stream answers it, and an artifact with no events is silver no matter how good it looks or how long it took.

Tier is derived, which is why there's no tier field

The rule that keeps all of this from collapsing into vibes: tier is read from an artifact's lineage. An artifact whose lineage includes a production-ship event is gold. One composed from multiple golds is platinum. Systems compute the answer by walking the chain.

The alternative fails in a way anyone who's worked with a data catalog will recognize immediately. A tier column that humans set is a column that's accurate the week it's populated and decorative within a year. People tier things optimistically, nobody demotes anything, and the field drifts into a record of what somebody once hoped rather than what's true. A derived tier can't drift, because there's no field to drift.

This is where the three properties stop being theory and start being load-bearing, and the dependency runs in one direction. Deriving tier requires total lineage, which is the provenance section. Total lineage requires that bronze never changes, which is the append-only argument. Knowing which events happened when requires transaction time, which is the bitemporal section. Pull out any one of the three and the ladder becomes a field somebody maintains by hand.

One chain, walked end to end

Walking one chain makes the derivation concrete. An interview recording gets ingested and stored exactly as received, which makes it bronze, and it stays bronze permanently no matter what happens downstream. A transcript is generated from it: a changed copy, so silver. The transcript gets chunked and embedded, producing more silver. A researcher pulls a set of those chunks into a brief, still silver. A draft comes out of the brief, silver again, and so does every revision of it.

Then the piece publishes and readers start hitting it. Retrieval events, engagement events, and any downstream conversion begin accruing against that artifact, and at that moment it becomes gold. Nobody promoted it. The system walks its lineage, finds an event stream attached, and reports gold. The identical artifact sitting unpublished in a drafts folder is silver, and the difference between the two states is entirely the presence of events, which is exactly what makes the distinction resistant to opinion.

Notice what the chain gives you that a folder structure cannot. Every one of those silver steps knows which bronze it descends from and which artifacts descend from it. When the interview subject later retracts a claim, the retraction has a path to travel: up through the transcript, the chunks, the brief, the draft, and into the published piece, flagging each one. In a folder, a retraction is an email asking whether anybody used that interview.

Credit propagates back down the chain

Now the part that pays for the whole architecture, and it's an analytics capability rather than a governance one.

Because tier is lineage-derived and lineage is total, every event against a gold artifact propagates credit backward: through the silver artifacts that shaped it, down to the bronze sources that fed it. A reader engaging with an article credits the drafts that produced it, the research that informed those drafts, and the original documents the research drew from.

Run that across a whole corpus and questions become answerable that flat storage cannot even represent:

• Source-utility ranking. Which documents ultimately produced work that performed, as opposed to which documents got cited most or read most.
• Template win rates. Which prompts and templates earn their slot in a pipeline, measured by what came out of them downstream rather than by how often they ran.
• Enrichment return. Which processing steps actually improved outcomes, which is the question that decides whether an expensive enrichment stays in the pipeline.

That's attribution, pointed at a knowledge supply chain instead of an ad account. Part I opened on an attribution failure: a business that couldn't say which campaign produced revenue, because the join between click and order was never recorded. This is the same question with the same structure, and it comes out differently only because the lineage was recorded this time. Credit assignment is solvable when the chain exists and unanswerable when it doesn't, in marketing and in knowledge work alike.

The capability exists only because bronze is immutable, lineage is total, and gold is event-defined. Remove immutability and the chain has gaps. Remove total lineage and credit stops partway. Remove event definition and there's no signal to propagate in the first place.

Platinum and diamond, described as what they are

Being precise about which rungs are built and which are aspiration is itself part of the discipline, so: the top two are future state.

Platinum is composed from multiple gold artifacts, and the reason gold has to be event-defined becomes obvious here. Event streams tell you which gold performed, so platinum composes from measured winners rather than from whatever looked good. A playbook distilled from pieces whose events proved them is platinum; the same playbook assembled from drafts somebody liked is silver wearing a nicer name. The lineage chains make this derivable once a corpus matures.

Diamond means mission-critical in a specific sense: it doesn't break. Every world model carries its own trilemma, three mutually opposed constraints that ordinarily can't all hold. Diamond means that entity's particular trilemma is satisfied on all three legs at once, which is the expensive corner and the reason it holds up. Identifying what an entity's trilemma actually is takes real world-model work, and treating diamond as a named aspiration rather than a current state is the accurate description today.

Where these converge [summarize]

The three properties are annotations on relationships

Something has been true of all six preceding sections without being said directly. Time, provenance and contradiction are properties of relationships rather than properties of things.

A customer doesn't have a validity window. The claim that this customer belongs to that segment has one. A number doesn't have a source. The derivation connecting a number to its inputs does. Two values don't contradict each other in isolation; two assertions about the same subject do, and the contradiction is a relationship between them. Every property this article argues for attaches to an edge.

Which explains why they fit badly into a table. A table stores things in rows, and a relationship gets represented as a foreign key, which is a pointer with nowhere to hang an annotation. Adding "when was this true" to a relationship means promoting the relationship into its own table, and then adding "where did it come from" means another table, and adding "what disagrees with it" means the relationship needs an identity that other relationships can point at.

The shape where that's native is a metagraph, and the RAG article already defines it as knowledge held as a navigable structure of concepts and relationships with time, contradiction and provenance as first-class properties. No point restating that here. The relevant piece for this article is the ladder underneath it. A graph gives nodes and edges. A hypergraph lets one edge bind many participants at once, which is what a real event needs, since a real fact has many participants bound into one occurrence. A metagraph makes the edges themselves addressable, so you can point at a relationship, annotate it, connect it to other relationships, and reason about it. Rules about rules. Confidence on the confidence.

Every fact carries a confidence, a provenance (the episode that produced it), and a bitemporal validity window recording when it became true and when it was superseded. That sentence is the whole article compressed, and it's only expressible in a shape where facts are addressable objects rather than cell values.

Worth naming one thing this buys, because it answers the standard objection. Annotating a graph usually means reification: promoting a relationship to a node and hanging binary edges off it, which multiplies the node count and makes every query longer. Treating an annotation as a value over the existing structure avoids that. A confidence or an attention weight is a number attached to a shape rather than a new object in it, which is why this approach scales where triplet bookkeeping drowns.

Four axes at once

The payoff shows up when something has to navigate all of it, and the capability is specific enough to enumerate. A system working over a corpus shaped this way can move along four axes simultaneously.

It moves vertically, from a whole-domain summary down to a specific paragraph and back, which a human expert can also do, more slowly. It moves horizontally, following cross-connections the graph makes explicit, arriving at a bridge between distant regions that a person would need a flash of insight to notice. It moves through time, distinguishing what was believed then from what's believed now, which people do poorly and a bitemporal structure does natively. And it moves through confidence, weighting a well-grounded claim differently from a speculative one by propagating trust through the support structure rather than treating every retrieved sentence as equally true.

No working memory holds four axes of a large domain at once. The interesting consequence is where the capability actually lives: the model doing the navigating is rented, commodity, and identical to everyone else's. The territory is the part that's yours, built once, compounding, and impractical to copy without redoing the ingestion that produced it. My own knowledge base runs past 5 million words across hundreds of mostly Markdown files, and the words are the least interesting part of it. The structure over them is the asset.

Compounding and accumulating are different operations

There's a loop here that only closes when lineage is total, and it's the reason to build structurally rather than as a search index.

Work produced from the corpus gets ingested back into it. An article written out of a harvested set of sources becomes, once published and chunked and connected, part of the material available to the next piece of work. The corpus that produced the article now contains the article, and it contains it with the derivation intact, so the new node connects things that were previously only near each other.

A search index accumulates and a structured corpus compounds, which is the difference between a folder that gets bigger and a knowledge base that gets smarter. Accumulation adds documents. Compounding adds documents plus the connections they created plus the credit signal flowing back to whatever fed them. The medallion ladder is what makes the second one measurable, since without event-defined gold there's no way to tell which additions were worth anything.

What ships today, and what doesn't

An accurate accounting matters here, because this subject attracts more enthusiasm than it does working systems.

Available now, off the shelf. Bitemporal modelling is standardized in SQL:2011 and implemented natively by several major databases, with a well-understood build pattern for the ones that lack it. Provenance has a settled vocabulary in the W3C Recommendation and a rigorous theory in the semiring literature. Graph databases with property edges are mature and boring in the good sense.

Buildable, with real work. Confidence propagation through support and attack edges is implementable today and mostly isn't implemented, because it needs the provenance layer underneath it to be worth anything. Lineage-derived tiering needs total lineage, which is an engineering commitment rather than a product you buy. Credit propagation back down the chain follows from those two and is where most of the differentiated value sits.

Aspiration, and worth labelling as such. Platinum composition from measured winners needs a corpus mature enough to have measured winners. Diamond is a named target. Saying so plainly costs nothing and it's the same discipline the rest of this article argues for: record the claim at its actual altitude instead of rounding it up.

The formal lineage of the term is worth one sentence, since it gets used loosely. Metagraph has a real definition in the discrete-mathematics literature as a generalization where edges connect sets to sets, and OpenCog's Atomspace adopts it for a structure where every atom, node or link alike, has a handle and can itself be the subject or object of other links. That handle is the technical content of "edges pointing at edges," and Part II covers what that lineage actually claims in more detail.

What to do tomorrow morning [summarize]

The first move is much smaller than a rebuild

Nothing in this article requires re-architecting a company. The useful version of this work starts narrow and the narrowness is what makes it survive contact with a roadmap.

Start by listing the reports and automated decisions that actually matter. Board reporting, commission, pricing, routing, whatever a regulator or a customer can ask about. For each one, write down the attributes it groups by or branches on. That list is almost always under ten columns, and those columns are the only ones that need history.

Then three concrete changes, in this order, because each makes the next one cheaper:

1. Convert those columns to Type 2. New row on change, effective dates, current flag. Leave every other attribute on Type 1, since versioning a display name buys nothing and makes every join worse.
2. Turn on transaction time where it's free. On SQL Server, Db2 or MariaDB, system-versioning is a DDL change. On Postgres, it's a trigger writing to a history table for those same few columns, with an exclusion constraint so periods can't overlap.
3. Record activity and agent, not only entity. Every pipeline job writes one row saying what it read, what it wrote, and which version of itself did it. That's the cheapest provenance available and it converts "where did this come from" from an investigation into a query.

None of that is a platform. It's a few columns, a trigger, and a log table, and it can ship inside a normal sprint alongside whatever else is going on.

The test that tells you where you stand

Before any of it, run one diagnostic, because it costs an afternoon and it tells you whether you have a problem worth spending on.

Take a report from last quarter and reproduce it exactly. Same query, same parameters, and compare the output line by line against the version that was distributed at the time. Then do it for a report from a year ago.

Three outcomes, and each one is actionable. The numbers match, in which case the dimensions those reports touch are stable or already versioned and you can stop reading. The numbers differ and you can explain every difference, which means you have partial history and a documented restatement, which is a healthy place to be. Or the numbers differ and nobody can say why, which is the common result, and it means the answer to "what did this business believe last year" is currently unavailable at any price.

That third outcome is Part I's ceiling diagnostic pointed at the time axis. Trace each question back to the event that would have had to be captured, find where the trace breaks, and the break is the ceiling. Finding it on a Tuesday by choice is considerably cheaper than finding it during due diligence.

Who should do none of this

Most businesses reading this should skip the entire article, and being specific about which ones is more useful than another paragraph of encouragement.

Three conditions make this work pay. There's more than one source producing values for the same field. There's an outside party who can compel an answer, which means a regulator, an auditor, an acquirer, a court, or a customer with a contract. Or there's an automated decision that has to be explained after the fact. If none of those three apply to you, stop here and spend the money on something that moves revenue. A single-product business running one CRM, selling through one channel, with no regulator and no disputes, genuinely has one version of the truth, and a flat table with Type 1 dimensions is the correct engineering answer.

I'd also send away the team that wants event sourcing because it sounds like the right way to build things. It's a real architecture with a real bill, and adopting it on taste rather than on a named requirement produces a half-migrated system where a log and a mutable table disagree and neither is authoritative. That failure costs more than the Type 1 overwrite it was meant to prevent.

And if you've never once needed to reproduce a number from last year, I'm probably not your guy for this. Come back when somebody asks and you can't answer, which is the moment this stops being architecture and starts being the thing standing between you and a bad week.

What a fact is worth

There's a book on my desk I've had for years. Its contents are worth the cover price and anyone can buy the same words tomorrow. What makes that particular copy valuable is everything around the text: where I was when I read it, what I was working on that made a passage land, the argument it settled, the work it went on to produce. The knowledge was never only the pages. It was the connections, the sources, the record of what came from it.

That's the whole argument of this article, applied to a single object.

A fact is worth what its biography makes it worth. A number with no history, no source, and no record of what disagreed with it is a number you can buy anywhere and defend nowhere. The same number carrying when it was true, where it came from, and what contests it is evidence.

Three parts, one claim. Part I argued that the shape of your data sets a ceiling on the questions you can ever ask, and that the ceiling is usually set by whoever built the first version of the checkout. Part II walked the five shapes and made the case that choosing one is choosing what to be structurally bad at. This part covered the three properties that decide whether your data can testify about itself, and all three have the same character: cheap to build in, impossible to add later, and invisible until the day somebody asks.

Every system reading this data is going to get better. That part is nearly free and it arrives on someone else's roadmap. The territory is the part that improves only if you build it, and it improves on its own once you do, as long as the outputs make it home. Build the map. The models to explore it are already here and they're getting cheaper every quarter.