The Shape of Data III: Time and Provenance. Past decisions need the version of the evidence available when they were made.

Explaining a past decision requires its historical inputs.

A commission report can change after the money has already been paid. Finance corrects a deal amount, someone runs the original query again, and the new result no longer explains the payment. The correction may be right. The regenerated report may also be using the wrong version of history.

A system that has to explain that payment needs to retain what it asserted at the time, when the corrected amount took effect, and the evidence behind the correction. Those are choices about what a fact contains. A current value alone can't answer a question about a version it replaced.

This is the third part of The Shape of Data. Part I examined the questions a system loses when it never captures the necessary information. Part II compared the shapes that make different questions possible. Time and provenance determine whether those shapes can explain a decision after its inputs have changed.

A corrected report can stop explaining the payment[summarize]

The deal amount changed after commission ran

Consider an illustrative deal. It closes on March 3 and enters the system at $80,000. A commission run uses that amount on March 15. Finance discovers a missed contractual discount on May 12, so the correct amount was $72,000 from the close date onward. These are teaching figures, not a reported client result.

The business now needs two answers. A restatement should use $72,000 for March 3. An explanation of the March payment should show that the system held $80,000 when commission ran. The first answer uses the correction. The second reconstructs the information available to the payment process.

Replacing $80,000 with $72,000 preserves the corrected amount and loses the earlier assertion from that row. The database still answers a present-day question. It can no longer answer the historical question from the current row alone.

The same distinction applies to a pricing rule, an account assignment or a report sent to a board. Reproducing a decision means recovering the inputs that the decision consumed, including inputs later found to be wrong. Correcting the inputs and explaining the earlier decision are separate jobs.

A new column records only the history supplied to it

A team can add a validity column to an existing table. The migration creates somewhere to put future history. It doesn't supply the old values, the intervals they covered or the source that produced them.

Recovery depends on what survived elsewhere. A retained event log, historical extract, document or backup may contain part of the missing record. A history that was overwritten everywhere and retained nowhere can't be reconstructed by adding a timestamp. Backfilling every row with the migration date makes that date an observation boundary. It doesn't make it the date each value became true.

That limit is the practical meaning of a data-shape ceiling. More computation helps with information a system has. It can't determine which of several possible pasts occurred when the surviving records don't distinguish them. A reconstruction can be useful, but it needs to carry its sources and uncertainty rather than appear as captured history.

My data-engineering work grew out of moving information between agency and customer accounts, then helping other agencies make use of their data. The lineage question was already there in attribution. A purchase after an ad impression doesn't establish that the last ad caused the purchase. The earlier contacts matter, and the chain has to be available before anyone can examine it. The same requirement applies when the output is a commission payment rather than a sale.

Keep history beneath a usable current view

An ordinary current-state design gives each business key one active value. That can be exactly what the application needs. An account screen should normally show the current segment without asking its user to choose among historical versions.

The loss happens when the same row also becomes the only durable record. An update replaces the old value, and any source or disagreement attached only to that value disappears with it. A uniqueness rule over the business key then makes concurrent assertions impossible to store in that table.

A system can keep the convenient current view while retaining versions underneath it. The extra work belongs in the write path and the historical query contract. Consumers that need the present get a deliberate projection; consumers explaining a past decision get the history appropriate to that question.

The design starts with a distinction the commission example makes unavoidable. March 3 is when the corrected amount applied to the deal. May 12 is when the system learned about it. Those dates describe different events, so they need different places in the record.

A fact can have an effective date and a recording date[summarize]

The two clocks answer different questions

Valid time records when an assertion applies in the world being modeled. A price takes effect on the first of a month. An employee's promotion applies from an agreed date. A deal has a contracted amount from its close date. The business supplies the meaning of that period.

Transaction time records when the database held that assertion. A price change might enter the system after its effective date. A correction to a deal might arrive after commission has run. Transaction time preserves the sequence of recorded knowledge, including the lag between an event and its capture.

Martin Fowler describes the distinction as actual history and record history. Actual history describes the history as currently understood. Record history describes how that understanding changed. A correction to the first history becomes a new entry in the second.

An updated_at field usually records the latest write. It doesn't by itself say when the value became applicable, what the previous value was or how long the database held it. A table needs enough information to distinguish both timelines. Naming a timestamp without defining which event sets it leaves that requirement unresolved.

The distinction also limits what an audit can establish. A database recording date shows when a particular system recorded a claim. A person may have known something earlier, an upstream service may have stored it elsewhere, or a delayed import may have arrived later. The database's history is evidence about that database. An organization-wide account needs the other relevant records too.

The commission example needs intervals on both axes

The illustrative deal was recorded at $80,000 from March 3 until the correction on May 12. The corrected assertion says $72,000 applies from March 3. A bitemporal design preserves both statements with the periods needed to tell them apart.

The old assertion can be described as effective from March 3, recorded from March 3 until May 12. The replacement is also effective from March 3, but recorded from May 12 onward. Asking for the deal's effective value on March 3, using knowledge available on March 15, selects the old assertion. Asking for the same effective date using knowledge available after the correction selects the new one.

A period needs a boundary convention. A half-open interval includes its start and excludes its end. That convention lets one version end exactly when the next begins without counting both at the boundary. Whatever convention the implementation uses, writers and readers have to agree on it.

The correction doesn't make the old assertion true in the world. It preserves the fact that the system asserted it. That's enough to explain an input to the payment process, though explaining the payment also requires its rule, rate, rounding and any other input the process used. A deal history alone isn't the whole commission audit.

This gives the reader four useful question forms. They share an entity, but select different histories.

The last question explicitly fixes both axes. An audit table can answer it if the table retains enough old values, effective periods and recording history. Bitemporality describes that information requirement. It doesn't require a particular product or a table with a particular name.

Historical inputs are part of debugging

A rule can behave differently against a corrected dataset. Re-running today's code on today's data answers what that combination produces now. It may say little about the input that caused a past decision.

A historical investigation therefore needs to identify which version of the rule ran as well as which data it received. A pricing engine might use an account classification that was later corrected. Replacing that input before reproducing the decision removes a possible cause from the investigation.

Time-aware data makes this investigation possible at the granularity captured. It doesn't automatically retain an external API response, an unrecorded manual override or a model version. Those are separate inputs to identify. The useful question is always concrete: which values and rules could have changed this output, and which of them were retained?

Temporal features still need a query contract

SQL:2011 distinguishes application-time periods from system-versioned history. Those correspond to effective time supplied by the application and history maintained by the database. Having one mechanism doesn't automatically supply the other.

Microsoft documents system-versioned temporal tables as a current table with an associated history table. The engine maintains period columns and stores previous versions when rows change. Business-effective dates still need their own meaning and representation when the question requires them.

A deployment using Db2, MariaDB or Oracle needs its own feature check. Verify which timelines the chosen version retains and which write paths it covers. Check extensions and managed services against the actual deployment too. A feature listed in a language standard may still require application code in the selected database.

A hand-built history design also needs atomic writes, deletion handling, writer coverage and concurrency rules. A trigger that captures updates but misses another write path leaves a gap. PostgreSQL documents range types and constraints for non-overlapping ranges, which are useful tools for temporal modeling. The constraint has to match the model: versions of the same assertion shouldn't overlap accidentally, while genuinely competing assertions may intentionally cover the same effective period.

The immediate reporting risk is counting versions as entities. A query that omits its recorded-time filter may return the original and corrected deal amounts together. The SQL can execute successfully and still answer a question nobody intended to ask.

Give ordinary consumers explicit views for current state and corrected history. Reserve the two-time query for callers that need it, with its parameters named. This is the same shared-history, multiple-projections discipline used for other data shapes. It keeps a richer record from making every report harder to write.

Versioning also changes storage growth. The history grows with changes as well as entities, and repeated synchronization writes can create versions that add no business information. Retention, partitioning and the definition of a meaningful change belong in the design. The next question is which attributes deserve that history in the first place.

A dimension update can reclassify unchanged sales[summarize]

Historical reports depend on the dimension they join

A warehouse fact records something the model measures, such as an order or payment. A dimension supplies descriptive context, such as the customer segment, product category or sales territory. A report combines the two when it groups revenue by segment.

That join makes the dimension part of the report's input. Preserving order amounts and dates is insufficient if every historical order joins to a customer's current classification. A report can change while every fact row stays untouched.

An illustrative account cleanup shows the effect. A company classifies customers as SMB, Mid-Market or Enterprise. Sales operations later updates some classifications. The update may reflect real growth, a corrected error or a revised segmentation policy. Those cases already mean different things, and the report needs to know which interpretation its author wants.

A Type 1 slowly changing dimension replaces the stored attribute. Re-running the earlier revenue-by-segment query now uses the new classifications. The total revenue can remain unchanged while its distribution between segments changes. The query has a changed input that the report's author may not have noticed.

Revenue by segment is only one example. The same issue can affect pipeline by territory after reassignment, retention by plan after regrouping, or margin by category after a product moves. Each report has to decide whether it wants the classification current at the event or the classification current at the time of the query.

Both can be legitimate. A sales manager comparing today's territories may want all historical revenue restated into today's map. A commission dispute may need the territory assignment that applied to the original sale. A data model should make that choice visible instead of making the latest dimension row answer both questions by accident.

Type 2 preserves versions, and Type 3 preserves a limited comparison

The familiar SCD choices concern how much attribute history a dimension retains. Type 1 overwrites. Type 2 adds a versioned dimension row, with the dates or keys needed to associate facts with the appropriate version. Type 3 adds space for a previous or alternate value, giving a limited comparison rather than an unrestricted sequence.

Ralph Kimball and Margy Ross describe these patterns in The Data Warehouse Toolkit. Microsoft's current dimensional-model guidance describes Type 1 overwrites and Type 2 versioning. The numbering is a vocabulary for design choices. Higher-numbered hybrids combine techniques for particular reporting needs; their existence doesn't make a simple choice obsolete.

A Type 2 customer dimension normally gives each version its own identifier. A fact can retain the identifier of the dimension version applicable at the fact's business-effective date. Another design can join a durable customer identifier to an effective period. Either way, the relationship between fact and dimension version must be explicit. Adding rows without changing a join that assumes one row per customer creates duplicated facts instead of historical accuracy.

The model also needs to define when a new version starts. A genuine customer move belongs at its effective date. A correction discovered later may need both that date and the date the correction entered the warehouse. Type 2 often supplies effective history, but it isn't automatically bitemporal. A correction that overwrites the effective-history record can still lose what the warehouse previously reported.

The commission distinction therefore survives inside the dimension. The business can need historical classification as currently corrected, historical classification as previously reported, and a restatement under current classification rules. Those are different query contracts over related data.

Backups preserve recoverable states under a different contract

A backup can be valuable evidence. An archive that retains the relevant state may recover a value lost from the live table. The limitation is whether it captures the necessary moments, remains available and can be inspected at the required granularity.

Restoring a database image to recover one attribute is an operational procedure. An analyst needs a history query they can repeat without restoring an entire system. A backup archive and a historical model can contain overlapping information while serving different jobs.

Retention is another boundary. A recovery policy may preserve the period needed after a failure and still omit the period needed for a business dispute. There's no universal retention window to assume. The actual policy determines what remains.

Scheduled snapshots can miss intermediate states. A value that changes and changes back between snapshots may leave no trace in either image. A recovery arrangement that retains intervening logs can preserve more than snapshots alone. The question is what that arrangement captured, not whether the organization uses the word backup.

An archive also needs an access path across time. Determining when an attribute changed can require comparing retained states. A versioned history can represent that change directly. Restoration may supply evidence for a one-off reconstruction, but it doesn't automatically become a reportable, indexed sequence of assertions.

Keep recovery and historical explanation as separate requirements. A successful restore proves the captured system state can be recovered. It doesn't prove that every prior report can be reproduced or that every decision input was captured.

Some attributes can still be overwritten

A misspelled display name can be a reasonable Type 1 correction. The question is whether the old spelling is material to a record the business needs to explain. A name used in a signed document, matching rule or disputed communication may have a history requirement even though the value was a mistake.

Follow the consumers to decide which attributes need history. Did a compensation rule, pricing tier, routing process, report or model read the attribute and act on it? A decision that consumed the attribute creates a reason to preserve that version. An attribute used only for a replaceable display may need less treatment, subject to the business's records and privacy obligations.

A blanket ban on updates would make ordinary systems harder to run. A blanket default to Type 1 can erase the explanation for a consequential decision. Attribute-level decisions give the team somewhere more useful to work between those extremes.

Record the required questions alongside the model. A customer segment might need an effective history because reports group by it. A source identifier might need versioning because an enrichment provider changes its classification. Another field can remain a current value. The table name doesn't decide the requirement for every column in it.

That selective approach keeps history tied to an identified reporting or decision requirement. Some domains need more than attribute history, though. They need to reconstruct the sequence of operations that produced the state, and the durable record changes shape again.

An event log can make state reproducible[summarize]

A fold derives the balance from the entries

A history table retains versions alongside a current record. Event sourcing makes the retained sequence of events authoritative, with current state computed from that sequence. Martin Fowler's account of event sourcing and reconstruction describes the distinction directly.

A balance is a simple way to understand it. Start with an initial state, apply each recorded deposit and withdrawal in order, and carry the result forward. The balance at the end is a computation over the entries. A fold is the operation that walks that sequence while carrying the accumulated state.

The computation might be simple enough for a running total. A richer domain can use rules that change order status, inventory reservations or account permissions. In every case, the model needs to define what each event means and how applying it changes state.

The sequence gives an investigation more than a final value. It can expose the operations that led to that value, the state at a retained position in the sequence and the first transition that violated a chosen invariant. An invariant is a condition the model is supposed to maintain, such as a constraint on available stock.

Those capabilities depend on what the log captured. Replaying an incomplete sequence won't recover an omitted event. Replaying with changed rules may compute a corrected state rather than the state the old software produced. A useful replay procedure has to distinguish reconstructing an old result from recalculating history with a new rule.

The ordering rule matters too. A timestamp alone may not settle the order of concurrent events. An event stream needs the sequencing or causal information its domain requires. Recording an event's effective time and its arrival time can distinguish a late delivery from an operation that occurred later in the business.

A correcting entry preserves the mistake and its correction

A correcting entry can preserve the earlier entry and record what reverses or adjusts it. A reader can then distinguish the original operation from the later response.

That comparison supports an append-and-correct discipline. It doesn't make every event-sourced application an accounting system, and double-entry bookkeeping and event sourcing aren't identical concepts. Double entry imposes its own accounting relationships. An event stream can record operations with no debit-and-credit interpretation at all.

Replacing an entry leaves only the revised value in that position. Retaining a correcting entry preserves both the earlier operation and the later adjustment. The ledger comparison applies to that preservation rule; an event stream still needs its own domain rules.

An append-only rule also needs enforcement. A record that application code normally appends can still be altered by an administrator or a faulty migration. Permissions, retained versions and audit controls determine what evidence survives. Storage format alone can't establish that nobody changed the log.

The cost of preserving a record includes more than bytes. Someone has to define correction, decide who can issue it and keep old events readable. Those responsibilities remain even when storage is cheap. A complete authoritative record and the appropriate computation can rebuild the balance.

Replay can locate the transition that broke inventory

Consider an illustrative inventory problem. A quantity shows negative three for a physical SKU. The number may violate the application's inventory rule, but the current quantity doesn't explain which operation produced it.

A retained sequence lets an investigator follow the running state. The first transition below the allowed quantity becomes an inspection point. Its event identifier, type, inputs and ordering can reveal whether the system accepted a duplicate, applied an operation in the wrong order or failed to enforce a reservation rule.

The explanation still needs the domain. A return might add stock, remove a prior allocation or trigger a separate inspection process. A duplicate return doesn't universally cause negative inventory. Event names and signs have to be interpreted against the actual reducer and stock definition. These are hypotheses to check against a recorded sequence, not a diagnosis of a particular warehouse.

A useful investigation can then ask whether the same faulty transition occurs elsewhere. The team has a reproducible condition over recorded events instead of a collection of isolated current-value repairs. The investigator can apply the same condition to other items in the retained history.

An inventory system may also record legitimate backorders or delayed reconciliation as negative availability. That makes the invariant a business definition to confirm, not a fact established by the minus sign. The software needs to say which quantity it represents before an engineer declares the state impossible.

Replay needs to isolate outside effects

An old event may have caused a payment, notification or external request. Reconstructing internal state shouldn't repeat that outside action. Fowler explicitly discusses the external-system problem during replay. A replay boundary needs to distinguish calculation from a new consequential operation.

External responses can also be inputs. An event that depended on a service's answer can't necessarily reproduce the old result by calling the service again. Its response may have changed or the service may no longer exist. Capturing the response used in the original operation can be part of the history requirement.

A log therefore buys reconstructability under stated conditions: the necessary events and inputs survive, their order is meaningful, their semantics remain available and replay doesn't repeat side effects. An event-sourced system still has to meet each condition. The benefits are substantial where those questions matter, and so are the obligations that follow from making the log authoritative.

Event sourcing adds contracts the team must keep[summarize]

Old events need an interpretation

A persisted event becomes an input to future software. Changing today's class definition doesn't change the meaning of an event already stored. The team must retain an interpretation for old versions or carry out a controlled transformation that preserves the required evidence.

Versioned event types and upcasters are one response. An upcaster translates an older representation into a form current code can consume. The translation itself becomes part of the reconstruction procedure. A replay that silently guesses a missing field can produce a plausible state with an unsupported input.

Old events must remain interpretable for as long as the retained history requires. A system can retire a version through an explicit migration or retention decision, but that operation has to account for the evidence it changes. Immutability is a contract the system enforces, not a physical inability to rewrite storage.

A schema version also doesn't capture every semantic change. The same field can acquire a different business meaning without changing its type. An amount might switch its tax treatment, or a status might stop implying an inventory reservation. Reproduction needs the relevant meaning and rule version as well as a parseable payload.

Snapshots reduce replay work and create a validation job

A complete rebuild applies the retained events from an initial state. As history grows, rebuilding everything can become an operational burden. A snapshot stores an intermediate state with the position in the log it represents, so replay can continue from that point.

A snapshot is a derived result. Its usefulness depends on the state calculation and stream position being correct. A checkpoint associated with the wrong event boundary can skip an operation or apply one twice. The procedure also needs to identify which rule version produced it.

A valid snapshot doesn't undermine the authority of the log. The snapshot accelerates a calculation the log still supports. The failure is allowing an unexplained snapshot and the log to become competing authorities. A documented rebuild or comparison procedure gives the team a way to settle a disagreement.

Snapshots also affect the questions the system can answer economically. A checkpoint suitable for current-state recovery may not contain the metadata needed to explain an older output. The historical contract still determines what must remain accessible underneath it.

A projection can lag behind a completed write

A read model often updates after the event is accepted. A user can complete an operation and still see the old result in a list or dashboard. The product needs to make that state understandable.

The team might show the operation as pending projection, use a version boundary when reading after a write or provide a view of accepted events alongside the derived display. The choice depends on the user action and its consequences. Support should have an observable status to inspect rather than an instruction to assume the data will catch up.

The same distinction applies between internal consumers. A report drawn from a projection needs to know the point through which that projection has processed the log. Comparing two projections without their processing positions can turn ordinary lag into an apparent data conflict.

Coarse events limit what the log can answer

An event called order_updated may record only that something changed. An event that captures an address correction can identify what changed and why, if those details are supplied and retained. The name alone contributes nothing without the payload and semantics behind it.

Event granularity is the shape decision inside event sourcing. A log that omits the distinction needed by a later question has the same information limit as a current-state row. Keeping every event is useful only at the resolution those events represent.

The right granularity follows the decisions and disputes the domain needs to reconstruct. Recording every technical write may add volume without explaining a business operation. Recording only a broad business label may hide the inputs needed for diagnosis. The team has to choose and document the useful boundary.

A simpler history can satisfy the actual requirement

A business needing historical customer segments may be served by a versioned dimension. A business needing to explain selected changes may be served by an audit history with the necessary context. Neither requirement automatically calls for event sourcing across the application.

Finance, compliance, dispute resolution and safety can create strong reconstruction requirements. They still don't mandate one architecture in every system. The question is whether an authoritative event sequence supplies information or operational guarantees that the simpler alternatives can't provide for the named job.

Team capability belongs in the decision. Someone must own schema evolution, projections, replay, recovery and the distinction between accepted events and derived state. A partial migration that leaves a mutable table and a log disagreeing with no declared authority can be harder to investigate than the original system.

Choose the smallest history that answers the consequential questions. That may mean declining event sourcing while retaining the lineage of a report. History explains how the system reached a state. The next requirement is to explain which sources and transformations support the state it reports.

A result needs a record of how its inputs were used[summarize]

A source list can leave the dependency ambiguous

An analyst can know which files fed a report and still be unable to explain a particular figure. The files identify possible inputs. The derivation identifies the records, transformations and rules that produced the output.

That difference becomes consequential when a source is corrected or withdrawn. A file-level link may say that a report used an extract somewhere. It may not say which figure used the affected record, whether another source independently supported the same result or whether the output needs recalculation.

The aim is to make those questions queryable from the record. A person who remembers the pipeline can help reconstruct it, but their memory should be recorded as a reconstruction with an origin. It shouldn't silently become the same kind of evidence as a retained execution record.

Database provenance distinguishes several questions that are easily flattened into one source list. Why-provenance asks which input records support an output. How-provenance preserves more of the way those records combine. Where-provenance asks where an output value came from, such as the source location associated with a copied cell.

These terms help define the lineage a system needs. A table-level dependency can support impact analysis at the table level. A question about a cell's derivation may need a finer record. Recording the coarse relationship doesn't imply that the fine one can be reconstructed later.

Joint inputs and alternative derivations behave differently

Todd J. Green, Grigoris Karvounarakis and Val Tannen's Provenance Semirings formalizes the distinction between inputs used together and alternative ways to derive the same result.

Suppose a result requires records A and B jointly. Removing B breaks that derivation. Suppose another result can be derived independently through A or through B. Removing B can leave the derivation through A intact. A plain list containing A and B doesn't express that distinction.

The semiring treatment represents alternative derivations with addition and joint use with multiplication. The notation is useful because it preserves how ordinary relational operations combine annotated inputs. A join combines dependencies differently from an operation that offers another route to the same output.

A retraction must follow those semantics. Flagging every descendant of a source can identify candidates for inspection. It doesn't mean every candidate has become false. Some results have another derivation, some need recalculation and others need a changed caveat. A system that immediately deletes every reachable output would discard the distinction the provenance was meant to preserve.

The compositional claim concerns annotated inputs and positive relational algebra under the stated semantics. It doesn't establish automatic correct lineage for arbitrary application code, a model-generated summary or a pipeline that drops the annotations. A system has to preserve the derivation through the operations it actually performs.

Provenance belongs in the production path. The transformation can record its input identities, rule version and output identity while all are available. A later catalog can organize those records. It can't infer an exact dependency merely because two files appeared in the same folder.

PROV separates what existed, what happened and who was responsible

The W3C's PROV data model gives provenance a vocabulary that can travel between systems. Its core distinctions are an entity, an activity and an agent. An entity is the thing whose fixed aspects are being described. An activity acts on or with entities over a period. An agent bears responsibility for an activity, an entity or another agent's activity.

A report version can be an entity. The calculation that generated it can be an activity. The responsible person, service or organization can be an agent. Recording only the input and output files leaves out the operation that connects them and the responsibility attached to that operation.

An execution record can therefore identify which input versions an activity used, which output version it generated and which agent was associated with the activity. The same vocabulary can distinguish a document revision from a fresh document that happens to share its name.

The standard is a model, not a required product. A relational store can represent those entities and relationships. Another system can serialize them differently. The useful decision is whether the record preserves the distinctions the question requires, with identifiers precise enough to join across components.

Responsibility needs care. A service account can identify the process that ran without identifying the human who approved a consequential action. An operator can approve a report without generating its inputs. PROV's distinctions help keep those roles separate instead of treating one created_by field as the answer to every responsibility question.

Record the activity version behind a job name

A job name that points to changing code is an incomplete description of a past transformation. The record needs enough version information to identify what the job meant at the time. Its configuration can matter as much as its source code.

The same requirement applies to inputs. A path that now contains a different extract doesn't identify the extract the report used. A retained version, immutable object identifier or content hash can distinguish the input, provided the bytes or other necessary evidence remain retrievable. A hash helps identify bytes; it doesn't recover missing bytes.

The producer should also distinguish an observed event from a claim about a prior event. An import can record that it received a source at a particular time. It may only have the source's assertion about when the underlying business event happened. Retaining both keeps provenance connected to the two-clock model rather than collapsing it back into one date.

An acquirer asking how a retention figure was calculated may need the cohort definition and exclusions. An engineer investigating a pricing decision may need the model and feature versions. A client checking a technical claim may need the primary passage and its qualification. Each request asks for a different part of the support structure.

A useful lineage design makes that structure available at the required level. It can turn a remembered explanation into an inspectable chain. It still has to distinguish evidence that an activity occurred from evidence that the activity produced a correct conclusion. That distinction governs how much trust the chain deserves.

Provenance makes support inspectable without declaring a claim true[summarize]

A correction needs evidence too

A technical label attached to a researcher can look invented. Removing the attribution without checking the researcher's writing can introduce an error while appearing to improve rigor. Accepting the attribution without checking can preserve an error for the opposite reason.

A correction needs the same source check as the statement it changes. A skeptical tone doesn't supply provenance.

A recorded source relationship lets another reader inspect the attribution. The record should point to the passage that supports the term, the version consulted and any qualification the original author supplied. A bare edge to an entire book may identify somewhere to search, but it doesn't establish that the book supports the claim.

The same caution applies to the derivation itself. A pipeline may have connected the wrong input, a model may have cited a source it didn't use, or a human may have recorded a mistaken explanation. Provenance can itself be questioned. W3C PROV includes a way to describe provenance of provenance, which is useful when the record's own origin affects its standing.

A chain-of-custody record can help establish an artifact's identity and integrity by recording its handling. Its evidentiary significance depends on the use and applicable rules. The data model preserves that record without deciding legal admissibility.

Different kinds of support need different annotations

A billing figure, a sales forecast and a model estimate can share a currency unit while representing different kinds of claim. The unit alone doesn't tell a consumer whether a number records a transaction, expresses a person's expectation or comes from a predictive calculation.

The model needs somewhere to carry that distinction. A useful annotation can identify the source, method, effective period, recording time and whether the output is observed, inferred or proposed. The vocabulary should match the domain instead of turning every difference into one generic confidence score.

An assessment should identify how much and what kind of support a claim has. An established result with a primary source, a synthesis attributed to its author and an unresolved research claim shouldn't become indistinguishable just because each can fit in a text field.

This doesn't require rating every sentence. It requires preserving the distinctions a consumer might act on. A report excluding modeled inputs needs those inputs marked. A review of superseded figures needs the source versions and supersession relationships. An investigation of a disputed forecast needs to distinguish what was estimated from what was observed later.

A source change creates an inspection task with a reason

A retracted source can affect a downstream claim without automatically settling it. The claim may retain independent support, may require a weaker formulation or may no longer have enough evidence to use. The lineage should identify the dependency and make that decision inspectable.

A practical result can therefore carry both an applicable value and a support status. A system may still need to display the figure used in a historical decision after the source has been discredited. Deleting the figure would erase part of the explanation. Retaining it without the retraction would mislead a new consumer.

The two-clock model helps here. The system can preserve when it relied on a source and when it learned of the retraction. The current assessment can change while the earlier reliance remains part of the record.

This record can explain why a claim was used, what evidence challenged it and how its standing changed. It also needs room for competing assertions to coexist. A table that accepts only one value per subject can lose the disagreement before any of this reasoning begins.

Competing assertions can coexist with a deterministic answer[summarize]

The same headcount field can contain different claims

Consider an illustrative account whose sales rep records 45 employees, whose enrichment provider returns 120 and whose signup form reports 60. A current customer row has room for one headcount. A process must choose a value before that row can be written.

Last-write-wins chooses by arrival order. A source-priority rule chooses by a predetermined ranking. Taking the first non-null field chooses by column order and availability. Each rule can be deliberate and useful. Each also makes a decision about how a consumer will interpret conflicting inputs.

The problem is losing the other assertions and the reason the winning value was selected. A downstream dashboard can present the chosen number without revealing that anyone disagreed. Another account where every source reported 120 can then look identical to the contested account.

Agreement and conflict are different evidence. Agreement still needs context: several sources can repeat the same upstream record, so their agreement may not be independent corroboration. Conflict also needs context: one source may count a subsidiary and another a whole group, or their observations may concern different dates.

Before marking the headcounts contradictory, compare subject, definition and effective time. A change from 45 to 120 may represent hiring between observations. A difference between employees and contractors may be a definition mismatch. Retaining those qualifiers lets the system distinguish genuine disagreement from values that answer different questions.

Store assertions before applying the consumer's policy

An assertion needs an identity. It says what was claimed about which subject, under what definition, by which source and for what period. The recording history says when the system received or adopted it. A replacement can supersede the earlier assertion without erasing that earlier record.

A consumer can then apply a resolution policy. An operational screen may choose the most recent approved estimate. A model may take several source values as separate features. A reconciliation process may refuse to choose and request a source check.

A department might choose a conservative or optimistic estimate for a particular task. Those policies need a reason tied to the decision. Finance may need a legally defined count. Sales may need a verified count because pricing depends on it. The record should make the policy explicit and appropriate to the decision instead of baking an assumption about a department into the data.

A deterministic result is still possible. The query or projection can name the policy version and return its chosen value with the assertion identifier behind it. Two consumers can choose differently without maintaining two disconnected accounts of what the sources said.

Query-time resolution doesn't require recalculating every policy on every page load. A materialized projection can apply the policy and retain its provenance. The separation is logical: preserve the assertions, make the resolution rule identifiable and let the read path serve the result its caller needs.

Resolution also has operational consequences. A payment or access decision may require one authoritative result at the moment of action. Retaining disagreement doesn't permit the application to leave that action ambiguous. The system should preserve the choice it made and the inputs it had when it made it.

Disagreement can be an investigation signal

A conflict can prompt an investigation of hiring, acquisitions, rebrands or data-entry defects. Each is a hypothesis to check. A spread among sources doesn't prove any one of them, and treating disagreement as a predictive feature needs validation against the actual task.

The duration of an assertion can add context. A correction shortly after entry may be a typo or an early discovery of a serious error. A value that stood for years may describe a stable reality or a mistake nobody noticed. Time distinguishes those histories; it doesn't settle their causes.

That distinction matters for modeling. A system can compute time-to-correction and compare it with other evidence. It should avoid turning the measured interval into a causal label without support. Keeping time and source together makes the question testable instead of building a conclusion into a heuristic.

Support and attack edges need a defined computation

Addressable assertions can refer to one another. A claim can support another claim, dispute it or supersede an earlier version. Those relationships allow a system to compute an assessment from its evidence rather than ask a person to maintain a floating score by hand.

A confidence-propagation design can pass priors along support and attack edges. Convergence depends on the computation it defines. A computation needs a defined rule, treatment of cycles, update behavior and a stopping condition. Some rules can oscillate or amplify circular support instead of converging to a useful assessment.

Independent evidence also has to remain independent in the calculation. A source copied by several documents shouldn't receive several votes merely because it has several descendants. Provenance is what makes that duplication visible. A graph without it can turn repetition into apparent corroboration.

A computed score should state what it measures and how it was obtained. It might summarize support under a policy. That doesn't automatically make it a calibrated probability that the claim is true. The distinction between a useful ranking and a validated probability belongs next to the number.

Use one value where the field has one authority

An assigned invoice identifier can have one authoritative value in its issuing system. A completed payment can have an authoritative processor record, while disputes about the surrounding business meaning remain separate assertions. Adding competing claims to every column would make the ordinary read path needlessly difficult.

The selection test is whether multiple sources can legitimately supply different answers and whether retaining their disagreement helps a consequential decision. Firmographics, enrichment, attribution and model outputs often make that question worth asking. A deterministic identifier usually has a different contract.

Time distinguishes when an assertion applied and when it was recorded. Provenance identifies what supports it. Competing assertions preserve what challenges it. Keeping all three allows a system to change its current assessment while retaining the explanation for an earlier one.

My medallion scheme derives an artifact's tier from its record[summarize]

The tiers describe different relationships to evidence

A source document, a processed draft and an article someone has used are different kinds of record. My medallion scheme distinguishes them through lineage and tracked events. The tiers are bronze, silver, gold, platinum and diamond.

These definitions belong to my design. Other data platforms can use the tier names differently. The useful part is the observable condition behind each label. A person shouldn't have to decide that something looks valuable before the system can classify it.

Bronze is source material retained as ingested, with its origin recorded. A document received from someone is a bronze artifact. The record of its arrival identifies the version being preserved. Later processing produces another artifact rather than replacing the original.

Here, immutability is a preservation rule. A corrected source can arrive as a new version related to the earlier one. The old version remains the record of what was received, subject to the applicable retention and protection rules. Calling a file bronze doesn't suspend those obligations or make the source's contents true.

Silver means processing has occurred. A transcript generated from a recording is silver. Chunks, embeddings, enrichments, research notes, templates, drafts and evaluations can also be silver. Each needs a relationship to the inputs and activity that produced it.

The distinction prevents effort from becoming evidence of use. A carefully edited draft may be excellent and still have no recorded event showing that anyone used it. The design preserves its derivation without treating the amount of processing as a result in the world.

Gold needs tracked events against the artifact

Gold is defined by association with events the system tracks. A reader accesses an article. An agent retrieves from a knowledge base. A prompt runs in a tracked pipeline. A maintainer reacts to a pull request. The relevant occurrence is captured against the identifiable artifact.

Publishing is one way to make such events possible. It isn't the entire definition. An internal tool can produce meaningful tracked use without a public audience. An uploaded public page can remain unobserved if the system captures no use against it.

A retrieval proves a retrieval occurred under the record's capture conditions. A run proves a run occurred. Neither establishes that the retrieved material was correct or that the run helped anyone. Success, failure and usefulness require their own definitions and observations.

The event-backed condition gives the system something firmer than a human-maintained quality badge. It can answer whether the relevant event exists and which version it concerns. The assessment of what that event means still belongs to the task. A page hit and a completed purchase shouldn't become interchangeable evidence because both are events.

The same artifact can therefore be gold under the use definition and poor under a quality assessment. Keeping those questions separate is what allows the model to learn from an unsuccessful artifact. Deleting or relabeling failures to keep the tiers looking impressive would remove evidence the system needs.

Recorded derivations connect the interview to the article

Follow an illustrative interview through the production chain. The ingested recording is bronze and remains the captured source. A generated transcript is silver. Chunks and embeddings derived from that transcript are also silver, as is a research brief assembled from them. A draft and its revisions continue the derived chain.

Each processing step needs to identify the input version it used. A draft based on an earlier transcript needs to point to that transcript, even if a later correction exists. Retaining only the newest transcript would make the draft's actual inputs harder to explain.

The published article begins to receive tracked reader or retrieval events. Those events make it gold under this scheme. The identical text retained only as an unused draft remains silver in the illustrative case. An internal draft used by a tracked process can meet the gold condition too; the definition follows events rather than the folder name.

Now suppose the interview subject retracts a statement. A complete chain identifies the transcript passage, chunks, brief, draft and article that may depend on it. The retraction can generate a review task for those descendants. Their handling still depends on what each one used and whether it has other support.

The original recording remains evidence that the subject made the statement. It doesn't remain evidence that the statement is currently endorsed. Time, attribution and supersession let the system preserve both facts without rewriting what the recording contains.

A derived tier still depends on complete capture

The tier is computed from lineage and events rather than assigned as an editable badge. A system can materialize that answer for reads, provided it can explain the derivation and recompute it when its inputs change.

A missing event or broken lineage edge can still produce an incomplete classification. Derivation prevents one kind of drift, the manually maintained label that no longer reflects its rules. It doesn't remove the need to validate the records those rules read.

The definition also has to identify the artifact's boundary. A page may contain several versions of a document over time. If events attach only to the URL, the system may be unable to distinguish which version performed. Version identity is part of the analytics contract as well as the preservation contract.

Platinum and diamond remain named future states

Platinum composes specialized material from multiple gold artifacts, with the intent of drawing on measured performance. A playbook assembled from pieces with observed outcomes has a different basis from one assembled from drafts someone liked.

The qualifying outcomes have to be specified. A high retrieval count may identify useful reference material, or material an agent repeatedly retrieves because it never resolves the question. A composition needs the evidence appropriate to its purpose before the label can mean measured success.

Diamond is the design's mission-critical aspiration. Its language refers to satisfying an entity's own opposed requirements, sometimes expressed as a trade-off triangle or trilemma. That's an ambition to define and test against concrete failure conditions. It isn't evidence that a current artifact can't break.

Those upper tiers require mature outcome records and explicit operating criteria. The documented scheme labels them as future state. Preserving that boundary keeps a proposed capability from acquiring the authority of something already demonstrated.

The lower tiers already define a usable preservation chain. Retain the original, identify each derivation and record real events against the resulting version. That chain supports a further question: when a result is used, which earlier sources and processing steps contributed to it?

Lineage can carry credit back to the sources[summarize]

A downstream event can identify its upstream dependencies

A reader event belongs to the article version the reader received. The article's lineage can identify the draft, research, transformations and source records that fed it. That gives an attribution system a route for associating the event with earlier contributions.

The direction is the reverse of production. Sources feed derived work. Observed use of that work can be related back to the sources. The chain makes that relationship inspectable instead of asking someone to remember which research was valuable.

The association needs an allocation rule. An artifact may have several inputs, some essential and some incidental. A source list doesn't by itself say how much credit each one deserves. A derivation can supply more detail, but the system still has to define what it measures.

A source-utility ranking might count how often retained sources fed work with the chosen outcome. A template comparison might group outputs by the template version that produced them. An enrichment assessment might compare downstream results for artifacts that received a processing step. These are different questions with different denominators.

A prompt that ran often has demonstrated use. Its win rate needs a definition of a win and a record of eligible runs, including failures. An enrichment step associated with high-performing outputs has a relationship worth examining. Its return on investment needs cost records and a defensible account of benefit.

The lineage enables those analyses by preserving the join. It doesn't choose the outcome metric or solve the causal question for the analyst.

Attributable use is different from causal improvement

The distinction is familiar from advertising. A customer may encounter several contacts before a purchase. The last visible ad is part of that record, but its position doesn't establish that it caused the purchase.

A knowledge-production chain has the same problem. A research source may appear in a successful article because it helped. It may also appear because a strong team chose it, because the subject was already popular or because every article in that category cites it. The event and lineage establish recorded participation. A causal claim needs an appropriate comparison and the assumptions that make that comparison informative.

A source-ranking dashboard can turn a count of appearances into an apparent measure of contribution. The numbers may be computed correctly while the label claims something the calculation never established.

The retraction example shows why the full derivation matters. A source that was one of several independent supports has a different role from an input required jointly with every other input. Credit allocation can need those distinctions too. A simple reachability walk knows which artifacts are connected, but it doesn't know the counterfactual value of removing each one.

Returning outputs to the corpus needs to preserve their origin

A completed article can become input to later work. Its usefulness grows when the next process can navigate from the article to the sources and reasoning behind it. The corpus then contains both the synthesis and the record that lets another reader inspect it.

Re-ingestion also creates a risk. The article may repeat a source claim, and a later draft may retrieve both the article and that source. Counting them as independent support would manufacture corroboration. The derivation record should identify that shared origin.

A correction should travel through the same relationships. A source change can flag the article and later work that reused it. A new article can add a useful connection between topics without erasing the distinction between original evidence and its descendants.

This is the concrete difference between adding documents and adding reusable structure. More text increases the available material. Retained relationships can also improve the questions a later process asks about that material. The improvement depends on accurate capture and useful retrieval; it doesn't happen simply because the store is called a graph.

The medallion scheme gives the analytics somewhere to start: preserved sources, identifiable derivations and tracked events. A database can only propagate credit through the links it retained. The next design question is how to represent those links so they can carry time, support and disagreement together.

An addressable relationship can carry time, source and disagreement[summarize]

A customer-segment assertion needs its own identity

A customer is an entity. The assertion that the customer belongs to a segment is a relationship with a period, a source and possibly a competing assertion. Attaching one validity window to the customer would miss the fact that the same customer can have many relationships with different histories.

A derivation also connects identifiable things. A report version uses an extract version through a particular calculation. A disagreement connects assertions about a subject. Giving these relationships identities lets another record refer to them precisely.

Time, provenance and contradiction need to refer to a particular assertion or derivation rather than a floating value. A current cell holding a segment name doesn't supply that identity by itself.

Relational storage can represent the same requirements

A relational design can promote the relationship into a row of its own. An assertion table can identify subject, predicate, value and source, with effective and recording periods. Other tables can identify derivations or disagreements by referring to assertion identifiers.

A foreign key isn't inherently unable to participate in that structure. The limitation is a design that stores only the pointer and omits the relationship record. Choosing to model the association explicitly is how a relational system gives it somewhere to carry annotations.

A graph-oriented design can make navigation among those relationships more direct for some workloads. Its advantage depends on the queries, storage implementation and indexing. A scaling claim needs evidence from the actual workload. A data shape describes what's represented; it doesn't supply a benchmark.

Use the representation the system can operate reliably while preserving the distinctions the application needs. A graph database that omits source versions has lost provenance. A relational database that retains addressable assertions can preserve it. The product name settles neither question.

Metagraph terminology needs its definition alongside it

A graph models nodes and edges. A hypergraph can use a relationship involving more than two participants. I use metagraph here for a richer structure in which relationships are addressable and can themselves participate in relationships.

That vocabulary has several technical lineages. A sets-to-sets definition in discrete mathematics and the addressable-link approach associated with OpenCog's AtomSpace are distinct technical uses. These shouldn't be treated as interchangeable formal definitions simply because the same word appears near them.

Here, the implementation requirement is specific enough to state without settling the word for every field: identify the assertion or relationship, attach its annotations and let other assertions refer to it. Confidence in an assertion, provenance of a derivation and a challenge to a source assessment then have identifiable objects to describe.

Reification is one way to do that, by representing a relationship as an entity with its own links. Another implementation may store annotations as properties or values over existing objects. The choice changes query complexity and storage behavior. It doesn't eliminate the need to define what the annotations mean.

A confidence number attached to a relationship is still a computed or assigned value under a method. A relationship pointing to that assessment can record its origin. More levels of annotation don't make the initial evidence stronger; they make the assessment process more inspectable.

The corpus can be navigated in several directions

A structured corpus can support navigation through levels of detail, cross-connections, time and evidential support. Each direction needs corresponding relationships in the record.

A reader can move between levels of detail, from a domain summary to the paragraph or record supporting it. That requires relationships between the summary and its material. A heading hierarchy alone supplies organization, but not necessarily evidential support.

A reader can follow cross-connections, such as a shared source or a relationship between concepts discussed in separate documents. The usefulness of that navigation depends on what the links denote. A merely similar passage is different from an independent source supporting the same claim.

A reader can move through time, asking what a system asserted at a retained point and what it asserts after correction. That depends on the temporal record already described. It's especially useful when a retrieved passage has been superseded but remains relevant to an old decision.

A reader can inspect support, distinguishing an observed value from a forecast or an unresolved claim. A retrieval system can use that distinction to select material appropriate to the task. It should show enough of the method for the resulting confidence treatment to be understood.

These directions describe useful access paths. They don't establish that a model traversing them reasons correctly. A more structured corpus can make evidence available; the application still needs to use it faithfully and expose uncertainty when the record doesn't settle the question.

Separate available components from the proposed assembly

Temporal tables, retained event sequences, provenance vocabularies and addressable relationships are established engineering ingredients. Their availability doesn't mean a particular system already combines them into complete time-aware lineage, contradiction handling and outcome attribution.

The assembly has to capture each activity, preserve relevant versions, connect assertions and make its resolution policies visible. Confidence propagation needs a defined computation. Lineage-derived tiers need the corresponding events. Reverse credit needs an allocation method and outcome definitions.

Platinum and diamond remain future ambitions in the documented medallion design. A current use of the word metagraph doesn't prove those ambitions have been reached.

A team can now state the requirement without buying a new platform: retain the particular history and relationships that its decisions require. The adoption question is which missing records prevent a real answer today, and what the smallest change would preserve from this point onward.

Start with one report the business needs to explain[summarize]

A reproduction attempt can expose a missing input

Choose a retained report whose historical meaning matters. Keep the distributed version alongside the query, parameters and any available input versions. Re-run the report under the intended historical interpretation, then compare the outputs at the level the business relies on.

A report from last quarter and another from an earlier period can expose different retention boundaries. The useful choice depends on the actual reporting obligation. The exercise should establish which histories survive, not satisfy an arbitrary age requirement.

A match is evidence about the sampled report. It may mean the relevant values didn't change, that history was preserved or that changes canceled in the aggregate. It doesn't establish that every input to every report has a usable history. Inspect the relevant detail before declaring the broader problem solved.

An explained difference can be healthy. A documented restatement, changed classification policy or intentional correction may produce a new answer. The record should identify the difference and preserve what the earlier report represented.

An unexplained difference identifies an investigation. Trace the output to its inputs and rules. Look for a dimension joined at current state, a replaced source extract, a changed calculation or a manual adjustment whose origin wasn't retained. The missing record determines what can be recovered and what must remain unknown.

The test should also distinguish an inaccessible record from one that never existed. A retained backup or archived document may support a bounded reconstruction. That reconstruction should identify its inputs and gaps. A team shouldn't turn missing access into a claim of permanent impossibility, or turn an inferred history into a captured one.

Follow the attributes that decisions consume

List the reports and automated decisions that need historical explanation. Commission, pricing, routing, board reporting and contractual reporting are useful places to look when they exist in the business. For each one, identify the attributes and rule versions it consumes.

That inventory gives history a scope. Some attributes may need only an effective sequence. Others may also need recording time because corrections arrive later. A source identity or calculation version can be just as consequential as the visible numeric input.

The number of columns is whatever the dependency trace finds. Scope the change to the attributes needed by the selected questions, then expand it when another requirement is demonstrated.

A practical implementation order follows the dependencies already exposed.

1. Define what the historical question means, including effective date, knowledge date and the output to reproduce.
2. Preserve the required attribute versions and connect facts to the intended versions. Keep a deliberate current-state view for ordinary consumers.
3. Record the relevant source versions and transformation activities, including responsibility and the rule or configuration version.
4. Preserve competing assertions where a consumer needs to understand disagreement, with a named policy for the operational answer.

A team can introduce these records selectively. Native temporal support may reduce the work, but activation still needs a write-path and query review. A trigger-based design needs atomic history capture, writer coverage and deletion handling. Neither should be described as free merely because its first statement is short.

State the boundary of the history being created

A new history starts at an observable capture point. Existing values can enter as an initial observation with that limitation recorded. Earlier periods should be populated only from evidence that supports them.

The same rule applies to provenance. A new activity record can identify which input and output versions it observed. It shouldn't claim complete upstream lineage if an earlier transformation left no record. An incomplete chain with a marked boundary is more useful than a complete-looking chain assembled from guesses.

Retention and access also belong in the contract. Historical records can contain sensitive information, and preserving every value indefinitely may be inappropriate. The business needs to decide what must be retained, who can inspect it and how lawful deletion or correction affects dependent records. Append-only engineering has to operate inside those obligations.

A successful first change leaves an inspectable result: the team can explain the selected report or name the precise missing input, and future changes to the relevant attributes enter a defined history. That's a concrete outcome against which to judge the work. A new database or architecture label isn't an outcome by itself.

Decline the larger architecture when the questions don't justify it

A small system with one authoritative source and no meaningful historical or dispute requirement may be well served by current-state tables. A display correction can remain a Type 1 update where its prior value isn't required elsewhere. A limited version history can satisfy a report without an event-sourced application.

The disqualifier is the absence of a named question that the richer record needs to answer. Multiple sources, outside audit or contractual demands, and explainable automated decisions are reasons to inspect the requirement. They're signals to investigate, not an exhaustive list of every reason a history may matter.

I wouldn't choose event sourcing because it sounds like a more serious architecture. It introduces event contracts, projections, replay and recovery responsibilities. The team should be able to name the question or guarantee that pays for those obligations before taking them on.

A larger system can still start with one bounded dependency chain. A selected report, the attributes it consumes, the transformations behind it and the versions that were available give the team a coherent unit to preserve. Generalizing that unit can follow demonstrated need rather than a platform rewrite.

The corrected amount and the earlier payment can both be explained

Return to the illustrative deal. A current statement can say that its effective amount was $72,000 from March 3. A historical explanation can say that the March commission process used the $80,000 assertion available at the time. A source record can explain the correction, and a retained calculation record can explain how the payment followed from its inputs.

Those statements can coexist because each identifies what it claims and which history it uses. A correction improves the current account without removing the evidence needed to explain the earlier action.

A useful fact record therefore includes more than its latest value when the decision requires more. It preserves when the assertion applied, where it came from and what later challenged it. Start with the decision that needs an explanation, then keep the information that explanation depends on.