The Shape of Data: Questions Lost at Capture. The shape you give data decides which questions you can ever ask of it.

The Shape of Data, Part I: The Ceiling

A checkout can ship every order correctly and still leave the business unable to connect a sale to the ad click that preceded it. The order exists. The click exists. The link between them may never have been recorded. A better dashboard can make both records easier to read without supplying that missing link.

That's the ceiling I mean. The distinctions a system preserves determine which questions its records can settle. A missing distinction can remain invisible through years of successful transactions, clean backups and accurate reports, because none of those jobs requires the question somebody eventually asks.

A successful checkout can leave a marketing question unanswered[summarize]

Consider an illustrative business with an orders table and a separate advertising system. Each order carries an identifier, customer email, total, timestamp and shipping status. Each click carries a campaign identifier and information about the visit. Checkout doesn't carry the click or session identifier into the order.

The shipping team has what it needs. It can send the right parcel, check whether payment cleared and handle a return. The marketing team asks which campaigns were associated with the orders that produced revenue, and the available records don't contain a reliable join between the two systems.

An email address might help. A timestamp might narrow the candidates. Neither becomes an observed click-to-order relationship merely because an analyst writes a convincing matching rule. The analyst has introduced an inference, and the quality of that inference needs to remain visible wherever the resulting number is used.

The reports can disagree before anyone makes an arithmetic mistake. An advertising report might count purchases inside an attribution window under the platform's rules. A finance report might count settled payments, net of refunds, over a different period. Those are different definitions and different populations. Reconciling the definitions is necessary before diagnosing a missing identifier, and even a perfect reconciliation won't create a relationship the systems never captured.

There's another distinction hiding inside the marketing question. A recorded path from click to purchase shows that the events were connected. It doesn't establish that the ad caused a purchase that otherwise wouldn't have happened. Someone may click an ad while already intending to buy. The causal question needs a defensible comparison, often designed before the campaign runs. An identifier solves a linkage problem. It can't, by itself, solve incrementality.

A repair can be useful without answering every version of the question. Carrying a session identifier through checkout may make an observed-path report possible. It gives the business something concrete to inspect, and it lets an analyst distinguish a recorded association from a modeled one. Claiming that it proves which campaign created the money would ask the field to carry more than it contains.

The missing distinction belongs to the data model

The checkout engineer may have designed a perfectly reasonable system for fulfilling orders. Campaign identity wasn't part of that job. The later reporting requirement exposes a gap between two jobs, which is why blaming the engineer or replacing the analyst misses the mechanism.

A slow query usually produces a visible symptom. A lost distinction can produce ordinary-looking output. The table validates, the job completes, the dashboard updates, and nothing in those success signals says whether the business could answer a question about a relationship that isn't represented.

Data cleaning helps when addresses are malformed, currencies are mixed or an order appears twice. Those are genuine problems. Cleaning the surviving values can't restore an absent event or tell which of several possible histories produced the same rows. A team can finish an excellent cleanup and remain unable to make the decision that commissioned it.

What would have had to be observed for this answer to be justified? That question belongs beside the business decision, before a conversation about vendors, warehouses or model quality.

Shape describes what a record means[summarize]

A logical model defines what one record stands for, which things have their own identity and which relationships can be represented. One row might mean an order, an order line or a customer's activity over a day. A relationship might say that a person approved an invoice, or it might also preserve whose authority they used and which policy applied.

Shape includes those choices. It includes the keys captured with an event and the distinctions retained when data moves between systems. It also includes whether a fact is an observation, a later inference or a claim that someone may dispute.

Storage is a different set of choices. An index helps a database find rows. A partition scheme organizes work. An engine determines how queries execute. Those choices can change the cost and practicality of an answer without changing what the rows mean.

Storage independence preserves the questions you already have

Codd's physical data-independence principle is useful here: applications should be insulated from changes in physical storage. An index can change without requiring the business to redefine an order. That separation is a major benefit of database systems, and it explains why a storage improvement can be substantial while leaving the logical model intact.

The distinction isn't a claim that every migration is easy or every storage choice is reversible. Moving data can involve outages, lost metadata, changing semantics and costs that make reversal impractical. The narrower point is about information. A faithful physical change preserves the distinctions it receives. It has no reason to invent missing ones.

A logical model can also change successfully. New events can be captured, relationships can gain their own records, and an old summary can coexist with finer detail. The historical limit remains: a new field doesn't establish what its value would have been before anyone recorded it.

That separates two promises often bundled into a modernization project. One promise is that the business can run its existing questions more reliably or affordably. The other is that it can answer questions it previously couldn't. The first can be delivered through storage engineering. The second requires identifying what new evidence or representation makes an answer possible.

A faithful migration preserves missing information too[summarize]

Moving an unchanged logical model from one engine to another can improve concurrency, reliability and query cost. The same move preserves a missing campaign identifier, an over-coarse sales total and an approval record with no policy attached. Migration fidelity protects existing information. It doesn't repair information that wasn't there.

An illustrative warehouse move makes the distinction concrete. The source has daily sales totals by location. The target receives the same dates, locations and totals, and the team reconciles the transfer. The reconciliation supports the claim that the migration preserved those values. It says nothing about whether either system can answer which trading hours should be staffed.

A team should judge the move against its actual purpose. Better reliability can be worth paying for. So can reducing the cost of a query that the business already knows how to express. Those benefits don't need a second promise that the new engine will reveal historical detail absent from the source.

Ask which distinction changes

A practical architecture-review question is: after this change, which business question gains or loses supporting evidence? A proposal that only changes how existing records are accessed belongs mainly to the storage discussion. A proposal that captures a previously absent event or preserves a relationship belongs to the logical-model discussion as well.

Some changes do both. A platform with a different data model might make a relationship easier to represent, and a migration might deliberately introduce that representation. The distinction is a way to examine the work, not a rule that every project must fit one box. Ask the team to identify the new record and show how it supports the question.

The checkout repair from the opening is an example. Carrying an observed session identifier into an order creates evidence for a join. The complete change may require work across a browser, a payment provider, consent handling and multiple checkout paths. Calling it a column doesn't make the end-to-end capture problem disappear.

A successful repair also needs a meaning for missing values. Some purchases may have no identifiable session or may come through a channel where collecting one is inappropriate. An explicit unknown preserves that limit. Filling the field with a guessed identifier just to make a completeness report look better corrupts the distinction the repair was supposed to establish.

Cleanup can remove a future question

Changes that remove information deserve the same question review as changes that add it. A field with no current query traffic might support a dispute, an audit or a future analysis. Traffic records which questions somebody has already implemented. It doesn't establish which distinctions the business can safely discard.

Several ordinary maintenance operations can change answerability:

• A rollup replaces detailed events, and the retention policy deletes the source rows.
• A customer merge rewrites identifiers without preserving the evidence behind the match.
• A table consolidation erases a distinction between roles that currently happen to coincide.
• A field removal deletes the only surviving link between two events.

Each operation can also be done safely in a suitable design. A rollup can retain its inputs. A consolidation can preserve role information. A merge can retain the original identities and represent the proposed equivalence separately. The review has to examine what survives, rather than treating the operation's name as a verdict.

Consider an illustrative customer merge based on a shared email address. The records might represent duplicate accounts for one person. They might represent two household members, a business address used by several buyers or separate accounts whose history needs to remain distinct. A matching rule makes a judgment about identity. That judgment should have an author, evidence and a way to be revised.

Rewriting every order to point at the surviving customer makes the judgment convenient to consume. Deleting the original identifiers makes it difficult to inspect. A later analyst can no longer tell whether the customer was observed as one person or declared to be one by a matching job.

Keep the reason for a deletion inspectable

Preservation has costs and limits. Personal data should have a lawful purpose and an appropriate retention period. Security, contractual duties and deletion rights can require removal. An unused field isn't entitled to survive indefinitely simply because an unknown future question might need it.

The decision should name what's being given up. A retention policy can say that hourly staffing questions remain answerable for a defined period, while older data supports only daily reporting. A privacy decision can deliberately restrict linkage. Those are defensible constraints when the business understands them, and they should remain visible to anyone interpreting the reports.

The harmful case is accidental loss presented as a purely physical cleanup. Nobody decided that a question should become unanswerable, yet the supporting evidence disappeared. A deletion review that includes grain, identity, relationships and history gives the business a chance to make that decision deliberately.

The next step is diagnosing which of those distinctions a question needs. A missing total, a missing key and a missing relationship call for different repairs, even when all three first appear as an empty dashboard cell.

A total can't reveal the detail it replaced[summarize]

The grain of a record is what one record means. Kimball and Ross make declaring that grain a starting point in dimensional design, and the reason is practical: a table's name doesn't tell an analyst which distinctions survive inside it. A table called sales could contain order lines, orders, daily customer totals or monthly channel totals.

Each grain supports different questions. An order total can show the value of a purchase. It can't reveal which product contributed the margin if the line items are gone. A monthly channel total can show reported performance over a month. It can't show which day an interruption occurred unless some finer record survives elsewhere.

A thumbnail is a useful comparison. A full photograph can produce a thumbnail, but a thumbnail doesn't uniquely determine the photograph that produced it. An image model can create a plausible enlargement. The invented detail remains different from recovered detail. A sales rollup has the same problem when several different event histories could produce the same total.

That's the first way a question loses its answer: the surviving grain is too coarse.

Match the grain to the action

Consider an illustrative restaurant staffing decision. The owner wants to know whether particular opening hours cover the costs associated with keeping a location open. Daily sales totals are relevant, but they don't isolate the hours being considered. Hourly receipts, labor assignments and the treatment of shared costs matter to the comparison.

Even hourly revenue alone won't settle it. A busy hour may require preparation earlier in the day. Closing an apparently weak period may move purchases into the neighboring period or lose customers who came for convenience. The decision needs a model of the operating change as well as records at the appropriate resolution.

The data question can still be stated precisely. Does any surviving source identify when sales occurred and which labor or other costs belong to the candidate periods? That's a narrower investigation than asking a BI contractor to improve restaurant reporting. It tells the contractor which distinctions the analysis needs.

A daily total permits several incompatible hourly histories. Sales might be concentrated in one part of the day or spread across it. Both patterns can produce the same total while supporting different staffing choices. A query can't choose between them using only the total, because the distinguishing information isn't in its input.

An estimate can use other evidence. Similar locations, recent periods with better capture or a temporary observation exercise may support a decision. The result needs to say where the pattern came from and how far it can be transferred. Applying a pattern from one location to another doesn't convert the older daily totals into observed hourly records.

Search below the report before declaring the detail lost

A coarse report doesn't prove that the whole system is coarse. The detailed events may still exist in the application database, a staging layer, a vendor export, a retained log or a backup that predates the rollup. The report may hide detail that the producer still holds.

The investigation should find the finest relevant record that actually survives. It should also establish the period covered. Recent hourly data and older daily totals create different limits for different cohorts. Writing one global statement that the business either has or lacks hourly data would erase the boundary the investigation found.

Recoverability has practical constraints too. A file can exist but be unreadable, inaccessible under the available permissions or too incomplete to support the decision. Those conditions belong in the finding. They're different from proving that nobody ever recorded the event, and they may have different remedies.

The distinction matters when a vendor export window closes. The useful response is to preserve an authorized copy of the needed detail while it's still available, subject to the retention and privacy obligations that apply. Buying a warehouse with more capacity after the source has discarded the detail won't reopen that window.

Retaining the inputs keeps a rollup open to inspection

Aggregation is often the right design. Detailed rows can be expensive to scan, and routine reporting may need only a summary. Keeping a summary and keeping its inputs are separate decisions. The first buys convenience. The second determines whether someone can later inspect the detail or change the grouping.

A business doesn't have to retain every event forever. It does need to connect retention to the questions it promises to answer. The policy should identify the point at which a detailed question becomes an estimate or becomes unavailable. Reports can then state that boundary instead of quietly extending recent evidence across the whole history.

Grain explains why a complete-looking total may be insufficient. The next failure is less visible: both detailed records survive, but the relationship between them doesn't.

Detailed records still need identity and relationship roles[summarize]

A click and an order can each survive at full resolution while their connection remains uncertain. The same problem appears when acquired businesses have incompatible customer identifiers, or when a support desk identifies someone by an email address different from the one on the buying account. More detailed rows don't resolve identity on their own.

An identifier captured by the producer can record an intended relationship. A later matching process proposes one using surviving attributes. Both can be useful, and both can contain errors, but they make different claims about how the connection became known.

A matched identity should retain its uncertainty

Consider an illustrative acquisition. Each business has customer records, and the combined organization wants to know how many customers bought from both. Shared names and addresses provide evidence. They also create ambiguous cases: common names, changed addresses, household accounts and companies with several contacts.

The matching process needs rules for links, non-links and cases that remain unresolved. Fellegi and Sunter's record-linkage framework formalizes this kind of decision. Its usefulness doesn't remove the need to know whether a particular proposed match is trustworthy enough for its intended use.

A support analysis has a similar dependency. A customer writes from a personal address, while the account belongs to an employer. Counting the ticket against the right account requires some evidence of that relationship. A text match may propose a connection, and an authenticated support session may provide a different kind of evidence. Treating both as identical foreign keys hides a consequential difference.

The risk grows when a matching result is written into a warehouse column and consumed without its method. A dashboard joins on the new identifier. A churn model uses the joined ticket count. A manager sees a relationship between support load and cancellation. The original uncertainty may affect every step, even though only the matching job still mentions it.

A useful record keeps the proposed link, the method that produced it, the relevant evidence and its review status. Downstream reports can then distinguish confirmed links from inferred ones, show coverage and test whether conclusions change under another defensible matching rule.

A deterministic identifier also needs scrutiny. A shared field can be reused, truncated, copied incorrectly or attached to the wrong event. The question is how the producer establishes identity and preserves it through the workflow. The presence of _id in a column name isn't evidence that the join means what an analyst assumes.

This is the second failure mechanism: the records don't preserve a dependable connection at the point the decision needs one. Matching may reduce the uncertainty, and a surviving source may resolve it completely. A team should investigate those possibilities before declaring the past permanently unknowable.

One field can conceal several different relationships

The third failure concerns the relationship's structure. Every named participant may have a record, yet the schema may have no way to state how those participants took part in the same event.

An illustrative employee table has one manager_id. That field can represent one defined managerial role. A person with a functional manager and a project lead needs two relationships with distinct meanings. Choosing one for the column doesn't make the other role disappear from the organization. It only removes the role from that representation.

A second generic manager column might preserve more values while still failing to explain them. Does the first field mean payroll approval, professional development or project allocation? Which period does each relationship cover? A useful repair starts by naming the roles and their validity, rather than adding enough blank slots to accommodate today's org chart.

An approval event makes the same point more sharply. Consider this illustrative sentence: Alice approved an invoice on behalf of Bob under an exception policy during a reporting freeze. The decision may depend on who acted, whose authority they used, what they approved, which policy permitted it and when the approval applied.

An approved_by value preserves only one part of that account. It may identify Alice correctly while leaving an auditor unable to establish delegation or policy scope. There needn't have been wrongdoing. The record may simply be insufficient to review what happened.

Counting participants in the sentence is a useful prompt, but it isn't a literal equation between nouns and columns. Some facts belong in related records. A policy can have its own version and validity period. The reporting freeze might be derivable from a reliable calendar. Check whether the whole relationship can be reconstructed from authoritative records, with each role intact.

A richer relationship doesn't require a particular database brand

A relational design can represent the approval as an event linked to its participants, with explicit roles. A document design can preserve those roles in a suitable structure or refer to other documents. A graph can represent them through a relationship entity or another explicit construction. Each approach brings different enforcement and querying work.

The engine doesn't excuse an omitted relationship. Buying a graph database for records that never captured delegated authority won't establish who authorized an old approval. Storing a large JSON object won't help if the application never writes the policy version into it.

Document locality is useful when related information is commonly read together. Cross-document references require a deliberate plan for integrity. Relational decomposition requires care too: splitting a relation should preserve the information needed to reconstruct it. Lossless reconstruction and preserving enforceable dependencies are related design concerns, not interchangeable names for one guarantee.

Normalization can improve the relations a team has actually represented. It can't discover a business role that nobody modeled. Denormalization can be safe when it preserves an authoritative source and a way to reconcile derived copies. It becomes dangerous when aggregation, lost keys or competing copies remove distinctions without anyone noticing.

The diagnostic is to write the fact in ordinary language, identify the roles and time boundaries, and trace where each is recorded. A populated row only proves that the fields in that row have values. The business question may require a relationship the row has no way to express.

These three mechanisms can coexist. An approval summary might use a coarse monthly grain, a guessed employee match and a single approver field. Fixing only one would leave the others intact. The repair starts with the missing evidence, then asks whether that evidence survives somewhere else.

Hard, buried and absent evidence call for different work[summarize]

A question can remain unanswered for several reasons. Its inputs may exist but require expensive computation. They may be scattered across systems nobody has connected. Or the distinction needed to settle the question may be absent from every source the investigation can establish.

Those conditions can look identical in a backlog. Each gets a ticket saying that reporting is blocked. Giving them the same treatment wastes effort in different directions: a team can abandon recoverable evidence, or keep commissioning analysis of a distinction no surviving record contains.

Hard questions have relevant inputs

A large scan can be hard. So can a deep graph traversal or an optimization problem with too many possible combinations for an exact search to be practical. Better algorithms, more resources or a bounded approximation can make useful progress when the inputs needed for the chosen formulation exist.

More money doesn't guarantee an exact answer. Computational complexity, poor specifications and practical limits still apply. The useful distinction is that the obstacle lies in processing or assembling relevant evidence, and the team can identify what progress would look like.

An illustrative subscription business wants gross margin by plan. Product revenue lives in one system and allocable costs in another. If the relevant records survive and the allocation rule is defensible, connecting them is a bounded engineering and accounting job. The team should state the allocation assumptions, but it isn't trying to recover an event nobody observed.

Buried evidence needs a recovery boundary

A second illustrative question asks whether trial users encountered a rate limit before canceling. The warehouse may have cancellation dates but no rate-limit events. Application logs might contain those events for recent users, while older logs have expired.

Any resulting answer then has a coverage boundary. Recent cohorts may support a direct analysis. Older cohorts may support only an estimate, or no reliable conclusion at all. Combining them into one chart without distinguishing those statuses would conceal the investigation's main finding.

Recovery should have a stopping rule based on named sources and periods. Check the producer, raw exports, relevant archives and available backups. Record what was inspected, what couldn't be accessed and which retention policies govern each. An unmounted backup is an unresolved recovery possibility. An established policy that overwrote the relevant events is a different kind of evidence.

This approach limits the search without pretending to know the contents of systems nobody has examined. A finding such as 'the available warehouse and retained application logs don't contain the event for these cohorts' is useful and reproducible. 'No record exists anywhere' requires much stronger authority.

Different possible histories expose missing evidence

A third illustrative question asks which onboarding email changed a customer's decision to convert. Records of sends and purchases establish that both happened. They may not establish whether the customer read the message, and even an observed click doesn't reveal what that person would have done without the email.

The causal problem needs more than a join. A comparison design, its assumptions and the way exposure was assigned determine what effect can be estimated. An analyst can't observe both outcomes for the same customer under the same circumstances. Better capture supports a better design, but it doesn't turn every causal question into a direct lookup.

A useful test is to construct two possible histories consistent with the available evidence. In one, the email changes a purchase decision. In the other, the customer would buy anyway. If the surviving records are identical in both histories, those records alone can't distinguish the answers.

The same test works without causal claims. Two hourly sales patterns can produce the same daily total. Two different people can share the attributes used by a matching rule. A permitted approval and an unauthorized one can leave the same approved_by value when delegated authority was omitted.

The test identifies what additional evidence or assumption is needed. It doesn't prove that every possible source in the world has been exhausted. Its strength comes from making the ambiguity explicit and stating the information boundary against which the conclusion holds.

A structural check can narrow the investigation. Mathematics often uses an invariant, a property unchanged by allowed transformations, to rule out an equivalence without trying every transformation. Different invariants can distinguish objects. Matching invariants don't automatically prove that the objects are the same.

The operational version is modest: examine the information required by the question instead of trying an indefinite sequence of tools. A bounded audit can establish that a particular set of surviving records is insufficient. It can also identify the exact outside source, measurement or experimental design that would change the finding.

A plausible number can hide the category

An estimate may be the best available support for a decision. The problem begins when its conditions disappear. A modeled number copied into a spreadsheet looks like a measured one unless the report carries the distinction with it.

The decision-maker should be able to see what was observed, what was inferred and which changes in assumptions would alter the action. A number with those limits attached can be useful. An apparently precise answer to an unresolved evidence gap may close the investigation before the gap has been understood.

Before buying more analysis, ask which of these jobs it will perform: process surviving inputs, recover a named source, estimate under explicit assumptions or begin collecting better evidence. That's also the question to ask when the proposed analyst is a model.

Models can extract facts or estimate under assumptions[summarize]

A model can recover a useful field from material outside conventional reporting. A support transcript may state an account identifier. A scanned invoice may contain a purchase-order reference. A free-text note may explain which policy authorized an exception. The information is present, and extraction makes it available in another form.

Extracting a recorded field differs from establishing a fact for which the model has no evidence. A generated value can be a prediction, an imputation or an unsupported guess. The wording alone doesn't tell the reader how the value was produced.

A proxy can predict without identifying the missing fact

Consider a model estimating campaign credit when the historical records lack dependable campaign-to-order links. Device, geography, time and landing page may correlate with campaign exposure. Those variables can help a prediction. They don't establish which campaign a particular order was connected to, much less whether the campaign caused it.

A model might perform well on an evaluation that uses the same patterns. Its performance can change when targeting, product mix or customer behavior changes. A proxy that once tracked campaign identity may stop doing so. The output needs an evaluation that reflects the decision and the conditions under which it will be used.

Predictions can remain valuable without identifying every causal mechanism. A demand forecast can support inventory planning without reconstructing each buyer's reason. The claim has to match the job: a useful forecast is different from a verified historical attribution.

A model may also bring information from outside the narrow table being inspected. Training, tools or another source can supply facts the warehouse lacks. That doesn't violate the information boundary. It expands the evidence available to the system, and the source of the added information needs to be inspectable if the result will be treated as a fact.

The relevant question is therefore whether the determining evidence is available anywhere in the actual workflow. 'The field is missing from this table' and 'the fact can't be recovered from any authorized source' are very different findings.

Matching thresholds decide which errors the business accepts

A model can propose record matches that a simple rule would miss. The proposal still needs an acceptance policy. A false link combines records belonging to different entities. A missed link leaves records for one entity separated. Those errors can have different consequences in marketing, support, finance and access control.

Changing a threshold on a fixed score trades sensitivity against selectivity. A more permissive threshold may accept additional true links and additional false ones. A stricter threshold may reject false links while missing valid ones. Better evidence or a better method can improve both error rates, so the trade-off isn't a law that all matching systems are forever equally limited.

The commercial decision is what consequences the organization accepts. A false merge in an audience analysis can distort reach. A false merge in an account-access workflow can cross a trust boundary. Using one default threshold for both would treat different risks as though they were the same task.

Someone with authority over the decision should own the threshold and the unresolved cases. That owner needs measured performance on relevant, reviewed examples, including the kinds of records likely to be ambiguous. A single aggregate accuracy score can conceal failures concentrated in one segment or one source system.

Changing the threshold can change reported channel performance too. The direction depends on the data and matching method. It isn't defensible to promise that looser matching always benefits broad-reach channels. The useful check is to compare the budget conclusion under plausible threshold choices and show which unresolved links drive the difference.

A review queue is a legitimate outcome. Some cases should remain unresolved until a person or a new source supplies the missing evidence. Forcing every pair into a final identity makes the dataset easier to join while hiding the cases most likely to change a proposed match.

Keep extraction, linkage, imputation and retrieval separate

These jobs use some of the same tools, but they warrant different evidence:

• Extraction identifies a value in an existing source and should preserve a way to inspect that source.
• Candidate linkage proposes relationships and should retain the matching method, uncertainty and review outcome.
• Imputation fills a gap under a statistical model and should state the missingness assumptions and relevant uncertainty.
• Retrieval ranks existing material and should identify what was searched and what the returned material supports.

A business can commission any of them deliberately. The dangerous arrangement is a pipeline that performs all four, writes the outputs into ordinary fields and drops the distinctions before the report reaches its reader.

An extracted invoice reference and an imputed invoice reference can look identical in a cell. The record needs to say whether the value was read from an invoice, inferred from another record or supplied by a reviewer. Preserving that origin at write time is easier than reconstructing it after downstream systems have copied the value.

There are practical checks inside the distinction. An extraction can point to a page or passage. A proposed identity can point to reviewed evidence. An imputation can expose the model and assumptions. Those records don't guarantee correctness, but they let a consumer inspect the kind of claim being made.

A model can help repair a representation that obscures evidence already present. It can't establish a particular missing historical fact merely by producing an answer-shaped string. The same boundary applies to vector retrieval, where a ranked candidate often arrives looking more authoritative than its score warrants.

Similarity doesn't establish direction or completeness[summarize]

Vector retrieval is useful because it can find relevant material without requiring the query and the source to use identical wording. A text embedding represents a passage as a vector, and a search compares that representation with other vectors. The returned neighbors are candidates for the job that follows.

A neighbor is different from an assertion that has been checked. A passage may concern the right subject while describing an exception, an obsolete rule or the opposite of the proposition the reader asked about. The system still has to establish what the passage says and whether it applies.

A symmetric score doesn't encode a directed claim

Cosine similarity gives the same result when its two input vectors are swapped. That symmetry doesn't mean an embedding model must represent 'A caused B' and 'B caused A' with the same vector. A model can represent their wording and meaning differently. The limit is narrower: the scalar similarity between two vectors isn't itself a record of who caused what.

A directed claim needs roles. A supports B, A contradicts B and A supersedes B are different relationships, even when the passages share most of their vocabulary. A retrieval score can help find the passages. Some further representation or reasoning has to preserve the direction and meaning of the relationship.

Negation deserves the same care. A system should be tested on the distinction between a permission and its denial, or a rule and its exception. Failures observed in a particular language-model experiment don't prove that every current embedding system is unable to represent negation. They do give a reason to avoid treating semantic proximity as proof of agreement.

The requirement follows from the business question. Finding passages about refunds is a retrieval job. Establishing whether a particular customer was entitled to a refund requires the relevant policy, its effective period and the facts of that customer's case. Similar words can identify candidate evidence without settling those conditions.

Missing records require a known population

An illustrative compliance question asks which policies have never been reviewed. Search can find review reports and policy documents. A list of relevant passages doesn't establish the full set of policies, and a failed search doesn't establish that a review never happened.

An answer needs a defined population: which policies belong to this inquiry, which versions count and what period is covered. It also needs a defined review record. A review might concern an older version, remain incomplete or be recorded in another system.

With those definitions in place, the team can compare the policy inventory against qualifying review records. A policy with no matching review is then a specific finding within that inventory and record set. The conclusion should preserve its scope. Missing evidence of a review may require investigation before being reported as proof that nobody performed one.

This is why a manifest matters. The painted outline on a tool wall makes absence visible because the expected tools are already specified. A search result supplies things it found. It doesn't supply an authoritative account of everything that should exist.

The same reasoning applies to uncontacted accounts, missing approvals and jobs that never completed. The system needs a population and a definition of the expected event. A query can then test for absence against that scope. Better ranking can't replace the missing inventory.

Preserving distances doesn't guarantee preserving meaning

The Johnson-Lindenstrauss lemma concerns preserving pairwise distances for a finite collection of points under an appropriate lower-dimensional mapping. That's a claim about geometry under stated conditions. It doesn't certify that a learned text representation preserves every fact, logical relationship or distinction that a business might ask about.

Nor does fixed dimensionality alone prove that every distinct input of interest must collide in a useful system's representation. Capacity, finite precision, the domain being represented and the learned mapping all matter. The practical concern is which distinctions the actual representation and retrieval process preserve well enough for the task.

A team can test that concern directly. Can the system distinguish current policy from superseded policy? Can it retain source references? Does it find the clause that establishes an exception instead of only the paragraph describing the general rule? Those questions evaluate the representation against the answer it's supposed to support.

The promise should remain proportionate to the evidence. A search system can rank useful candidates very well and still need explicit identity, validity and relationship records for reliable answers. Choosing those records begins with the reader's job, not with an assumption that every problem should become a vector search.

Derive the representation from the job it must support[summarize]

The nouns in a customer conversation are tempting schema names. Documents, tickets, campaigns and accounts sound concrete because people handle them every day. Each may deserve a record. None establishes the grain or relationships needed to solve the customer's actual problem.

I start with the problem independently of a proposed solution. Someone needs to make a decision or perform a job under conditions that currently prevent it. The useful future is an observable change in that situation. The representation should follow from what has to change.

A request for a better document system may conceal a question about which policy claim supports an answer. A request for one customer view may conceal several different uses of identity, each with its own tolerance for a false merge. Starting with a table called documents or customers leaves those questions unresolved.

Each design step should produce something inspectable

The derivation in this series runs through eight steps. It's a route from a job to a representation, not a claim that every design requires eight meetings. A small job may fit the whole chain in one short record.

1. Establish the problem independently of a particular tool. Name the decision or activity that's currently failing.
2. Describe the story in which it occurs. Identify the person, the relevant situation and what happens when a requested answer is wrong or unavailable.
3. Write the world model. State which facts and relationships would explain that situation, including what can actually be observed.
4. Compare solutions. Identify ways to change the outcome and the evidence each would need.
5. Define the features that let the person perform the job. A feature should have an observable use, such as inspecting the passage behind an answer.
6. Identify the systems that provide those features. Retrieval, synthesis and citation are different jobs even when one interface presents them together.
7. Identify the components those systems need. A source reference or a validity interval may be reused across several kinds of record.
8. Define the entities and their relationships at the grain those jobs require.

The chain also runs back upward as a check. An entity should have a reason to exist in the job above it. A proposed world model should survive contact with what can be observed. A feature that requires unavailable history may need a narrower claim or a new capture process.

That check prevents two opposite mistakes. A team can omit a relationship because nobody traced the job far enough. It can also build an elaborate catalog of entities for questions nobody needs to answer. Both mistakes start with representation detached from use.

My retrieval work changed how source material was divided

My own RAG Knowledge Engine had to work across Markdown writing, PDF textbooks and YouTube material. A single static chunking approach didn't suit that mixture. I built on Archon and changed the ingestion pipelines by document type, with semantic chunking and task-specific embedding choices.

The point of that work was where one concept ended and another began. A code block, a quoted passage and a narrative segment don't necessarily fit the same arbitrary boundary. A system can retrieve a relevant-looking fragment while leaving behind the condition that determines what it means.

That's an example of changing how existing evidence is exposed. The source material was already present. The work improved the units available for retrieval rather than claiming that the model had reconstructed an undocumented event.

A policy assistant provides an illustrative extension. Its answer might need a source passage, its scope, its effective period and a relationship to a superseding passage. A document identifier is part of that record, but it doesn't locate the particular claim supporting a sentence.

Storing claims and source spans doesn't guarantee a correct answer either. The system still has to interpret scope, retrieve the relevant evidence and cite what each answer actually uses. A finer grain makes those jobs inspectable. It doesn't abolish them.

This is also why 'smaller chunks' is an incomplete prescription. A fragment that loses its definition, negation or exception may be worse than a larger passage. The unit should retain the context needed for the question, with a way to recover surrounding material.

Give the logical model one reviewable home

My Python systems work uses Pydantic data models as a shared source of truth. Here, the important property is that the model has an address. Engineers can inspect what an object means, what properties it carries and which constraints apply before separate interfaces develop incompatible interpretations.

I use a typed catalog as the first build artifact after the job has been understood. Storage definitions, APIs and consuming code should use that contract or have an explicit mapping to it. A frontend and a warehouse don't have to store identical bytes, but their different representations shouldn't silently redefine a customer or an approval.

Types help make the agreement inspectable and enforce parts of it. They can't establish that a recorded value is true. A well-typed guessed identifier is still guessed, and a valid timestamp may still refer to the wrong event. The catalog needs meanings and provenance as well as accepted shapes.

I also use an entity-component approach: identity belongs to the entity, information attaches through components, and behavior is kept separate from the data. That gives a system room to represent overlapping roles without forcing each object into one exclusive branch of a class hierarchy.

A customer can also be a supplier. A document can also be a contract. These illustrative roles show why one exclusive hierarchy may be a poor fit. Composition is useful when the roles need to coexist and change independently. It carries its own complexity, which the job still has to justify.

The catalog should record which question requires each consequential field, how that field is produced and what happens when it's unavailable. An engineer encountering a new checkout path can then see why a session link matters. Without that connection, a future cleanup can remove the field while every local test still passes.

That leaves a practical deliverable for a business that hasn't done this work: a small register of decisions, required evidence and the first point where the evidence fails.

Trace a decision to the record that would support it[summarize]

The diagnostic starts with someone who owns a consequential choice. A data team can inspect tables, but it shouldn't have to invent which business decisions justify the inspection. The person responsible for a channel contract, a staffing plan or a discount policy supplies that part.

Use a small working set of priority questions. Five is the working number in this method because the exercise needs a selection, not an inventory of every metric the business might display. The number isn't a statistical threshold. A genuinely important sixth question shouldn't be discarded to preserve it.

Write the actions for alternative answers

A question earns its place by changing what someone does. Write the alternatives in plain language: if the evidence supports one conclusion, this person takes one action; if it supports the other, they take another. The format exposes metrics collected without a clear use.

An illustrative channel decision might be to renew a contract, reduce it or stop spending there. A staffing decision might change opening hours or labor allocation. An onboarding decision might justify another support role, or direct effort toward a product defect that staffing wouldn't solve.

The alternatives don't have to be binary. A decision can include postponing action, collecting evidence or choosing a smaller commitment. The useful constraint is that an answer has an identified consumer and a consequence. If every plausible answer produces the same action, ask whether the question is useful for monitoring, compliance or a later decision before removing it.

That last distinction avoids another shortcut. A measure can be worth collecting without immediately changing a budget. The exercise is about questions blocking a named decision, not a universal rule that every observation must trigger an action today.

Name the evidence in the business before naming a system

For each decision, describe what would have to be observed. Use participants, actions and timing. An order associated with a recorded visit is a business event. A row in an analytics table is one possible representation of that event.

Writing the event without naming a vendor helps separate the business requirement from the system currently available. The requirement might span several observations rather than one. An approval question needs the act, its authority and the applicable policy. A staffing question needs sales and relevant costs over the periods under consideration.

The discount question makes the causal requirement visible. Comparing revenue before and after a promotion mixes the promotion with seasonality, changing traffic, product mix and anything else that moved. An estimate of incremental effect needs a defensible comparison between exposure conditions, with the assumptions and assignment process made explicit.

A deliberately designed holdout may supply that comparison. Other designs may be possible under appropriate assumptions. Writing 'customer saw discount and bought' is insufficient to establish what would have happened without the discount. The diagnostic should expose that design gap before anyone promises a causal dashboard.

Run four checks against a defined scope

Name the systems, sources and period covered by the investigation. Then examine the required observations in order. An unresolved access restriction remains visible, and a discovered failure doesn't excuse assuming the rest of the chain is sound.

1. Was the event captured at all? Check the producer and authorized surviving sources, including raw logs, vendor exports and archives. Distinguish a search that found nothing from a producer whose capture behavior is known.
2. Is the grain sufficient? Compare the distinction needed by the decision with the finest relevant evidence that survives. Check participant roles as well as time and aggregation level.
3. Are the connecting identities and relationships dependable? Trace how each key was established, whether it was observed or inferred, and what uncertainty accompanies it.
4. Does the record survive for the required period? Examine retention, overwrites, validity changes and recoverability. A backup helps only if the relevant record can actually be restored and interpreted.

The first failure identifies the first repair dependency. Record it, then note any remaining checks that couldn't yet be assessed. Several failures can coexist. Adding a key won't fix a capture path that also overwrites history, and discovering a retention problem doesn't prove that the original relationship was complete.

The structural failure from earlier belongs inside this trace. A relationship can be 'captured' in a weak sense while omitting a role the question requires. The grain and relationship checks must compare the whole business fact against the representation, not merely confirm that an event row exists.

The deliverable names a gap, an owner and a next action

A ceiling register is a table of decision limits. It should let an owner distinguish a missing fact from a missing query and see what work would change the situation. Keep the table small enough to use; attach the detailed source trace where the team normally keeps technical evidence.

The following rows are illustrative, not findings from a client audit. Each combines the gap and its bounded response so the reader can see why different questions produce different work.

A real register should also carry the investigated period, source locations, current status, repair estimate and the date from which new evidence will accumulate. The responsible engineer supplies the estimate after inspecting the capture path. An estimate needs the actual write paths and integration constraints.

An absent event calls for prospective instrumentation. Buried evidence calls for a recovery job. Insufficient grain calls for preserving finer observations. An uncertain identity calls for a better connection or an explicit matching policy. A causal gap may call for a different experiment. Those are different purchases.

A repair starts a new history

A system that begins preserving a session link now can support better observed-path reporting for events captured after the repair. The older period remains subject to whatever evidence survived from it. The first report should identify that boundary instead of suggesting the improved schema has repaired the past.

The field also has to survive future changes. A platform migration needs an explicit mapping. A second checkout path needs to produce the same kind of evidence, or state why it can't. Capture, transfer and retention are each part of the contract.

The register gives the business a way to prioritize those repairs without waiting for an unrelated platform decision. It also gives the business a way to decline them. A named question and a known gap still have to justify the full cost of changing the system.

Pay for the questions that change the business[summarize]

Shape work consumes engineering capacity that could support other customer needs. It can require capture changes across several systems, a transition between definitions and continuing maintenance after the original project ends. A more expressive model also asks every new engineer to learn more before making a safe change.

Those costs belong beside the decision the work is meant to improve. A register that names missing information but omits the operating burden is incomplete. The business needs a reason to collect, protect and maintain a distinction, as well as a reason it might be useful.

Specify which report governs during a transition

A period with old and new pipelines can produce two defensible numbers. One may count orders while the other counts settled purchases. One may use inferred customer merges while the other preserves distinct identities. Both can be correctly computed under their own rules and remain unsuitable for direct comparison.

Running both pipelines can be a sensible way to inspect the change. It needs an explicit account of which output governs each decision, how differences are reconciled and when the old route retires. Without that plan, dual writing can replace a known reporting limit with uncertainty about which report the business should use.

A new schema may permit a historical restatement that removes double-counting when the necessary source detail survives. It can't restore detail that's gone. The restatement should say which periods were recomputed, which sources supported them and which figures still depend on estimates.

The people who reported the earlier numbers need the explanation too. A changed number can affect budgets, incentives and assessments of past work. Giving those readers a traceable definition change is part of the engineering delivery. Quietly replacing the number leaves them to discover the disagreement in a later decision.

Preserve the reason a field exists

A repaired key can disappear when somebody adds a new write path. An approval role can be omitted from an import that only copies the fields visible in a screen. A retention change can delete the source detail that a report still claims to represent.

The defense is a record connecting the field to its purpose and enforcement at the boundary where the evidence enters. The system should reject a record that lacks a required value. Legitimate unknowns should have explicit meaning and remain visible in coverage reporting. Treating every null as a defect would encourage fabricated completeness.

The contract also needs a consumer. A check that nobody owns can fail repeatedly without changing anything. The responsible team should know which decision is affected, what a failure means for current reports and how to restore valid capture.

Composition and typed catalogs help make those duties inspectable. They also carry a maintenance tax. More entities, more roles and more validity rules create more opportunities for incompatible interpretation. A team should add them because a job requires them, then document and enforce the meanings that make the addition useful.

A flat table may already answer every question that matters[summarize]

A small business can run the diagnostic, find no consequential gaps and leave its model alone. A simple product range, a limited set of channels and decisions tolerant of broad uncertainty may not justify a richer representation. An owner may also have relevant knowledge that hasn't yet become a software requirement.

That knowledge still has a continuity risk if other people need to act without the owner. The right next step might be documenting a rule or recording a small set of events. It needn't be a metagraph, a platform migration or a full redesign.

Even a single acquisition channel doesn't settle every causal marketing question. The useful simplification is that a business may have fewer identity and allocation problems, and those problems may not change its actions enough to justify additional machinery. Simplicity has to be evaluated against the actual questions.

I'm not proposing shape work for a business that can already make its important decisions with adequate evidence. A richer schema with no decision behind it creates a maintenance obligation without a demonstrated return. Keep the plain table when it supports the job.

Repeated decisions make the trade-off visible

A recurring unanswered question is one signal to revisit the model. The owner keeps facing the same choice, different analysts return different estimates and the distinguishing evidence is still absent. Prospective capture may then be more useful than another round of retrospective modeling.

Recurring manual reconciliation is another signal. A person repeatedly connects orders to payments, tickets to accounts or approvals to policy records. Inspect what that person actually does before calling the work automatable. They may be applying judgment the software doesn't represent, enforcing a legitimate separation or resolving source errors that a foreign key alone wouldn't fix.

When the work is repeatedly establishing a stable relationship, it may belong in the capture process. When it requires judgment, the system may need a review record with evidence and authority. Both responses are more precise than telling the person to keep reconciling indefinitely or promising to replace them with an automatic join.

A proposed material investment supplies a third occasion. Before launching a campaign or changing an onboarding process, specify what evidence would support the later decision about it. Capture can begin with the activity, and the comparison design can be chosen while alternatives are still available.

The opening checkout can continue shipping parcels correctly while the campaign question remains unresolved. Fixing that gap means deciding which claim the business needs to make, recording the evidence that supports it and preserving the limits of the resulting answer. The choice is concrete enough to price and narrow enough to decline.

Part II, Five Shapes, examines the representations available for those jobs. Part III, Time and Provenance, takes up the history needed to tell when a fact applied, where it came from and what disputes it. The useful starting point for either is a decision whose required evidence can be named.