The Shape of Data II: Five Shapes. Every shape answers one class of question cheaply and another class not at all.

Choosing a shape is choosing what to be bad at [summarize]

There is no general-purpose shape

Every data vendor sells a general-purpose product. None of them sells a general-purpose shape, because a shape is a set of operations you get for free and a set you don't. Relational engines give you set operations over tuples. Document stores give you one addressable tree per key. Vector indexes give you nearest neighbours under a distance function. Graph engines give you traversal along typed edges. Each of those is a different algebra, and the questions each algebra can express are different questions.

So when a store claims to handle everything, it's telling you something specific about its priorities. A general-purpose claim is a claim that some question class doesn't matter, and it's usually the class the vendor's benchmark doesn't cover. Postgres with a vector extension is a fine product and it's still a relational engine with a nearest-neighbour index bolted on, which means it inherits relational strengths and the approximate-search caveats both. Neo4j is a fine product and it's still a poor warehouse. Nobody is lying. The marketing just rounds a structural trade down to a feature checkbox.

Shape decisions also arrive as defaults, which is why so few teams remember making one. A web team picks relational because the framework ships with an ORM and a migration tool. A mobile team picks documents because the sync SDK speaks JSON and the offline story is already written. An AI team picks vectors because every tutorial in the category starts with an embedding call and a similarity search. Each of those choices is correct for the problem the team had that week, and none of them gets recorded anywhere as a choice, which is why the design review three years later has nobody in it who can say why the data looks like this.

The reflex answer to a shape problem is to add another shape. That reflex is right, and the cost sits somewhere people don't look. Standing up a second store takes a few days. Backfilling it takes forever or takes nothing, depending entirely on whether the facts it needs were ever written down. You can copy a relational table into a graph in an afternoon. You cannot copy in the edge that says which salesperson introduced which account if nobody ever recorded the introduction, and no traversal engine invents it for you. A shape decision gets paid for years after you adopt it, in the gap between the questions you now want and the events you happened to capture.

Part I of this series established that a shape sets a ceiling on what a system can ever know, and that "we can't answer that" is usually a schema fact rather than an analytics failure. The Ceiling is the argument; this part is the survey.

What survives the transformations you don't care about

There's a cleaner way to state what a shape does, and it comes from a branch of mathematics built for exactly this question. Topology is geometry with the rules loosened. You may stretch, bend and squash a shape however you like, and you may never cut it or glue it. Two shapes count as the same when one can be mushed into the other under those rules, which is why the standing joke says a topologist can't tell a coffee mug from a donut. One hole each.

Proving two shapes are the same takes one witness: exhibit the deformation and you're done. Proving two shapes are different means ruling out every deformation that might have worked, and there are infinitely many, so nobody can check. Mathematics solved that asymmetry with an invariant. An invariant is a fingerprint computed from a shape by a fixed recipe, engineered so that no legal deformation can change it. Compute the fingerprint of A, compute the fingerprint of B, and if they differ then the shapes differ, permanently, without anybody examining a single deformation.

The feature they picked to fingerprint was holes, and the reason they picked it is the design question sitting underneath every schema decision anybody has ever made: what property survives the transformations you don't care about, so that any change in it must reflect a change you do care about? Size doesn't survive, because stretching changes it. Curvature doesn't survive, because bending changes it. Angles, lengths and areas are all destroyed by legal mushing, so all of them are worthless as fingerprints. Holes survive. You can stretch a donut, dent it, paint it and wear it as a hat, and the hole is still there, because creating or destroying a hole needs tearing or gluing and those are the forbidden moves.

Now transfer it. A data shape is a transformation you apply to reality on the way in. Reality arrives as events with participants, sequence, context, sources and duration, and the shape flattens all of that into rows, or trees, or coordinates, or edges. Whatever survives that flattening is the complete set of things you can ever ask about, and everything else is gone the moment the write commits. So the design question is which properties need to survive the trip, rather than which store handles your load, because a question about a property the shape destroyed has no answer waiting anywhere downstream.

Here's the whole part in one table, as a map for what follows.

Bad at is a technical claim with four settings

"Bad at" gets used loosely, so it's worth pinning down. When a question meets a shape, one of four things happens, and only two of them are fixable with money.

The middle two settings cause most of the damage, because both of them return an answer. An expensive query returns the right answer slowly, which is annoying and visible. An unenforceable relationship and an inexpressible question both return something that looks like an answer and carries no error bar. A document store will happily let two copies of a customer record disagree. A vector index will happily return the nearest passage to a question that has no answer in the corpus. Neither one raises an exception, so neither one shows up in a postmortem as a data problem. They show up as "the numbers feel off," which is where data problems go to become political problems.

How to read the five sections

Five sections follow, in rough order of how much structure each shape carries: the table, the document, the vector, the graph, the metagraph. Each one runs the same four beats. What the shape actually is, stated precisely enough to be useful to somebody who has to defend the choice in a design review. One real question it answers cheaply, with the mechanics of why it's cheap. One real question it can't answer, with the mechanics of the failure. And what the workaround costs, since everybody eventually builds the workaround.

You can read the five in any order. The ladder does have a direction: each shape carries more structure than the one before it, and pays for that structure in write cost, query complexity, and the number of people on your team who can reason about it fluently. More structure buys more expressible questions. It never buys them for free, and the last section of this part is about the fact that nobody in production runs one shape anyway.

The table [summarize]

What normalization actually preserves

A relational table is a set of rows over a fixed list of typed columns, with keys that identify rows and constraints the engine checks on every write. E. F. Codd published the model in A Relational Model of Data for Large Shared Data Banks (CACM 13(6), 1970), and its most useful property for this argument is the separation it draws between the logical shape and the physical storage. Indexes, file formats and access paths can all change without touching a single query. The logical model can't, because changing it changes which questions are expressible at all.

Normalization is usually taught as hygiene, which undersells it. It's a theory of safe decomposition, and it answers one question precisely: when can I split this table into two smaller tables without losing anything? Two conditions carry the answer. A decomposition is lossless-join when the attributes shared by the two projections functionally determine at least one of them, which is what guarantees the natural join reconstructs the original rows exactly, with no invented combinations. A decomposition is dependency-preserving when every constraint that held on the original can still be checked against one of the pieces, without a join. Miss the first and your data quietly gains rows that were never true. Miss the second and your constraints stop being enforceable at write time, which means they stop being enforced.

Those two conditions don't always co-operate, and the way they fail is the reason working schemas look the way they do. Boyce-Codd normal form is always achievable losslessly and sometimes costs you dependency preservation. Third normal form always preserves dependencies and pays for it with a little residual redundancy. Most production schemas sit at 3NF for exactly that reason, and the choice is a real engineering trade rather than a failure to finish the homework.

Two footnotes on that table, because both get repeated wrongly. Ronald Fagin's 1979 SIGMOD paper Normal Forms and Relational Database Operators is the fifth-normal-form source. His 1981 TODS paper introduces domain-key normal form, which is a different and non-equivalent idea, so citing 1981 for 5NF is a citation error that has propagated through a decade of blog posts. And normalization theory only ever describes safe decomposition. There's no matching theory that tells you which tables are safe to merge, which is why denormalization loses answerability without anybody noticing. It loses it three ways: by aggregating to a coarser grain with no detail retained, by dropping the keys needed to reconstruct a join, and by co-locating independent facts so that no copy is authoritative any more.

Grain is the commitment you can't take back

Kimball's rule is to declare the grain before anything else, and the rule earns its place because grain is the one decision in a warehouse that behaves like a ratchet. The grain of a fact table is what one row means. One row per order line, one row per order, one row per customer per day, one row per channel per week. Whatever you pick sets the maximum resolution of every question that will ever be asked of that table, forever, because aggregation runs one direction only: from a sum you can't recover the addends or how they were distributed.

Here's what that costs in practice. A subscription brand I'll keep anonymous ran a fact table at order-line grain, which was by far its largest table. A nightly job rolled that up into daily revenue by channel, and a retention policy dropped line-grain rows after 90 days to control storage. Both decisions were signed off as cost work and both were locally reasonable. Seven months later the merchandising lead wanted to know what share of orders contained both the starter kit and a refill, because she wanted to build a bundle and price it. That's a basket question. It needs two lines that share an order id. Daily channel rows can't produce it at any price, the raw lines were gone, and the answer to a question worth real margin was a shrug plus a proposal to start capturing it now and revisit in a quarter.

Nothing was broken. Every query ran, every dashboard populated, and the number the business needed had been deleted on purpose by people optimizing storage against a question nobody had asked yet. This is the same failure Part I calls the grain-too-coarse death, and the reason it lands in the table section is that tables are the shape where grain is declared explicitly, which makes them the shape where the loss is most preventable. A single retained line-grain partition in cold storage would have kept the question alive for the price of a rounding error. The data pipelines entry covers where in the pipeline each grain decision gets locked.

The question a table answers before you finish typing it

Give a warehouse a fact table and two dimensions and ask for revenue by channel by week for the last 18 months, split by new versus returning customers. That query is a scan, two joins on integer surrogate keys, a group-by and a sort. The engine has been engineered for exactly that operation for 40 years: columnar layouts so the scan touches only the columns named, statistics so the planner knows which join to run first, partition pruning so the 18-month window skips most of the disk. Adding another slice doesn't change the shape of the query. That's the native class, and the star-schema pattern behind most working dashboards exists to make it as cheap as possible.

Aggregation is only half of what you're buying. The other half is enforcement. A relational engine is the only shape in this survey that will refuse a write because it breaks a relationship. A foreign key means an order cannot reference a customer who doesn't exist. A unique constraint means two rows cannot claim the same invoice number. A check constraint means quantity can't go negative. Those guarantees hold no matter which application, script, intern or agent is doing the writing, and they hold at three in the morning when the retry logic is confused. Every other shape in this article moves that responsibility into application code, where it is implemented once per writer and correct in most of them.

Where the table stops: variable depth and ragged records

Relational operations work on sets of rows, a fixed number of relations at a time. A path of length two is a self-join. A path of length three is two self-joins. A path of unknown length is neither, so you reach for the fixpoint: SQL:1999 added recursive common table expressions, and they do express the traversal correctly. This is an expensive question rather than an inexpressible one, and the distinction matters because people reach for a graph database when a recursive CTE would have done.

Then they hit the second wall. Each iteration of a recursive CTE joins the frontier against the whole edge table, so cost scales with fan-out per level rather than with the answer size. Optimizer cardinality estimates degrade fast across iterations, because the planner is guessing about a set it hasn't built yet. Ranking, weighting or bounding the traversal means hand-rolling logic the engine has no operator for. On a supply chain with wide fan-out, "which suppliers sit upstream of this SKU at any depth" goes from a clean recursive query to a scheduled job with a materialized closure table, and a closure table is a hand-built index of every path, maintained by application code, in a shape the engine can't validate.

Ragged records are the second weakness, and they're the reason the next section exists. A table declares one column list for every row in it. When 12 percent of your records need three fields the rest don't, you either add sparse nullable columns that mean nothing for most rows, or you add a JSON column and drop the extras in there. That JSON column is a document store hiding inside a table, with none of the constraint help the surrounding columns get. Insurance claims, medical intake, product catalogues across categories and anything shaped by a form somebody keeps editing all live here.

One more limit worth naming and deferring. A table row records what is currently true and nothing about when it became true, who asserted it, or what it replaced. You can model history in tables perfectly well, so this is a weakness of how tables get used rather than of the relational model, and it's the whole subject of Part III.

The document [summarize]

What locality buys you

A document store keeps self-describing trees under keys, and reads or writes each tree whole. Martin Kleppmann frames the trade precisely in Designing Data-Intensive Applications (chapter 2, O'Reilly, 2017): documents buy locality and pay in relationship expressiveness. Locality means related data sits together on disk and arrives in one read, and that's worth more than it sounds.

Take an order confirmation. Modelled relationally, it's an order row, some line rows, a shipping address row, a tax breakdown, a promotion application and a payment result: six tables, five joins, and rendering code that assembles a tree the database just finished taking apart. Modelled as a document, it's one key and one read. The line items are nested inside the order because they belong to the order and get read with it every time. A document store wins the cases where the shape of the record and the shape of the read are already the same shape.

Two more strengths get undersold. Ragged records are the first: insurance claims, medical intake, product catalogues across categories, anything shaped by a form somebody keeps editing. Add a field on Tuesday and old records simply don't carry it, with no migration, no lock and no nullable column that means nothing for 88 percent of the rows. The second is atomicity. A single document write is atomic in every serious document store, so the boundary of your document is the boundary of what you can keep consistent without a distributed transaction. An invariant that lives inside one document ("line totals sum to the order total") is trivially enforceable. An invariant spanning two documents isn't enforceable at all, which makes document boundaries a design decision about consistency rather than a decision about convenience.

One correction on vocabulary, because it causes real damage. "Schemaless" describes where the schema lives rather than whether one exists. Every consumer of a document carries an implicit schema in its head: the reporting job knows there's a total field, the mobile client knows address.zip is a string, the finance export knows the currency code is uppercase. Schema-on-read means the schema moved out of the database and into every reader, and there are more readers than anybody counts. A field rename that would have been a migration in a relational store becomes six silent breakages discovered at different times by different teams.

The second hierarchy has nowhere to live

A tree branches one dimension at a time. That's the definition rather than a limitation of any particular product, and it's where the shape stops. Andy's own ECS note puts it bluntly: a tree can only branch along one dimension at a time, and reality is not a tree, reality is a matrix.

Here's the concrete version. A consultancy has people, projects and clients. Nest people inside projects and "which projects has this contractor touched" turns into a scan across every project document, because the contractor exists only as a repeated fragment inside documents keyed by something else. Flip it, nest projects inside people, and the client-facing question dies instead. Store both and you've got two writes that can disagree, with no constraint anywhere in the system that says which one is right. The second hierarchy always exists, and the document shape gives it nowhere to live.

The same limit shows up in knowledge work, more sharply. A body of knowledge has horizontal edges: a theorem appears in chapter three as a lemma and returns in chapter nine as the engine of a major result, the Kalman filter shows up in a time-series text and again in a control-theory book and a third time in a quantitative-finance text applying it to price. A human expert sees one idea wearing three costumes. A tree sees three unrelated leaves whose only connection runs all the way up to a shared ancestor and back down. The most valuable structures in a corpus are the cross-domain bridges, and a hierarchy is exactly the shape that can't hold them. Filing systems are trees for a good reason: people retrieve things by remembering where they put them. Representing how the ideas connect was never part of that job.

References are joins without a referee

Everybody eventually reaches the same workaround: stop nesting and store an id instead. A person document carries an array of project ids, and the application resolves them. That works, and it's worth being precise about what just happened. A cross-document reference is a join, executed by application code, with none of the guarantees the word "join" implies in a relational engine.

Three things go missing at once. Nothing enforces that the referenced document exists, so deleting a project leaves a dozen person documents pointing at an id that resolves to nothing, and every write along the way was accepted as valid. Nothing makes the resolution one operation, so the read becomes a fetch plus N follow-up fetches, which is a nested-loop join running across a network with no planner and no statistics. And nothing makes the two directions agree, so if you also stored a member list on the project, you now have two representations of one relationship and a standing possibility that they disagree with no way to detect it except a job somebody writes and nobody maintains.

That's the unenforceable setting from the frame section, and it's the most expensive of the four, because the system keeps running. A missing foreign key throws an error at write time in a relational engine. A dangling reference in a document store throws nothing, ever. It surfaces months later as a report with a slightly wrong count, or as a customer who exists in billing and doesn't exist in support, and by then the bad writes are spread across a year of history and there's no log of which write was the wrong one.

All four of those are legitimate and I've shipped all four. The one that reliably goes wrong is the first, because it's the only one that feels free at the time. Duplicating in both directions costs nothing on day one, costs a reconciliation script in month three, and costs a data-quality project in year two once enough disagreement has accumulated that nobody can say which copy was right. Pick the workaround whose cost lands early, since the ones that defer the cost are also the ones that compound it.

None of this makes document stores a bad choice. They're the correct choice for a large class of systems, and the class is well defined: the record is read and written whole, the invariants you actually care about fit inside one record, and the relationships that cross records are few enough to manage by hand. Check those three against your own system before the next design review. If two of the three fail, you're using a document store as a general-purpose store, and the section on hybrids at the end of this part is about what to do instead.

The vector [summarize]

What a vector index actually stores

An embedding model reads text and returns a list of numbers, a point in a space of a few hundred to a few thousand dimensions. A vector index stores those points and answers one question: which stored points sit nearest this one. That's the entire mechanism, and its usefulness is real.

Here's the case it wins outright. A customer writes in: "it won't charge overnight." Your manual says "the unit fails to draw power while docked." No shared vocabulary worth matching on, so keyword search returns nothing and the customer gets told there's no article on their problem. An embedding puts those two sentences close together, because the model was trained on enough text to place them in the same neighbourhood. Vector search finds meaning the writer and the reader phrased differently, which is a job no other shape in this article does at all.

Its second advantage is operational rather than semantic. Point it at a pile of documents on Friday and you have retrieval by Monday, with no schema, no key design, no entity resolution and no ingestion contract. That speed is why the shape spread across every company in two years, and it's worth naming clearly, because most arguments against vector search are really arguments against skipping the modelling work that the vector index let you skip.

What you get back is a ranking by proximity and nothing else. Every other property, direction, negation, quantity, recency, authority, has to be carried by something outside the geometry. The rest of this section is about why that "nothing else" is structural rather than a gap the next model closes.

An embedding is a one-way door

The map from text into coordinates is many-to-one. Different inputs can land on the same point, or close enough that no downstream component can tell them apart. Many-to-one means not invertible, so you can't recover the input from the vector, and you can't recover any property of the input that the map didn't preserve. Most people know the first half. The second half is the one that costs money.

Apply the invariant question from the top of this part: what survives the transformation? Rough neighbourhood in meaning survives. That's the fingerprint. Sentence order survives weakly, sentence length not at all, and the logical operators barely register. Three contract clauses illustrate it:

• "Liability is capped at two times fees paid."
• "Liability is not capped."
• "Liability is capped at two times fees paid, except in cases of gross negligence."

Those three sentences live in the same tiny region of embedding space, because they share almost every content word and the whole topic. They also describe three different companies from a risk perspective, and the difference between them is exactly what a general counsel is paid to notice. A retrieval system asked "is our liability capped" returns whichever of the three sits marginally closer to the query, with a similarity score that gives no indication that the other two exist and contradict it. The geometry preserved the topic and discarded the operator, and the operator was the answer.

One symmetric number can't carry direction

Cosine similarity between two vectors is a single number, and it's symmetric: the score for A against B equals the score for B against A, by construction, since the formula can't tell which argument came first. That's arithmetic rather than a model-quality issue, and one consequence deserves to be stated on its own line.

"A caused B" and "B caused A" are indistinguishable to any symmetric metric. So are "we acquired them" and "they acquired us," "the vendor owes us" and "we owe the vendor," and every other fact whose meaning is carried by the direction of the arrow.

Negation fails in the same family of representations. Nora Kassner and Hinrich Schutze probed this directly in Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (ACL, 2020), and their finding is stated plainly in the abstract: "We find that PLMs do not distinguish between negated ('Birds cannot [MASK]') and non-negated ('Birds can [MASK]') cloze questions." Quantification degrades the same way. "All of our suppliers passed audit," "some of our suppliers passed audit" and "none of our suppliers passed audit" occupy nearly the same coordinates while describing three different companies.

The strongest evidence that this is structural comes from the field that tried hardest to fix it. Knowledge-graph embedding models exist because asymmetric relations need structure that plain similarity doesn't provide, and their history reads like a series of admissions. TransE models a relation as a translation between entity vectors. DistMult uses a bilinear diagonal form, which is symmetric, so it provably can't represent an asymmetric relation at all: score "Alice manages Bob" and you've scored "Bob manages Alice" identically. ComplEx moves the embeddings into complex numbers specifically to break that symmetry. Every one of those design choices is the same admission: if you want direction, you have to build it into the representation, because distance never carries it.

What Johnson-Lindenstrauss actually guarantees

Somebody in the design review will invoke the Johnson-Lindenstrauss lemma to argue that the embedding is basically lossless. It's worth knowing exactly what that result says, because the usual reading of it is backwards.

Johnson and Lindenstrauss (Extensions of Lipschitz mappings into a Hilbert space, Contemporary Mathematics vol. 26, 1984) proved that n points can be mapped into a space of O(eps^-2 log n) dimensions with all pairwise Euclidean distances preserved to within a factor of (1 ± eps). It's a beautiful result and it's genuinely load-bearing for dimensionality reduction. Read the statement carefully and notice its subject: the lemma preserves distances that already existed between points you already had. It says nothing about semantic content, nothing about logical structure, and nothing about entailment.

So invoking it as reassurance inverts what it promises. If two contradictory sentences already sat close together in the encoder's output, the lemma guarantees they stay close after any dimension reduction you apply. It preserves your problem with a proof attached. A theorem about preserving distances can't manufacture a distance that encodes a distinction the encoder never made, and the distinction between "capped" and "not capped" is precisely one the encoder never made. Anybody arguing from Johnson-Lindenstrauss to "the vectors keep the meaning" has skipped from a claim about geometry to a claim about semantics without noticing the step.

The questions with no passage to retrieve

Top-k retrieval scores every passage against the query independently and returns the k best. That procedure has four failure classes, and they're worth separating because only one of them is a tuning problem.

Multi-hop questions need evidence that lives in two places. "Which of our suppliers is owned by a company we've already blacklisted" requires an ownership fact and a blacklist fact, and neither passage is especially similar to the question. Three benchmark datasets exist because this failure is reproducible: HotpotQA (Yang, Qi, Zhang, Bengio, Cohen, Salakhutdinov and Manning, EMNLP 2018), MuSiQue: Multihop Questions via Single-hop Question Composition (Trivedi, Balasubramanian, Khot and Sabharwal, TACL vol. 10, 2022), and Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (Ho, Nguyen, Sugawara and Aizawa, COLING 2020). The mechanism is the same in all three: the retriever carries no representation of the path connecting two passages, so the second hop stays invisible until you already have the first, which retrieval by similarity has no way to know.

Global questions ask about a corpus rather than about a passage in it. The GraphRAG paper (Edge, Trinh, Cheng, Bradley, Chao, Mody, Truitt, Metropolitansky, Ness and Larson, arXiv 2404.16130, 2024) states the problem exactly: RAG fails on global questions directed at an entire text corpus, such as "What are the main themes in the dataset?", since this is inherently a query-focused summarization task rather than an explicit retrieval task. That paper is a preprint rather than a peer-reviewed benchmark result, and it reports improvements in comprehensiveness and diversity over a conventional baseline without attaching a percentage, so I'd characterize it the way its authors do.

Negative-space questions are the sharpest failure and the one operators hit most often. "Which of our 400 policies were never reviewed" is a set difference: it needs the full set of policies, the set of reviewed ones, and a subtraction. Nearest-neighbour search returns neighbours among things that exist. No passage in the corpus says "this policy was never reviewed," so the correct answer is a document that doesn't exist, and no amount of retrieval quality produces it. Absence requires an authoritative manifest of the whole domain, which is a relational property rather than a search property.

Ask a vector store what's missing and it returns the closest thing that's present, with a confident-looking score attached.

Whole-document questions want the arc of a document rather than a piece of it. Retrieval bounded by k is structurally local, so "what's the argument of this book" returns the five paragraphs most similar to the words "the argument," which are almost never the paragraphs carrying it. RAPTOR (Sarthi, Abdullah, Tuli, Khanna, Goldie and Manning, ICLR 2024) fixes that by manufacturing the missing altitude: chunk and embed the leaves, reduce the dimensionality, cluster with a Gaussian mixture that picks its own cluster count, have a model write an abstractive summary of each cluster, then embed and cluster and summarize those, repeatedly, until the corpus collapses into a handful of top nodes. Retrieval draws from every level at once, so a thematic query pulls a summary and a factual query pulls a leaf. The paper's reported result: "By coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20% in absolute accuracy."

One detail in that construction matters more than the benchmark. The clustering is soft, so a chunk carries a probability of belonging to several clusters rather than being filed under exactly one, which means chunks end up with multiple parents. A hierarchy whose nodes have several parents is a graph, and RAPTOR's own mechanism concedes the point the previous section made. Knowledge doesn't partition. Every method that forces it to is buying tidiness with accuracy, and the good ones tell you which one they bought.

The instrument that measures the whole cloud

Retrieval only ever asks local questions of an embedding space: what's near this query, who are this document's neighbours. Nobody asks the global question, which is what shape the whole cloud is in. That question has an answer, and it comes from the same branch of mathematics that gave us the invariant at the top of this part.

The bridge from a pile of points to a measurable shape is one idea. Grow a ball around every point at the same radius, starting at zero. When two balls touch, draw an edge; when a group is mutually connected, fill in the face between them. At any radius you have a structure whose holes the hole-counting machine can compute. Rather than picking a radius, sweep it from zero upward and watch features appear and disappear. Every feature gets a birth radius and a death radius, and the collection of lifespans is a barcode: long bars are structure that persists across scales, short bars are noise that flickered and drowned. Nobody has to choose the scale, which is fortunate, because in a corpus of embeddings nobody knows the scale.

What makes it usable in production is a guarantee. A stability theorem, due to Cohen-Steiner, Edelsbrunner and Harer, says that perturbing the input by at most epsilon moves the barcode by at most epsilon, so no long bar can be created or destroyed by wiggling the data. Set that against the tools most teams already run on embedding clouds. K-means reshuffles whole clusters on a reseed. T-SNE draws a different picture every run and dares you to say which one lied. A barcode is contractually incapable of that behaviour, and there aren't many instruments in applied data analysis that offer a guarantee of that shape.

What it measures maps onto the failures above. The count of connected pieces tells you how many separate islands your knowledge sits in, which is a real answer to "is our documentation one body or six." Loops are chains of concepts that circle back on themselves, which in a corpus usually means one structure being rediscovered in several domains. And voids are the interesting ones: a void is a region the corpus circumnavigates and never fills, a question everything around it implies and nothing in it answers. That's negative space, measured, in the one shape whose defining weakness is negative space.

There's a measured result about language here worth putting in front of anybody who thinks embeddings are semantically complete. Tulchinskii, Kuznetsov, Kushnareva, Cherniavskii, Barannikov, Piontkovskaya, Nikolenko and Burnaev (Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts, arXiv 2306.04723) estimate the intrinsic dimension of a text's embedding cloud, and their abstract reports the average intrinsic dimensionality of fluent natural-language text hovering around 9 for several alphabet-based languages and around 7 for Chinese, with AI-generated text in each language roughly 1.5 lower. That's their published result rather than anything of mine, and it illustrates the point precisely: a global property of the cloud carries information that no individual similarity score contains, and it's hard for a generator to game, because no local edit moves a global invariant very far.

Limits, stated plainly, because this is the least-deployed idea in the article. Tooling for the standard case is mature and Python-native, the computation gets expensive fast as point counts grow, and the discipline is subsampling and landmark points rather than heroics. More to the point, it answers questions about your corpus rather than questions from your user, so it belongs to whoever owns the knowledge base rather than to the retrieval path. Nobody should replace a vector index with a barcode. The instrument tells you what your corpus is shaped like, which is a question your retrieval stack can't ask and your business keeps needing answered.

The graph [summarize]

What traversal buys

A graph stores nodes and typed, directed relationships between them, and its engineering trick is index-free adjacency: each node holds direct references to its own relationships, so following one is a pointer dereference. The cost of a hop scales with the local degree of the node rather than with the size of the graph, which is the mechanical reason depth is cheap here and expensive in the table section.

Fraud is the cleanest worked example. A payments company flags an account and wants every account connected to it by a shared device, phone number, address or funding instrument, out to four hops. In a graph that's one query with one shape, and a fifth hop means changing one number. In a warehouse it's four self-joins against an edge table that keeps growing, a fifth hop means editing the query text, and the planner's estimates get worse with each level. Same data, same answer, and the difference in effort is entirely a property of the shape.

Introductions are the second example, and this one runs in most service businesses. "Who introduced us to the client who introduced us to our largest account, and how many hops away are they?" The answer to that question is a path rather than a set of rows, and graph engines return paths as first-class results. Every other shape in this article can produce the endpoints and loses the route, which matters because the route is what you'd act on: it names the person to thank, and the person to call when you want another one like it.

Underneath both examples is a class of algorithms that exist only here. Reachability, shortest path, cycle detection, community detection, centrality: decades of literature, mature implementations, and all of them awkward at best expressed in SQL. If your questions sound like "connected how," "how far," "through whom" or "is there a route at all," this is your shape and the argument is over.

Property graphs, RDF, and the fact with four participants

Two models dominate. A property graph gives nodes and relationships both a type and a bag of key-value properties, with relationships directed and first-class. RDF makes everything a triple of subject, predicate and object, with globally unique identifiers, which makes merging data across organizations tractable in a way property graphs never quite manage. Pick between them on whether your graph is private and query-driven or shared and vocabulary-driven.

Both rest on binary edges: two endpoints, one relationship. Plenty of real facts have more participants than that. "Alice gave Bob a book on Monday" has a giver, a receiver, a thing and a time, and only two of those four fit the endpoints. A property graph parks the other two as properties on the edge, which works until you want to ask about the book independently, at which point you discover the book is a string on a relationship rather than a node you can traverse to. Anything you store as an edge property has been removed from the graph as a thing and demoted to an attribute of a connection.

Classic RDF can't even do that, since a triple has no room for annotation, so the standard answer is reification: promote the relationship to a node of its own (a GivingEvent) and hang four binary edges off it. That's the correct move and it isn't free. One hop becomes two. Every query and every consumer has to know the intermediate node exists. The intermediate node needs a type, a naming convention and an identity policy, so two engineers modelling the same domain produce two incompatible sets of event nodes. And a reified query read out loud takes four times as long to explain, which is a real cost in a team that has to maintain it.

A hypergraph skips that tax by letting one edge bind arbitrarily many vertices at once, which is the shape a four-participant fact actually has. That's the middle rung of Andy's own ladder, graph then hypergraph then metagraph, and the next section climbs the rest of it. Here's the same trade across all four options.

Direction is the whole signal

Direction is the property this shape preserves and the vector shape destroys, and it's worth sitting with how much of your data is asymmetric to its bones. A citation points. A derivation points: this claim follows from that one, never the reverse. An acquisition points. A payment points. A temporal fact points, earlier to later. In a business graph, the arrow usually carries more of the meaning than the two endpoints do, since knowing that two companies are connected is trivia and knowing which one bought the other is the fact.

Most analysis drops it anyway. The standard machinery for measuring the shape of a network starts by building a structure from the graph, and that construction begins by forgetting which way the arrows point. Run it on your knowledge graph and you've analyzed the graph's shadow, with the most informative structure in the building amputated at the door.

There's a fix, and it's the kind of thing worth knowing exists even if you never build it. Grigor'yan, Lin, Muranov and Yau constructed a homology theory native to directed graphs in 2012. The machinery is a faithful port of the classical version with one change that carries everything: chains are formal sums of directed paths, where every consecutive hop has to be a real edge respecting its real direction, and any face that breaks directedness gets discarded rather than counted. Two networks with identical undirected skeletons and different arrow patterns get different fingerprints, which is exactly the distinction a temporal knowledge graph exists to record. Chowdhury and Memoli later built the persistent version for weighted digraphs, with the same stability guarantee as the barcode from the previous section.

What it finds is operationally interesting. A one-dimensional class in that theory is a directed cycle of relationships that can't be collapsed through anything else in the graph, which means a genuine feedback loop. In a market graph that's a reflexive dynamic captured structurally rather than statistically: sentiment drives price drives sentiment. In an operations graph it's support load driving churn driving support load. In a knowledge graph it's a ring of claims that support each other, which is either a coherent self-supporting argument or a circular dependency wearing that costume, and telling those two apart is a first-class piece of epistemic work. Meanwhile the incidental triangles, three facts that happen to interlock inside a dense region, die in the quotient. The machine sorts your cycles into load-bearing and incidental using nothing but connectivity, with no model and no training run.

The practical limit: there's no off-the-shelf tool for this the way there is for the undirected case. The theory is proven and the algorithms are published, and the implementations are research code, so this is a real engineering effort against a bounded subgraph rather than an install. I'm naming it because the shape argument in this series has a direct consequence here. Direction survives the graph shape and dies in nearly every other one, so if the arrows matter in your domain, the decision to store them is the decision that keeps those questions alive.

What a graph is bad at

Aggregation, badly enough that it's the standard reason a graph deployment gets quietly supplemented within a year. "Revenue by channel by month" over a property graph means walking every relevant node and summing as you go, with no columnar layout to skip untouched attributes, no partition pruning, no precomputed rollups and no optimizer statistics worth the name. Graph engines chase pointers, and a scan-and-aggregate workload is the one shape of work that makes pointer-chasing the wrong primitive. A question a warehouse answers with one scan becomes a walk over millions of nodes, and the gap widens as the graph grows.

Set operations land in the same bucket. Anything the relational optimizer does well (large joins with good statistics, group-by with hash aggregation, window functions over sorted partitions) is work a graph engine does slowly or does not do. Teams discover this in the same order every time: the connection questions get answered brilliantly for six months, finance asks for a monthly report off the same data, and somebody builds a nightly export to a warehouse. That export is correct and it's also the moment the deployment became a hybrid, which is the subject of the last section here.

The second weakness is subtler and it's the doorway to the next shape. Property graphs let you attach properties to an edge, so simple annotation works: a confidence number, a timestamp, a source string. What you can't do is point at a relationship from somewhere else in the graph. You can't say "this relationship was asserted by that document," or "this relationship contradicts that relationship," or "this rule applies to that relationship," because the target of an edge has to be a node and a relationship isn't one. The moment you want to talk about a relationship, you reify it, and then it isn't a relationship any more. That limitation is a modelling ceiling of exactly the kind Part I describes, and it's where the ladder has one more rung.

The metagraph [summarize]

Edges you can point at

The RAG knowledge engines entry already defines a metagraph as a navigable structure of concepts and their relationships, with time, contradiction and provenance as first-class properties. That definition stands and I won't repeat it. What belongs here is the structural move underneath it, because the move is one sentence and it dissolves the ceiling the previous section ended on.

A graph gives nodes and edges. This relates to that. A hypergraph lets one edge bind many participants at once, because a real fact has many participants bound into one event. A metagraph makes the edges themselves addressable: you can point at a relationship, annotate it, connect it to other relationships, and reason about it. Rules about rules. Confidence on the confidence. Where a fact came from, what it depends on, when it was true, what it contradicts.

Technically the property is small. Every element in the structure, node or link alike, has a handle, and anything with a handle can be the subject or object of another link. That single property removes the reification tax from the previous section. You stop promoting a relationship to a node in order to say something about it, because you could already point at it, and the thing you're pointing at is still a relationship rather than a converted copy of one that every query now has to route around.

The consequence worth holding is about what survives the write, which is the question this whole part runs on. A table keeps values. A graph keeps connections. A metagraph keeps the assertion itself as an object, so everything you'd want to say about an assertion has somewhere to live. Who asserted it. What it depends on. When it became true. What contradicts it. Those are the properties every other shape in this article drops on the floor, and they're the ones a business discovers it needed at the exact moment somebody disputes a number.

Where the term comes from

The word gets used loosely enough that it's worth knowing it has a real lineage. In the discrete-mathematics literature (Basu and colleagues) a metagraph is a generalization of a graph where edges connect sets to sets rather than points to points. That's the formal ancestor, and it's older than the current wave of interest by decades.

The usage most people are actually encountering comes from OpenCog. Its Atomspace is a typed structure where every atom, node or link alike, carries a handle and can itself be the subject or object of other links, which is what "edges pointing at edges" means technically. MeTTa programs in that system are themselves metagraphs rewriting metagraphs, so code and knowledge sit in one substrate rather than one being a program that operates on the other.

Ben Goertzel's argument for why that substrate is the right one is worth reading at its actual altitude. The load-bearing paper is Patterns of Cognition: Cognitive Algorithms as Galois Connections Fulfilled by Chronomorphisms On Probabilistically Typed Metagraphs (2021), and behind that title is a specific thesis: the algorithms an intelligence needs (logical inference, evolutionary program learning, pattern mining, clustering, probabilistic modelling) each pair a specification of what would count as a solution with a search that constructs one, each formalizes as the same kind of adjoint pair, and the searches decompose into folds and unfolds over a directed metagraph with probabilistically typed edges. One substrate, one grammar of recursion, many cognitive processes as instances of it.

One calibration note, since claims from this corner of the field circulate at the wrong confidence in both directions. The convergence claim you may have heard attached to this program, that dynamic programming and fluid dynamics and quantum dynamics read as one substrate, is Goertzel's own stated position in his own primary sources under his own name for it, and it rests on pairwise mathematics that's established and older than his use of it. The unified version is a research program awaiting the work that would make it a theorem. Treat it as an attributed and unproven synthesis, which is neither a theorem nor a rumour, and hold the same standard for any vendor who arrives quoting it at you.

What becomes answerable

Three question classes turn from projects into queries, and all three show up in businesses that never intended to build anything exotic.

Retraction propagation. A source you relied on gets retracted, corrected or superseded. Which of our conclusions rest on it, at any depth? In a metagraph each assertion is addressable and carries edges to the assertions and sources it depends on, so this is a traversal over dependency edges and the answer is a list somebody can work through on Thursday. In a property graph you'd need to have reified every assertion in advance, and if you didn't, the question is dead in the way Part I means dead. Any business that publishes numbers has this problem the first time a number turns out to be wrong.

Contradiction held open. Two facts disagree: the CRM says the contract renews in March and the signed PDF says June. Most schemas enforce one value per key, so the system resolves the disagreement at write time, keeps one value and destroys the evidence that there ever was a conflict. A metagraph holds both assertions, links them with a contradiction relationship, and hangs confidence and provenance on each, so the resolution becomes a query rather than an overwrite. Part III is entirely about this and I'll leave the depth there.

Rules about rules. A rule is an addressable object, so a policy can govern which rules apply under which conditions, and a change to a rule is a fact with a date and an author rather than a deploy nobody logged. Compliance work lives here, and so does any pricing engine that has ever had to answer "which version of the discount policy applied to this order."

Andy's own framing of the target state is compact enough to quote: every fact carries a confidence, a provenance (the episode that produced it), and a bitemporal validity window, when it became true and when it was superseded. Model the world once, as a metagraph that knows what it knows, and every asset becomes a view over the same source. That last clause is the bridge into the final section of this part, because "every asset becomes a view" is the only version of the multi-shape story that stays coherent over years.

The better instrument that lost the job

I'd want somebody to make this argument against me before I built one, and it comes from the same mathematics as the rest of this part.

Two great families of invariants exist for measuring shape. Homotopy attacks holes directly, studying the maps themselves with all their structure intact, and it's strictly more sensitive than homology. The demonstration is stark: homology inspects a 2-sphere and reports one piece, no loops, one enclosed void, nothing above that. Homotopy inspects the same sphere and finds an infinite family of genuinely distinct ways to wrap a 3-sphere around it, real structure that shows up in physics, and the coarser instrument is completely blind to it.

So why isn't everyone using the sharp one? Because the sensitivity is purchased with intractability, and the intractability is proven rather than merely observed. Deciding whether an element of a general finitely presented group equals the identity is undecidable in the hard sense, which puts it in the same category as the halting problem: no cleverness fixes it, no hardware fixes it, no scale fixes it. Homology is what you get when you deliberately break the sharper instrument to make it computable. It becomes linear algebra, the linear algebra becomes matrices, and matrices surrender to row reduction.

Working mathematicians priced those two instruments against each other for a century and the result wasn't close. The coarse computable one became the daily driver of geometry, number theory and physics. The sharp one became a specialist discipline that borrows the coarse one's tools to make progress at all. A perfect fingerprint you can never take is worth less than a partial fingerprint you can take at scale, and that's the single most transferable lesson available to anybody choosing a data shape.

Apply it here without flinching. The metagraph is the expressive instrument in this article. It can represent everything the other four represent and several things they can't. That expressiveness is exactly what makes it slower to query, harder to hire for, thinner on tooling, and easy to model two incompatible ways: there's no widely deployed query language with the ergonomics of SQL or Cypher, and modelling freedom means two competent engineers hand you two different models of one domain and both are defensible. The most expressive shape is not automatically the right one, and the graveyard is full of theoretically superior models nobody could compute in finite time.

So, plainly: if you run 12 tables and 40 people and nobody has yet asked a question your schema couldn't answer, don't build this. If you can't name three questions right now that require pointing at a relationship, you're buying expressiveness you won't use and paying for it in query complexity you will. Most businesses should stop at a relational core with a graph projection beside it, and that combination will serve them for a decade.

What earns a metagraph is corpus-scale knowledge work where provenance, contradiction and time are the product rather than metadata about it. Research operations, regulated environments where "which version of this claim did we hold in March" is a filing requirement, and agent systems that have to say where an answer came from and what would change it. In those cases the expressive instrument is the only one that can hold the question, and the cost of running it is the price of being able to ask.

Four questions, five shapes [summarize]

One dataset, four questions

Take a consultancy. It has clients, engagements, people, invoices, documents and introductions, which is a few dozen entities of each kind and nothing exotic anywhere. Four people in that business ask four questions in the same week, and every one of the questions is reasonable.

1. Finance asks: what did we bill, by client, by quarter, for the last two years?
2. Delivery asks: show me this engagement exactly as the client submitted it, every field, including the ones we stopped collecting last year.
3. Sales asks: which past engagements were about the same problem as this new one, even though nobody involved used the same words?
4. The founder asks: who introduced us to the client who introduced us to our largest account, and how many hops back does that chain go?

Same company, same week, same underlying facts. Nothing about the business differs between those four questions. Only the shape you stored the facts in differs, and it decides which of the four cost a query and which cost a quarter.

Which shape wins each one, and why

Each question has a shape where it collapses into one operation, and the reason is mechanical every time.

Run the same four questions against the wrong shape and the failures are the ones this part has already traced. The billing question against a graph is a walk over every invoice node with no columnar scan to help. The as-submitted question against a normalized warehouse is a six-way join that rebuilds a form somebody already had, minus the fields the schema dropped. The same-problem question against a table is a keyword search that misses every engagement whose consultant wrote "throughput" where this one says "backlog." The introduction chain against a document store is an N+1 crawl in application code with no guarantee the ids still resolve.

Reading a table like that is worth something. Watching it happen to your own facts is worth considerably more, which is what the piece below is for.

Which of the four you can still add, and which you can't

Notice who asked. Finance, delivery, sales and the founder, four departments in one week, four different shapes implied. A company running a single shape has silently decided which of its departments gets an answer and which gets a project plan, and nobody in the building experienced that as a decision.

The repair costs differ enormously, and the difference has nothing to do with which store is harder to install. Three of these four questions can be added years late to a business that never planned for them, because the facts they need survived in some form. One can't.

• Similarity is always recoverable. The engagement write-ups still exist as prose, so embedding them is a weekend of work whenever somebody finally asks. Text that was stored is text you can re-shape.
• Aggregation is usually recoverable. Invoices exist as rows with amounts and dates, so building the warehouse is a modelling exercise. It stops being recoverable at exactly the point the table section described: whatever grain you rolled up and then deleted.
• Whole-record depends on one earlier decision. If you kept the submitted form, loading it into a document store is straightforward. If your ingest normalized it and dropped the fields nobody mapped, the record as submitted no longer exists anywhere, and no store recovers it.
• The introduction chain is the one that dies. Nobody records introductions. An introduction happens as a dinner, an email thread, or a sentence in a call nobody transcribed, and none of those lands in a table.

That last one deserves the detail, because it's the pattern that repeats across every business I've audited. There's no field anywhere that says this client came from that client. Install a graph database and it arrives empty of exactly the edges you wanted, because a traversal engine traverses relationships that were recorded and invents none. What you can do is approximate: mine the CRM notes with a model, guess from account creation timing, ask the partners what they remember. Every one of those produces a link with an error rate rather than a fact, which is the domain Fellegi and Sunter formalized in A Theory for Record Linkage (JASA 64(328), 1969), and their framework is explicit that under optimal probabilistic matching some links stay uncertain and have to be declared as possible matches carrying stated error rates.

So the practical version of this whole part fits in one question to ask when somebody proposes a new store. Are we missing a shape, or are we missing a fact? A missing shape costs a sprint. A missing fact costs a guess with an error rate attached, forever, and the guess is what gets presented to the board as a number.

The fifth question

Here's the one the table above can't answer anywhere. Which of the claims in our published case studies rest on a number we later corrected?

Trace it through the five shapes. A table holds the current number, so the corrected value overwrote the original and nothing records that a claim depended on it. A document holds the case study as written, with the number embedded in a sentence. A vector index finds case studies that talk about numbers, which is useless. A graph can connect a case study to a metric node, and the moment you want to say "this claim depends on that value as of that date, and the value was superseded," you're pointing at a relationship from outside it, which is the wall the graph section ended on. Only the fifth shape holds the question, and it holds it because an assertion there is an object with edges to its supports.

That question class is where this series goes next. Time, provenance and contradiction are the three properties every system defers and none can retrofit, because the data that would support them stops existing the moment somebody overwrites a row. Part III is about all three, and about the ladder that turns lineage into a valuation of your own data.

The hybrid that actually ships [summarize]

One write path, many projections

Every production system I've worked on runs three or four of these shapes at once, so the choice was never which shape. The decision that matters is which shape owns the write path and which shapes are derived views of it. Teams that make that decision explicitly stay coherent for years. Teams that don't get there by accretion and spend those same years in reconciliation meetings.

The write path belongs to whichever shape can enforce the invariants you can't afford to break. For most businesses that's a relational core, because it's the only shape here that refuses a write that breaks a relationship. For systems where the sequence of what happened is the product, it's an append-only log, which is Part III's subject. Either way the choice is made on enforcement rather than on query convenience, because convenience is what projections are for.

What keeps the projections in agreement is defining the data once, above all of the stores. Andy's rule for this is worth quoting because it's stricter than how most teams work:

Everything starts as Pydantic as IR. The first build artifact of any system is the Pydantic data catalog. Storage backends, API layers, frontends, and agents all derive from it; none of them get to redefine what the data is.

That last clause carries the weight. When each store owns its own definition of a customer, you have four definitions and a standing argument. When one typed model defines it and every store is an adapter, a change to the model breaks the consumers at the type checker rather than in a quarterly report six months later. The intelligence engineering entry covers the pattern in depth, and the RAG entry shows the retrieval-side version, where lexical, dense and graph stacks sit behind one API and one reranker.

The test for a derived view

One question separates a projection from a second system of record. Could you delete this store entirely and rebuild it from the source, with a command that actually runs today? If yes, it's a view and you can treat it casually. If no, it's authoritative whether or not anybody decided that, and it needs the backups, the access control and the audit trail you gave the database you think of as primary.

Most vector indexes fail that test, and the failures are boringly consistent. Somebody enriched the chunks during ingestion with a step that lives in a notebook. The source PDFs got cleaned up after the load. The chunker changed twice and nobody recorded which version produced which vectors. Now the index holds information that exists nowhere else, and it's a system of record that nobody backs up, nobody can audit, and nobody can reproduce.

Two habits stop a projection from becoming a system of record by accident, and both are cheap:

• The rebuild runs on a schedule. Rebuild a sample in CI, weekly, and compare it against the live store. A rebuild path that hasn't executed in six months doesn't exist, in the same way an untested backup doesn't exist.
• Every projection has an owner and a staleness budget. Name how far behind the source it's allowed to run, measure it, and alert on the breach. Unmeasured staleness turns into "the graph says one thing and the warehouse says another," and that sentence has never once been resolved in the meeting where it was first said.

The data pipelines entry defines each stage by the shape the data takes as it moves, which is the operational companion to this section.

The failure where everything is authoritative

Here's the shape of the failure, and most readers will recognize it. The CRM holds a customer record. Billing holds a customer record. The warehouse holds a customer dimension. Each has an edit surface, each has a team that treats it as the system of record, and the three disagree about the same customer's name, plan and start date. Nobody chose this. It happened one integration at a time, and each integration was correct on its own terms.

The cost shows up in three places. There's the reconciliation meeting, which recurs monthly and never converges. There's the number nobody can defend, which is what turns a board deck into an argument about data quality instead of a decision. And there's the analyst who keeps a private spreadsheet because they've learned not to trust any of the three, which is a person heroically compensating for a structural failure, and the most reliable signal that one exists.

The expensive answer is a master data management program. The cheap answer, which works, is to assign the write path per entity rather than per system. Customers get written in exactly one place. Invoices get written in exactly one place. Engagements get written in exactly one place. Every other surface reads, and any surface that needs to change an entity it doesn't own submits an event to the owner instead of writing directly. That's a week of interface work and a month of arguing about who owns what, and the arguing is the valuable part, because the argument is what surfaces the three teams who each believed they owned customers.

What to do on Tuesday

A procedure, in the order it actually works:

1. Write down five questions the business needs answered this year. Not the queries you already run. The questions people ask in meetings and don't get answers to.
2. Classify each one as aggregation, whole-record, similarity, path, or claim-about-a-claim.
3. Match each class to the shape that serves it and check which of those shapes you actually have.
4. For every class you can't serve, ask Part I's question before you buy anything: is the underlying fact captured at all? A missing shape is a weekend of work. A missing fact is unanswerable at any budget, and no store you install this quarter changes that.
5. Pick the write path by invariants, then make every other shape rebuildable with a command that runs in CI on a schedule.

Step four is the one people skip, and it's the one that saves the money. Half the shape projects I've seen were procurement exercises aimed at a question whose underlying event was never recorded, which means the new store arrived, got populated with the same collapsed data, and answered exactly as poorly as the old one while costing a migration.

And the disqualifier, out loud, because this part has spent 11,000 words making shapes sound consequential. If nobody in your company has asked a question your current store couldn't answer, you don't have a shape problem. You have a reporting backlog, and adding a second store will lengthen it. Fix the backlog, keep one database, declare your grain properly, and come back when somebody asks a question that gets a shrug instead of an answer. That's the moment this article becomes worth acting on, and it arrives on its own schedule.

The through-line, in one line: a shape is a transformation you apply to reality on the way in, and the properties that survive it are the only ones you'll ever be able to ask about. That's the whole argument. What comes next is the three properties almost everybody assumes they can add later.