The Shape of Data: How Five Representations Answer Different Questions. A daily revenue total can't reveal which products shared a customer's basket.

A daily revenue total can't tell you which products shared a customer's basket. That information lives in the order lines and their shared order identifier. A graph, a vector index and a larger warehouse all face the same limit once those records are gone. Choosing a data shape means deciding which distinctions to retain, which questions to make convenient, and which obligations the system must enforce.

This is part two of The Shape of Data. Part I explains the ceiling that capture decisions set. Here the question is how that ceiling changes across tables, documents, vectors, graphs and metagraphs. The names describe representations. A single database product may support several of them.

A different store can't recover a discarded fact[summarize]

A revenue total omits basket membership

Consider an illustrative subscription shop selling a starter kit and a refill. Its source records identify each order, each product and each line's amount. Finance can total revenue by day from those records. Merchandising can also ask which orders contained both products, because the shared order identifier connects the lines.

Now suppose the shop keeps only daily revenue by channel and deletes the order lines. Finance's report continues to work. The basket question has lost its evidence. Several different collections of baskets can produce the same daily total, and nothing in that total identifies which collection occurred.

The database hasn't failed. It returns the information it holds. The loss happened when the pipeline kept a sum and discarded the distinctions needed for another question. Installing a new store changes how the surviving records can be queried. It doesn't tell the shop how customers combined the products.

A retained source export could change that answer. So could an order archive elsewhere in the business. The boundary is what the whole system preserved, not what happens to be available in the reporting table. Check for that evidence before declaring a question permanently unanswerable.

Preservation gives the comparison a useful test

Topology offers a way to think about preservation. A coffee mug and a doughnut can count as the same kind of shape under an idealized set of transformations that permits stretching and bending but forbids tearing or gluing. The handle's hole survives those allowed changes. Length and curvature don't.

An invariant is a property that stays unchanged under the transformations being considered. Different values of an invariant can prove that two shapes differ. Matching values don't necessarily prove that the shapes are the same. A fingerprint can miss distinctions.

The data-design question follows the same pattern. Which changes should leave a fact meaningfully unchanged? Which distinctions must survive because someone will act differently when they differ? Renaming a display label may leave a customer identity intact. Merging two customers because their labels match can destroy the very distinction a billing question needs.

A schema and its ingestion rules select what survives. An order identifier preserves basket membership. A nested record preserves which submitted fields belonged together. A directed relationship preserves who introduced whom. An assertion identifier lets another record refer to a particular claim, including the claim later found to be wrong.

This analogy has a limit. A database isn't literally a topological space, and a schema isn't a theorem proving that every useful property survived. The analogy supplies a design question. The schema, retained sources and actual queries have to answer it.

An awkward question can fail in different ways

A team that says a store is bad at a question should name the failure. The remedy changes with it.

The question may be native to the chosen representation and access path. Revenue grouped by channel fits a warehouse's scan-and-aggregate workload. A source record retrieved by its key fits a document read. Those operations can still be slow when the implementation is wrong; native describes the fit, not a performance guarantee.

The question may be expressible but expensive. Recursive SQL can follow a chain of suppliers through an unknown number of levels. Indexing, fan-out, cycles and the requested result determine the cost. The existence of a graph-shaped problem doesn't establish that a second database is necessary.

A relationship may be recorded without an adequate enforcement rule. An application can write a project identifier into a person's document even when nothing checks that the project exists. A successful write then proves only that the value was accepted. The integrity obligation remains somewhere else.

Finally, the evidence may be absent. A daily total without order membership can't answer the basket question. An introduction path without recorded introductions can't establish who introduced an account. More compute won't distinguish histories that the retained data makes identical.

Those failures shouldn't share one shopping list. A slow query needs a cost diagnosis. An unenforced relationship needs an integrity rule and an owner. Missing evidence needs better capture, a surviving source, or an explicitly uncertain reconstruction.

Five shapes preserve different useful distinctions

The comparison below names each representation's useful emphasis. None of the rows grants a product exclusive rights to the operation.

More expressive modelling can retain more distinctions, but these shapes don't form a simple ranking. A vector representation specializes in a different operation from a table. A metagraph doesn't become a better revenue warehouse because it can describe an assertion's provenance. The choice starts with a question and the evidence it needs.

A table makes that relationship unusually visible. One row has a declared meaning, and changing that meaning changes what can be counted.

Tables make grain and constraints explicit[summarize]

A safe split must reconstruct the same facts

A relational table organizes rows under named attributes. Keys distinguish records, and declared constraints restrict the states the database will accept. E. F. Codd's relational model separates that logical organization from physical access paths. A table's meaning can remain stable while its indexes or storage layout change.

That separation is useful when discussing shape. Adding an index can improve a query without recovering any missing attribute. Changing what a row represents can change a query's result even when every query still runs.

Normalization concerns how to divide relations without creating misleading combinations or unnecessary repetition. Suppose, in an illustration, an order record contains an order identifier, customer identifier and customer name. Separating customer details from orders can avoid repeating the same name for every order. The customer key preserves the connection.

Splitting on an attribute that doesn't identify the intended match can create false combinations. Two people can share a name. A join that uses the name as though it were a unique identifier can combine an order with the wrong person. Each resulting row is syntactically valid. Its claim about the business is wrong.

A lossless decomposition guarantees that joining the pieces reconstructs the original relation without such extra combinations. For a binary decomposition governed by functional dependencies, the shared attributes determining one of the component relations is the familiar condition. A functional dependency says that a value, or combination of values, determines another value. A customer key determining the customer's recorded name is one example.

Dependency preservation asks a different question. Can the relevant dependencies still be checked on the decomposed relations without reconstructing their join? A split can preserve information while making a rule harder to enforce locally. Keeping these tests separate prevents a design review from treating every normalized schema as interchangeable.

Normal forms address particular kinds of dependence

Third normal form and Boyce-Codd normal form address functional dependencies at different levels of restriction. A suitable third-normal-form synthesis can preserve dependencies and permit a lossless join. A lossless Boyce-Codd decomposition can sacrifice dependency preservation. The trade concerns which dependencies remain locally checkable, not whether the designer finished enough normalization homework.

Fourth normal form addresses multivalued dependencies. Think of independent sets of attributes being repeated together in combinations. Fifth normal form addresses join dependencies that require a more general decomposition. These names identify different anomaly conditions; they aren't a universal ladder to climb regardless of workload.

The practical review is smaller than a tour of every normal form. Identify the facts that repeat, the keys that relate them, and the rules that must remain true. Then check what the proposed split preserves and how the application will enforce the remaining obligations.

Denormalization needs the same scrutiny. Joining data into a wider record can be reversible when the source keys and values survive. Keeping a materialized view beside authoritative detail can be a sensible read optimization. The destructive changes are more specific: losing keys, replacing detail with an aggregate, or allowing copies of one fact to diverge without an authority rule.

Calling all of those changes denormalization hides their different risks. The basket example loses answerability because it discards membership. A wide table retaining the order and line identifiers hasn't necessarily done that. Inspect the transformation instead of deciding from the label.

Grain determines which distinctions remain countable

The grain of a fact table states what one row means. An order line, an order and a customer-day describe different populations. A query can sum each of them correctly and still answer a different question from the one the business asked.

Revenue by channel by week fits a fact table with the required amounts, dates and channel relationships. A warehouse can scan the needed columns, join dimensions and group the result. Columnar storage, partition pruning and statistics can reduce that work. The physical choices matter, but they work on the distinctions the logical model retained.

Counting customers from order lines needs an identity rule. Counting orders needs an order key. Comparing new and returning customers needs a definition and enough history to apply it. A report heading doesn't supply those facts. Each one is a requirement on the records underneath the calculation.

The grain commitment is irreversible only where finer evidence has been discarded everywhere. A daily summary backed by immutable order lines remains a view. A daily summary that becomes the only surviving record has made the total the maximum available resolution.

Relational constraints can protect part of this meaning at the shared write boundary. A foreign key can reject an order referencing an absent customer. A unique constraint can prevent duplicate invoice identifiers. A check constraint can reject an invalid quantity. These protections depend on declaring and enabling the appropriate rules. A relational database doesn't infer the business's invariants from column names.

The important benefit is that all writers using that protected boundary face the same rule. An agent, an import job and an application can't each adopt a private interpretation of whether a customer reference is valid. Other database models can also provide transactions or constraints; evaluate the actual engine's guarantees instead of treating shape as a product specification.

Tables can traverse paths, but the workload changes

SQL can express variable-depth traversal through recursive common table expressions. A supplier chain doesn't become inexpressible merely because its depth is unknown. A recursive query repeatedly extends a frontier until a stopping condition is met.

The cost depends on indexes, local fan-out, depth, cycle handling and how much of the result the query materializes. It isn't inevitably a full-table scan at every level. A small, well-indexed relationship table may serve the question adequately. A broad traversal with many possible paths may require substantial work in either a relational or graph engine.

A closure table can precompute reachability. That exchanges read work for storage and maintenance work. The team must define how inserts, removals and corrections update the closure, and whether the table records reachability or actual distinct paths. Those are different answers.

Ragged records create another trade. Sparse columns, related subtype tables and JSON columns can each be reasonable. JSON doesn't automatically lose every form of validation, and a typed column doesn't automatically capture the meaning of an unfamiliar field. The design should specify which attributes need database enforcement and which need flexible preservation.

I start with platform exports, map them into a standardized schema and analyze them, sometimes using a local SQLite database. The useful habit is defining the records before calculating over them. The named store can change without changing that discipline.

A table can also preserve history, sources and conflicting assertions when they're modelled explicitly. A current-value row alone doesn't do so. That's a reason to inspect the schema, and a reason to keep the distinction between a model's capabilities and the way a particular application uses it.

Documents make a different choice conspicuous: which pieces of a record belong together on a read and a write.

Documents keep a record together[summarize]

The read can match the submitted record

An order confirmation arrives with a structure. It has line items, delivery details, a tax breakdown and a payment result. A document can keep those related fields nested under one identifier, ready to retrieve together.

A normalized relational model might distribute that information across several relations. The application assembles a confirmation by joining the pieces. Neither representation is automatically more faithful. The difference is whether the normal access pattern wants the assembled record or independently managed facts.

Locality is the document's useful emphasis. A read can retrieve the information stored together without resolving every nested piece through a separate lookup. The document boundary also gives the designer an explicit unit to consider for updates, caching and consistency.

The physical behavior varies by engine. Partial updates, indexes and storage layouts complicate any claim that every document store always reads and writes a whole tree. The design reason survives that variation: information used together can be modelled together.

A submitted engagement form is another good fit. Different clients may submit different fields, and a form may evolve. Keeping the original record can preserve an older field even after the current application stops asking for it. A new required field in today's interface doesn't have to erase what yesterday's client actually supplied.

Preserving a submitted form and maintaining the current customer record are different jobs. An address on a historical order may need to remain the address used for that order. The customer's current address can change independently. Treating both as duplicate copies of one current fact creates unnecessary confusion about which one should update.

The boundary needs an explicit answer: is this a snapshot of an event, or the current state of an entity? A document can represent either, but its nesting alone doesn't tell a reader which one it is.

Flexible fields still create obligations for readers

A record can omit a field without requiring every old record to be rewritten immediately. That flexibility is useful for product catalogues, intake forms and other uneven inputs. It also means consumers must handle the records they actually receive.

A reporting job may expect total. A mobile client may expect a string at address.zip. An export may expect an uppercase currency code. Those expectations form a schema even when the database accepts additional fields.

Changing a field name can therefore be a migration distributed across readers. Some will fail visibly. Others may substitute a default or omit an amount, leaving an apparently successful report with different contents. The absence of a database migration command doesn't remove the migration work.

Record versions, validation at the write boundary and explicit consumer contracts help make that work inspectable. The right combination depends on the system. A document model can coexist with schema validation, typed application models and database transactions. Flexibility describes which changes the representation accommodates; it doesn't exempt those changes from coordination.

Consistency has the same boundary question. Updating an order and its nested lines as one atomic unit can prevent a reader from observing a partially applied update. Atomicity alone doesn't establish that the arithmetic is correct. A transaction can atomically store a wrong total unless a validation rule rejects it.

An invariant spanning records needs guarantees that actually span those records. Some engines provide suitable transactions or other mechanisms. Others leave the application to coordinate updates. The shape doesn't settle the matter. Name the invariant, the enforcement point and what happens during a failed or concurrent write.

A tree has to choose where a shared thing belongs

Consider an illustrative consultancy with people, projects and clients. Nesting people inside projects makes the project roster easy to retrieve. A question about every project a contractor has touched now crosses those project documents.

Nesting projects inside people reverses the convenience. Duplicating both directions makes both reads convenient, but the relationship now has two editable representations. Adding the contractor to a project and adding the project to the contractor must stay in agreement.

The tree's difficulty is the shared entity. One contractor can participate in several projects, and a project can involve several contractors. A nesting hierarchy has one parent position for each nested instance. Representing the many-to-many relationship requires repetition or a reference beyond that position.

Knowledge collections have the same issue. A concept can belong in several subjects and connect to concepts outside its filing branch. The location where a document is stored can be a good retrieval aid without being a complete representation of its ideas' relationships. A folder hierarchy and a concept network answer different questions.

References require a shared integrity policy

Storing an identifier instead of another full copy is often the right response. A project document can refer to a person kept in one authoritative record. The application then resolves the identifier when it needs the person's details.

That choice creates a join operation, wherever the operation runs. It may happen inside the database, through batched reads, or in application code. A naive loop can cause a follow-up request per reference. Batching or an engine-supported lookup can change that cost. Count the actual access pattern rather than assuming every reference requires a network round trip.

The integrity question remains separate. What stops a project referring to a person who has been deleted? What updates both sides when the system stores both directions? What distinguishes a historical reference that must survive from an invalid current membership?

Several designs can answer those questions. One authoritative membership record avoids independently editable copies. An appropriate transaction can coordinate changes. A relational membership table can enforce references, with a document view assembled for the read. A graph projection can make repeated relationship questions convenient.

Each option has an operating cost. Duplicated facts need a synchronization rule. References need resolution and integrity handling. A second store needs a rebuild path and a freshness policy. The apparently small choice to copy a field becomes expensive when nobody owns its meaning after the first write.

A document remains a sensible choice when its boundaries fit the main reads, the important invariants are protected, and cross-record relationships have a manageable policy. A new relationship question calls for inspecting those boundaries. It doesn't make the records already preserved less useful.

Vectors offer a different kind of convenience. They can retrieve a relevant record when the person asking and the person writing used different words, but relevance still leaves the record's exact claims to be checked.

Vectors retrieve candidates without establishing their truth[summarize]

Different wording can describe the same problem

An embedding model maps an input to numerical coordinates. A vector index uses a distance or similarity function to find nearby stored representations. The model determines which distinctions influence those coordinates; the index determines how it searches them.

Consider an illustrative support query: a customer writes that a device won't charge overnight. A manual describes a unit failing to draw power while docked. A retrieval system that relies only on exact word overlap may miss the connection. An embedding model trained to associate related expressions may bring the relevant passage into the results.

That's a useful capability even when the model is imperfect. Sales notes, support tickets and technical documentation don't share one controlled vocabulary. Semantic recall can find candidate material without requiring every writer to anticipate every reader's wording.

The output is still a candidate. A nearby passage may concern a different device, an outdated procedure or the opposite condition. The system needs the original text and any applicable product, date or access restrictions to decide whether the passage can support an answer.

I built on Archon and changed ingestion for different document types, using semantic chunking and task-specific embeddings in a local retrieval workflow. That work dealt with preparing the material before the agent searched it. A chunking strategy still needs to be judged against the documents and questions it will serve.

The transferable lesson is that retrieval begins before the search call. An embedding of a severed explanation differs from an embedding of the explanation with its condition intact. Choosing document boundaries, keeping source references and preserving the original text remain modelling work even when the index accepts every vector successfully.

A similar topic can contain a different instruction

These illustrative contract clauses concern the same subject:

• Liability is capped at two times fees paid.
• Liability isn't capped.
• Liability is capped at two times fees paid, except in cases of gross negligence.

Their shared vocabulary doesn't settle their legal effect. A semantic search might retrieve several of them, and a particular encoder might distinguish them well or poorly. No scores have been measured here. The example shows why a topic-level match can't be accepted as a determination of which condition applies.

A similarity threshold doesn't turn that ranking into entailment. The closest available passage can still fail to answer the question. The system must be able to say that it hasn't found adequate evidence, and it must inspect the exceptions that determine which answer applies.

An embedding is also a different representation from its source. A typical retrieval pipeline doesn't provide a lossless inverse that reconstructs the original document from its coordinates. Even where some input information can be inferred from an embedding, that isn't a reliable substitute for retaining the source. The operational rule is to store the original, its identity and the transformation details needed to reproduce the derived representation.

The preservation question is useful again. Does the pipeline retain negation in the source text? Does it attach the right effective date? Can retrieval return the full clause instead of a chunk ending before the exception? Those are concrete checks. Declaring all meaning preserved because a vector has many dimensions avoids them.

Symmetry describes the comparison, not word order inside a sentence

Cosine similarity is symmetric. Comparing vector A with vector B gives the same score as comparing B with A. That mathematical property doesn't imply that an encoder gives the sentences "A caused B" and "B caused A" identical vectors.

The encoder can react to word order. The resulting vectors can differ, and each can receive a different score against a third query vector. Reversing the arguments of a similarity function and reversing the words inside an input are different operations.

A symmetric proximity score alone doesn't specify a directed business relationship. A large similarity between two company representations doesn't tell the system which company acquired the other. That fact needs to be encoded and interpreted somewhere, such as in source text, a typed relationship or a model designed to score directed relations.

Knowledge-graph embedding methods make this design choice explicit. TransE represents a relation as a translation between entity vectors. DistMult uses a symmetric diagonal bilinear score. ComplEx uses complex-valued representations to support asymmetric scoring. Their differing mathematical forms shouldn't be collapsed into a claim that all embeddings behave like plain document similarity. The relevant review question is what the actual score means and which asymmetries the chosen model can represent.

Negation and quantity need the same care. A passage saying all suppliers passed an audit differs from one saying some did. A retriever can help locate the statements, but answering still requires inspecting their quantifiers, scope and evidence. Model quality can improve this task without changing the responsibility to verify it.

Preserving distance doesn't prove that distance means the right thing

The Johnson-Lindenstrauss result concerns reducing the dimensionality of a finite collection of points while approximately preserving pairwise Euclidean distances. Its guarantee starts from distances already defined in the original space.

That's different from proving that the original distances faithfully represent contractual meaning, authority or logical implication. If an encoder puts two materially different clauses near each other, a distance-preserving reduction can keep them near each other. The reduction hasn't repaired the encoding decision.

The distinction also applies without dimensionality reduction. A retrieval benchmark measures performance on a specified task and population. It can't establish that every future question can be answered by the nearest passage. A strong score on matching paraphrases says something useful about paraphrases. It doesn't supply a manifest for counting omissions or a dependency model for following a chain of ownership.

A vector index earns its place when approximate semantic recall is the job and the surrounding system preserves the material needed to judge the candidates. Exact filters, lexical retrieval, reranking and source inspection can all contribute. The resulting system is already combining representations.

Some questions need more than better candidate ranking because their answer isn't contained in one local passage at all.

A nearest passage can't answer every corpus question[summarize]

A path needs evidence from several records

Top-k retrieval returns a bounded set of highly ranked candidates. That procedure is useful when a relevant passage contains an answer. It supplies no guarantee that the returned set contains every fact needed for a more complex question.

Consider an illustrative procurement question: which suppliers are owned by a blacklisted company? One record identifies a supplier's parent. Another identifies the parent's blacklist status. The answer depends on joining those facts through a correctly resolved company identity.

A search may retrieve both by good fortune or by careful query construction. The retrieval mechanism alone doesn't establish the connection. A multi-step system can retrieve an ownership record, extract a parent identifier, look up the parent's status and then inspect the supporting sources. A graph can make the recorded path explicit. Both approaches require the evidence and the identity mapping to exist.

Increasing k can improve coverage while also adding irrelevant material. It doesn't guarantee that a chain has been completed. The stopping condition needs to concern the question: have the required ownership links and status records been established, and are they applicable at the relevant time?

HotpotQA, MuSiQue and the multi-hop dataset described by Ho and colleagues test questions assembled from multiple facts. That research direction is useful here because it separates finding a plausible passage from completing a reasoning chain. No benchmark result is being claimed for this consultancy illustration or for my own retrieval system.

A global answer needs a view of the collection

A question about the main themes in a documentation corpus asks for coverage across the collection. Returning the passages closest to the phrase "main themes" can overrepresent one region and omit another. The ranked sample and the whole collection are different populations.

A global answer needs a method for surveying that population. The system might aggregate over a document manifest, group material, summarize groups and inspect how well those groups cover the sources. Each step creates another opportunity to omit or misstate something, so the resulting summaries need source links and a way to inspect their scope.

GraphRAG develops community summaries for global, query-focused answers. Its significance for this comparison is the change in the represented information. A system creates and stores a view above individual passages, then uses that view when the question needs broader coverage.

That doesn't make every summary correct or every community boundary meaningful. A method that compresses a corpus must still explain what it compressed, which sources it used and how an operator can check an answer. Adding a graph doesn't remove the obligations inherited from summarization.

Absence needs a complete expected set

Now consider an illustrative policy register. An operator asks which policies have no recorded review. An exact answer requires the expected set of policies and the set with qualifying review records. The operation is a set difference.

A search across policy prose doesn't necessarily know the first set. It may omit an old policy, an inaccessible document or an item never ingested. A search across review prose doesn't necessarily know which records qualify, whether they refer to the same policy version, or whether every review was captured.

An answer's wording needs to match those limits. "No review recorded in this register" is supportable when the register has no matching record. "Never reviewed" also requires confidence that reviews couldn't have occurred outside the captured process. That's a claim about completeness, not a similarity threshold.

A relational anti-join is a natural way to compute the difference once both populations are defined. A graph or an application can perform equivalent reasoning if it preserves the same identities and completeness assumptions. The representation doesn't confer knowledge of things nobody recorded.

This distinction prevents a subtle substitution. A relevant passage about review policy may be good advice, but it isn't a list of missing reviews. Any answer must be evaluated against the operation requested, even when the returned prose reads fluently.

A book's argument needs more than its closest paragraphs

A whole-document question has another scope. The reader may want the progression of an argument: what problem the author starts with, which premises accumulate, which objections change the conclusion, and where the reasoning ends. A few locally similar passages can miss that progression.

One approach is to preserve the document's structure and read it at suitable levels. Section summaries can point back to paragraphs. A broader summary can point to those sections. The reader can then move between a whole-document interpretation and the specific evidence that supports it.

RAPTOR recursively clusters and summarizes material to create several levels available for retrieval. It illustrates how a system can manufacture an additional level of representation instead of expecting a flat passage index to provide it automatically.

The distinction between a tree and overlapping membership matters here. A passage can contribute to several themes. A design that permits multiple memberships can represent that overlap instead of forcing the passage into one exclusive branch. The structure may then be graph-like even when the interface presents a tidy hierarchy.

The derived material adds maintenance work. A changed source can invalidate a passage summary and the summaries built above it. A generated summary can also introduce a claim its source didn't make. Keeping the underlying passages and dependency links allows a reviewer to trace those changes rather than treating the generated layer as independent evidence.

These question classes suggest different repairs. A path question needs connected evidence. A corpus-wide question needs coverage. An absence question needs an authoritative expected set. A whole-document question needs enough structure and scope to preserve the argument.

A retrieval system can combine all of those mechanisms. Calling the result vector search doesn't make the extra work disappear, and calling it a knowledge graph doesn't prove that the work has been done.

There's another global question behind this discussion: what does the collection's geometry look like before anyone asks it for an answer? That question leads back to topology, with a different role for the result.

The whole embedding cloud has a measurable shape[summarize]

A local score leaves the collection out of view

A similarity query describes a relationship between a query representation and candidate representations. It doesn't describe the geometry of the collection as a whole. A corpus can contain separated groups, connecting regions or loops that no individual query score reports.

Topological data analysis studies such properties across scales. The connection to the earlier mug-and-doughnut analogy is preservation: compute a description that ignores some small changes while retaining structures of interest. The object being measured is now a point cloud derived from documents.

One construction starts by placing a ball around each point and increasing their common radius. At small radii the points sit apart. As the radius grows, nearby regions connect, loops can form and empty regions can later fill.

There are different ways to turn those neighborhoods into a mathematical complex. A construction based on common intersections of balls and one based on pairwise distances aren't identical. The chosen construction and its distance rule determine what's being measured. A picture of growing circles is an explanation of the idea, not a complete algorithm specification.

Persistence records when a feature appears and disappears

Persistent homology tracks features as the scale changes. A feature has a birth and, when it ceases to persist, a death within the filtration. A barcode or persistence diagram records those intervals.

A feature that persists across a wide range of the chosen scale is less dependent on one arbitrary threshold than a feature visible only briefly. That makes persistence useful for finding candidates for further inspection. It doesn't establish that every long-lived feature is meaningful or that every short-lived one is noise.

A long-lived separation in an embedding corpus might reflect a real division between subjects. It might also reflect language, writing style, source formatting or the behavior of the encoder. A brief feature could be a rare, relevant topic. Geometry provides evidence about the representation, and interpreting it as knowledge requires another step.

Stability theorems make this family of instruments more useful. Under specified assumptions and suitable measures of perturbation, small changes to the input function or space bound changes in a persistence diagram. The exact guarantee depends on the construction and theorem. It isn't a blanket promise that editing any document leaves every semantic conclusion unchanged.

Cohen-Steiner, Edelsbrunner and Harer's stability result formalizes this kind of bounded change. Its practical lesson is to ask for the perturbation model and the distance used to compare diagrams before quoting a stability guarantee. A theorem with those conditions has a different evidential role from a visually consistent plot.

A geometric void doesn't prove a missing answer

Connected components can reveal separated regions at a chosen scale. An operator might inspect whether those regions correspond to separate document collections, vocabularies or access boundaries. That can inform a corpus audit without deciding its result in advance.

Loops and higher-dimensional holes also describe the constructed space. A void surrounded by embedded documents might suggest a topic worth investigating. It doesn't identify a missing policy, prove that a question is unanswered, or supply the expected set needed for an absence query.

The policy-register problem still needs its manifest. The topological instrument can suggest where to look; the register establishes which required item lacks a recorded review. Substituting one for the other would confuse a research lead with an operational finding.

Research by Tulchinskii and colleagues uses intrinsic dimension to study generated language through global geometric properties. Such a result concerns a defined dataset, embedding method and estimator. It doesn't establish a universal authorship detector or a property that a generator can't change. No dimensions or detection performance are being claimed here.

The question to carry into a design review is therefore specific: what decision would change if this measurement changed, and what additional evidence would make the interpretation defensible? A chart whose geometry can't be connected to a decision may remain interesting research without belonging in the production retrieval path.

The instrument belongs beside retrieval

Computing these structures can become expensive as the collection and the requested dimensional detail grow. Subsampling and landmark methods can make an experiment tractable, while changing which features it can detect. Record that choice instead of presenting the result as a complete measurement of the original corpus.

The measurement also depends on representation choices upstream. Changing the embedding model, chunking scheme or normalization can alter the geometry. Comparing two barcodes is hard to interpret if the pipeline changed and nobody retained the earlier transformation details.

A bounded corpus experiment should therefore name its source population, encoder, metric, construction, scale policy and sampling method. It should show the records behind an interpreted feature. That lets an operator reopen the same question rather than inherit a plot with no reproducible meaning.

This is an analytical instrument for the person maintaining the knowledge base. It can complement checks for coverage, duplication and disconnected collections. It doesn't replace a vector index, a source manifest or the reasoning needed to answer a user's question.

The representation has retained geometry that local retrieval wasn't asking about. Graphs make another retained property explicit: the named, directed connections between particular things.

Graphs preserve named connections and return paths[summarize]

The route can be the useful answer

A graph represents entities as nodes and relationships as edges. A typed, directed edge can distinguish a supplier owned by a parent company from a parent company owned by the supplier. The connection has a meaning that proximity alone doesn't specify.

Graph-oriented storage can make following those relationships convenient. An engine with index-free adjacency keeps direct access to a node's relationships, reducing the need to rediscover them through a general join at every step. That's an implementation technique, not a guarantee attached to every graph product.

Local access doesn't make every traversal cheap. The number of relationships examined grows with fan-out and the requested depth. Returning every possible path can be expensive even when each hop is efficient. A query needs a stopping rule, a policy for repeated nodes and a clear definition of what counts as a result.

An illustrative fraud investigation makes the distinction concrete. A flagged account shares a device, phone number or funding instrument with other accounts. An investigator wants the connected accounts and the evidence for the connection within a chosen number of hops.

The graph can return the paths as well as the endpoints. That's useful because sharing a household device and sharing a payment instrument may deserve different interpretations. A connected account is a candidate for investigation, not proof of fraud. Recording the relationship type and its source lets a human inspect why the account appeared.

Introductions give the same operation a different business purpose. A founder asks who introduced the client that later introduced a large account. The path identifies the people and events that connect the accounts. A list of accounts alone would omit the action the founder can take with that information.

A relational edge table can also preserve and return the route. The point is the explicit relationship model and the access pattern. Graph tooling makes path-oriented operations central instead of requiring every consumer to reconstruct what a path means.

Reachability, shortest paths, centrality and community detection serve different questions. None of those algorithm names establishes that the input edges are correct. A centrality score over guessed introductions is still a calculation over guessed introductions.

A multi-participant event needs more than two anonymous endpoints

A property graph typically permits properties on nodes and edges. RDF expresses statements as subject-predicate-object triples, with identifiers and vocabularies that can support exchange across systems. Both approaches can model business relationships, but their conventions and tooling differ.

Consider the illustrative event "Alice gave Bob a book on Monday." It includes a giver, recipient, object and time. A binary edge has two endpoints. Storing the book's title and the date as properties on a giving edge may serve a narrow query, but that choice doesn't automatically make the book an independently traversable entity.

An event node is one solution. Give the event its own identity, then connect it to Alice, Bob, the book and an appropriate time representation. Queries now need to understand the event node and its role edges. The model can preserve the whole event without pretending it fits a single binary relationship.

This choice is often described as reification when a relationship or statement is represented as a thing about which other statements can be made. The naming doesn't eliminate the work. An event needs an identity policy: what distinguishes two similar giving events, what counts as a correction, and how several sources refer to the same event?

A hypergraph offers a representation in which an edge can connect more than two participants. Role conventions still matter. An unordered set containing Alice, Bob and a book doesn't identify the giver. More participants in one connection doesn't remove the need to preserve their distinct roles.

RDF-star-style statement annotation is another family of approaches discussed in this territory. The precise semantics and support depend on the specification and implementation. It shouldn't be treated as a universally available way to erase every statement-modelling problem.

These are alternate representations of a requirement. The useful comparison asks whether the system can identify the event, reach each relevant participant, annotate its evidence and explain the result to the team maintaining it. A shorter diagram isn't automatically a simpler operating model.

Direction survives only when the analysis uses it

A directed graph can preserve which claim cites which source, which organization acquired which company and which event preceded another. An analysis that turns those connections into undirected edges intentionally discards that distinction.

Sometimes that's acceptable. An operator may want a broad neighborhood of related records regardless of direction. It's wrong when the question depends on direction, such as what conclusions depend on a withdrawn claim. Following supporting evidence backward and following dependent conclusions forward are different operations.

Path homology investigates structure while retaining directed-path information. Persistent path homology extends this analysis across varying resolutions. These are research methods; no deployment or causal result is being claimed for the illustrative business graphs here.

Its operational interpretation needs restraint. A directed cycle can be a dependency pattern worth inspecting. Connectivity alone doesn't establish that sentiment causes price, that support load causes churn, or that a circular argument is valid. Those are claims about the meaning and effects of the relationships, which the graph's shape can't establish by itself.

A model can retain arrows and still mislabel them. It can also retain correct arrows and apply an analysis that ignores them. Check both boundaries: what the capture process writes and what the chosen algorithm reads. Preserving information is useful only when the answering process uses the relevant distinction.

Aggregation and assertion references create different demands

Revenue by channel by month is a different workload from a path query. A graph engine may support the aggregation, but visiting many records and summing values can be less suitable than a warehouse scan organized for that purpose. Actual performance depends on the product, indexes, data layout and query.

A graph projection and a warehouse projection can therefore coexist without either being a failed design. One serves connection questions; the other serves repeated analytical scans. They need a common account of which facts they derive from and how far behind those facts they're allowed to run.

Another requirement appears when a query needs to say something about an edge itself. A confidence value or timestamp stored on a relationship is an annotation. Connecting a source document to that specific assertion, or connecting one assertion to another as a contradiction, needs an addressable target.

An ordinary node-to-node relationship model may require an assertion node or another statement representation for that target. Reification remains a valid solution; it doesn't destroy the ability to answer the question. It adds conventions that every relevant writer and query must follow.

The next shape makes that convention central. It treats assertions as things the rest of the system can identify and discuss.

A metagraph makes assertions addressable[summarize]

The system can refer to a particular claim

A customer record may say that a contract renews in March. A signed document may say June. Overwriting March with June can produce a current value, but it doesn't preserve who supplied the earlier claim, why the system used it or which decisions depended on it.

An assertion-centred model keeps identities for the claims as well as the entities they concern. Another record can then refer to the March assertion, the June assertion and the evidence supporting each. A contradiction can remain visible while the business decides how to resolve it.

I use metagraph here for the modelling move that makes such assertions and links addressable. The important property is a stable target for another relationship. A source, a dependency, a correction or a confidence assessment can point to the assertion itself.

This is compatible with several implementations. A relational assertion table can identify a claim. A property graph can represent it as a node. A statement-oriented system may give links their own addressable identity. The capability doesn't belong exclusively to a product bearing the metagraph label.

The shape matters because the model treats the assertion as something worth preserving. A current-value schema that never records the assertion can't answer questions about its history. An assertion-aware schema in a conventional database may do so perfectly well.

Corrections can follow recorded dependencies

Consider an illustrative published case study containing a revenue claim derived from an analysis. The analysis uses an imported source figure. That figure is later corrected.

Finding the affected material requires a dependency chain from the source figure to the analysis and from the analysis to the published claim. A text search for the old number can help locate candidates. It misses derived claims that use a ratio, rounded wording or a conclusion that never repeats the number.

Addressable assertions let the system record those dependencies when the work is done. A correction query can follow them and return the material that requires inspection. It doesn't prove every dependent conclusion is now false. A source correction may leave a conclusion intact. The returned set is a review queue with reasons for inclusion.

Completeness still matters. A missing dependency edge can hide an affected claim. A mistaken edge can add unnecessary review work. A metagraph doesn't infer a perfect history from the existence of an assertion identifier.

Contradiction needs similar care. Two renewal dates may genuinely conflict, or they may concern different contract versions. The system should preserve the context needed to distinguish those cases before labelling the assertions contradictory. Holding both claims open is useful because it keeps evidence available; an automatic rule for declaring a winner is a separate decision.

Rules can be addressable too. A pricing decision can refer to the policy version used, its effective period and the inputs that satisfied its conditions. A later policy change then creates a new record instead of silently rewriting the explanation of an earlier invoice.

This is where time, provenance and identity become part of an answer. Part III develops what a system must retain about when a fact held and where it came from. An annotation field called source is only a start if nothing specifies which source version it identifies.

The name covers several related ideas

Metagraph has more than one technical usage. The discrete-mathematics lineage associated with Basu and colleagues includes edges connecting sets of vertices. OpenCog's AtomSpace is associated with typed nodes and links that can themselves participate in further structure. These usages shouldn't be treated as identical merely because both extend an ordinary graph.

The requirement in this article is the addressable assertion, including its dependencies and evidence. A design review should ask an implementation to demonstrate that requirement. A definition broad enough to include everything doesn't establish the semantics of a particular operation.

Ben Goertzel has proposed a common mathematical account of several cognitive processes in his research on probabilistically typed metagraphs. It doesn't establish a general theorem that a business knowledge system gains intelligence by choosing one representation. That stronger claim would need its own argument and evidence.

The same discipline applies to a personal wiki, a passage-retrieval system and a multilevel knowledge engine. Each can be appropriate for its job. Their visible diagrams don't establish which questions they answer, and an impressive representation isn't evidence of a useful operating loop.

A convincing demonstration starts from a retained source, performs a consequential query and shows how its answer depends on that source. An operator can then correct the source and inspect what changes. The capability is in that behavior, not the number of nodes drawn on screen.

More expressive information can cost more to operate

Topology supplies a counterweight to the temptation to preserve every possible distinction. Different invariants observe different structure. Ordinary homology of a two-dimensional sphere records a connected component, no one-dimensional holes and a two-dimensional class enclosing the sphere's interior. Homotopy can distinguish kinds of mappings that this homological description doesn't capture.

A more detailed invariant may require substantially harder mathematics or computation. General undecidability results in topology concern precisely stated problem classes. They don't prove that every homotopy calculation is impossible, just as the expressiveness of a data model doesn't prove that every query over it's intractable.

Homology is useful partly because it turns questions about a chosen complex into algebraic calculations. The coarser representation deliberately gives up distinctions to gain an instrument that can be computed and compared. The engineer's version is choosing the least complicated representation that preserves the distinctions the actual job needs.

An assertion-aware system adds genuine work. Writers must distinguish an entity from a claim about the entity. Queries must know whether they want the latest accepted value, an older value, competing claims or the evidence behind a decision. Identity and versioning conventions need to agree across consumers.

Tooling and hiring costs depend on the chosen implementation. A custom model can be hard to teach even when the underlying database is familiar. A specialized query language can add another maintenance burden. None of those costs disappears because the representation is theoretically expressive.

A team whose current questions are served by a relational schema doesn't need to adopt a metagraph to demonstrate architectural ambition. A team publishing disputed claims or maintaining rules with effective dates may need assertion-level records, but it should first check whether extending the existing model satisfies that requirement.

The added representation has to earn its cost through named questions. The same business can now test that claim against the whole set of shapes, using the same underlying facts each time.

One consultancy can need all five kinds of question[summarize]

The business stays the same while the required evidence changes

Consider an illustrative consultancy with clients, engagements, people, invoices, documents and introductions. Finance, delivery, sales and the founder ask ordinary questions about the same work. Each question needs a different property of the records.

Finance asks what was billed by client and quarter. That needs invoice amounts, dates, client identities and a rule for credits or cancellations. A table organized for aggregation is a natural fit. A document containing an invoice can preserve the same facts, but the reporting process still has to extract and group them consistently.

Delivery asks for an engagement exactly as the client submitted it. That needs the original record, including fields the current form no longer collects. A document can preserve the submitted structure. A relational system retaining the raw submission can also serve it. A model that kept only the currently recognized fields can't reconstruct discarded answers.

Sales asks which earlier engagements concerned a similar problem expressed in different words. That needs surviving descriptions and a retrieval method that can compare meanings. A vector index can help find candidates. The salesperson still needs to inspect whether an engagement's constraints, context and outcome make it a useful precedent.

The founder asks who introduced the client that later introduced another account. That needs recorded introduction events and stable participant identities. A graph makes the path convenient to query and inspect. Neither an account creation date nor semantic similarity between client notes establishes that an introduction happened.

The comparison concerns required operations, not a rule that every department must buy a different database. One engine may support the necessary representations well enough. A modest relational schema with retained submissions and a search index may cover the business's actual needs.

The design becomes clearer when every question names both its operation and its evidence. This table is a compact version of that review.

The last question is the assertion-level requirement. A case study can discuss revenue without repeating the exact value used in its analysis. Finding related prose is useful, but tracing dependence needs the link between that particular claim and that particular source value.

A missing representation can be added when the source remains

The consultancy can build semantic retrieval later if it retains the engagement descriptions. It may need better extraction or chunking, and it must evaluate the resulting matches. Those are tractable modelling and retrieval tasks because the evidence is still available.

It can build a revenue view later if invoices retain the necessary detail. The same caution about grain applies: totals alone can't recover line-level distributions. A source archive may restore that detail; a more expressive query language can't.

The as-submitted record depends on retention. Keeping a normalized interpretation beside an original payload preserves the option to revisit fields the first parser ignored. Keeping only the interpretation makes that parser's omissions part of the permanent record unless another source survives.

Introductions require event evidence. The consultancy may have explicit introduction fields, emails, CRM notes or another reliable source. If none recorded the event, a graph database has no established relationship to ingest.

A model can extract a candidate relationship from notes. A partner can recall an introduction. Timing can suggest a lead worth investigating. Those methods differ in evidential strength, and the resulting link should preserve its source and uncertainty.

A remembered introduction shouldn't silently become indistinguishable from a recorded introduction email. An inferred link may be useful for investigation while being unsuitable for a compensation calculation. The policy for using it belongs with the decision, not hidden inside an ingestion prompt.

The same distinction applies to identity matching. Two records that probably concern the same company can be linked provisionally without overwriting the original identities. Probabilistic record linkage is relevant background, but no generic matching theory supplies an error rate for this particular extraction. That rate would need measurement on the actual process.

A corrected figure tests the model's history

The fifth question forces the consultancy to distinguish several things it may previously have called one fact. There's a source value, a version of that value, an analysis using it and a published claim derived from the analysis.

A current-value table can store the latest figure and still omit the earlier value. A case-study document can preserve the wording without identifying what supported it. A vector index can retrieve relevant passages without knowing their derivation. A graph linking a case study to a metric can omit which metric version the claim used.

Any of those implementations can be extended with assertion identities, versions and dependencies. The decisive change is recording those distinctions. A store's category doesn't prevent it from holding the required records, and a new category doesn't create records that never existed.

A correction also doesn't justify deleting the original assertion. Keeping both versions allows an operator to explain what was believed, what changed and which uses need review. Access and retention policies may constrain what can be kept, but the policy should make the loss explicit instead of describing an overwrite as complete history.

The consultancy's design review can now ask a more useful question than which shape is best: what exact evidence supports each requested answer, and where is that evidence authoritative? That question leads directly to the hybrid's write path.

A hybrid needs an owner for each authoritative write[summarize]

Several representations can describe one business

A warehouse, a document archive, a vector index and a graph can all derive from the same source events. Each makes a different operation convenient. The difficult part is deciding which writes establish facts and which stores merely present them for a particular use.

An invariant gives that decision a practical basis. An invoice must refer to an existing customer. A source version must remain identifiable. An order total must satisfy the business's accounting rules. The authoritative write path needs to enforce the applicable rules or coordinate with the boundary that does.

A relational core may fit those obligations. An append-only event log may be appropriate when preserving the sequence of events is central. A document system with suitable transactional guarantees may protect the record boundaries the application needs. Choose from the actual requirements and guarantees rather than granting one shape authority by habit.

The other representations can then be projections. An invoice event contributes to an analytical view. A submitted engagement contributes to a searchable corpus. An introduction contributes to a relationship view. The projection changes how the information is used while retaining a trace to the source that authorized it.

Define the data before each adapter interprets it

I standardized Pydantic data models across APIs, pipelines and agent tools in my Python systems work. I start with the data objects because the functions, tests and logging depend on what those objects mean.

That's the useful part of a typed data catalogue. A customer, invoice or source assertion gets a definition that consumers can refer to instead of inventing independently. An adapter maps that definition into a particular storage model or interface.

A shared definition doesn't guarantee every consumer agrees. Pydantic validates data at the boundaries where it actually runs. A Python model doesn't automatically make an independently written SQL transform or a TypeScript client fail when its interpretation changes. Generated contracts, explicit mapping and checks are needed where those boundaries matter.

The distinction is worth keeping because it changes the engineering work. Declaring a source of truth is an organizational statement. Connecting each writer and consumer to that definition is an implementation obligation. A copy of a model that can drift without detection is another definition, whatever the diagram calls it.

A useful record definition therefore includes meaning as well as type. An amount needs a currency and an accounting interpretation. A date needs an event or validity meaning. An identifier needs an identity policy. A string accepted by every parser can still mean different things in billing and delivery.

The data-pipeline discussion follows these representations through their processing stages. Each transition should state what it preserves, derives or intentionally drops. That turns a pipeline from a chain of successful jobs into an inspectable account of how an answer was produced.

A projection must be reproducible from what remains

A store qualifies as derived when the system can rebuild its required contents from retained authoritative inputs and recorded transformations. A claim that the rebuild is possible someday isn't enough to establish that property.

The rebuild has to include more than the obvious rows. A vector index may depend on source text, extraction, chunking, an embedding model and metadata enrichment. A graph may depend on entity resolution and relationship extraction. If unique human corrections exist only in the projection, the projection contains authoritative information too.

That information deserves its own retained record. A correction can be promoted into the source process and replayed during rebuild. Alternatively, the store must be acknowledged as authoritative for the correction and protected accordingly. Calling the store a cache doesn't make its only copy of a decision expendable.

Reproducibility also has degrees. A model-driven transformation may not reproduce identical coordinates or summaries after an upstream model changes. The team should define whether a rebuild must be byte-identical, preserve specified records, or meet a measured answering requirement. Retain versions and outputs where the job requires exact historical reconstruction.

Run the rebuild on a bounded sample as an operating check. Compare the result using the requirement actually promised. An existence check on a rebuild script proves less than a successful reconstruction from the retained inputs.

Each projection also needs a freshness budget. Name the lag it can tolerate, the event used to measure that lag and what happens when it's exceeded. A graph and a warehouse can disagree because one is behind, because their definitions differ or because a transformation is wrong. An explicit freshness rule helps distinguish those cases.

Write ownership belongs to entities and facts

Consider an illustrative customer record edited in a CRM, a billing system and a warehouse interface. Each team can have a legitimate reason to change what it sees. Allowing each interface to redefine every shared field creates several competing authorities.

The ownership rule can be narrower than choosing one whole system to win. Billing may own invoice state. The CRM may own prospect qualification. A governed customer-identity record may own the identifier linking them. Each fact needs an authority and a route for requesting a change.

An interface that doesn't own a fact submits a command or event to the owner. The owner validates the change and publishes the resulting state. Other stores update from that state, under their declared lag and recovery rules.

The mechanics still need failure handling. Repeated delivery shouldn't create duplicate invoices or introductions. A partial update shouldn't silently leave two conflicting accepted values. The exact solution depends on the system's transaction and messaging guarantees, but the ownership decision supplies the rule those mechanisms must preserve.

Teams may disagree about whether two fields describe the same fact at all. A historical shipping address and a current contact address shouldn't compete for one authority because they concern different questions. Resolve meaning before building synchronization around an accidental name match.

A coherent hybrid keeps those distinctions visible. It records what happened, protects what must remain consistent and creates the views needed to answer particular questions. The final design review can be conducted against those obligations without beginning with a vendor shortlist.

Review the unanswered question before adding a store[summarize]

Name the evidence needed to answer a question

A useful design review starts with a question the business can't presently answer. Write it in the terms the requester will use to judge the result. Revenue, billed amount and collected cash can imply different records. A sentence precise enough to expose that difference is more useful than a category called analytics.

The procedure follows what an answer depends on. Each step should leave something the next person can inspect.

1. Record the question and the decision it supports. Identify the population, period and relevant definition. A report that counts the wrong population can be technically correct and operationally useless.
2. Classify the required operation. Does answering the question require aggregation, a preserved record, semantic recall, a path or a dependency between assertions? A question can require several operations.
3. Locate the source evidence. Name the records, keys, grain and versions needed. Look for surviving archives before deciding that the evidence was lost. Mark inferred relationships separately from directly recorded events.
4. Diagnose the current failure. Distinguish a costly query, an unenforced rule, a missing representation and missing capture. Choose a correction that addresses that failure.
5. Assign the authoritative write. State which invariant it protects and which changes require another owner. Define any derived view's rebuild and freshness obligations.

The first correction may be smaller than a new store. A missing index, an explicit key or a retained raw submission may solve the actual problem. A recursive query may be sufficient for a bounded path question. An assertion table may satisfy a provenance requirement without a new graph platform.

A second representation becomes justified when the required operation remains a poor fit after the evidence and model are clear. The proposal can then name its value: which question becomes supportable, which operation changes and what maintaining the projection costs. That's a decision another engineer can challenge with the same records.

Keep the existing system when it serves the job

A database that answers the required questions and protects the required invariants doesn't need replacement merely because another model is more expressive. Reporting work can remain reporting work. A team can improve its definitions and retention without adopting every shape described here.

A current dashboard working correctly doesn't establish that tomorrow's question is answerable. The relevant test is whether the retained evidence contains the distinction the new question needs. A familiar interface can conceal a permanent capture loss just as easily as an unfamiliar graph can conceal an unsupported inference.

The basket question makes a useful closing check. Find an order, its lines and the identifier tying them together. Identify where the daily total was derived and whether the original membership survives. If it does, changing the query or adding a projection remains an option. If only the total survives, label the missing evidence and change capture for future orders.

That check separates two promises a storage project can make. It can make a retained fact easier to use. It can also preserve a distinction in future records. It can't truthfully promise to recover an unrecorded past merely by changing the representation.

The next question is what those records need to remember about themselves: when a claim became valid, which source supplied it and what later replaced or contradicted it. Part III follows time and provenance through those changes.

The companion entries approach the same design from other jobs: combining retrieval methods in a knowledge engine, deciding which intelligence work is worth doing, and presenting analytical results through dashboards. Keep the unanswered question beside whichever design follows. It's the reason to preserve one more distinction, and the test of whether that work helped.