A metal-art client wanted better advertising because a competitor's ads looked better. I exported their Facebook data, organized it in Google Sheets, and compared cohorts by creative, copy, age and device. Women over 45 were a valuable audience. Family and home mattered to the offer. The work became useful when those findings changed which campaigns I ran and what the ads said.
That's the practical test I use for intelligence engineering. Information has to reach a person who can use it, support a decision, and leave evidence of what happened afterward. A correct report is one part of that job. The job also includes finding the problem worth reporting on, deciding what the system should reward, and checking whether anyone's work improved.
A buying pattern changed the creative decision
The request pointed toward the competitor
The metal-art business was copying a competitor's style. That competitor had a fitness-influencer presence and a male-model appearance, which was part of why his advertising worked. Reproducing the visible format couldn't reproduce the relationship he already had with his audience.
I hadn't learned Python yet. I used exports, macros and pivot tables to compare groups of ads and buyers. Which creative and headline combinations worked? Which audiences responded? Did performance hold when I compared different periods, or was I looking at one promising result in isolation?
The analysis gave me a reason to separate campaigns and change the creative. Family-oriented messages such as "Family is Forever" worked. An ordinary customer photo of a metal-art piece on a barn was also among the stronger ads. The offer could be about someone's home and the things they cared about, without trying to recreate the competitor's persona.
The buying story needs its qualification. The case summary emphasizes gifts, but my fuller account says many people bought for their own homes. Gift giving was part of the business. It wasn't the explanation for every purchase. The same care applies to the device finding: the iPhone targeting approach worked in that historical setting, and I later warned that changes in the data available to Facebook made it unsuitable as a recipe to repeat unchanged.
The spreadsheet mattered because the campaign changed
The case record describes growth from roughly $10,000 a month to roughly $150,000 a month in about 90 days. That was an engagement result involving analysis, positioning, creative, testing and campaign structure. A cohort table alone didn't produce the revenue, and the figure isn't a forecast for another business.
The transferable part is the decision sequence. The initial request suggested copying an ad. The evidence suggested changing who the offer addressed and how the campaigns reached them. I could then test those choices against subsequent performance. Keeping the same ads and filing the cohort report would have left the analysis unfinished.
This distinction matters when commissioning a data system. A warehouse can preserve records the business needs. A dashboard can make those records readable. An agent can retrieve an answer. Each capability can work while the decision at the end remains unchanged, because that answer arrives too late, nobody trusts its definitions, or the person receiving it has no authority to act.
Intelligence engineering includes that last part of the work. It connects the source event, its representation, the interpretation and the operator's next action. The system needs enough technical discipline to preserve the evidence and enough commercial understanding to ask a worthwhile question.
The first question I want answered is which decision better information would change. That answer determines what needs collecting, what has to be reliable, and what would count as success. It also gives the operator a way to reject work that looks sophisticated but contributes nothing to the decision.
The metal-art spreadsheet was enough to support a useful intervention. Another decision might require a production data service with monitoring and recovery. Choosing between them starts with the work someone must do with an answer.
Information needs an operator who can act on it
A business can collect accurate data and still struggle to decide what to do. The missing work becomes easier to locate when we separate the stages between something happening and someone acting on it. I use five layers for that explanation. They describe the path to a decision; they aren't five software products to purchase.
Each layer answers a different question
Consider an illustrative sales process in which a prospect submits an application, books a call and eventually buys. An application submission is an event. A row containing its identifier and submission time is data about that event. A count of applications by campaign is information. A finding that one campaign produces applications the sales team can't qualify is an insight. A budget change based on that finding is an operating decision.
The same sales process can fail at any of these layers.
| Layer | What it contributes | What the operator still needs |
|---|---|---|
| Event | Something occurred in the business | A reliable way to capture it |
| Data | A structured record of that occurrence | Meaning, definitions and usable context |
| Information | A calculation, join or summary | An interpretation relevant to the work |
| Insight | A pattern that could change an action | An owner able to decide and act |
| Intelligence in use | Evidence informs a consequential decision | Follow-up on the action and its outcome |
A problem can sit between any pair. Applications might never reach the CRM. A campaign name might change halfway through a reporting period. Sales and marketing might use different definitions of a qualified prospect. A correct finding might reach a manager after the budget decision has already been made.
Building a more detailed report only addresses some of those failures. The right fix could be a capture repair, an agreed definition, a change in delivery, or a decision owner. That's why I want to know which part of the path has failed before choosing technology.
Design backward, then check the evidence forward
Start with the action. An operator needs to decide whether to increase a campaign's budget, change an offer, pause acquisition, or investigate a discrepancy. Name the alternatives and the evidence that could distinguish them. A request to "understand performance" leaves too much unresolved to design against.
Work backward from there. A budget decision may depend on qualified appointments and sales capacity. Those measures depend on consistent CRM outcomes. The CRM outcomes depend on records that connect the original application to what happened during follow-up. The design reaches the source events because the decision needs them.
Then walk forward again. Does the business capture those events? Can the records be joined without guessing? Does enough time pass before a meaningful sales outcome exists? Can the operator inspect the evidence behind a recommendation? This second walk catches a design that assumes information the business doesn't have.
The smallest useful pipeline may already exist in pieces. A disciplined export and reconciliation can reveal whether a larger integration is warranted. It can also reveal that a desired answer is currently unknowable. That finding should stay visible. Filling the gap with an estimated success value would turn an incomplete pipeline into a misleading one.
Infrastructure has legitimate purposes beyond immediately changing one decision. Records may need retaining for operations, accountability or later analysis. The decision-first test helps scope an intelligence engagement; it doesn't make all other data work worthless. It requires us to identify what the business is buying and how we'll judge that particular investment.
I start by separating the customer's problem from the service I might sell. The problem exists independently of the consultant's catalog. A company can need relief from an operating failure without needing the service the consultant hoped to sell. Existing data shouldn't dictate the question any more than an available product should dictate the problem.
A written account of that problem comes before the system design. Otherwise every promising dataset becomes a reason to build something, and the operator still has to work out what to do with it.
Diagnose the work people keep doing by hand
A buyer's request is useful evidence about what they want to change. It can also name a solution before the problem is clear. "We need an agent" might mean that someone spends their evenings reconciling records. "We need better reporting" might mean two teams disagree about the same revenue figure. The requested product leaves those differences hidden.
I begin by defining the problem independently of anything I sell. That discipline keeps an AI engineer from treating every complaint as a reason to build an agent, and keeps a data engineer from treating every disagreement as a missing dashboard.
Follow the recurring consequence
The diagnostic questions in that method are deliberately concrete. Each question needs an answer the operator can connect to actual work.
- "Where is the money actually catching on fire right now?" Look for the loss or waste the phrase refers to, then inspect the records behind it.
- "Who gets yelled at when this breaks?" Find the person absorbing the consequence and ask what fails before it reaches them.
- "What is the one email you dread opening every morning?" Use the answer to locate a recurring problem the operator is handling personally.
- "What happens if we do absolutely nothing for 90 days? Does the business survive?" Ask what consequence is expected, what supports that expectation, and when it would become observable.
Those are my recorded diagnostic questions, not a validated instrument that replaces interviews or financial analysis. An answer can be incomplete, mistaken or politically convenient. The next step is to look at the work it identifies.
A recurring workaround is particularly useful. An analyst may reconcile a report by hand because the official report omits a source. A founder may approve every exception because nobody else has permission. A finance lead may keep a separate spreadsheet because the formal revenue definition doesn't match the cash question they need to answer.
I call the pattern heroic failure in my method: people keep compensating for a process that can't operate without them. The effort is evidence that something needs investigating. It doesn't establish the cause. The parallel spreadsheet might be necessary, or it might duplicate a capability nobody has explained. Watch the task, inspect its inputs and ask what would happen if the workaround stopped.
Write the constraint before agreeing on a solution
Write a problem statement identifying the current reliance, the pain it causes and the consequence of leaving it unchanged. A genuine time boundary belongs there when the business has evidence for it. An invented deadline adds pressure without improving the diagnosis.
An illustrative statement might read: "The sales team allocates follow-up from application counts that include unqualified prospects. Reps spend their available calling time on records that can't become customers, while suitable prospects wait." This is a composed example, not a report from a client. It gives the investigation something to establish and the intervention something to change.
That statement supports several possible responses. Better qualification data might help. So might clearer ownership of follow-up or a change to the offer. The decision stays open until the evidence distinguishes them. Naming a new dashboard in the problem statement would have chosen the product before doing that work.
The same statement controls scope after the project begins. A requested chatbot belongs in the engagement only if it addresses the agreed failure. A CRM integration may belong because it carries the required outcome data. Each request needs the same explanation of how it contributes. A separate useful idea can remain a separate project.
The operator also has obligations. They may need to provide source access, agree definitions, authorize a workflow change or assign someone to act on the result. Those changes connect the current situation to the future the buyer wants. Paying for software doesn't make those changes happen automatically.
The diagnostic conversation should end with a constraint the operator recognizes and evidence both parties can inspect. It should also identify whose account of the problem is still missing. A founder's explanation of customer behavior is one account. The customers' own words can challenge it.
Customer language tells you what to investigate
A buyer can describe a product in language its customers never use. The resulting research may be tidy and internally consistent while missing the question that determines whether anyone buys. My response is to collect the words customers use around the problem and keep them connected to their original context.
I collect the customer's exact words as research material. The useful part is the underlying evidence: exact phrases, where they came from, who was speaking and what prompted the statement. A collection of polished summaries loses the language it was meant to preserve.
Keep the complaint and its context together
The sources I describe in that method include Reddit discussions, comments under competitor videos, one-star reviews and support tickets when access is authorized. They complement interviews. An interview can reveal a workflow a public comment can't, and a review can reveal a failure a satisfied customer wouldn't mention. Neither source is automatically representative or truthful.
A useful research record preserves the original words alongside the product, use case and surrounding exchange. It distinguishes an isolated complaint from a repeated one. It also keeps disagreement. Customers can want incompatible things, and combining their phrases into one imaginary buyer would conceal the choice the business needs to make.
The next step is to connect language to a consequence. A complaint about unclear ingredients could indicate a need for better explanation, a product-quality concern, or a claim the brand can't substantiate. Copy is an appropriate response only to some of those problems. Research should be able to tell the business that it needs to change the product.
The temptation is to take a phrase that sounds persuasive and use it everywhere. That discards the diagnostic value of the work. The phrase needs to belong to the audience and the situation being addressed. Its source also needs to support any factual claim placed beside it. Customer vocabulary doesn't authorize a medical promise, an invented guarantee or a claim about all buyers.
The supplement sequence separated education from selling
The fitness-supplement engagement shows how audience research can guide a sequence. The brand had a substantial Amazon business but little direct relationship with its buyers. We treated the email audience as cold and began with education rather than an immediate hard pitch. The concerns in my account included safety, what went into the product and where it came from.
The sequence explained testing and product use in language the audience could understand. We watched opens and clicks to distinguish levels of engagement, then used a later conversion sequence for the warmer audience. The first decision was therefore about what kind of conversation to have with someone. The system needed to retain enough behavioral evidence to make the next decision differently.
The case record reports roughly $50,000 a month in additional direct email revenue from a previously dormant channel. It also reports roughly $100,000 a month in additional paid-ad performance associated with the audience and pixel work, carried out with the PPC team. My fuller account describes new campaigns using the audiences and events we supplied.
The account doesn't provide a controlled estimate separating email copy, site changes, audience creation and the PPC team's work. Crediting the entire improvement to one subject line would make the story more dramatic and less accurate. An email program can contribute both direct sales and evidence used by another channel.
Preserve the historical limit
The historical list came from email addresses extracted from Amazon order data. My later account explicitly calls the method contrary to Amazon's terms and says I couldn't encourage anyone to repeat it. That warning belongs beside the example. The case isn't permission or a present-day acquisition recipe.
Apply the teaching through data the business is authorized to collect and use. Establish access, consent and applicable messaging requirements before building a sequence. A useful marketing result doesn't erase the obligations attached to the source.
The same account limits the measurement. We used opens and clicks to segment engagement; we couldn't directly cross-reference individual time on site in the way the question proposed. A system design should preserve that limit instead of silently turning an available proxy into an unavailable observation.
Customer language can make a message recognizable because it addresses a concern someone already has. I use the term echo effect for that recognition in my method. It remains something to investigate and test. A buyer recognizing a phrase doesn't prove trust, a sale or a particular emotional response.
The lexicon improves the questions the team can ask and the messages it can test. It doesn't finish the diagnosis. Repeated complaints tell us where to look; the work still has to establish what produced them and which part the business can change.
Keep drilling until the constraint can be acted on
"I need more sales" names a desired result. It leaves the mechanism unresolved. More traffic, better qualification, a different offer or more delivery capacity could each be relevant. The diagnostic work has to establish which explanation fits before an intervention can be justified.
I investigate the causes behind that opening complaint. It moves through ads that aren't converting, acquisition cost, targeting, the absence of buyer analysis, and reluctance to inspect the data. The progression shows why a plausible surface fix can leave the operating constraint untouched.
Treat the next explanation as something to check
The source uses different layer counts for this exercise. Its examples matter more than a promise to reach a particular numbered answer. Asking why a fixed number of times doesn't certify that the last answer is true. A person can become more candid during a conversation, but they can also become more agreeable to the interviewer.
Each move needs evidence. A complaint about conversion needs an agreed definition and a view of what happens after the click. An acquisition-cost concern needs the cost, the acquired outcome and a period that both sides mean in the same way. A targeting explanation needs some connection between audience and purchasing behavior. Missing analysis needs to remain an admitted gap until someone does it.
Reluctance to look at data may be part of the problem. It should be discussed as a possibility the operator can confirm or reject. A consultant can't infer fear from every objection to a reporting system. The operator might have found an actual data-quality defect, lack authority to make the suggested change, or know about a cost the consultant hasn't considered.
A diagnosis becomes useful when it identifies an observable constraint and an intervention that could change it. "The client is afraid of the truth" gives the consultant an explanation that's difficult to challenge. "Nobody has agreed who can pause an unprofitable campaign" identifies work both parties can do.
Technical complaints can have organizational causes
A slow dashboard illustrates the same discipline at the data layer. The queries could be inefficient. The data model could serve a different workload. A source could be late. A proposed repair could also be waiting on procurement or on someone willing to own a migration. These are alternative explanations to inspect, not a single story to impose.
The original request might be to buy a faster tool. A query trace could establish that changing one calculation is sufficient. An access review could establish that the engineer can't reach the source required for the repair. A conversation with the owner could establish that the proposed measure doesn't answer the decision they make. The investigation has to stay open long enough to discover which problem exists.
The operator needs to commit to the change. An operator who wants reliable reporting may need to accept consistent definitions and validation. A team that wants an agent to prepare decisions may need to supply usable knowledge and name which decisions the agent can't make. Those obligations should be explicit before the technical system becomes responsible for an outcome it can't control.
The stopping point can be a bounded intervention, a request for a specific missing record, or a decision not to proceed. The consultant should be able to explain what evidence would change that decision. More probing after the cause and next action are clear can become its own form of delay.
Agreement on the diagnosis still leaves a design question. The new system will put some information first, make some actions easier and interrupt people about some events. Those choices can reinforce the behavior the engagement was meant to change. The interface needs the same scrutiny as the data underneath it.
Every default encourages a way of working
A CRM can tell its users to maintain an accurate forecast while putting total pipeline value in the largest type on the page. Those instructions can conflict. Adding a speculative deal improves the prominent number immediately. Removing an unlikely deal can make a rep's screen look worse even when the forecast becomes more useful.
This is an illustrative design problem, not an account of a particular client. It shows what I mean by the hidden curriculum of an operating system. The explicit instruction is what the team says the system is for. The implicit instruction is the pattern of behavior its measures, defaults and rewards encourage.
Read the screen as advice to the operator
A team can understand the intended purpose and still respond to a different incentive. A default list ordered by the largest deal may direct attention toward size while a renewal at risk remains below it. An alert on every minor change may train people to ignore notifications. A report that celebrates content views may encourage topics that attract viewers who can't become customers.
The system repeatedly makes one measure visible or one action convenient. Managers may use that same measure in reviews. Staff learn which activity gets recognized and can adapt their work to it. The interface is part of that process, alongside compensation, management and the operator's own judgment.
That's a risk to examine, not a law that the largest number always wins. A team can ignore a dashboard, challenge its definitions or compensate for it with a separate process. Those responses are themselves evidence about how the design fits the work.
The useful review asks what happens if someone follows the screen's apparent advice consistently. A pipeline-value target needs deal quality and likely close timing beside it. A lead-volume report needs the definition of a qualified lead and some account of sales capacity. A content report needs a connection to the outcome that justifies producing the content.
One dominant visual element can help a reader find the main decision. It doesn't require reducing the business to one unconstrained number. A screen can emphasize an action while keeping the conditions that could make that action wrong visible.
Audit defaults as part of the operating process
The original method includes four checks that turn this concern into design work. Apply them to a dashboard, an alert or a workflow trigger with the operator who will use it.
- Identify what attracts attention first. Ask whether it supports the decision the operator is responsible for, and which qualification must remain visible beside it.
- Inspect the default order. Check whether the first records deserve attention for this job, rather than simply having the largest value or newest timestamp.
- Inspect interruptions. Name what action each notification permits and whether the operator can take that action when the notification arrives.
- Examine repeated use. Ask what behavior a team would adopt by following these defaults over an extended period, then compare that prediction with actual use after delivery.
A color choice belongs in that review when it changes how a state is understood. Marking an uncertain estimate green can imply approval the data doesn't justify. Hiding incomplete records from the main view can make a queue look finished while leaving work unresolved. Those are meaning decisions expressed through design.
The review also needs the underlying records. An interface can encourage a sensible action and still present a wrong total. Better behavior doesn't excuse inaccurate data, just as accurate data doesn't excuse a design that encourages the wrong action. Each failure needs its own evidence.
After deployment, observe the actual task. Does the operator inspect the intended evidence? Do they need to leave the system to resolve an uncertainty? Do they act on an alert, dismiss it or wait for someone else? The answer can identify a training need, an authority gap or a misleading default. It should change the next version of the system.
These are engineering requirements because a data service has to preserve the distinctions the interface communicates. Missing, failed and zero can't all arrive as the same value. A decision screen can only explain the evidence its producers retain.
Preserve evidence through every handoff
The operator should be able to follow an unexpected answer back to the records and rules that produced it. That requirement gives the production system its shape. The original intelligence-engineering method separates capture, storage, transformation, serving, observability and the decision interface. Each responsibility addresses a different way the evidence can be lost or misrepresented.
These are responsibilities, not a requirement for six services. A small application can perform several of them in one process. The design needs to make their contracts clear enough that the team can find a failure without reconstructing the entire business from a dashboard total.
Capture and storage establish what the system knows
Capture records what happened at the source. The record needs identity, relevant time and enough context to connect it to the business event. A retry should have a defined effect. Idempotency means that repeating an operation has the same intended effect as performing it once; a capture design might use a stable event identifier to prevent a retried delivery from creating a duplicate.
A failed capture must stay distinguishable from a successful capture of zero activity. The operator may respond differently to "no sales occurred" and "we couldn't retrieve the sales." Substituting zero for the second state can make an outage look like a business result.
Failure reporting also needs a path that can survive the failure being reported. If a destination database is unavailable, promising to write every capture failure into that same database is incomplete. The design must say where the failed attempt remains observable and how someone can recover or replay it. The appropriate mechanism depends on the consequence of losing the event.
Storage preserves records under explicit definitions. Validation can reject a malformed timestamp or an unsupported status before it becomes indistinguishable from an accepted record. Rejection needs handling: retain the evidence needed to investigate, identify who owns the correction, and avoid treating the rejected record as successfully processed.
A schema also needs meaning. A field named revenue could refer to an order total, collected cash or an amount after refunds. Declaring it a number doesn't settle that choice. The definition, currency, period and source must fit the decision it supports.
Transformation and serving preserve the agreed meaning
Transformation turns accepted records into the information the operator needs. Calculations should have explicit inputs, output definitions and failure behavior. Separating calculation from external side effects makes the calculation easier to examine and test. The result should retain enough source and version context to explain how it was derived.
A transformation that joins applications to sales, for example, needs a rule for unmatched records. Discarding them could conceal a tracking failure. Counting them all as failed sales could conceal a join failure. The system should expose the unresolved state and the rule used to handle it.
Serving delivers the result through a contract a consumer can rely on. That consumer may be a dashboard, an agent or another service. It needs to know what a field means, which states can occur, when the result was produced and how a breaking change will be introduced. An API response that still parses can nevertheless change meaning. Consumers need protection against both kinds of change.
My documented Python practice uses Pydantic data models and FastAPI interfaces for these jobs. The tools make an approach repeatable. The obligations remain when another stack is more appropriate: validate inputs, make definitions explicit and preserve the contract between producer and consumer.
Observability and the interface support a real operator
Observability records the execution evidence needed to investigate the system. My Python stack uses Logfire for structured instrumentation. The design question is which trace would let someone explain an unexpected decision: which source arrived, which version of the calculation ran, where processing stopped, and what the consumer received.
Instrumenting activity without those questions can produce a large log that answers little. The records need correlation, meaningful states and an owner able to respond. They must also respect the data's access boundaries. Debugging doesn't require making every customer record visible to everyone who can read a log.
The decision interface completes the path. It supplies an answer in the workflow where an authorized operator can use it, with uncertainty and failed states visible. An alert needs an action. A recommendation needs its relevant evidence. A missing input needs a recognizable state that prevents someone from mistaking absence for approval.
The system can be designed backward from this interface while being built as a small end-to-end path. Capture one real input, preserve it, derive the needed answer, let the operator act and retain the result. That slice reveals whether the definitions and handoffs work together before the team expands the system.
A shared data model can keep those handoffs consistent. It gives everyone a place to inspect the assumptions being passed along, provided the team understands what the model can and can't guarantee.
Shared models make assumptions inspectable
I start technical design by asking what data objects will move through the system. My Python-stack account describes Pydantic as the place to define those objects and their properties. The point is to make the data shape explicit before separate functions, storage adapters and interfaces develop incompatible ideas about it.
A shared model can act as an intermediate representation: the agreed form a record takes between the systems that receive, store and use it. The surrounding adapters translate between that form and their own requirements. This is a way to organize responsibility. It doesn't make every destination equivalent.
A type describes only the rules someone supplied
An illustrative decision record might identify its source, subject, relevant time, status, calculation version and responsible operator. Its amount fields would need units and definitions. Its states would need to distinguish an accepted value from a failed or incomplete calculation. The exact fields follow from the consumers' jobs.
My data-modeling approach separates data from the functions acting on it. An entity supplies identity, components hold relevant data, and processing functions operate on the data they require. That separation can help a small pipeline without requiring a full game-engine-style Entity Component System runtime.
The same separation helps clarify missing data. A function that displays a source excerpt might need only its text and source reference. A calculation using that excerpt may require an additional accepted classification. The ability to display the record doesn't establish that it can support the calculation. Each consumer should state what it requires.
Pydantic validation can check rules expressed in the model when validation runs. It can't establish that an accepted sales amount is true merely because the amount has the right type. A timestamp can be syntactically valid and refer to the wrong event. A field can be required and still carry a convenient false value.
A shared Python model also doesn't automatically cause every frontend or agent consumer to fail a type check when the model changes. That protection needs an actual integration: generated or shared contracts, validation at the relevant boundary, compatibility checks and consumers that use them. A schema change should be traced through its real dependents.
The storage benefit has the same limit. Keeping domain meaning outside a database adapter can reduce coupling. Moving between storage systems still requires work on queries, indexes, constraints, transactions, history and operational behavior. The shared representation gives that work a stable reference; it doesn't perform the migration.
Examples and properties test different claims
The Python Systems Stack case describes Hypothesis property-based tests alongside data models and instrumentation. An example test supplies a chosen input and checks its expected result. A property test supplies a rule and explores generated inputs looking for a case that violates it. Both are useful when their claims match the problem.
A known calculation with a known answer deserves an example. A rule about replaying events deserves a property: processing the same identified event again mustn't change the accepted total a second time. The test needs a generator that includes repeated deliveries and the states that make the rule meaningful.
Another candidate property concerns accounting periods. If the business assigns each event to one reporting period, an event at a boundary should enter exactly one period under the declared timezone policy. That's an illustrative invariant, not a claim about a bug I found. The boundary, timezone and definition of an event all belong in the test's assumptions.
Broad rules can be wrong. "Revenue is never negative" fails for a model that includes refunds or adjustments. "Timestamps are never in the future" may reject a scheduled event the business legitimately needs to store. "No duplicate IDs" needs to distinguish separate events from multiple representations of the same entity. Domain definitions determine the invariant.
A property framework can generate cases and reduce a failure to a smaller reproducible example. Its search still depends on the generator and the property supplied. Passing doesn't prove that every possible input was explored, that the stated property is the right one, or that the business objective is sensible.
Make the check explain what failed
The practical combination is a clear specification, examples for expected behavior, properties for meaningful invariants, and execution records that connect a failure to its inputs. A developer should be able to see which rule broke and reproduce the case. An operator should be able to see which business result remains uncertain.
The cost of a wrong answer determines how much assurance the system needs. My stack account distinguishes an experiment used for learning from software real people will use and pay for. The latter needs evidence appropriate to the consequence. Adding tools by name can't substitute for that judgment.
These checks establish whether the machinery preserves declared rules. A bidding system can satisfy a spend calculation while being rewarded for buying the wrong outcome. Choosing what deserves a reward remains a separate design decision.
A reward needs business limits and outside evidence
An advertising system told to maximize conversion volume can improve its assigned measure by spending more. The business may need profitable acquisition, enough cash to operate, and customers it can actually serve. Those objectives overlap only under conditions someone has to specify.
This is an illustrative reward-design problem. The failure doesn't depend on a particular bidding algorithm. A person following a target can make the same mistake as software: improve the measured quantity while accepting a cost the target leaves out.
Keep the objective and its constraints distinct
I usually start agent design with reward and constraints. What counts as good? What counts as bad? What should the system be unable to do? The answers determine the behavior worth encouraging and the boundaries that must hold even when a result looks attractive.
A campaign objective might prioritize growth while the company enters a new market. Later, the same company may prioritize cash generation. A system needs a way to represent that change, identify who authorized it and retain which objective governed an earlier decision. Otherwise a historical result can be judged against a rule that didn't exist when the action was taken.
Multiple measures help expose the trade. Acquisition volume, cohort profitability and a longer-term value estimate tell different stories. A delayed estimate also carries more uncertainty than an observed payment. Combining them requires a declared policy about time, uncertainty and priorities.
A weighted combination remains a single score. It can conceal a tradeoff just as a simple metric can. A high growth score might compensate mathematically for a loss the business can't afford. Some boundaries therefore belong outside the score: a spend ceiling, a minimum evidence requirement, an approval step or a stop condition. The design should say which values can trade against one another and which can't.
The interface then needs to communicate both the preferred action and the limits on it. A recommendation to increase spend should remain visibly conditional on the evidence and authority required. Making a constraint small or hiding it behind a tooltip can undo the care put into the objective.
An attractive proxy can accompany a failed business test
The Garden Metal Art case is a useful counterexample from my own work. The products were visually striking, and campaigns produced high engagement and substantial traffic. We tried different structures, audiences and creative approaches. The combination of product, price, audience and site never produced a profitable, repeatable path for cold traffic, and after the test rounds we stopped.
The ads could attract attention without establishing that the business could acquire customers profitably. A report organized only around clicks would have supported the wrong conclusion. Calling the experiment unsuccessful preserved a decision the positive intermediate metric couldn't justify.
The result also has limits. It doesn't prove the product could never sell, or that a different channel or operation would fail. It establishes what those tests failed to produce under their constraints. The record can inform a future decision without becoming a universal verdict.
Check changed actions and outcomes separately
The original intelligence-engineering method calls its outside-world test verification by calendar. The calendar stands for the operator's actual work and consequences: calls held, deals closed, renewals, collected cash, time spent on a task. A system's internal success report needs comparison with the event it claims to describe.
Record the decision before the intervention. Name the evidence available, the action taken and the expected consequence. Then retain the later records that could support or contradict the expectation. A dashboard can display those records; the requirement is that the verification refers to the underlying outcome rather than merely repeating the system's derived success flag.
A changed decision and a better outcome are different findings. A manager might use the new information and choose a sensible action, while market conditions worsen. A business result might improve even though nobody used the new report. Before-and-after movement alone doesn't isolate what the system caused.
Compare the timing, other changes and the relevant baseline. Use a controlled comparison when it's feasible and necessary for the claim. Where the evidence supports only a contribution or an association, state that. The supplement result illustrates why attribution needs this care even when the overall engagement was useful.
Outcome evidence can take longer to arrive than execution evidence. The team still needs intermediate checks for failed capture, wrong calculations and unsafe actions. Waiting for a renewal outcome doesn't justify leaving an incorrect pipeline unattended. Separate the technical check, the changed workflow and the eventual business result, and assign responsibility for each.
The agreement between consultant and operator should preserve those distinctions. Both sides need to know what's being promised, what's being investigated and who has the authority to make the next move.
A commitment needs an owner who can keep it
A transformation can expect clean records from storage while storage expects capture to supply them. Both components can appear healthy when neither actually checks the assumption. The failure becomes visible only when a consumer tries to use the result.
The original method brings Mark Burgess's promise-theory framing into this problem: consider what each participant commits to about its own behavior. Used as a design lens, the framing moves a vague expectation into a statement someone can inspect. It doesn't make a distributed system reliable by declaration.
State what the producer supplies and the consumer checks
A capture component can commit to preserving the source identifier and reporting incomplete retrieval. Storage can commit to applying its declared validation. A calculation can commit to a specified treatment of retries and unmatched records. A consumer can check whether the result satisfies the contract it relies on.
The contract needs a boundary and a response to failure. A promise to keep data fresh needs a definition of freshness for this job and a visible stale state. A promise of stable output needs a change policy. A promise to report an error needs an owner and a reporting path that exists when the normal path is unavailable.
The consumer still has work to do. It can reject an unsupported schema version, show an incomplete result as incomplete, or stop an action that lacks required evidence. Accepting a producer's success flag without checking the relevant contract would leave the original assumption intact.
Those commitments should be testable through the actual path. Inspect a failed handoff and confirm that the receiving component exposes the failure in the form the operator needs. A contract that exists only in a diagram offers no evidence that the running system honors it.
The engagement has the same dependency problem
The consultant can promise a defined pipeline and the checks agreed for it. The operator can commit to access, definitions, decision ownership and the workflow changes the intervention requires. Both can agree which evidence will determine whether the engagement worked.
Neither party should promise an outcome they can't control. A technically correct recommendation can't make an unauthorized employee change a budget. A consultant can't guarantee a business result whose dependencies include an offer, fulfillment capacity or market conditions outside the engagement. The contract should name those dependencies instead of hiding them inside a broad success claim.
Qualification must include the people who control those dependencies. Budget authority, access authority and the willingness to accept an inconvenient finding can matter more than enthusiasm for the technology. The team needs to know whose commitment is required and whether that person has actually made it.
My Web3 experience supplies a painful limit. I worked on an anonymized project where community collaboration and communication continued through severe instability, and the investor ultimately betrayed the community and the team. My recorded account says we kept pushing for the developers to build what had been promised, while the investor's conduct threatened the community and the team.
The lesson I drew was to investigate the investors behind a project as well as its visible leadership. Communication with the community couldn't make an untrustworthy funder honor the same commitments. A well-organized team still had a dependency on someone outside the behavior the team could control.
The case record includes market-cap growth during that period. It doesn't establish that my communication caused the growth, and the later outcome prevents treating the peak as evidence that the whole arrangement was sound.
For an intelligence engagement, put the practical version in writing. Identify who can authorize the action, who supplies the evidence and who responds when the result contradicts expectations. An unanswered dependency can remain a reason to investigate or to stop that part of the work. It mustn't quietly become a promise the engineer is assumed to have made.
A bounded forensic audit is one way to make such an agreement concrete. It starts with records both sides can inspect and ends with a decision the operator has authority to take.
Reconcile the report with the outcome it claims
The Forensic Ad Audits case describes brands and agencies with substantial Facebook or Google activity, cluttered accounts and reporting that couldn't explain what worked. My job was to pull the account exports, segment performance and reconcile the findings back to revenue. The deliverable was a prioritized action plan the operator could use.
The case reports that in many accounts I found 10 to 30 percent of spend that could be cut or reallocated without hurting revenue. That describes findings across those accounts. It isn't an estimate of waste in an unexamined account, a promise of savings or permission to cut spend without checking the consequences.
Establish which claims can be compared
An audit starts with a bounded question. A campaign's reported return may conflict with the operator's cash position. A growing lead total may conflict with what sales can close. The investigation needs to define the quantities before deciding that either report is wrong.
Platform-attributed revenue and collected cash can differ for legitimate reasons. Attribution windows, refunds, payment timing, duplicated credit and reporting periods can all affect the comparison. The auditor should list the differences and reconcile them against source records. An unexplained gap is a finding to investigate, not automatic evidence of fraud.
Pull the records at a level that can support that work. Campaign objective, audience, device and funnel stage are useful segmentation dimensions from my case account. They allow the team to locate where aggregate performance conceals a different pattern. The dimensions still need consistent definitions and enough data to support the comparison being made.
A segment can look weak because it's new, small or waiting for outcomes. A segment can look strong because it receives credit for customers already likely to buy. The audit should show what the records establish, what remains uncertain and which alternative explanation would change the recommendation.
Connect the finding to a bounded change
A useful recommendation names the source evidence, the proposed action and the expected consequence. It also identifies a reversal or stop condition where the action carries risk. "Improve targeting" leaves the operator to design the intervention. A specific account-structure or qualification change gives them something to approve, implement and examine.
The priority follows the business constraint. A measurement defect may need fixing before a budget decision can be justified. An offer problem may make campaign optimization premature. A fulfillment limit may make additional acquisition undesirable even when the acquisition report looks attractive.
This is where the problem statement earns its place. It keeps the audit from expanding into an unrelated redesign of everything the auditor notices. A finding outside the question can be recorded separately. The agreed intervention still needs to reach a real action and a follow-up, rather than ending as a catalog of observations.
Verification returns to the relevant outcome records. Did the approved change occur? Did spend move as intended? What happened to qualified demand, sales and the other constraints the operator agreed to protect? The answer can support keeping, modifying or reversing the action. The follow-up record should also include other changes that complicate attribution.
Transfer the questions while respecting the domain
The Lead-Gen Funnel Overhaul case describes a related change in what the team measured and pursued. The work tightened the audience and offer, rebuilt the form and follow-up, and added sales feedback. A cheaper form submission would have been an incomplete success criterion if reps still couldn't contact or qualify the prospect.
The agency self-funnel cases connect the same reasoning to operations. An agency could deliver for customers while depending on referrals and a weak internal CRM. Improving its own acquisition meant clarifying an offer, establishing a usable sales path and making follow-up happen. In my fuller agency account, sales recordings and delivery systems mattered alongside ads.
The diagnostic questions transfer to technical work too. A warehouse audit can compare reported query performance with the task users are waiting to complete. An agent audit can compare a success message with the action actually recorded by the receiving system. A support audit can compare ticket closure with the customer's unresolved issue.
The records and expertise differ. Ad-account experience alone doesn't qualify someone to diagnose every database or agent failure. The reusable discipline is to inspect sources, define the outcome, reconcile the claim and test a bounded intervention. Each domain still requires its own knowledge of what could make the evidence misleading.
Retaining that reasoning makes an engagement useful beyond its immediate result. The next operator should be able to see why the action made sense, where it failed and which conditions would make it worth trying again.
Keep the reasoning that changes the next decision
A reusable result contains more than a successful tactic. It preserves the question, source evidence, constraints, action and outcome. Without that context, the next team can copy the visible move while losing the reason it worked.
The original intelligence-engineering argument calls this accumulated calibration. A diagnostic method becomes more useful as the operator learns where it applies, what can mislead it and which counterexamples should stop an automatic recommendation. Documentation matters because those distinctions otherwise remain in one person's memory.
Retain the condition attached to the lesson
A customer-language collection can give the next research job a starting point. It still needs checking against the new audience. A typed model can supply an established definition. It still needs a consumer whose requirements match that definition. An audit procedure can preserve a useful sequence of questions without predetermining any conclusions.
The record should include failed interventions. An intervention that didn't improve the outcome can expose a wrong hypothesis, insufficient evidence, an implementation failure or a business constraint the first diagnosis missed. Those are different lessons. Recording only the final metric makes them difficult to distinguish.
This is also why reuse needs a source trail. Someone should be able to find the record behind a claim and see when it was true. A confident summary detached from its source can become less useful than the original messy notes. Later systems may repeat the summary as if repetition were corroboration.
My RAG Knowledge Engine work addressed a version of that problem. I built on an existing platform, Archon, and changed the handling of different document types rather than building a whole retrieval system from scratch. My account describes semantic chunking and task-specific embedding choices so agents could retrieve material from the corpus I actually wanted them to use.
The engineering lesson is to preserve a source's useful units and evaluate what retrieval returns for the real task. A text chunk that severs a qualification from a claim can make a generated answer misleading even when the retrieved words are genuine. More stored text doesn't establish that the system can retrieve the right evidence.
Agents can assist within an explicit decision boundary
Agent Redwood is my assessment framework for designing agents. It names concerns such as mission, constraints, memory, evaluation, tools, reward and integrations, then helps prioritize how much each needs for the particular job. It's a blueprint for decisions about a system, not a replacement for an orchestration library or proof that the resulting agent works.
My account emphasizes that every component needn't be built to the same depth. The choice depends on the work, the available resources and the consequence of error. Reward and constraints are often where I start, because they clarify what the system should attempt and what it must leave alone.
An intelligence workflow can give an agent a bounded mechanical task: retrieve authorized records, prepare a reconciliation or propose findings with source references. The operator retains whatever authority the workflow requires for diagnosis, customer commitments and consequential changes. Tool access should reflect that division. An agent asked to explain a discrepancy doesn't automatically need permission to change the account.
The output needs checking against the actual source and receiving system. A plausible finding can be wrong. A claimed action can have failed. Keeping the evidence and the decision separate lets a person accept useful assistance without treating the agent's account as its own proof.
Start with a decision record someone can reopen
Choose a recurring decision that better information could change. Write down who makes it, what alternatives are available and which evidence would justify choosing differently. If those fields remain unclear, return to the diagnostic work. A new pipeline can't settle an undefined decision for the business.
Then specify the smallest information path needed to support it. Identify the source events, the meanings that must survive each handoff and the failed states the operator must see. Agree the action boundary and the outside outcome record before treating a new report as evidence of success.
Keep the resulting decision with its evidence. Record the action taken, the conditions under which it made sense and what happened afterward. A later reader should be able to reopen the record, challenge the interpretation and decide whether it still applies.
The metal-art work began with ordinary exports and a question about who was buying. The reusable part was the reasoning that connected that evidence to a different campaign. Choose the next decision with the same care, and name the observation that would make you change your mind.