AI tasks passed quality and source checks before release
A bounded repair loop handled quality issues. Work without adequate sources went to a person.
A quality check allows up to three repair tries, and a person gets any work the claims check can't source.
Context
Teams were excited about AI and stuck between flashy demos and systems that reliably do useful work.
Problem
Proofs of concept impressed leadership but stalled before production.
Approach
I treated AI projects like any other production system. I scoped the narrowest problem with the biggest payoff, designed the agent's workflow, then wired in the LLMs, tools and guardrails it needed. Most of the work went into monitoring, failure modes and the experience of the person running it.
Stack
- LLMs (Claude, OpenAI)
- APIs
- Python
- TypeScript
- Queues and schedulers
- Existing SaaS tools
Result
Agents and automations handled research, data preparation and repetitive operations. Each task went through a quality check with up to three repair tries, then a check of its claims against sources. Work that failed the second check went to a person instead of shipping.
Impact
Each agent's limits were set before it ran: up to three repair tries, a check of every claim against sources, and a person for anything it couldn't source.
Lessons
Budget as much time for guardrails and monitoring as for prompts.
Who this is for
For operators who want AI agents doing real work, with clear rules for when an agent stops and when a person takes over.
Get a quoteThe method behind this result
- Agent SystemsAI agent operators need action histories they can inspect after interruption. Each attempted tool call should retain its arguments, permission check and result.
- The AI Agent Industry Has Automated the Handoff Back to YouThe agents finish their parts. The buyer still has to make the result usable.