Two incidents changed how I think about letting AI agents do real work, and neither of them was catastrophic. That’s actually what made them useful. Catastrophic failures get noticed and fixed immediately. These two were small enough to nearly get shrugged off, and that’s exactly the kind of failure that repeats.
Incident one: the test that reached a real person
We were testing a workflow that forwards email replies through an automated pipeline before they land in a shared inbox. As part of testing a downstream change, a simulated reply was sent through that pipeline to check the routing logic. The pipeline worked exactly as designed. It also forwarded that simulated reply to real, external contacts, because the “test” traffic and “real” traffic ran through the same live endpoint, and nothing in the system distinguished between them.
Nobody meant for a fake message to reach a real inbox. But intent doesn’t matter to the person who receives an email that looks like it’s part of an ongoing conversation and isn’t. The system had no concept of “this is a test,” so it treated the test exactly like production, because from its point of view, there was no difference.
Incident two: the live edit that drifted
Separately, a workflow got a small live edit directly in its production environment, the kind of change that feels too minor to justify going through source control first: adjust a threshold, tweak a field mapping. It worked. Nobody thought more about it. Weeks later, when the same workflow needed a larger update pushed from the actual source repository, that push overwrote the live edit, because the repository had no record it had ever happened. Several workflows had quietly drifted out of sync with the code that was supposed to be their source of truth, and the drift was invisible until something tried to reconcile the two and found they disagreed.
The pattern underneath both
Different systems, different failure modes, same root cause: nobody had written down, in advance, what an agent or a quick manual shortcut was allowed to do on its own, versus what needed a deliberate, separate approval step first. Both incidents were technically avoidable with a rule that, in hindsight, sounds almost too obvious to need stating: don’t let test traffic touch a live external-facing endpoint, and don’t let a live edit bypass the same review a normal change would go through. The trouble is that “obvious in hindsight” is not the same as “obvious in advance,” and nobody had sat down to write the rule before the two incidents forced the question.
Building the actual framework
So I built one. Every action an agent might take gets scored on two axes.
Leverage: how much time it saves, multiplied by how often it happens. High-frequency, low-effort actions (reading files, checking status, running a read-only lookup) sit at the top of this axis, because the value of automating them compounds fast.
Risk: how bad it is if it goes wrong, multiplied by how reversible that outcome is. An action that’s easy to undo (a draft that hasn’t sent, a branch that hasn’t merged) carries very different risk from one that isn’t (a message that’s already delivered, a record that’s already deleted).
Cross those two axes and you get a rough ladder:
- High leverage, low risk: the agent just does it. No prompt, no delay. Reading, searching, checking status, working in an isolated sandbox.
- Medium risk with a real, checkable gate available: the agent proceeds automatically, but only once that specific gate passes. Not “the agent decided it looks fine,” but a concrete, reproducible check that either confirms the change is safe or it doesn’t run.
- Anything that reaches another human, spends money that can’t be clawed back, deletes something with no backup, or changes a system-level permission: stop, state the blast radius in one line, and wait for an explicit yes before doing anything.
- Anything catastrophic and irreversible with no legitimate reason an agent would ever need to do it: never, full stop, no override.
Why this matters more than the model
It’s tempting to think the interesting challenge with AI agents is capability, what can the model actually do. In practice, the interesting challenge is almost entirely this: given that the model can now do a lot, what should it be trusted to do without someone watching? That question doesn’t get easier as models get better. If anything it gets harder, because a more capable agent can cause more damage per unit of trust extended to it, just as easily as it can save more time.
Writing the ladder down didn’t stop every possible failure. What it did was turn “should this need approval?” from a judgement call made in the moment, often under time pressure, into a lookup against a rule that was written calmly, in advance, with both incidents already accounted for. That’s the actual value: not perfect foresight, just moving the decision to a moment when you can think clearly about it, instead of a moment when you’re mid-task and the path of least resistance is to just let it run.
This is the unglamorous part of working with AI agents that doesn’t get talked about much. Not the model, not the prompt. The boundary around what it’s allowed to touch on its own.