For months, one of our own automations was flagging our own domain as a lead.
Not a client’s. Ours. Every time someone on the team replied to an internal thread, a reply-classification workflow scored it as a positive reply from a prospect and queued it for follow-up. By its own numbers, our internal domain was one of our best-performing accounts. Nobody built it that way on purpose. It just kept happening, quietly, in the background, for long enough that it stopped looking like an anomaly and started looking like normal output.
What the system was supposed to do
The workflow in question reads inbound email replies across a set of outbound campaigns and classifies each one: positive, neutral, out-of-office, wrong person, and so on. Part of that classification checks whether the sender’s domain matches a “known” list, our own domain and a handful of others, so that internal test traffic and team replies don’t get treated as real prospect activity. That check is a small, unglamorous piece of the pipeline. It is also exactly the kind of piece nobody re-reads once it works the first time.
Where it broke
The matching logic used a string check that looked for the company name inside the sender’s domain, rather than matching the domain as a whole word or an exact value. It was written to catch a domain like “ourcompany.com” and correctly exclude it. What it also caught, without anyone intending it to, was any domain that happened to contain the same sequence of letters somewhere inside a longer string. A partial match read as a full match. The check that was supposed to say “this is us, ignore it” would sometimes fail silently and let internal traffic straight through as if it were an external reply.
The result wasn’t a crash, an error log, or a failed run. It was a workflow that kept executing successfully, on schedule, producing output that looked completely normal. Reply counts went up. Classification categories filled in as expected. Nothing about the dashboard said “something is wrong here.” The only way to notice was to go looking at who was actually generating the replies being classified, and cross-check that against who should never appear in that list at all.
Why it took so long to catch
This is the part that stuck with me more than the bug itself. A pipeline that’s producing an error is easy to notice, because it announces itself. A pipeline that’s confidently producing the wrong answer looks identical, from the outside, to one that’s producing the right answer. The reply rate looked healthy. The lead counts looked plausible. Everything downstream of the bug, dashboards, reports, weekly numbers, inherited the mistake without any of those layers having a way to detect it. Nobody was lying. The system was, structurally, incapable of noticing its own error, because the thing that would have caught it (checking whether “our own domain showing up as a lead” makes any sense at all) wasn’t a check anyone had written.
The fix
Once found, the fix itself was small: change the matching logic from “does this string appear somewhere inside that string” to “does this exact domain match, respecting word boundaries.” A few lines. The kind of change that, read on its own, looks almost too minor to have caused months of quietly wrong output. That gap, between how small the code change is and how long the wrong behaviour ran undetected, is the actual lesson.
What I took from it
Two things, and I think they generalise well beyond this one workflow.
First: an automation’s own reported numbers are not evidence that it’s working correctly. They’re evidence that it ran. Those are different claims, and it’s easy to quietly substitute one for the other once a pipeline has been live for a while and nobody’s had a reason to doubt it.
Second: the checks that guard against obviously-wrong output (a hard crash, a validation error, a missing field) are the easy half of reliability. The harder half is building in the checks that catch plausible wrong output, the kind that passes every test you already wrote because none of those tests were written with this specific failure in mind. That second kind of check usually only gets written after you’ve been burned once by not having it.
I’d bet most automated pipelines running today have at least one version of this sitting in them right now, quietly correct in every obvious way and wrong in one that nobody’s gone looking for yet. Worth an afternoon, every so often, of asking “what would it look like if this were subtly wrong, and would I actually notice?”