In the last piece I described a failure that ran in production for two months while I checked on it most weeks and concluded, every time, that everything was fine.
The family it belongs to is called silent failure: code that produces wrong behaviour with no crash, no error, and no page. It's been named and studied for years; this isn't a gap in the literature. The specific member I kept hitting is the one where an AI writes the wiring: the integration code between parts of a system it can't see end to end. I started calling that silent wiring: the wire is there, you can read it and trace it and watch it light up in your monitoring, but what's moving through it is absent, empty, or fake.
This piece is the part I promised at the end of the last one. There isn't one shape. There are four. And the reason they cost me five months instead of an afternoon is the property that ties them together: each one is invisible to the check that catches the one below it. You fix the first, feel clever, and walk straight into the second, which your shiny new check reports green on, because it genuinely is green on the thing that check measures.
Here's the ladder, bottom to top.
Type 1: the handler that never fires
The entry-level lie. A handler that exists everywhere it's supposed to (registered, configured, discoverable in your architecture) and never actually runs. The event goes out. Nothing picks it up. Sometimes the subscription was never wired. Sometimes it's wired fine and the events are being eaten upstream: in my case, a reasonable-looking except RuntimeError: pass in the event publisher that swallowed every event when the worker threads had no event loop to run on. Different mechanism, same outcome. What they share is silence.
Here's why your monitoring won't help you. Your monitoring is watching for failures: errors, timeouts, latency spikes, the pager going off at 3 a.m. A handler that never fires produces none of those. It produces an absence. And absence doesn't page anyone. In an event-driven system, silence is what normal sounds like, so a handler that has gone quiet is indistinguishable from a handler with nothing to do.
The check is embarrassingly simple: one timestamp per handler, last_event_at. If it reads never, you have a phantom. This is the whole job of a liveness probe, and it's a different question from the one your health check answers. Your health check tells you the process is up. A liveness probe tells you this specific handler did something, recently, at all. Same process, same memory, same database connection. Different question, and only one of them is the question that matters.
Somewhere in your system is a handler whose last-event time nobody has ever looked at. It's probably fine. "Probably" is the whole problem.
Type 2: the handler that fires and does nothing
Now the handler runs. You can see it in the logs. The health check confirms it's active, the event counter is climbing, and by every surface measure this code is working.
It just doesn't produce anything useful. It hits a silent catch block and returns. Or it runs a conditional that's always false and skips the actual work. Or (the purest version) it writes its output to a table that no consumer reads. The rows land, correctly formatted, and sit there like mail delivered to an empty house.
Mine was the pure version. The handler wrote to one table; every consumer read from another. A naming convention had drifted between two AI coding sessions: the code that wrote the data used the old name, the code that read it used the new one. Both names were valid. Both tables existed. Every line was correct and nothing threw an error, because nothing was an error. The failure was purely relational: the writer and the reader disagreed about where the data lived, and neither one complained.
Every check you built for Type 1 passes clean here, because the handler is firing; your liveness probe is green and it's right to be. To catch this you have to stop asking "is it running?" and start asking "did the right thing actually arrive?" That's a semantic canary: send a known input on the real wire, and check that the known output comes out the other end. It's the difference between pressing a fire alarm's test button and setting off the sensor with smoke. The button proves the speaker works. The sensor proves the whole chain works. A liveness probe presses the button; a semantic canary trips the sensor.
Your handlers are firing. Are they writing to the table your consumers are actually reading?
Type 3: the handler that fires, writes, and lies
This is the one that keeps me up at night.
The handler fires. It writes. And the values look real: right table, right format, one row every few minutes, month after month. The kind of table you'd pull up to prove to your manager that a system is healthy. But the handler wasn't doing the work. A fallback path (a cold-start safety net built for first boot, when there's no history to average yet) had quietly become the only thing writing to that table. With no real input to compute from, it did what a moving average does when the input stops: it decayed. Smoothly. Plausibly. It drifted downward the way a real instrument settles as it finds its level, and then it died in place and stayed there.
Here's what makes it worse than a hardcoded default: every value was different from every other value. Nothing wrote a constant. If you'd gone hunting for the obvious tell (a column of identical numbers, a flatline, a suspiciously round figure), you'd have found variety, nodded, and closed the tab. Every freshness check reported green, and reported it correctly, because the data genuinely was fresh. It was fresh and fake at the same time.
This is the type where your own diligence is the weapon turned against you. You checked. There was data. You moved on. The only way to catch it is to stop asking whether the value is wrong (on live data you have no ground truth to be wrong against) and start asking whether the value is the product of computation at all. You answer that with spread: real work produces variance, a decayed fallback doesn't. And be careful which check you reach for. A check that pages when the number matches the fallback's value catches almost nothing, and catches it weeks late, because the fallback drifts through perfectly plausible territory before it settles. A check that simply asks whether the number has any spread at all catches it in hours. The trap is the absence of variance, not any particular value.
Your handlers are firing. They're writing output. But is that output computed, or a fallback value wearing a fresh timestamp?
Type 4: the handlers that all quietly agree to be the same
Every type so far had something you could point at: a missing subscription, an empty table, a flatlined value. This one doesn't. Every individual output is valid. Computed, not a default. The values vary, and they change over time. Every check you built for Types 1 through 3 passes clean.
The failure only exists at the population level. I had an engine whose whole job was to evolve a diverse, rugged landscape: strong candidates surviving, weak ones culled, a competitive middle where the interesting work happens. Instead, ninety-six percent of its population had collapsed into a band about three cents wide. Dozens of distinct values, which sounds like diversity right up until you notice nearly all of them are crammed into the same tiny range. Flat where it should have been rugged. And a flat landscape gives natural selection nothing to work with.
No per-row check catches this, because no individual row is wrong. To see it you have to know what the distribution should look like (how much variety should exist), and that answer is different for every system and every column. It's the only one of the four that demands genuine domain expertise, which is exactly the expertise that evaporates when an AI writes a module nobody on the team fully understands. The AI writes each pixel correctly. Who's checking the picture?
The check is distribution, not values. Pull ten records from your most critical table (not the newest one, ten across a span) and line them up. If real work should produce variance and you're not seeing variance, you have something to investigate. And if you can't tell what "real" even looks like for that column, that is the finding: your team currently can't distinguish the system working from the system faking it.
Pull ten records and line them up. If you can't tell them apart, neither can your system.
Why this is a ladder and not a checklist
The types don't just escalate in nastiness. They mask each other. Three of my four failures shared a single upstream cause: that one swallowed-event bug starved every handler at once. The empty execution and the counterfeit output physically couldn't manifest while nothing was arriving. Fix the delivery, and only then can you see the next door. It's like peeling off a bandage to find another bandage underneath, except each one is hiding a different wound.
So the rule I climbed my way to is unglamorous and load-bearing: fix-and-check, not fix-and-ship. Repair a phantom subscription, and within the hour run an output check on the same handler. Get a clean output check, and follow it with a value check. You are not done when the first fix works. You're done when you've climbed the whole ladder and every rung holds weight.
And there's a last twist, the one that made me stop debugging and start writing a book. The checks themselves can go silent. A freshness check that loops over every flow, finds each one returns nothing or comes back stale, and skips every single iteration will report green while looking at exactly nothing. A check built to catch fresh-but-fake output can be fresh-but-fake itself. So make it count what it actually evaluated, and if that number is zero, make it say so out loud. You want to know the guard is asleep before you start trusting its silence.
None of this required a cleverer team than yours. It required asking a question most monitoring never asks (not is this handler running, but is anything real coming out of it), and asking it at four different altitudes, because the answer hides at a different one each time.
I've turned all four into paste-and-go checks (the code, the thresholds, the false alarms you'll hit in the first week) in a field manual attached to a book called Silent Wiring, on Leanpub shortly. The prompts I use to make a coding assistant surface its own integration seams, instead of quietly papering over them, are in a free prompt pack. And if you'd rather have someone run this over your system than run it yourself, that's a thing I do.
Working code that isn't working doesn't look broken. It looks like a table full of rows. The only question that matters is whether anything real put them there.