Naming them turned out to be the most practically useful thing I did, because once you can tell which type you are looking at, you immediately know which check would have caught it. And, more expensively, which check will not help.
The property that ties them together is the one that cost me five months: each type is invisible to the check that catches the one below it. A handler timestamp catches the first in seconds. The second sails past it, because the handler genuinely is firing. An output check catches the second. The third sails past that, because the output is genuinely there. And so on, up the ladder.
| Type | The question | The signal | What to check |
|---|---|---|---|
| 1. Phantom subscription | Is it firing? | last_event_at reads never | Handler timestamps, bus registrations |
| 2. Empty execution | Is it writing? | Target row count is zero | Output tables, consumer-side reads |
| 3. Counterfeit output | Is the output real? | Variance collapsed against expected | Expected ranges, distribution over time |
| 4. Monoculture | Is the output diverse? | Entropy near zero | Distribution analysis, population variance |
Type 1. The phantom subscription
The handler that never fires. It exists everywhere it is supposed to: registered, configured, discoverable in your architecture. It never runs. Sometimes the subscription was never wired. Sometimes it is wired correctly and the events are being eaten upstream, which is what happened to me: a reasonable-looking except RuntimeError: pass in the event publisher swallowed every event when the worker threads had no event loop to run on. Different mechanism, same outcome. What they share is silence.
Why monitoring misses it. Your monitoring watches for failures: errors, timeouts, latency spikes. A handler that never fires produces none of those. It produces an absence, and absence does not page anyone. In an event-driven system, silence is what normal sounds like, so a handler that has gone quiet is indistinguishable from a handler with nothing to do.
The check. One timestamp per handler, last_event_at. If it reads never, you have a phantom. This is a different question from the one your health check answers: that tells you the process is up, this tells you a specific handler did something, recently, at all.
Type 2. The empty execution
The handler that fires and does nothing. It runs. You can see it in the logs, the counter climbs, the probe is green and correct to be. It just produces nothing anyone uses. Sometimes it hits a silent catch block and returns. Sometimes it writes to a table no consumer reads, which is the purer version: there is no error handling to find, because every line is correct. Mine came from a naming convention that changed between two AI sessions. The handler wrote to one table, the consumers read another, both existed, both names were valid, and neither side complained.
Why the Type 1 check misses it. Every check designed to catch a phantom passes clean, because the handler is registered and is firing. Those are now true and no longer useful.
The check. Stop asking whether it ran and start asking whether it wrote anything anyone reads. Count rows on the target, and count them from the consumer's side rather than the writer's. The two sound like the same question right up until the day they are not.
Type 3. The counterfeit output
The handler that fires, writes, and lies. Right table, right format, one row every few minutes, month after month: the kind of table you would pull up to prove to your manager that a system is healthy. The handler was not doing the work. A fallback path, a cold-start safety net built for first boot when there is no history to average yet, had quietly become the only thing writing. With no real input to compute from it did what a moving average does when the input stops. It decayed, smoothly and plausibly, drifting downward the way a real instrument settles as it finds its level, and then it died in place.
Why the Type 2 check misses it. There is output. There is plenty of it. Every freshness check reports green and reports it correctly, because the data genuinely is fresh. It is fresh and fake at the same time. Worse than a hardcoded default, because every value differs from every other value: go hunting for the obvious tell, a column of identical numbers or a suspiciously round figure, and you will find variety, nod, and close the tab.
The check. On live data you have no ground truth, so do not ask whether the value is wrong. Ask whether it is the product of computation at all, and answer it with spread: real work produces variance, a decayed fallback does not. Be careful which check you reach for. One that pages when the number matches the fallback's value catches almost nothing and catches it weeks late, because the fallback drifts through perfectly plausible territory before it settles. One that asks whether the number has any spread at all catches it in hours.
Type 4. The monoculture
The handlers that all quietly agree to be the same. Every previous type has something identifiably wrong with it: a missing subscription, an empty table, a value that stopped moving. This one does not. Every individual output is valid, computed, and defensible. Mine was a fitness engine whose scores had collapsed into a three-cent band, with two independent bugs behind it: one starved the population of new material at the source, the other flattened whatever variety was left. Neither needed the other. Together they made it inevitable.
Why the Type 3 check misses it. The values are real. They were computed, not defaulted. They vary, and they change over time. Every check up to here passes honestly. The failure exists only at the population level, and no check that looks at one row at a time will ever see it.
The check. Distribution, not validity. What entropy should this population have, and what does it have? This is the only one of the four that needs genuine domain knowledge, because you have to already know what healthy looks like before you can recognise that something is sick, and that answer is different for every system. It is also, not coincidentally, the expertise that goes missing when a module is generated by a model nobody on the team has read closely.
It is a ladder, not a checklist
The types do not just escalate. They mask each other. Three of my four failures shared one upstream cause, a single swallowed-event bug that starved every handler at once, which meant the empty execution and the counterfeit output physically could not manifest while nothing was arriving. Fix the delivery and only then can you see the next door. It is like peeling off a bandage to find another bandage underneath, except each one is hiding a different wound.
Which gives the one operating rule worth taking from this page: fix and check, not fix and ship. Repair a phantom subscription, and within the hour run an output check on the same handler. Get a clean output check, and follow it with a value check. You are not done when the first fix works. You are done when you have climbed the whole ladder and every rung holds weight.
I fixed two of mine in the same commit, which collapsed the ladder and means I will never know what the second one looked like on its own. Fix one rung, check the next, then commit.
The long version of all four, with what each one was like to find, is the field note Four ways code can run, write a row, and still do nothing. If you want the term itself rather than the taxonomy, start with what silent wiring is.