A Monday in early summer. Two of us are grinding through a ticket queue that never empties, and my colleague closes a disk space ticket on a production database server because he reads it as a memory alarm. It takes him about two seconds, which was roughly the time budget each ticket had.
He was not careless. He was doing exactly what the system had trained both of us to do for months. This is what alert fatigue looks like from the inside.
At the time I was working for a global IT services provider, on the account of a car manufacturer’s plant in Turkey. Around 300 Windows servers. A monitoring platform run by another team, under what everyone called the global standard.
The standard that generated its own weather
The rule was simple enough to fit on a slide: monitor every Windows service set to automatic start, and raise an alarm when one of them is not running.
Google Chrome installs an update service. That service sets itself to autostart, and then Chrome starts and stops it whenever it feels like it. More than 100 of those 300 servers had Chrome on them. Browser updates on that estate came from SCCM anyway, so the updater had nothing useful to do except breathe in and out.
Our monitoring treated every one of those breaths as an incident worth a ticket.
Hundreds of “service stopped” tickets a day. Two of us closing them, including during meetings, one eye on the call and one hand on the queue. You learn a rhythm. You stop reading the ticket body and start reading its shape: subject line, server name, close.
The second source of noise
Same period, same account. The customer would not let us set a maximum memory limit on SQL Server. The position was stated plainly: “I paid for it, I will use all of it.”
So every night, while reports were generated, the database servers ran at 100% RAM and CPU for about ten minutes. The alarm fired in the middle of the night. We had no shift team. By the time anyone looked in the morning, everything had long since returned to normal, and the SLA clock had already charged us a penalty for responding late.
We asked for something modest. Raise the resource alarm only if CPU or RAM stays at 100% for at least an hour, and treat out-of-hours differently. The answer from the monitoring team was: “I can’t make a custom setting. If I do it for everyone, every region defines its own standard and I can’t manage it.”
I understand that answer better than I like to admit. A standard you can enforce everywhere is easy to govern. A standard nobody can adjust is not a standard, it is a weather system, and you dress for it instead of arguing with it.
What we dressed in was bulk closure. Three or four tickets for the same condition on the same server was normal. Closing tickets stopped being an action and became a reflex.
The signal that did exist
This is the part that still bothers me. The disk alarm was not missing. Disk space monitoring was in place, the threshold fired, the platform did its job.
It did its job exactly once. The alarm was converted into a ticket, and from that moment the platform considered the matter handled. The alarm itself never cleared, because the disk never got bigger. But a human closed the ticket, and nothing on either side ever checked whether the other side agreed.
No reminder after 24 hours. No escalation for an alarm that stays open for a week. No rule that closes a ticket when the alarm clears, and refuses to close it while the alarm is still active. An open alarm sat in a console for three months, quietly satisfied that it had already spoken.
Three months later
The disk on the production database server filled. SQL services stopped. Production in the plant stopped with them. The loss is reported to have run into thousands of dollars; that is how it was told to us, and I have no invoice to put next to it.
We extended the disk immediately, which felt useful and was not. The line had already been down.
Nobody blamed my colleague, and I want to be clear that this was not managerial generosity. The managers knew about the noise. They had been reading the same monthly service reviews we had, where the question “why did you respond late?” came up every month and received the same copy-paste answer: “the ticket arrived outside business hours, we don’t have a shift team.”
An unhappy customer, unhappy staff, and money gone. That is what the global standard bought.
Why nobody saw it: alert fatigue at work
The technical reason is that the platform alerted on state changes that were normal behaviour for the software involved, applied no duration threshold to resource alarms, created tickets in a fire-and-forget fashion, and never reconciled ticket state with alarm state.
The human reason is shorter. Two people cannot read hundreds of tickets a day, so they stop reading. Once you stop reading, every ticket looks like the last one, and a disk ticket looks like a memory ticket. The noise did not hide the signal from the monitoring system. It hid the signal from us.
Monitoring things that do not need monitoring is not extra safety. It is the thing that kills monitoring.
What would have caught it
- A noise budget. Set a weekly ticket-count ceiling per alarm class. Any class that goes over it is either suppressed or has its threshold rewritten, and that decision has an owner and a date.
- Duration thresholds on resource alarms. CPU or RAM at 100% for ten minutes during a nightly batch is a report running. For an hour, it is an incident. Encode the difference instead of paying SLA penalties for it.
- An explicit out-of-hours policy. If there is no shift team, say so in the contract and in the alert routing, and stop pretending a night alarm has a responder.
- Re-notification and escalation for alarms that stay open. If the condition is still true 24 hours after the first ticket, the system speaks again, louder, to someone else. “I already raised a ticket” is not a state anyone should be allowed to rest in.
- Automatic closure driven by alarm state. The ticket closes when the alarm clears, not when a tired human clicks. If the alarm is still active, the ticket reopens.
- A documented exception path for the standard. Not a favour asked in a chat window, but a request with a form, an approver and an SLA of its own, so “I can’t manage exceptions” becomes a process rather than a refusal.
That last one is the political item, and it is the one that actually decides whether the other five ever happen.
On Monday, count last week’s tickets by alarm class, find the loudest class, and go kill it before you tune anything else.
Sources
- First-hand account by the author. No external sources were used; company, customer, host names and exact dates are deliberately omitted.