The Method Behind Killing a Drowning Alert Queue

I dropped into Microsoft's commerce org in 2012 and inherited a queue nobody trusted. The volume wasn't the real problem. Nobody could tell, glancing at the queue, which alert actually meant something. Everything paged the same way. Everything looked equally urgent, which meant nothing was.

The fix was a sequence, and I've run some version of it more than once since.

Start with failure modes. An alert queue that grew organically usually has one alert per symptom someone noticed at 3am and decided to guard against. That's the wrong unit. I went failure-mode by failure-mode instead, asking what actually breaks, and whether that specific alert tells an on-call engineer something they can act on right now. If the answer was no, it either got tuned to a real threshold or it got cut. Most of the queue was symptoms of the same handful of underlying failure modes, firing separately.

Put severity thresholds in writing, before you touch the alerts. I wrote what became a CEN document — communication and escalation thresholds, spelled out: what counts as a page-someone-now versus a look-at-it-in-the-morning, and who gets told what, at what point. Skip that step and cutting alerts is just guessing at what matters. Do it first, and you're applying a decision someone already signed off on.

Fix the roster before you fix the noise. A properly triaged on-call rotation matters as much as the alerts themselves. The sharpest severity thresholds don't help if the person getting paged doesn't know whether it's actually their call to make.

Then measure it, out loud, every week, for six months. I built a weekly report across every team in the Xbox Operations Center showing each team's noisiest alerts and how their trendlines moved. Ranking a team's noise next to everyone else's got it fixed faster than a private ticket ever did. Six months of that consistently, and the volume dropped enough that the queue stopped being something people tuned out.

The trendline data itself became useful, past the reporting. Once alert history was clean and consistently reported, it became possible to auto-resolve tickets against known patterns instead of a human closing each one by hand. The reporting built the dataset that let the next layer of automation exist at all.

None of that was a one-time fix. I kept refining the same underlying practice for the rest of my time running the Xbox Operations Center's alerting. The queue I inherited in 2012 was just one instance of it. Later, the method grew more rigorous. I built statistical baselines per title, comparing a rolling window against the most similar historical windows and back-testing against longer windows before trusting a threshold. I split anomalies into two tiers: sustained deviation over a real window got treated as urgent, a single-interval blip got queued for next-business-day review instead of paging anyone at 2am. I explicitly excluded the traffic patterns that would have broken a naive threshold: the overlap where East Coast and West Coast login timing create a double-humped curve that a naive threshold would flag as an anomaly, and low-population regions where traffic is too erratic for a percentage threshold to mean anything.

I also wrote the runbooks so the method didn't live only in my head. Close to fifty of them, covering incident response and severity handling. They still run without me, according to people who still work there — other teams execute them, and I'm not around to explain anything.

More recently, some of that queue-watching work has become AI-assisted. Agents that read maintenance windows and suppress the alerts they'd predictably trigger, so a human doesn't get paged for planned work in the first place. It's automation applied on top of a method that was already sound. The signal-to-noise discipline came first, from failure-mode analysis, written thresholds, and six months of public trendlines. The automation just gets to build on a queue that was already trustworthy.

That ordering matters more than it sounds like it should. A team that points AI at a queue nobody's triaged yet is automating noise, faster. The method has to exist first.