Alert Fatigue: How to Cut Alert Noise So On-Call Only Gets Real Pages

Alert fatigue is what happens when on-call engineers get so many alerts, most of them needing no action, that they stop trusting alerts at all. Pages get skimmed, acknowledged and closed, and the one that matters gets the same treatment. The fix is not more discipline from whoever holds the pager. It is fewer, better alerts: page only for problems users feel and someone must fix now, send the rest to chat or a ticket, and turn the burst of alerts one failure sets off into one incident.

This guide covers why alerts get noisy, how to audit the alerts you already have, before-and-after rules for the most common kinds of noise, when a baseline beats a fixed threshold, and how to group, route and silence alerts. The examples use Atatus alerting, but the approach works with any monitoring tool.

Table of Contents

• What alert fatigue looks like
• Why alerts get noisy
• How to reduce alert fatigue: audit the alerts you have
• Page on symptoms, not causes
• Stop paging on short spikes and low traffic
• Fixed thresholds vs baselines
• Group the alerts one failure sets off
• Decide what pages, and silence planned work
• Keep it quiet with a review loop
• Reducing alert noise with Atatus
• An alert fatigue checklist
• Frequently asked questions

What alert fatigue looks like

Google's SRE book describes the effect plainly: "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page that's masked by the noise." (Monitoring Distributed Systems). In practice, you see:

• Pages acknowledged and closed with nothing changed.
• Alerts that resolve on their own before anyone opens them.
• The same alert at the same time every day, during backups, batch jobs or the traffic peak.
• One outage that sends a burst of pages from many hosts and services.
• Real incidents reported first by users or support, not by monitoring.

Why alerts get noisy

1. Alerts on causes, not symptoms. High CPU on one host behind a load balancer may hurt no one.
2. Thresholds picked, not measured. Round numbers like 80% or 1 second get copied from a template and never checked against the metric's normal range.
3. Windows that react to one bad minute. The rule fires on every deploy, garbage collection pause and cron job, then closes itself.
4. One failure, many alerts. The same rule fires once per host. A slow database also slows the API, then errors climb. Each alert is correct, and together they are one incident.
5. Fixed thresholds on metrics that follow traffic. A response-time threshold that suits the peak is wrong at night.
6. Everything goes to the pager. Nothing separates "fix now" from "fix this week".
7. Planned work and orphaned rules. Deploys and backups page, and rules for retired services keep firing.

How to reduce alert fatigue: audit the alerts you have

Before you write new rules, look at what fired. A month of alert history shows which rules make the noise.

1. Pull the last 30 days of incidents. In Atatus, set the Incidents page to All and Last 1 month. The Events page also logs issues opened and closed and notifications sent, newest first, with a link to each incident, but it has no date filter or export, so expect to tally by hand.
2. Group them by rule. The Incidents page shows each incident's policy and description, not a rule column, so work out the rule from those. For each rule, note how often it notified, how often someone acted on it, how often it fired outside working hours, and whether it usually closed by itself.
3. Ask Google SRE's questions. Does the rule detect a condition that is "urgent, actionable, and actively or imminently user-visible"? Could the action wait until morning? Are other people getting paged for the same issue?
4. Give each rule one decision from the list below.
5. Fix in this order: delete, merge, tune, reroute. Deleting is the cheapest fix, and it shrinks what is left to tune.

• Fires often, nobody acts: delete it, or keep the metric on a dashboard.
• Fires and closes itself within minutes: lengthen the window, or require every minute in it to breach.
• Fires at predictable times: switch to a baseline, schedule a maintenance window, or fix the job behind it.
• Fires with other rules for the same failure: group them, and keep only the symptom rule paging.
• Needs action, but not tonight: route it to chat or a ticket.
• Users are affected and someone must act now: keep the page, and name it so the responder knows what broke and where.

Page on symptoms, not causes

A symptom is what users feel: failed requests, slow responses, pages that don't load. A cause is why: high CPU, a full connection pool, a slow query. Page on symptoms, and use cause metrics to find out why. The SRE book's four golden signals (latency, traffic, errors and saturation) are a good starting set.

# Before: pages on a cause that may not affect anyone
cpu > 80% at any point in 5 min                -> page

# After: page on what users feel
checkout error rate > 5% for 5 min             -> page
checkout response time > 2 s for 10 min        -> page
checkout requests per minute < 100 for 10 min  -> page

# Keep the cause, on a longer window, where it can wait
cpu > 90% for 30 min                           -> team chat

The values are examples. Set yours from each metric's history. Don't skip the traffic rule: when requests stop arriving, the error rate stays low and an errors-only setup stays silent through the outage.

Some causes still deserve a page when the failure is close and certain, such as a disk that will be full within hours. In the SRE book's words, "when it comes to causes, only worry about very definite, very imminent causes." If you have SLOs, paging on how fast you spend the error budget is another symptom-based approach, covered in the SRE Workbook's Alerting on SLOs.

Stop paging on short spikes and low traffic

Many noisy rules are right about the metric and wrong about time.

# Before: one bad minute is enough
response time > 2 s, at least once in 5 min   -> page

# After: the whole window must breach
response time > 2 s, every minute for 10 min  -> page

A longer window delays detection, so choose it per metric: shorter for the error rate on a critical path, longer for metrics that swing, such as CPU.

Ratios on low traffic are the other trap. At 3 a.m., two failed requests out of ten is a 20% error rate. Page only when the rate is high and there are enough failures to matter:

A: error rate > 5% for 5 min
B: failed requests > 50 for 5 min
Page when: A AND B

Fixed thresholds vs baselines

Fixed thresholds suit metrics with a hard limit or a promise behind them: disk space, an error rate tied to an SLO, a response-time target. They are predictable and easy to explain at 3 a.m. Baselines suit metrics whose normal value moves with traffic, such as throughput or response time on a service with daily peaks. A baseline rule compares the current value with recent normal and fires on a large change.

# Before: right at the peak, wrong the rest of the day
response time > 1 s for 10 min

# After: compare with the last 2 hours, and ignore small changes
response time > 50% above its 2-hour average for 10 min
  AND at least 200 ms above it
  skip when traffic is under 10 requests per minute

The absolute condition stops tiny changes from paging: a jump from 40 ms to 70 ms is a 75% increase no user will notice. The traffic floor stops a handful of requests from setting off the rule.

Baselines have limits. A moving average follows slow change, so a leak that grows a little every hour can stay inside it, and if the last two hours were already bad, so is the baseline. Keep a fixed ceiling for hard limits.

Group the alerts one failure sets off

The same rule on many targets. If a CPU rule breaches on every host in a pool, you want one incident, not one per host. In Atatus, each alert policy has an incident preference: By Policy opens one incident for all issues in the policy, By Rule one per rule, and By Rule and Target one per rule for each target. Notifications go out when an incident is opened or closed, and new issues that join an open incident don't send another page. Email and chat channels also hear when it's acknowledged, but PagerDuty, OpsGenie and Jira don't, so acknowledge pages in your paging tool. The Alert Incidents docs have worked examples.

Different rules, one failure. A slow query raises one service's response time, then errors appear on every service that calls it. Put rules that tend to fail together in one policy set to By Policy, so they open one incident instead of several.

Grouping makes noise easier to read, but it doesn't replace deleting rules. If three rules always fire together, keep the symptom rule as the page.

Decide what pages, and silence planned work

Give every rule one of three tiers when you create it:

• Page: users are affected and someone must act now. Send it to your paging tool.
• Chat or ticket: it needs action, but it can wait for working hours.
• No notification: it helps with diagnosis. Keep it on a dashboard.

Who gets paged next when nobody responds belongs in your paging tool's escalation settings, not in extra alerts. In Atatus, notification channels are attached to an alert policy, and every channel on it receives each new incident. So split policies by urgency, for example "Checkout: page" with your paging channel and only symptom rules, and "Checkout: working hours" with chat or Jira. Note that by default a Warning threshold records an issue without opening an incident or sending a notification. Only a Critical breach notifies. For an early heads-up in chat, add a second rule with a lower Critical threshold to the chat policy.

Planned work is the last source of avoidable pages. In Atatus, Schedule Maintenance creates One Time or Recurring windows. A window covers the whole account: while it is live, alert rules are not evaluated, so no alert rule can page. Keep each window as short as the work. If you deploy many times a day, fix the rules that fire on deploys instead of opening a window for each one.

Keep it quiet with a review loop

• At every on-call handover, go through each page from the shift. Was it real, urgent, and did it need a human? Fix or delete at least one noisy rule a week.
• Track pages per shift over time. Google's SRE teams review page frequency, "usually expressed as incidents per shift", with management every quarter.
• After each incident, ask which alert told you first and which added nothing, as part of your incident response review.
• Each quarter, check that rules that never fired still watch the right targets, and remove rules for retired services.

Reducing alert noise with Atatus

1. Create one policy per service and urgency under Alert Policies. By Rule is a balanced incident preference to start with.
2. Add symptom rules with Create a new rule. For APM, HTTP Failure Rate, Web Response Time, Apdex Score and Web Throughput cover errors, latency, user satisfaction and traffic. For Web Response Time and Apdex Score, choose "for at least", which fires only when every minute in the window breaches, over "at least once in", which fires on one breaching minute. HTTP Failure Rate and Web Throughput are checked as one value for the whole window, so a longer window is what smooths out their spikes, and Web Throughput counts all the requests in the window, not requests per minute.

   HTTP Failure Rate  goes Above  5     for at least 10 Mins
   Web Response Time  goes Above  2     for at least 10 Mins   (seconds)
   Web Throughput     goes Below  1000  for at least 10 Mins   (requests in 10 min)

3. Use Baseline metrics for metrics that follow traffic, such as APM and Browser metrics. The rule compares the current value with a moving average over a period you set, up to 24 hours. Add an Absolute threshold with Fire when: Both breach, and under Advanced Baseline Options set Require sustained breach, plus Minimum requests per minute on response-time metrics. That floor doesn't apply to throughput or failure-rate baselines. The Baseline Alerts docs cover the basics.
4. Combine rules with Create composite rule. Pick rules from the same policy and alert when All of the members are firing (AND). Tick Only alert as part of a composite rule on the members, so they don't raise incidents of their own.
5. Attach channels by tier. Create them under Notification Channels, then use Add notification channels on each policy page, such as PagerDuty or OpsGenie for pages and Slack, Microsoft Teams (webhook), email or Jira for the rest. Once a channel is saved, use Send Test Notification on its edit page before you rely on it. Email channels don't have the button.
6. Add Schedule Maintenance windows for planned work.

The Alert Threshold docs cover durations and time functions in more detail.

An alert fatigue checklist

• Every page is about something users feel, and someone must act on it now.
• Cause metrics go to chat or dashboards unless a hard failure is close.
• Paging rules can't fire on one bad minute, and ratio rules need a minimum volume.
• Traffic-shaped metrics use baselines with a minimum absolute change. Hard limits keep a fixed ceiling.
• One failure opens one incident.
• Paging channels sit only on paging policies, and each one has been tested.
• Planned work is covered by short maintenance windows.
• Every rule has an owner, and each shift's pages are reviewed at handover.

Frequently asked questions

What is alert fatigue in DevOps?

It is the desensitization that sets in when on-call engineers get more alerts than they can treat as urgent, most of them needing no action. People start to skim, delay or ignore alerts, so real incidents are noticed later, and night pages wear the team down.

What causes alert fatigue?

Mostly alert design: rules on causes instead of user-facing symptoms, thresholds nobody measured, windows that fire on one bad minute, many alerts for one failure, and every alert going to the pager.

How do you reduce alert fatigue without missing real incidents?

Keep pages for user-facing symptoms such as error rate, response time and traffic, including a rule for traffic dropping, and move the rest to chat, tickets or dashboards. When you're unsure about a rule, demote it to chat for a few weeks before you delete it.

Are baseline or anomaly alerts better than static thresholds?

Neither is better everywhere. Baselines suit metrics whose normal changes with traffic, and static thresholds suit hard limits and SLO targets. Baselines get noisy on low traffic and small changes, so pair them with a minimum absolute change and, where your tool supports it, a traffic floor.

How many pages per on-call shift is too many?

Google's SRE book sets a maximum of two incidents per 12-hour on-call shift, because each one takes about six hours of follow-up work, with a likely median of zero. It also says that every page should be actionable and that every page response should require intelligence, and puts the limit from the on-call engineer's side: "I can only react with a sense of urgency a few times a day before I become fatigued." Track pages per shift, and treat a rising count, or pages nobody acts on, as rules to fix.

Atatus brings threshold and baseline rules, incident grouping, composite rules and notification channels together in Atatus Alerting.