{"id":297,"date":"2021-03-16T09:55:06","date_gmt":"2021-03-16T09:55:06","guid":{"rendered":"https:\/\/blog.spike.sh\/2021\/03\/16\/how-to-deal-with-alert-noise\/"},"modified":"2025-06-06T14:22:30","modified_gmt":"2025-06-06T08:52:30","slug":"how-to-deal-with-alert-noise","status":"publish","type":"post","link":"https:\/\/blog.spike.sh\/how-to-deal-with-alert-noise\/","title":{"rendered":"How to Deal with Alert Noise"},"content":{"rendered":"\n\n\n

Adding alerts across your monitoring tools is taking a proactive approach to reliability. But if there are too many alerts, then it can become counterproductive because team members will start ignoring alerts or remove the alerting altogether. Which is why you need a systematic approach to adding alerts and dealing with them. <\/p>\n\n\n

\n
\"\"
Alerts in Spike.sh dashboard<\/figcaption><\/figure>\n<\/div>\n\n\n

How many alerts is too many?<\/strong><\/h2>\n\n\n\n

To understand if your team is getting too many alerts, you should begin with deciding the maximum number of alerts that your team members can handle in a given shift. This can be based on the workload of the team members and the average time needed to deal with alerts. For example, Google targets for a maximum of 2 alerts in a 12 hour shift. <\/p>\n\n\n\n

Once you have set a target, start tracking the number of alerts and bringing them up during your regular team meetings. If alerts start crossing the maximum threshold regularly, you should discuss spending more time on fixing the causes of the alerts. <\/p>\n\n\n\n

Dealing with alert noise<\/strong><\/h2>\n\n\n\n

Here are some ways of dealing with alerts. <\/p>\n\n\n\n