{"id":2074,"date":"2025-06-30T10:29:56","date_gmt":"2025-06-30T04:59:56","guid":{"rendered":"https:\/\/blog.spike.sh\/?p=2074"},"modified":"2025-06-30T16:14:07","modified_gmt":"2025-06-30T10:44:07","slug":"postmortem-of-escalations-triggering-out-of-order","status":"publish","type":"post","link":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/","title":{"rendered":"Postmortem of escalations triggering out of order"},"content":{"rendered":"\n

On 27th June, 2025, we identified an incident<\/a> in our escalation engine where steps fired out of sequence. Escalation policies<\/a> containing more than four escalation steps\u2014and at least 2 alerts configured in Step 3 onwards \u2014occasionally triggered step 4 (and subsequent steps) earlier than their defined intervals. Although every notification still dispatched, later steps ran sooner than expected.<\/p>\n\n\n\n

Table of contents<\/span><\/p>\n\n\n\n\n\n

\n\n\n\n

Summary of the Incident<\/strong><\/h2>\n\n\n\n

We first became aware of the problem when one of our customers reported that step 5 of their escalation policy fired earlier than expected. They noted that, although step 4 correctly ran at its scheduled time, step 5 sometimes ran at the same time as step 4 rather than at its later timestamp.<\/p>\n\n\n\n

\"\"
Step 5 triggers earlier than expected<\/figcaption><\/figure>\n\n\n\n

To reproduce the issue, we set up a policy with five steps and configured multiple alerts in step 3. <\/p>\n\n\n\n