{"id":2074,"date":"2025-06-30T10:29:56","date_gmt":"2025-06-30T04:59:56","guid":{"rendered":"https:\/\/blog.spike.sh\/?p=2074"},"modified":"2025-06-30T16:14:07","modified_gmt":"2025-06-30T10:44:07","slug":"postmortem-of-escalations-triggering-out-of-order","status":"publish","type":"post","link":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/","title":{"rendered":"Postmortem of escalations triggering out of order"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">On 27th June, 2025, we <a href=\"https:\/\/status.spike.sh\/status-page\/61ac48c5c9e70f278642b5e9\/incident\/685e6501685fa1a1d6580f36\/preview\">identified an incident<\/a> in our escalation engine where steps fired out of sequence. <a href=\"https:\/\/spike.sh\/escalations\">Escalation policies<\/a> containing more than four escalation steps\u2014and at least 2 alerts configured in Step 3 onwards \u2014occasionally triggered step 4 (and subsequent steps) earlier than their defined intervals. Although every notification still dispatched, later steps ran sooner than expected.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span style=\"text-decoration: underline;\">Table of contents<\/span><\/p>\n\n\n\n<nav aria-label=\"Table of Contents\" class=\"wp-block-table-of-contents\"><ol><li><a class=\"wp-block-table-of-contents__entry\" href=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#summary-of-the-incident\">Summary of the Incident<\/a><\/li><li><a class=\"wp-block-table-of-contents__entry\" href=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#detection\">Detection<\/a><\/li><li><a class=\"wp-block-table-of-contents__entry\" href=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#recovery\">Recovery<\/a><\/li><li><a class=\"wp-block-table-of-contents__entry\" href=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#lessons-learned-and-next-steps\">Lessons Learned and Next Steps<\/a><\/li><li><a class=\"wp-block-table-of-contents__entry\" href=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#closing-notes\">Closing Notes<\/a><\/li><\/ol><\/nav>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"summary-of-the-incident\"><strong>Summary of the Incident<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We first became aware of the problem when one of our customers reported that step 5 of their escalation policy fired earlier than expected. They noted that, although step 4 correctly ran at its scheduled time, step 5 sometimes ran at the same time as step 4 rather than at its later timestamp.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"758\" data-attachment-id=\"2079\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/escalation-steps-incorrect-ordering\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering.png\" data-orig-size=\"1112,823\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"escalation-steps-incorrect-ordering\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering-1024x758.png\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering-1024x758.png\" alt=\"\" class=\"wp-image-2079\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering-1024x758.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering-300x222.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering-768x568.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/escalation-steps-incorrect-ordering.png 1112w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Step 5 triggers earlier than expected<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">To reproduce the issue, we set up a policy with five steps and configured multiple alerts in step 3. <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Step 3<\/strong>: fired at 9:00 AM<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Step 4<\/strong>: fired at 9:05 AM<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Step 5<\/strong>: scheduled for 9:15 AM, but also ran at 9:05 AM<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This behavior defied our design: Spike only ever queues the very next step after an alert goes out. That queuing logic prevents the system from overloading and lets us decide in real time which escalation step should run next. It\u2019s worth noting that each escalation step can contain multiple alerts, and in this scenario the mis-queue occurred even with multipe alerts in step 3. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After receiving the initial report on 26th June, we tried and failed to replicate the issue in our staging environment. That led us to pull logs from multiple active users\u2019 policies to see if others had been affected. We reviewed escalation\u2010queue entries and alert timestamps but didn\u2019t find any additional confirmed cases.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"detection\"><strong>Detection<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We first attempted to reproduce the issue in both staging and production by creating escalation policies with five and six steps, and by varying the number of alerts per step. Despite repeated tests, we saw no anomalies. To gain more visibility, we routed all queue and timestamp data through our internal debugging service, <strong>Warden<\/strong>, and augmented our logging to capture every enqueue operation. <\/p>\n\n\n\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"745\" data-attachment-id=\"2096\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/cleanshot-2025-06-30-at-10-13-03-2-2\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1.png\" data-orig-size=\"1103,802\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CleanShot 2025-06-30 at 10.13.03 2\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1-1024x745.png\" data-id=\"2096\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1-1024x745.png\" alt=\"\" class=\"wp-image-2096\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1-1024x745.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1-300x218.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1-768x558.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.13.03-2-1.png 1103w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"878\" data-attachment-id=\"2092\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/cleanshot-2025-06-30-at-10-12-16-2\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2.png\" data-orig-size=\"1117,958\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CleanShot 2025-06-30 at 10.12.16 2\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2-1024x878.png\" data-id=\"2092\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2-1024x878.png\" alt=\"\" class=\"wp-image-2092\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2-1024x878.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2-300x257.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2-768x659.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.16-2.png 1117w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"809\" data-attachment-id=\"2102\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/cleanshot-2025-06-30-at-10-12-59-2-2\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1.png\" data-orig-size=\"1113,879\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CleanShot 2025-06-30 at 10.12.59 2\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1-1024x809.png\" data-id=\"2102\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1-1024x809.png\" alt=\"\" class=\"wp-image-2102\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1-1024x809.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1-300x237.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1-768x607.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.59-2-1.png 1113w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"851\" data-attachment-id=\"2098\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/cleanshot-2025-06-30-at-10-12-05-2-2\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1.png\" data-orig-size=\"1109,922\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CleanShot 2025-06-30 at 10.12.05 2\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1-1024x851.png\" data-id=\"2098\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1-1024x851.png\" alt=\"\" class=\"wp-image-2098\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1-1024x851.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1-300x249.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1-768x638.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.12.05-2-1.png 1109w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"826\" data-attachment-id=\"2097\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/cleanshot-2025-06-30-at-10-11-56-2-2\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1.png\" data-orig-size=\"1113,898\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CleanShot 2025-06-30 at 10.11.56 2\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1-1024x826.png\" data-id=\"2097\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1-1024x826.png\" alt=\"\" class=\"wp-image-2097\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1-1024x826.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1-300x242.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1-768x620.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.56-2-1.png 1113w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<figcaption class=\"blocks-gallery-caption wp-element-caption\">Gallery of multiple escalations we tried during our investigation<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Early in the investigation, we hypothesized a feature-flag misconfiguration\u2014since the report came from a new customer\u2014but quickly ruled this out. Our escalation engine isn\u2019t gated by flags based on plan or customer tier; alerts and step-count limits vary only by plan, not by a feature toggle governing queuing behavior.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The enhanced logs revealed the true culprit<\/strong>: when step 3 contained multiple alerts, each alert invocation independently re-queued \u201cthe next\u201d step using the same base timestamp. The first alert in step 3 correctly queued step 4 at T+5 minutes, but the second alert also read the original timestamp and enqueued step 5 at T+5 minutes\u2014skipping the intended interval. The timeline below might illustrate this better:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Step 3 fires<\/strong> at 9:00 AM and has 2 alerts: A and B<\/li>\n\n\n\n<li><strong>Alert A<\/strong> queues <strong>Step 4<\/strong> for 9:05 AM<\/li>\n\n\n\n<li><strong>Alert B<\/strong> also queues its \u201cnext\u201d step\u2014<strong>Step 5<\/strong>\u2014for 9:05 AM instead of 9:15 AM<\/li>\n\n\n\n<li><strong>Result<\/strong>: Step 5 runs too early because each alert reused the original timestamp rather than chaining off the previous step.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This timestamp-queuing logic flaw only arises when a step has more than one alert and the step number is &gt; 2<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"737\" data-attachment-id=\"2100\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/cleanshot-2025-06-30-at-10-11-36\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36.png\" data-orig-size=\"1147,826\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CleanShot 2025-06-30 at 10.11.36\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36-1024x737.png\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36-1024x737.png\" alt=\"\" class=\"wp-image-2100\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36-1024x737.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36-300x216.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36-768x553.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/CleanShot-2025-06-30-at-10.11.36.png 1147w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Multiple alerts queueing incorrectly<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The internal service just pulls a lot more data and we built it to customise what we see in production. <em>Kinda like YouTube stats for nerds<\/em>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"615\" data-attachment-id=\"2081\" data-permalink=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/stats-for-nerds\/\" data-orig-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds.png\" data-orig-size=\"1146,688\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"stats for nerds\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds-1024x615.png\" src=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds-1024x615.png\" alt=\"YouTube stats for nerds\" class=\"wp-image-2081\" style=\"aspect-ratio:1.665065642010887;object-fit:cover\" srcset=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds-1024x615.png 1024w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds-300x180.png 300w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds-768x461.png 768w, https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/stats-for-nerds.png 1146w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">YouTube stats for nerds<\/figcaption><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"recovery\"><strong>Recovery<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once we confirmed the queuing-logic flaw, the team updated the escalation engine to enqueue only one \u201cnext\u201d step per escalation, regardless of how many alerts a step contains. We merged the fix, ran a quick smoke test in staging, and rolled it out to production. Post-deployment monitoring via Warden showed all steps queuing and firing in the correct order, and no further out-of-order executions have occurred. Service consistency was fully restored shortly after the patch went live.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"lessons-learned-and-next-steps\"><strong>Lessons Learned and Next Steps<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Strengthen Logging Around Queue Operations<\/strong><br>Enhance log granularity for enqueue\/dequeue actions and include contextual metadata (step number, alert ID) so future investigations surface root causes faster.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Automated Production Escalation Testing<\/strong><br>Leverage a daily Cron job in production to trigger incidents across varied escalation policies (different step counts and alert combinations). This live test will make sure our end-to-end queuing logic behaves correctly under real conditions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Anomaly Detection for Escalation Timing<\/strong><br>Map each alert\u2019s expected send timestamp per escalation step, then compare actual execution times on every escalation. If any step fires out of order, automatically open an incident in Spike and notify the on-call team.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"closing-notes\">Closing Notes<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This was a hard-to-catch issue, and we\u2019re grateful to the users who brought it to our attention. <strong>We apologize for any confusion it caused.<\/strong> As we strengthen our escalation engine, we\u2019ll keep improving our testing and monitoring. At Spike, we believe in honesty and transparency\u2014please subscribe to our status page for real-time updates. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thank you for your support.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-horizontal is-content-justification-center is-layout-flex wp-container-core-buttons-is-layout-cbad53a9 wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link has-base-color has-primary-background-color has-text-color has-background has-link-color wp-element-button\" style=\"border-radius:6px\">Subscribe to our status page<\/a><\/div>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>On 27th June, 2025, we identified an incident in our escalation engine where steps fired out of sequence. Escalation policies containing more than four escalation steps\u2014and at least 2 alerts configured in Step 3 onwards \u2014occasionally triggered step 4 (and subsequent steps) earlier than their defined intervals. Although every notification still dispatched, later steps ran [&hellip;]<\/p>\n","protected":false},"author":191914268,"featured_media":2120,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","_lmt_disableupdate":"","_lmt_disable":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[1424,1444],"tags":[1385,1391],"class_list":["post-2074","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-building-spike","category-postmortem","tag-incidents","tag-postmortem"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Postmortem of escalations triggering out of order - Spike&#039;s blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Postmortem of escalations triggering out of order - Spike&#039;s blog\" \/>\n<meta property=\"og:description\" content=\"On 27th June, 2025, we identified an incident in our escalation engine where steps fired out of sequence. Escalation policies containing more than four escalation steps\u2014and at least 2 alerts configured in Step 3 onwards \u2014occasionally triggered step 4 (and subsequent steps) earlier than their defined intervals. Although every notification still dispatched, later steps ran [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/\" \/>\n<meta property=\"og:site_name\" content=\"Spike&#039;s blog\" \/>\n<meta property=\"article:published_time\" content=\"2025-06-30T04:59:56+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-30T10:44:07+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1040\" \/>\n\t<meta property=\"og:image:height\" content=\"564\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Kaushik\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kaushik\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/\"},\"author\":{\"name\":\"Kaushik\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/#\\\/schema\\\/person\\\/b137e57ace218547f02b86fdcb2d0e64\"},\"headline\":\"Postmortem of escalations triggering out of order\",\"datePublished\":\"2025-06-30T04:59:56+00:00\",\"dateModified\":\"2025-06-30T10:44:07+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/\"},\"wordCount\":836,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/blog.spike.sh\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/postmorte-escalations-triggering-out-of-order.png\",\"keywords\":[\"incidents\",\"postmortem\"],\"articleSection\":[\"Building Spike\",\"Postmortem\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/\",\"url\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/\",\"name\":\"Postmortem of escalations triggering out of order - Spike&#039;s blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/blog.spike.sh\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/postmorte-escalations-triggering-out-of-order.png\",\"datePublished\":\"2025-06-30T04:59:56+00:00\",\"dateModified\":\"2025-06-30T10:44:07+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/#\\\/schema\\\/person\\\/b137e57ace218547f02b86fdcb2d0e64\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#primaryimage\",\"url\":\"https:\\\/\\\/blog.spike.sh\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/postmorte-escalations-triggering-out-of-order.png\",\"contentUrl\":\"https:\\\/\\\/blog.spike.sh\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/postmorte-escalations-triggering-out-of-order.png\",\"width\":1040,\"height\":564,\"caption\":\"postmorte-escalations triggering out of order\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/postmortem-of-escalations-triggering-out-of-order\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/blog.spike.sh\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Postmortem of escalations triggering out of order\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/#website\",\"url\":\"https:\\\/\\\/blog.spike.sh\\\/\",\"name\":\"Spike&#039;s blog\",\"description\":\"Learnings and opinions in a changing world\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/blog.spike.sh\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/blog.spike.sh\\\/#\\\/schema\\\/person\\\/b137e57ace218547f02b86fdcb2d0e64\",\"name\":\"Kaushik\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/c7ec6b633161978fc09ed325cefde9061797a65a730e4b98c0eb26bc6925bc81?s=96&d=robohash&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/c7ec6b633161978fc09ed325cefde9061797a65a730e4b98c0eb26bc6925bc81?s=96&d=robohash&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/c7ec6b633161978fc09ed325cefde9061797a65a730e4b98c0eb26bc6925bc81?s=96&d=robohash&r=g\",\"caption\":\"Kaushik\"},\"description\":\"Founder of Spike. I like sharing how we are building Spike and the intricacies of building a startup by waking people up for critical incidents.\",\"url\":\"https:\\\/\\\/blog.spike.sh\\\/author\\\/spikehq\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Postmortem of escalations triggering out of order - Spike&#039;s blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/","og_locale":"en_GB","og_type":"article","og_title":"Postmortem of escalations triggering out of order - Spike&#039;s blog","og_description":"On 27th June, 2025, we identified an incident in our escalation engine where steps fired out of sequence. Escalation policies containing more than four escalation steps\u2014and at least 2 alerts configured in Step 3 onwards \u2014occasionally triggered step 4 (and subsequent steps) earlier than their defined intervals. Although every notification still dispatched, later steps ran [&hellip;]","og_url":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/","og_site_name":"Spike&#039;s blog","article_published_time":"2025-06-30T04:59:56+00:00","article_modified_time":"2025-06-30T10:44:07+00:00","og_image":[{"width":1040,"height":564,"url":"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png","type":"image\/png"}],"author":"Kaushik","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kaushik","Estimated reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#article","isPartOf":{"@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/"},"author":{"name":"Kaushik","@id":"https:\/\/blog.spike.sh\/#\/schema\/person\/b137e57ace218547f02b86fdcb2d0e64"},"headline":"Postmortem of escalations triggering out of order","datePublished":"2025-06-30T04:59:56+00:00","dateModified":"2025-06-30T10:44:07+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/"},"wordCount":836,"commentCount":0,"image":{"@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png","keywords":["incidents","postmortem"],"articleSection":["Building Spike","Postmortem"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/","url":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/","name":"Postmortem of escalations triggering out of order - Spike&#039;s blog","isPartOf":{"@id":"https:\/\/blog.spike.sh\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#primaryimage"},"image":{"@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png","datePublished":"2025-06-30T04:59:56+00:00","dateModified":"2025-06-30T10:44:07+00:00","author":{"@id":"https:\/\/blog.spike.sh\/#\/schema\/person\/b137e57ace218547f02b86fdcb2d0e64"},"breadcrumb":{"@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#primaryimage","url":"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png","contentUrl":"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png","width":1040,"height":564,"caption":"postmorte-escalations triggering out of order"},{"@type":"BreadcrumbList","@id":"https:\/\/blog.spike.sh\/postmortem-of-escalations-triggering-out-of-order\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.spike.sh\/"},{"@type":"ListItem","position":2,"name":"Postmortem of escalations triggering out of order"}]},{"@type":"WebSite","@id":"https:\/\/blog.spike.sh\/#website","url":"https:\/\/blog.spike.sh\/","name":"Spike&#039;s blog","description":"Learnings and opinions in a changing world","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.spike.sh\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/blog.spike.sh\/#\/schema\/person\/b137e57ace218547f02b86fdcb2d0e64","name":"Kaushik","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/c7ec6b633161978fc09ed325cefde9061797a65a730e4b98c0eb26bc6925bc81?s=96&d=robohash&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c7ec6b633161978fc09ed325cefde9061797a65a730e4b98c0eb26bc6925bc81?s=96&d=robohash&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c7ec6b633161978fc09ed325cefde9061797a65a730e4b98c0eb26bc6925bc81?s=96&d=robohash&r=g","caption":"Kaushik"},"description":"Founder of Spike. I like sharing how we are building Spike and the intricacies of building a startup by waking people up for critical incidents.","url":"https:\/\/blog.spike.sh\/author\/spikehq\/"}]}},"modified_by":"Kaushik","jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/blog.spike.sh\/wp-content\/uploads\/2025\/06\/postmorte-escalations-triggering-out-of-order.png","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfMe4Q-xs","jetpack-related-posts":[{"id":4457,"url":"https:\/\/blog.spike.sh\/incident-postmortem\/","url_meta":{"origin":2074,"position":0},"title":"Incident Postmortem: How to Learn From Failures and Build Reliable Systems","author":"Samyati Mohanty","date":"27th November, 2025","format":false,"excerpt":"Incident postmortems help teams learn from outages without blame. This guide explains what they are, how to run them well, and how they strengthen reliability and continuous improvement.","rel":"","context":"In &quot;Uncategorized&quot;","block_context":{"text":"Uncategorized","link":"https:\/\/blog.spike.sh\/category\/uncategorised\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Getting-started-with-Incident-Management-1.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Getting-started-with-Incident-Management-1.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Getting-started-with-Incident-Management-1.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Getting-started-with-Incident-Management-1.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Getting-started-with-Incident-Management-1.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Getting-started-with-Incident-Management-1.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":4354,"url":"https:\/\/blog.spike.sh\/how-to-run-blameless-postmortem\/","url_meta":{"origin":2074,"position":1},"title":"How to Conduct a Blameless Postmortem","author":"Randhir Kumar","date":"20th November, 2025","format":false,"excerpt":"Incidents happen. A blameless postmortem is how your team learns from them without finger-pointing. This blog explains how to run an effective postmortem and build a resilient engineering culture.","rel":"","context":"In &quot;Post Incident&quot;","block_context":{"text":"Post Incident","link":"https:\/\/blog.spike.sh\/category\/incident-management\/post-incident\/"},"img":{"alt_text":"Blog cover titled \"How to Conduct a Blameless Postmortem\"","src":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/The-Top-10-On-Call-Management-Tools-for-DevOps.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/The-Top-10-On-Call-Management-Tools-for-DevOps.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/The-Top-10-On-Call-Management-Tools-for-DevOps.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/The-Top-10-On-Call-Management-Tools-for-DevOps.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":4114,"url":"https:\/\/blog.spike.sh\/jsm-review-for-incident-response\/","url_meta":{"origin":2074,"position":2},"title":"Jira Service Management (JSM) Review for Incident Response (2026)","author":"Sreekar","date":"12th November, 2025","format":false,"excerpt":"OpsGenie is shutting down, and Atlassian recommends Jira Service Management (JSM). But is JSM the right choice for incident response? I tested JSM thoroughly and reviewed it across four key criteria. Here's what I found.","rel":"","context":"In &quot;JSM&quot;","block_context":{"text":"JSM","link":"https:\/\/blog.spike.sh\/category\/comparison\/jsm\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-12.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-12.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-12.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-12.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-12.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-12.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":4174,"url":"https:\/\/blog.spike.sh\/jsm-vs-spike-for-incident-management\/","url_meta":{"origin":2074,"position":3},"title":"Jira Service Management (JSM) vs. Spike: Which is a Better OpsGenie Alternative in 2026","author":"Sreekar","date":"13th November, 2025","format":false,"excerpt":"Atlassian is shutting down OpsGenie, and if you are stuck between Jira Service Management (JSM) vs. Spike for incident management, this blog is for you. I signed up for both, ran identical tests, and compared them across key criteria.","rel":"","context":"In &quot;JSM&quot;","block_context":{"text":"JSM","link":"https:\/\/blog.spike.sh\/category\/comparison\/jsm\/"},"img":{"alt_text":"Blog cover titled \"JSM vs. Spike: Incident Management\"","src":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-3.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-3.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-3.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/11\/Basics-of-Incident-Management-3.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":3442,"url":"https:\/\/blog.spike.sh\/incidentio-alternatives-2026\/","url_meta":{"origin":2074,"position":4},"title":"5 Better Incident.io Alternatives (2026)","author":"Sreekar","date":"4th October, 2025","format":false,"excerpt":"Looking for Incident.io alternatives? I tested 5 better incident management tools and compared their alerting, on-call scheduling, and automation features. Read the blog and find the best Incident.io alternative for your team.","rel":"","context":"In &quot;Incident.io&quot;","block_context":{"text":"Incident.io","link":"https:\/\/blog.spike.sh\/category\/comparison\/incident-io\/"},"img":{"alt_text":"Blog cover titled \"5 better incident.io alternatives\"","src":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-7.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-7.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-7.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-7.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":3396,"url":"https:\/\/blog.spike.sh\/zenduty-vs-spike-detailed-comparison-2026\/","url_meta":{"origin":2074,"position":5},"title":"Zenduty vs. Spike: A Detailed Comparison (2026)","author":"Sreekar","date":"1st October, 2025","format":false,"excerpt":"This in-depth\u00a0Zenduty vs. Spike\u00a0comparison helps you choose the right incident management tool. I tested both platforms on alerting, on-call management, incident response, and pricing to see how they stack up. Find out which one is the best fit for your team's needs in 2026.","rel":"","context":"In &quot;Zenduty&quot;","block_context":{"text":"Zenduty","link":"https:\/\/blog.spike.sh\/category\/comparison\/zenduty\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-14.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-14.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-14.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.spike.sh\/wp-content\/uploads\/2025\/10\/Basics-of-Incident-Management-14.png?resize=700%2C400&ssl=1 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/posts\/2074","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/users\/191914268"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/comments?post=2074"}],"version-history":[{"count":23,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/posts\/2074\/revisions"}],"predecessor-version":[{"id":2123,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/posts\/2074\/revisions\/2123"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/media\/2120"}],"wp:attachment":[{"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/media?parent=2074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/categories?post=2074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.spike.sh\/wp-json\/wp\/v2\/tags?post=2074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}