Skip to content

Migrate from OpsGenie to Spike and never miss a critical alert.Get 20% off your first year 

Figuring out on-call

A weekly email for engineers who carry the pager.

Previously sent

Inbox · Figuring out on-call 27

No. 27 · Nov 19, 2026 · Response

How GitHub's incident came back after a rollback

The rollback worked, but that was only half the story.

A rollback usually feels like the end of an incident. The graphs look normal, so you resolve it and move on.

GitHub had a rollback like that on November 4. About 25 minutes into an incident, the team rolled back a config change to an internal event-routing service.

That change had paused several event streams, which delayed Actions runs and webhook deliveries. After the rollback, events flowed again and everything seemed fixed.

But the rollback only reverted the config. The code change behind it was still there, so a later deployment shipped it again.

So the same incident came back half an hour later and ran another 35 minutes.

That's because a rollback undoes what's running, but it doesn't touch what's waiting to ship.

So after a rollback, it's helpful to check whether the code change was reverted too. That way the next deploy ships the fix, not the incident.

See you next week.

Enter to open, arrows to move, Esc to close

Browse everything on spike.sh