Incident management is easily one of the most annoying things anyone has to ever deal with. There will always be only a handful of people who would ever want to walk into the building on fire to mitigate. That’s the same with most engineering teams. Only a handful are willing to get in, find the root cause, and mitigate the incident.
What the heck is an incident anyway?
Formal definition:: An incident is an event that is not part of normal operations that disrupts operational processes
- Website down? Yeah that’s an incident
- server running out of space? yup, that one too
- An increasing number of transactions failing? definitely an incident
Basically, anything that interrupts the smooth operations and needs you to look into it is qualified as an incident. Or if it’s not ideal than it’s perhaps an incident.
Below are some examples of incidents::
| Severity | Incident |
|---|---|
| Revenue impacting | Website / app crashes |
| Security vulnerabilities | |
| Server crash and burnouts | |
| Booking and transaction failures | |
| Leaving customers furious | SLA breaches |
| Delayed response times | |
| Dashboards not loading | |
| Incidents needing more attention | DB backups failing |
| Queue memory overloads | |
| DB queries are too slow | |
| Application errors | |
| Good to know incidents | CI/CD Alerts |
| CPU, Memory, I/O alerts | |
| Disk space alerts |
A good rule of understanding what incidents you could get would be – Imagine the most critical part of your application and then imagine if it fails to it’s job. If this ever happens, you wan to make sure that you are getting instant alerts either on phone call, sms, email, Slack, etc. The last thing anyone want is to miss out on these incidents until you start work the next day.
Frequently Asked Questions (FAQs)
What is the formal definition of an incident?
An incident is formally defined as an event that is not part of normal operations and disrupts operational processes: anything from a website going down to a server running out of space or a rising rate of failed transactions.
What are some examples of incidents?
Examples of incidents include a website going down, a server running out of space, and an increasing number of failed transactions. Basically, anything that interrupts smooth operations and needs attention counts.
How do you know if something qualifies as an incident?
Something qualifies as an incident if it interrupts smooth operations and needs attention, or if it isn’t the ideal state of the system. A useful test is imagining the most critical part of your application failing. If that would need an instant alert, it’s an incident-worthy scenario.
How should teams get alerted about incidents?
Teams should get alerted about incidents instantly, through channels like phone calls, SMS, email, or Slack, rather than discovering them the next business day.
