Handling incidents
1 like792 views
The document discusses how to handle incidents, downtime, and outages. It notes that the cost of downtime for companies in Q1 2015 was $2.9 billion, $870 million, and $4.1 billion. It recommends preparing for incidents by having on-call staff and documentation, responding quickly by following an incident response checklist and notifying stakeholders, and performing a postmortem within days to analyze what failed and how to prevent future issues.
Handling incidents
- 1. How to handle incidents, downtime & outages Devopsdays, Amsterdam 2015 David Mytton, Founder, Server Density
- 5. Cost of uptime? $2.9bn Q1: 2015
- 7. Cost of uptime? $2.9bn Q1: 2015 $870m Q1: 2015
- 9. Cost of uptime? $2.9bn Q1: 2015 $870m Q1: 2015 $4.1bn Q1: 2015
- 10. Cost of uptime?
- 11. How much are you spending?
- 12. Expect downtime Prepare Respond Postmortem
- 13. Prepare On call Primary/secondary
- 14. Prepare On call Primary/secondary Reachability
- 15. Prepare On call Off call
- 16. Prepare On call Off call Docs
- 17. Prepare On call Off call Docs Searchable
- 18. Prepare On call Off call Docs Searchable Independent
- 19. Prepare
- 20. Key info Team contacts Prepare
- 21. Key info Team contacts Vendor contacts Prepare
- 22. Key info Team contacts Vendor contacts Key credentials Prepare
- 23. Key info Unexpected situations Prepare Communication
- 24. Key info Unexpected situations Prepare Communication Internet access
- 25. Key info Unexpected situations Communication Internet access Support access Prepare
- 26. Respond First responder 1. Load incident response checklist
- 27. Respond First responder 1. Load incident response checklist 2. Log into Ops War Room
- 28. Respond First responder 1. Load incident response checklist 2. Log into Ops War Room 3. Log incident in JIRA
- 29. Respond First responder 1. Load incident response checklist 2. Log into Ops War Room 3. Log incident in JIRA 4. Begin investigation
- 30. Key response principles Log everything Respond
- 31. Respond Key response principles Log everything Frequent public updates
- 32. Respond Key response principles Log everything Frequent public updates Gather the team
- 33. Respond Key response principles Log everything Frequent public updates Gather the team Escalate!
- 34. Within a few days Postmortem
- 35. Within a few days Tell the story Postmortem
- 36. Within a few days Tell the story Appropriate technical detail Postmortem
- 37. Within a few days Tell the story Appropriate technical detail What failed, why? Postmortem
- 38. Postmortem How its going to be 鍖xed
- 39. Postmortem