Scaling humans - Ops teams and incident management
0 likes2,041 views
The document discusses the costs of downtime for companies and best practices for incident management teams to prepare for, respond to, and review incidents to minimize downtime costs. It notes that downtime costs companies billions per quarter and recommends teams prepare documentation and contact information, have on-call rotation schedules, log all responses to incidents, provide frequent status updates, gather teams to escalate incidents as needed, and conduct post-mortem reviews within days of an incident.
Scaling humans - Ops teams and incident management
- 1. Scaling Humans Ops teams and incident management dotScale, Paris 2015 David Mytton, CEO, Server Density
- 4. Cost of uptime? $2.9bn Q1: 2015
- 6. Cost of uptime? $2.9bn Q1: 2015 $870m Q1: 2015
- 8. Cost of uptime? $2.9bn Q1: 2015 $870m Q1: 2015 $4.1bn Q1: 2015
- 10. How much are you spending?
- 11. Expect downtime Prepare Respond Postmortem
- 12. Prepare On call Primary/secondary
- 13. Prepare On call Primary/secondary Reachability
- 14. Prepare On call Off call
- 15. Prepare On call Off call Docs
- 16. Prepare On call Off call Docs Searchable
- 17. Prepare On call Off call Docs Searchable Independent
- 18. Prepare
- 19. Prepare Key info Team contacts
- 20. Prepare Key info Team contacts Vendor contacts
- 21. Prepare Key info Team contacts Vendor contacts Key credentials
- 22. Prepare Key info Unexpected situations Communication
- 23. Prepare Key info Unexpected situations Communication Internet access
- 24. Prepare Key info Unexpected situations Communication Internet access Support access
- 25. Respond First responder 1. Load incident response checklist
- 26. Respond First responder 1. Load incident response checklist 2. Log into Ops War Room
- 27. Respond First responder 1. Load incident response checklist 2. Log into Ops War Room 3. Log incident in JIRA
- 28. Respond First responder 1. Load incident response checklist 2. Log into Ops War Room 3. Log incident in JIRA 4. Begin investigation
- 29. Respond Key response principles Log everything
- 30. Respond Key response principles Log everything Frequent public updates
- 31. Respond Key response principles Log everything Frequent public updates Gather the team
- 32. Respond Key response principles Log everything Frequent public updates Gather the team Escalate!
- 33. Postmortem Within a few days
- 34. Postmortem Within a few days Tell the story
- 35. Postmortem Within a few days Tell the story Appropriate technical detail
- 36. Postmortem Within a few days Tell the story Appropriate technical detail What failed, why?
- 37. Postmortem How its going to be 鍖xed
- 38. Postmortem