Flight training for DevOps & HumanOps - IncontroDevOps 2016
3 likes2,470 views
HumanOps is a set of principles which focus on the human aspects of running infrastructure. Talk in IncontroDevOps 2016.
Flight training for DevOps & HumanOps - IncontroDevOps 2016
- 1. Jorge Salamero Sanz <jsalamero@serverdensity.com> IncontroDevOps 1 April 2016 War Games - Flight training for DevOps
- 2. How to Monitor MySQL
- 3. Infrastructure automation Configuration automation Continuous testing Continuous deployment / delivery Monitoring Logs, error handling Feedback Human Ops DevOps lifecycle
- 4. Humans are part of any system Initial design, ongoing improvements Maintenance Upgrades Issues, Incident response Humans in DevOps
- 5. System issues = error rates + SLA + ... Human issues = alerts out of hours + interruptions + . System issues = Human issues Human issues = system issues
- 6. System health impacts human health Human health impacts system health Humans impact systems
- 7. Downtime = loss of users, reputation, revenue Downtime caused by unreliable systems Unhealthy teams reduce reliability Unhealthy teams = loss of users, reputation, revenue Humans impact business
- 8. Slip Lapse Mistake Violation (Always, again, again) Human risk
- 9. Prepare and practice Respond Postmortem Expect downtime
- 10. Real example (small war story, wont be long)
- 11. Power failure to half of our servers Automated failover unavailable (known failure condition) Manual DNS switch required Expected impact: 20 min Actual impact: 43min Incident example
- 13. Lesson learned?
- 14. Unfamiliarity with the process Pressure of time sensitive event (panic effect) Escalation introduces delays The Human Factor
- 15. Handling the Human factor
- 16. First responder, acknowledge alert Load incident response checklist Log into #ops-war-room in Slack Log incident into JIRA Begin investigation General response process
- 17. 1. Extended use of checklists Documented procedures
- 18. The limits of human memory and attention Complexity Stress and fatigue Ego Pilots, doctors, divers: Bruce Willis Ruins All Films (BCD, weights, releases, air, final) Pre-flight checklists
- 19. 1. Extended use of checklists 2. Not to follow blindly, use knowledge and experience 3. Independent system 4. Searchable 5. List of known issues and documented workarounds/fixes Documented procedures
- 20. Replica environment or mock command line Record actions and timing Multiple failures Unexpected results Realistic scenarios: War Games
- 21. Results
- 22. Team and individual test of response Run real commands Training the people Training the procedures Training the tools Results
- 23. Increase confidence Reduce panic Better coordination Trust relationships Improves time to resolution Humans results
- 24. Review Suggestions for improvements Do it again Scenario evolves People forget loop(): review and repeat
- 25. On call rotation design Alert prioritization Notification optimization What else?
- 26. Human Ops
- 27. 1. Humans are part of the system 2. Humans impact systems 3. Humans impact business 4. Human issues count as system issues Human Ops principles
- 30. Jorge Salamero Sanz Chief Developer Advocate @bencerillo @serverdensity our DevOps stories blog.serverdensity.com