Embracing Failure
0 likes470 views
The document discusses embracing failures in software and systems. It recommends monitoring for failures, conducting blameless postmortem analyses to determine root causes, simulating failures through "gameday" exercises before they occur, and designing systems using a "SafeMachine" approach to make failures safe rather than unsafe. The key ideas are to understand failures will happen, learn from them through root cause analysis and prevention practices, and design systems that can fail safely.
Embracing Failure
- 1. Embracing Failure (not my life story)
- 4. Setting the Mood Understand that they WILL happen Failures are not binary Impact determines importance deadlines for fixes are variable
- 7. What is Monitoring? Graphs. Everywhere. Alerts on failures phone calls texts Answers: Are we failing?
- 9. healthcare.gov Know when youre down before CNN
- 11. Postmortems (fool me once. shame on you. fool me twice. shame on me.)
- 12. Postmortems 1. Reconstruct the factual timeline 2. Root cause analysis 3. Remediation items
- 13. Postmortems Why did we fail? Blameless Moderated
- 14. Gamedays (You wouldnt wing a talk. Dont wing a hot fix)
- 15. Gameday Best defense is a good offense Simulate possible failures Do it in production
- 16. kill -9 1. Draw a block diagram 2. Cut every connection 3. Watch the fireworks
- 17. SafeMachine (like a state machine but safer)
- 18. Try, Try, Try again What if we could just retry failures? Side effects are the root of all evil Safe failures vs Unsafe failures
- 19. Whats in a SafeMachine Actions States START Computed File Uploaded File END compute upload record successful
- 23. Embracing Failure Monitor Postmortems Gamedays - you wouldnt wing a talk? SafeMachine
- 24. @chriswu_
- 25. Additional resources Postmortems https://codeascraft.com/2012/05/22/blameless- postmortems/ Gamedays - https://stripe.com/blog/game-day-exercises-at-stripe links at the bottom of this post are also great Error Tracking - https://getsentry.com/welcome/