際際滷

際際滷Share a Scribd company logo

DevOpsAllDay: Who Wants PIE? A series of post incident experiments

P
Paul Greig

1) The document discusses post-incident experiments (PIE) and ways to improve post-incident review processes. It explores input variables like emotion, experience, and exposure that can impact PIE. 2) Different approaches for each variable are examined, such as direct vs translated collection of emotional data, using challengers vs splitting review streams to improve experience, and looking externally vs internally for exposure. 3) The document advocates experimenting with PIE approaches to reduce repetition of incidents, prevent future incidents, improve action delivery based on emotion, and enhance action quality from experience. The goal is to continuously improve the incident lifecycle through experimentation with PIE methods.

1 of 12
Download to read offline
Who wants PIE? A series of post incident experiments
Hello! I am Paul Greig Here to share my thoughts and experiences on post-incident experiments. You can find me at @frogs_are_green 2
"Let all your efforts be directed to something, let it keep that end in view. It's not activity that disturbs people, but false conceptions of things that drive them mad." - Seneca 3
Incident Lifecycle 4 Each cycle reduces impact PHASE 3 Remediation PHASE 1 Incident PHASE 2 Post Incident Review
Experiment View any post-incident review processes as a guide to be improved upon 5
Input Variables Emotion Experience Exposure 6
Emotion - Why - Quantity Direct Received in person by SRE for certain companies or products Potential for bias, speed of collation is high. Translated Through anothers experience, an interview or support ticket. Reduced impact of bias but slow for collation 7
Experience - How - Quality Challenger Seeks to insert an individual to raise the quality after the PIR Requires additional training and trainers to generate challenger pool Does not scale Split Streams Seeks to parallelise PIR construction through multiple contributors Requires co-ordinator and training - significant overhead on each deployment Scales after initial investment 8
Exposure - What - Distribution External Remain aware of issues affecting companies or the more broader incident space Difficult in construction if details and impact of original incident are unclear Internal Understand other teams similarities yet removal from original remediation items Requires investment with no impact realised Needs careful selection for participation 9
Maturity and Experimentation 10 Quantity DistributionQuality
Incident Lifecycle Experiment Impact 11 Each cycle reduces impact PHASE 3 Remediation PHASE 1 Actual Incident PHASE 2 Post Incident Review Experiment Impact How (Experience) - Reduce Repetition What (Exposure) - Preventative for others Experiment Impact Why (Emotion) - Action Delivery How (Experience) - Action Quality
12 Thanks! Any questions? You can find me at @frogs_are_green GoRE : https://goo.gl/7jwysS paul.greig01@gmail.com

More Related Content

DevOpsAllDay: Who Wants PIE? A series of post incident experiments

  • 1. Who wants PIE? A series of post incident experiments
  • 2. Hello! I am Paul Greig Here to share my thoughts and experiences on post-incident experiments. You can find me at @frogs_are_green 2
  • 3. "Let all your efforts be directed to something, let it keep that end in view. It's not activity that disturbs people, but false conceptions of things that drive them mad." - Seneca 3
  • 4. Incident Lifecycle 4 Each cycle reduces impact PHASE 3 Remediation PHASE 1 Incident PHASE 2 Post Incident Review
  • 5. Experiment View any post-incident review processes as a guide to be improved upon 5
  • 7. Emotion - Why - Quantity Direct Received in person by SRE for certain companies or products Potential for bias, speed of collation is high. Translated Through anothers experience, an interview or support ticket. Reduced impact of bias but slow for collation 7
  • 8. Experience - How - Quality Challenger Seeks to insert an individual to raise the quality after the PIR Requires additional training and trainers to generate challenger pool Does not scale Split Streams Seeks to parallelise PIR construction through multiple contributors Requires co-ordinator and training - significant overhead on each deployment Scales after initial investment 8
  • 9. Exposure - What - Distribution External Remain aware of issues affecting companies or the more broader incident space Difficult in construction if details and impact of original incident are unclear Internal Understand other teams similarities yet removal from original remediation items Requires investment with no impact realised Needs careful selection for participation 9
  • 11. Incident Lifecycle Experiment Impact 11 Each cycle reduces impact PHASE 3 Remediation PHASE 1 Actual Incident PHASE 2 Post Incident Review Experiment Impact How (Experience) - Reduce Repetition What (Exposure) - Preventative for others Experiment Impact Why (Emotion) - Action Delivery How (Experience) - Action Quality
  • 12. 12 Thanks! Any questions? You can find me at @frogs_are_green GoRE : https://goo.gl/7jwysS paul.greig01@gmail.com