際際滷

際際滷Share a Scribd company logo

Command & conquer: Red alert

T
tadejm

The document discusses Tadej Murovec's approach to monitoring and alerting systems at HouseTrip. It recommends alerting on symptoms rather than causes, having teammates review alerts, preferring under-monitoring to over-monitoring, and using notifications to prevent alerts. It also provides an overview of the tools HouseTrip uses for monitoring, including New Relic, Datadog, Slack, and PagerDuty.

1 of 26
Download to read offline
Command & Conquer ! 息 Tadej Murovec, HouseTrip, 2015 1
! Hello Twitter: @tadejm GitHub: github.com/tadejm Work at HouseTrip. 息 Tadej Murovec, HouseTrip, 2015 2
! 息 Tadej Murovec, HouseTrip, 2015 3
! The debt 184 alerts per week => 1.095 alert per hour 息 Tadej Murovec, HouseTrip, 2015 4
息 Tadej Murovec, HouseTrip, 2015 5
! Rethink how to do alerting Alerts should be urgent, important, actionable and real. Over-monitoring is a harder problem to solve than under- monitoring. Symptoms are a better way to capture more problems more comprehensively and robustly with less effort.1 1 My Philosophy on Alerting Rob Ewaschuk 息 Tadej Murovec, HouseTrip, 2015 6
! Meet the Happy path * Guests can search for properties * Guests can browse properties * Users can login or register * Guests can send an enquiry or book a property * Guests can pay * Hosts can accept booking 息 Tadej Murovec, HouseTrip, 2015 7
The end user does not care about MySQL server being unreachable, but she does care about not being able to view a property. 息 Tadej Murovec, HouseTrip, 2015 8
Tools & services 息 Tadej Murovec, HouseTrip, 2015 9
息 Tadej Murovec, HouseTrip, 2015 10
New Relic IS AWESOME. 息 Tadej Murovec, HouseTrip, 2015 11
New Relic Application server response time Key transactions tracking with Apdex T Uptime monitoring Application segmentation into policy groups 息 Tadej Murovec, HouseTrip, 2015 12
息 Tadej Murovec, HouseTrip, 2015 13
Datadog IS AWESOME. 息 Tadej Murovec, HouseTrip, 2015 14
Datadog Monitoring scheduled errands Background queues and services health check Custom metrics that make sense for the business 息 Tadej Murovec, HouseTrip, 2015 15
息 Tadej Murovec, HouseTrip, 2015 16
Slack IS AWESOME. 息 Tadej Murovec, HouseTrip, 2015 17
Slack Chronological tracking, search Slack channels as urgency segmentation #alerts, #notifications 息 Tadej Murovec, HouseTrip, 2015 18
息 Tadej Murovec, HouseTrip, 2015 19
PagerDuty IS OK. And wakes you up at night. Sometimes. 息 Tadej Murovec, HouseTrip, 2015 20
PagerDuty Propagates alerts to duty engineers Manage duty rotation schedule Keep notes about alerts 息 Tadej Murovec, HouseTrip, 2015 21
! Progress and results 息 Tadej Murovec, HouseTrip, 2015 22
! 4 rules for efficient alerting Alert on symptoms not causes " Get your teammates to pair review the alerts # Prefer under monitoring to over monitoring $ Use noti鍖cations to prevent alerts 息 Tadej Murovec, HouseTrip, 2015 23
! Be a good citizen 息 Tadej Murovec, HouseTrip, 2015 24
息 Tadej Murovec, HouseTrip, 2015 25
That's all folks! Thanks! ! 息 Tadej Murovec, HouseTrip, 2015 26

More Related Content

Command & conquer: Red alert

  • 1. Command & Conquer ! 息 Tadej Murovec, HouseTrip, 2015 1
  • 2. ! Hello Twitter: @tadejm GitHub: github.com/tadejm Work at HouseTrip. 息 Tadej Murovec, HouseTrip, 2015 2
  • 3. ! 息 Tadej Murovec, HouseTrip, 2015 3
  • 4. ! The debt 184 alerts per week => 1.095 alert per hour 息 Tadej Murovec, HouseTrip, 2015 4
  • 5. 息 Tadej Murovec, HouseTrip, 2015 5
  • 6. ! Rethink how to do alerting Alerts should be urgent, important, actionable and real. Over-monitoring is a harder problem to solve than under- monitoring. Symptoms are a better way to capture more problems more comprehensively and robustly with less effort.1 1 My Philosophy on Alerting Rob Ewaschuk 息 Tadej Murovec, HouseTrip, 2015 6
  • 7. ! Meet the Happy path * Guests can search for properties * Guests can browse properties * Users can login or register * Guests can send an enquiry or book a property * Guests can pay * Hosts can accept booking 息 Tadej Murovec, HouseTrip, 2015 7
  • 8. The end user does not care about MySQL server being unreachable, but she does care about not being able to view a property. 息 Tadej Murovec, HouseTrip, 2015 8
  • 9. Tools & services 息 Tadej Murovec, HouseTrip, 2015 9
  • 10. 息 Tadej Murovec, HouseTrip, 2015 10
  • 11. New Relic IS AWESOME. 息 Tadej Murovec, HouseTrip, 2015 11
  • 12. New Relic Application server response time Key transactions tracking with Apdex T Uptime monitoring Application segmentation into policy groups 息 Tadej Murovec, HouseTrip, 2015 12
  • 13. 息 Tadej Murovec, HouseTrip, 2015 13
  • 14. Datadog IS AWESOME. 息 Tadej Murovec, HouseTrip, 2015 14
  • 15. Datadog Monitoring scheduled errands Background queues and services health check Custom metrics that make sense for the business 息 Tadej Murovec, HouseTrip, 2015 15
  • 16. 息 Tadej Murovec, HouseTrip, 2015 16
  • 17. Slack IS AWESOME. 息 Tadej Murovec, HouseTrip, 2015 17
  • 18. Slack Chronological tracking, search Slack channels as urgency segmentation #alerts, #notifications 息 Tadej Murovec, HouseTrip, 2015 18
  • 19. 息 Tadej Murovec, HouseTrip, 2015 19
  • 20. PagerDuty IS OK. And wakes you up at night. Sometimes. 息 Tadej Murovec, HouseTrip, 2015 20
  • 21. PagerDuty Propagates alerts to duty engineers Manage duty rotation schedule Keep notes about alerts 息 Tadej Murovec, HouseTrip, 2015 21
  • 22. ! Progress and results 息 Tadej Murovec, HouseTrip, 2015 22
  • 23. ! 4 rules for efficient alerting Alert on symptoms not causes " Get your teammates to pair review the alerts # Prefer under monitoring to over monitoring $ Use noti鍖cations to prevent alerts 息 Tadej Murovec, HouseTrip, 2015 23
  • 24. ! Be a good citizen 息 Tadej Murovec, HouseTrip, 2015 24
  • 25. 息 Tadej Murovec, HouseTrip, 2015 25
  • 26. That's all folks! Thanks! ! 息 Tadej Murovec, HouseTrip, 2015 26
AboutCopyright
息 2025 際際滷