Bloated Chefs: A Tale of Gluttony, and the Path to Enlightenment
1 like705 views
- The document discusses how PagerDuty's Chef runs became bloated and slow as the infrastructure grew in size and complexity. - Measurements found Chef was using over 500MB of memory and taking around 180 seconds to complete runs. - Several optimizations were made including step-through searches, partial searches, result memoization, and API tarpitting which reduced memory usage to 60MB and run time to 30 seconds, improving performance significantly. - The optimizations led to an 88% reduction in memory usage and 84% faster Chef runs.
Bloated Chefs: A Tale of Gluttony, and the Path to Enlightenment
- 1. 4/3/15 @evan2645 EVAN GILMAN Bloated Chefs A Tale of Gluttony, and the Path to Enlightenment
- 3. 4/3/15 Agenda BLOATED CHEFS 1. Chef resources in use at PD 2. Problems encountered as we grew 3. Measuring chef-client run 4. How we fixed it 5. How fast is it now?
- 4. 4/3/15BLOATED CHEFS CHEF @ PAGERDUTY
- 11. 4/3/15BLOATED CHEFS ALL WAS NOT WELL
- 12. 4/3/15 As we grew BLOATED CHEFS
- 13. 4/3/15 As we grew BLOATED CHEFS CPU spikes during chef-client runs
- 14. 4/3/15 As we grew BLOATED CHEFS CPU spikes during chef-client runs Awkward pauses at the beginning of the run
- 15. 4/3/15 As we grew BLOATED CHEFS CPU spikes during chef-client runs Awkward pauses at the beginning of the run chef-client run took several minutes
- 16. 4/3/15 As we grew BLOATED CHEFS CPU spikes during chef-client runs Awkward pauses at the beginning of the run chef-client run took several minutes chef-client OOM
- 17. 4/3/15 As we grew BLOATED CHEFS CPU spikes during chef-client runs Awkward pauses at the beginning of the run chef-client run took several minutes chef-client OOM
- 21. 4/3/15 Measuring Run Time BLOATED CHEFS
- 22. 4/3/15 Measuring Run Time BLOATED CHEFS https://github.com/joemiller/chef-handler-profiler
- 23. 4/3/15 Measuring Resources BLOATED CHEFS Total number of resources per run, by type Number of updated resources per run, by type
- 24. 4/3/15 Measuring Memory BLOATED CHEFS Gather proc stats with sys-proctable Gather GC stats Can be emitted as statsd
- 26. 4/3/15BLOATED CHEFS WHAT WE FOUND AND WHAT WE DID
- 29. 4/3/15 Step-through Searches BLOATED CHEFS From this To this
- 30. 4/3/15 Step-through Searches BLOATED CHEFS 417MB -> 190MB
- 31. 4/3/15 Step-through Searches BLOATED CHEFS 417MB -> 190MB ~54%
- 33. 4/3/15 Partial Searches BLOATED CHEFS Provide hash map of desired results
- 34. 4/3/15 Partial Searches BLOATED CHEFS Provide hash map of desired results Minimizes volume of node data returned/handled
- 35. 4/3/15 Partial Searches BLOATED CHEFS Provide hash map of desired results Minimizes volume of node data returned/handled hash2node
- 36. 4/3/15 Partial Searches BLOATED CHEFS Provide hash map of desired results Minimizes volume of node data returned/handled hash2node Two searches touched
- 37. 4/3/15 Partial Searches BLOATED CHEFS Provide hash map of desired results Minimizes volume of node data returned/handled hash2node Two searches touched 90s -> 60s
- 38. 4/3/15 Partial Searches BLOATED CHEFS Provide hash map of desired results Minimizes volume of node data returned/handled hash2node Two searches touched 90s -> 60s 30%
- 40. 4/3/15 Result Memoization BLOATED CHEFS Common search data
- 41. 4/3/15 Result Memoization BLOATED CHEFS Common search data API-backed LWRPs
- 42. 4/3/15 Result Memoization BLOATED CHEFS Common search data API-backed LWRPs Can be generalized
- 43. 4/3/15 Result Memoization BLOATED CHEFS Common search data API-backed LWRPs Can be generalized
- 47. 4/3/15 Other Nasties BLOATED CHEFS Too many conditional guards
- 48. 4/3/15 Other Nasties BLOATED CHEFS Too many conditional guards tmpfs storage
- 49. 4/3/15 Other Nasties BLOATED CHEFS Too many conditional guards tmpfs storage Multiple package resources (Chef 12)
- 50. 4/3/15 Other Nasties BLOATED CHEFS Too many conditional guards tmpfs storage Multiple package resources (Chef 12) Six seconds for twelve packages
- 53. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After:
- 54. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After: ~60MB
- 55. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After: ~60MB 88% less memory!
- 57. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After:
- 58. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After: ~30s/run
- 59. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After: ~30s/run ~84% faster!