200,000 Lines Later: Our Journey to Manageable Puppet Code
- 1. 2 0 0 , 0 0 0 L I N E S L AT E R O U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E D AV I D D A N Z I L I O C O N S TA N T C O N TA C T
- 2. W H O A M I ? Puppet Evangelist at Constant Contact Joined Constant Contact in 2014 SysAdmin background but more of a developer these days Second time speaking at PuppetConf
- 4. W H AT I S C O N S TA N T C O N TA C T ? B A C K G R O U N D
- 5. Founded in 1995 Headquarters in Waltham, MA Offices in San Francisco, NYC, London, among others ~ 1,500 Employees, ~ 350 developers, ~ 70 operations 160+ apps (SOA - microservices for hipsters) Were hiring! (See me after for details)
- 6. W H AT S T H I S TA L K A B O U T ? An overview of our environment and our challenges Our successes, failures, and lessons from trying to transform Puppet in a large enterprise Plans for achieving the Puppet singularity My quest to understand and affect change in a massively complex organization
- 7. R U N N I N G P U P P E T AT S C A L E I S H A R D T L ; D R
- 8. A L O N G T I M E A G O I N A G A L A X Y FA R FA R A WAY
- 9. E A R LY A D O P T E R S Y N D R O M E Started using Puppet in 2009 with version 0.24.8 You name the bug, weve probably seen it Forge didnt exist so we had to write everything Puppet talent was (still is) hard to come by Writing Puppet was a little harder back then
- 10. Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.x Dynamic Scope X X X X X Appending to attributes in class inheritance (+>) X X X X X X X X Multi-line C-style comments X X X X X X X Arrays of resource references allowed in relationships X X X X X X X Overrides in class inheritance X X X X X X X Appending to variables in child scopes (+=) X X X X X X X Class names starting with 0-9 X X X Regular expressions in node de鍖nitions X X X X X X Assigning expressions to variables X X X X X X Regular expressions in conditionals/expresions X X X X X X elsif in if statements X X X X X Chaining Resources X X X X X Hashes X X X X X Class Parameters X X X X X Run Stages X X X X X The in operator X X X X X $title, $name, and $module_name available in parameter lists X X X X X Optional trailing comma in parameter lists X X X X Hyphens/dashes allowed in variable names * X Automatic class parameter lookup via data bindings X X X Unless conditionals X X X Iteration over arrays and hashes X X The modulo (%) operator X X $trusted hash X
- 11. Lots of the code we wrote back then is still around The company was growing so fast, nobody had time to focus on Puppet Ops mandated that all configuration and app deployment had to be done with Puppet Lots of push back Puppet became a dirty word
- 12. T H E G O O D Everything is in Puppet Everything is in Git Our infrastructure is built to be ephemeral We can provision physical nodes about as fast as we can spin up a virtual machine Moving an app to OpenStack is just a matter of running Puppet
- 13. T H E B A D Stuck on Puppet 2.7.3; lots of the code was written before parameterized classes Most of our code lives in a single Git repo Testing is up to the developer No design beyond the initial implementation; most of the code just evolved organically Propensity for home-grown solutions, and a habit of thinking only in terms of the current use case
- 14. T H E U G LY Massive codebase (more on this in a second) Frightening complexity Tightly coupled codebase (massive understatement) Very few tests (effectively none) Very little documentation Too easy to get into production No dependency management
- 16. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E 8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 1 1 , 0 7 6 E R B 2 5 , 1 9 8 6 6 1 4 3 , 7 2 4 3 9 1 B O U R N E S H E L L 6 , 7 6 1 6 , 5 3 1 5 5 , 1 6 2 1 9 3 X M L 2 , 7 1 4 5 , 1 1 1 3 7 , 3 6 0 5 8 P E R L 4 , 1 9 9 6 , 5 2 0 2 3 , 8 6 6 5 4 2 R U B Y 4 , 9 2 2 1 , 7 2 0 2 2 , 6 6 8 5 1 P Y T H O N 2 , 9 5 4 1 , 6 2 2 1 1 , 2 6 8 6 7 S Q L 4 5 2 1 9 7 9 , 9 9 9 4 6 6 YA M L 6 2 3 8 4 2 8 , 7 9 9 1 0 0 O T H E R 2 , 5 3 9 6 , 0 3 4 8 , 3 2 8 1 1 , 2 1 1 T O TA L 9 3 , 4 2 1 5 0 , 5 7 6 5 2 0 , 0 7 5
- 17. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E 2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4 4 0 E R B 6 1 7 1 1 2 , 2 7 1 7 X M L 5 0 2 1 , 5 5 4 5 6 YA M L 1 6 4 1 , 5 1 1 3 2 B O U R N E S H E L L 4 0 2 4 7 9 1 , 6 3 8 1 6 J S O N 7 5 0 1 , 1 7 5 8 2 P U P P E T 1 5 9 4 6 6 9 4 2 3 S M A R T Y 3 8 0 3 6 9 2 L I S P 4 4 9 6 3 0 0 1 4 O T H E R 1 3 3 1 1 3 6 2 9 2 , 6 4 8 T O TA L 5 3 , 8 8 6 3 1 , 2 1 4 2 3 9 , 7 9 5
- 18. T H E J O U R N E Y B E G I N S
- 19. U N D E R S TA N D
- 20. I D E N T I F Y O U R U S E R S Consumers use Puppet to deploy their apps. They just want Puppet to work. Developers may write Puppet modules or profiles. They may consume Forge modules. Architects help shape the strategic direction of Puppet in the organization.
- 21. Consumers have a very different use case than Architects or Developers. Most of our users are Consumers, so we should focus on them. Consumers arent necessarily interested in learning Puppet, but they need to use it to deploy their apps. Interfaces are very important to these people.
- 22. T E A C H
- 23. T E A C H I N G P U P P E T Most people working with Puppet didnt really know the language well But they had to get shit done Rampant propagation of bad patterns Wrote a 4-hour intro class and started teaching Get everybody on a common baseline, a good foundation to build good habits Offer more advanced training to people who want it
- 24. U P G R A D E
- 25. A M O D E R N V E R S I O N O F P U P P E T Puppet 2.7.3 is buggy and slow Couldnt use any modules from the Forge out of the box because of trailing commas Puppet 2.7 was holding us back Offload file serving to Puppet 3 masters, cheap performance win: bit.ly/1VfSwYM Problem: code isnt compatible with Puppet 3
- 27. M I N I M U M V I A B L E R E FA C T O R
- 28. We knew the code was bad, but we didnt have the resources to focus on a redesign Refactor just enough to get to Puppet 3 Fix templates Eliminate dynamic scope Hiera played a big role in this Move hosts to Puppet 3 as code becomes compatible
- 29. TA K I N G T H E P L U N G E
- 30. We built out a Puppet 3 infrastructure alongside the legacy infrastructure Shared CA, so the clients could talk to both sets of masters Move hosts over as the code becomes compatible Turns out this is really hard when theres no standard way to classify hosts
- 31. We started with the easiest stuff: our own hosts Moved on to our CD apps In process of moving legacy apps now
- 32. M E A N W H I L E
- 33. We couldnt stop time and do all this work in a vacuum, life was still happening around us New modules are put into their own repos and deployed with r10k Weve kept Puppet 3 environment up to date. Started on 3.4.3, now on 3.8.3 New modules must have tests and documentation Were using roles and profiles for new things
- 34. CI workflow and infrastructure Testing environments, harnesses, and helpers Deployment tooling Knowledge of testing frameworks No official images
- 35. N E X T S T E P S
- 36. T H I N G S W E R E TA C K L I N G N E X T Node classification Roles and Profiles Decoupling our code Desire for bespoke everything Writing modules with a single entry point Dependency management The Forge test
- 37. T H E F O R G E T E S T / P U P P E T A P P R O V E D T E S T / D E F I N I T I O N O F D O N E Can this module be deployed to the Forge? Is there any proprietary data embedded? Is it modular enough to meet unforeseen use cases? Does it have a robust suite of tests? Does it conform to community style expectations? Is it fully documented?
- 38. I T S A L L A B O U T T H AT I N T E R FA C E Make it easier to use our modules Our consumers care about the interface, not the implementation Design better and fewer interfaces Modules with well defined interfaces for composability and better dependency management Documentation and tests as part of the definition of done
- 39. T E S T I N G Make testing easier Make testing automatic Make testing a part of the culture Teach people about BDD Show them how much easier and more reliable an automated test can be than a manual one Eventually, make testing required
- 40. L E S S O N S L E A R N E D
- 41. Know your audience Dont try and please everybody Its always going to take longer than you think Dont forget about the supporting infrastructure People, process, and technology. In that order Its never going to move as fast as you want it to. Walk, dont run
- 42. Dont let yourself become paralyzed by complexity Take notes, write down things you need to work on, youll get to them eventually Always take the easy wins Focus on high impact improvements, help as many people as you can Small improvements add up
- 43. Theres nothing more successful than success
- 44. L E S S O N S T O B E L E A R N E D
- 45. How do we get people to feel invested in the design? How can we reenforce good behaviors? How can we make sure our massive codebase is being regularly evaluated? How can (should?) we hold people accountable for their Puppet code? How can we get operations folks to realize theyre software developers?
- 46. T H E E N D
- 47. H O W T O F I N D M E @djdanzilio on Twitter danzilio on Freenode (you can usually find me in #puppet-community, #puppet-dev, and #puppet) danzilio on GitHub and The Forge david (at) danzil (dot) io blog.danzilio.net
- 48. I M A G E C R E D I T S The space photos are from the NASA Image Galleries and are either in the public domain or licensed under CC-BY 2.0 All other photos are mine or my employers