�ݺ�ߣ

Testing Infrastructure as Code
Best Practices and Common Mistakes
Given by Derek C. Ashmore
STAREAST 2022
April 24-29, 2022
©2022 Derek C. Ashmore, All Rights Reserved 1

Who am I?
• Professional Geek
since 1987
• AWS since 2010
• Azure since 2017
• Specialties
• Application
Transformation
• Infrastructure
Automation
• Yes – I still code!

Discussion Resources
• This slide deck
– /derekashmore/presentations
• �ݺ�ߣ deck has hyper-links!
– Don’t bother writing down URLs
• Put questions in the chat – we’ll address them at the end

Agenda
Intro and
Level Set
Testing
Challenges
Best
Practices
and Anti-
Patterns
Summary /
Q&A

Infrastructure as Code (IaC)
• Manual changes
– Increase errors
– Increase unwanted differences
between environments
– Increase admin workload
• Scripted/Coded changes
– Larger upfront cost, but…..
– Less busywork
– Leverage Others Work
– Decreases Errors
– Errors fixed in one place
– Eliminates unwanted differences
– Change history (with source control)

Infrastructure Code Categories
➢Network / non-application specific infrastructure
▪ Virtual Networks/VPCs and subnets
▪ Route tables, Network peering
▪ Security groups / NSGs
➢Application infrastructure
▪ Relational databases
▪ Serverless constructs
➢Security privileges and policies
▪ IAM Roles and privilege grants
➢Virtual machine image production
▪ Produce machine images for teams to use

Infrastructure Code Testing
➢IaC is code!
▪ Housed in source control
▪ Often changed and released
▪ Needs testing like any other code
➢IaC change can have negative impact
▪ Environment outages
▪ End-user internet connectivity outage
▪ Application outages
▪ Testing team delayed for four days
➢Testing IaC can minimize negative impact

Infrastructure Code Testing Differences
➢IaC != Application Code
▪ IaC requires external resources (e.g. Cloud) to run
▪ In-process unit testing often not possible
➢Limited localized (in-process) testing
▪ Generally limited to syntax checks
▪ Terraform validation
▪ Ansible Dry Runs
▪ IDE syntax checks
➢Most testing is “integration” testing

Agenda
Intro and
Level Set
Testing
Challenges
Best
Practices
and Anti-
Patterns
Summary /
Q&A

Infrastructure Code Testing Challenges
➢Friction
➢Dependencies
➢Testing costs
➢Manual intervention requirements
➢Shifting Sand Problem

IaC Testing Friction
➢Larger effort to write tests
▪ Tooling varies widely
▪ RSpec style testing [awspec (AWS), inspec
(Azure), etc.]
▪ Terratest
▪ Tooling requires additional expertise
▪ Ruby for Rspec
▪ GoLang for Terratest
➢Hard to make comprehensive
▪ Sheer number of conditions and
attributes to check

Friction Mitigation
➢Make IaC code easier to test
▪ Code in smaller reusable modules
▪ Small number of dependencies
▪ Self-contained
▪ Allows testing in Sandbox environment
▪ Re-usable Terraform module example
➢Focus testing on greatest areas of risk
▪ Let defects drive test code activity
▪ Watch testing scope
▪ No substitute for app teams/testers for
changes
▪ Must follow SDLC with changes

Reusable Module Test Cases
➢Example reusable modules
▪ Kubernetes Cluster
▪ Virtual Machine
▪ Virtual Networks and Subnets
▪ S3/Storage accounts
▪ Serverless services/functions
➢70+ Modules in all
▪ Used in 200+ pipelines
➢Tested in merge to master

IaC Testing Dependencies
➢Dependencies are the heaviest type of friction
▪ Examples include
▪ Active Directory
▪ DNS services
▪ Virtual networking
▪ Labor costs money, but most of it is up front
▪ Don’t strive for 100% coverage
▪ Spend testing labor time wisely
▪ Limit testing to our code, not infrastructure tech stack functionality
➢The cost for not testing doesn’t save money
▪ The business impact for deployed defects usually greater than testing runtime
costs

IaC Testing Costs
➢All tests for IaC cost money
▪ Test in small, disposable units
▪ We always test in a Sandbox
▪ All resources deallocated after each test
▪ Labor costs money, but most of it is up front
▪ Don’t strive for 100% coverage
▪ Spend testing labor time wisely
▪ Limit testing to our code, not infrastructure tech stack functionality
➢The cost for not testing doesn’t save money
▪ The business impact for deployed defects usually greater than testing runtime
costs

Manual Intervention Requirements
➢Some companies require manual intervention
▪ Often dictated by company policy
▪ Examples include
▪ Requiring DNS entries to be manually entered
▪ Separate group allocates security privileges
▪ Cloud functionality changes for Azure storage groups
▪ On-premises connectivity
➢IaC depending on manual intervention cannot have automation tests
▪ Localize the manual intervention requirements

Accommodating Manual Processes

Shifting Sand Problem
➢Changes occur outside of IaC
▪ Common Sources
▪ Automatic dependency updates
▪ Cloud backend updates
▪ Right-hand/Left-hand issues
▪ Managing cloud assets from multiple
sources
➢Makes IaC Code not Repeatable!
▪ Causes to break in future runs
▪ Recipe for unplanned work!
➢Further reading

Avoid Automatic Updates!
➢Perceived as a “convenience”
▪ Avoids upgrade work
▪ Causes unplanned work
➢Examples include
▪ “Latest” dependencies
▪ Terraform cloud provider version
▪ Common IaC Code version
▪ VM automatic updates
▪ Security updates not avoidable!
➢Convert “unplanned” work to
“planned” work

Early Detection Strategy
➢Scheduled testing in a sandbox
environment
▪ Detect/fix shifting sand problems before
users do!
➢Addresses
▪ Required automatic updates
▪ Cloud back-end changes
▪ Uncontrollable actions by other groups
➢Disadvantages
▪ Work is still unplanned
▪ Scheduled testing costs
▪ Labor to set up testing costs

Agenda
Intro and
Level Set
Testing
Challenges
Best
Practices
and Anti-
Patterns
Summary /
Q&A

Infrastructure Code Testing Best Practices
➢Don't expect perfection
▪ IaC output is much more complex than most application code
➢Concentrate Automated Testing for reused modules
▪ Tests cover the most widely used infrastructure code
▪ Defects for widely used modules will have the most negative impact
▪ Example: Some common modules get used by dozens of pipelines
▪ Completely self-contained
▪ Test create all dependencies the reused code needs
▪ Easier because of smaller blast radius
▪ Less for the test to set up in the sandbox
▪ Easy to make completely disposable
▪ Use sandbox environment
▪ Drop after test (save $)
▪ Coding mistakes have less impact

Infrastructure Code Testing Best Practices (con’t)
➢Let defects guide automated test writing
▪ Focuses test writing labor on the most important
➢Watch the blast radius
▪ Blast radius correlates with test code complexity
➢Make IaC more easily testable
▪ Keep small and targeted
▪ Reduce dependencies

Infrastructure Code Testing Anti-Patterns
➢No testing in Sandbox
▪ Need a place to test/edit without affecting others
▪ Environment must be disposable
➢No peer reviews
▪ IaC is complex
▪ IaC requires large breadth of skills
➢Inadequate testing in lower environments
➢No substitute for real world contact with others
➢Automated testing of IaC is not comprehensive

Summary
➢Infrastructure code must have automated tests too
▪ Infrastructure code has defects too
▪ Defects can cause outages
▪ Untested code cannot be trusted
➢Automated testing needs to be funded
▪ You will get a positive ROI
▪ You don’t need to be perfect
▪ Don’t let perfection prevent progress!
➢Discipline is Key
▪ Changes starting out will take longer
▪ Will be faster after developers are up to speed

Thank you!
• Derek Ashmore:
– Blog: www.derekashmore.com
– LinkedIn: www.linkedin.com/in/derekashmore
• Connect Invites from attendees welcome
– Twitter: https://twitter.com/Derek_Ashmore
– GitHub: https://github.com/Derek-Ashmore
– Book: http://dvtpress.com/
• Please fill out the evaluation form!

�ݺ�ߣ

Testing Infrastructure Code Best Practices and Common Mistakes

More Related Content

Testing Infrastructure Code Best Practices and Common Mistakes