際際滷

際際滷Share a Scribd company logo
VINCENT KOK | ENGINEERING MANAGER, TRELLO | @VINCENTKOK
Releasing the Monolith
On a daily basis
@vincentkok
About me
Trello
Engineering Manager on the
Trello team
Dutch
You¨ve probably heard that
already ;)
Dev opstalks 2018   releasing the monolith on a daily basis
Dev opstalks 2018   releasing the monolith on a daily basis
http://geekandpoke.typepad.com/geekandpoke/2011/08/hudson-status-monitor.html
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Cool stories
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Got issues?
https://?ic.kr/p/fSopv4
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Journey
https://?ic.kr/p/8WQutt
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Monolith
https://?ic.kr/p/nA4S49
FOCUS
FOCUS
Agenda
Confluence pipeline
Culture
Microservice envy
Four problems
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Confluence pipeline
https://?ic.kr/p/jus4Bt
15,000
builds per week
81,000
CI jobs
31,992
automated tests
1,000
build agents
NaN
flakes
3
size of the Pipeline team
72
steps to release Confluence :(
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Culture
https://?ic.kr/p/DsetEn
Dev opstalks 2018   releasing the monolith on a daily basis
Experience
The way you¨ve been
shaped
Beliefs
Based on your
experience
Actions
Based on your beliefs
Results
The outcome feeding
your experience
The Oz Principle
Early results
Theory in real life
MomentumClear goal
Good intentions don¨t work
ASANKA JAYASURIYA, HEAD OF ENGINEERING STRIDE
GreenDex
GreenDex
Adoption
Drive adoption by i.e. training
people
Mechanisms
Audit
Does it move the needle?
Process
What process can improve
the situation?
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Microservice envy
https://?ic.kr/p/b7ak8e
Clear ownership
Microservice traits
Independent
lifecycle
Small
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Monolith
https://?ic.kr/p/nA4S49
< 1 hour
< 1 day
Release
cadence
Ownership Test
automation
Complexity
Four problem areas
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
#1: Complexity
https://?ic.kr/p/cFM3cd
A TYPICAL PIPELINE
ReleaseInner Outer
Publish Deploy
SOME FLAWS
ReleaseInner Outer
Publish Deploy
Staging deploy
Manual release branch
Hard to understand
.
Impact of complexity
Inconsistent
feedback
Manual tasks
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Fix the model
https://?ic.kr/p/8ykpkW
FORK & JOIN
Release
Publish Deploy
Inner Outer
Fork
Join
FORK & JOIN
Release
Publish Deploy
Inner Outer
Fork
Join
FORK & JOIN IMPLEMENTATION
Build times
Iterate
Build stabilityInner loop
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
#2: Ownership
https://?ic.kr/p/bfRuoX
Getting the model wrong
Getting the model wrong
Broken windows
Impact
No improvementsFire fighting
Feedback loop
Even more impact
MTTRDev speed
OWNERSHIP
Sustainable
environment
Pipeline team - BuildOps
SREEngineering teams
WHERE DOES YOUR TEAM SIT?
Busy work
Keeping the lights on
Improving team life
Enforce, don¨t fix
SUPPORTING TEAMS
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Sustainable system
https://?ic.kr/p/awVLLs
Early results
Visibility
Create awareness across
the team
Measure
Gather data to allow data
driven decisions
Move the needle
Don¨t boil the ocean, focus
on a set of smaller results
Dev opstalks 2018   releasing the monolith on a daily basis
Dev opstalks 2018   releasing the monolith on a daily basis
Dev opstalks 2018   releasing the monolith on a daily basis
Enforce by
automation
Iterate
Shared ownershipDefine rules
SHARED OWNERSHIP
PROVIDE INSIGHT
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
#3: Testing
https://?ic.kr/p/cBbxuQ
Unit
Service
UI
Unit
Service
Balance
Balance
Duplication
Impact
FlakesSlow
The impact of flakes
Build times
Restarts cause build times
to increase drastically
Culture
The culture of green goes
out of the window
Broken windows
It starts with one. Soon, there
will be a lot
Retries as a painkiller
Automatic retries relieve some pain.
This is an early result, so see it as a
work-around.
Dev opstalks 2018   releasing the monolith on a daily basis
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Fix the root cause
https://?ic.kr/p/J44Nm
Test strategy Flake policy
Iterate
CREATE VISIBILITY
Detect Raise ticket Ignore Fix or remove
Flake policy
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Deleting tests, seriously?
https://?ic.kr/p/8BYXV4
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
#4: Releases
https://?ic.kr/p/91MFkh
RELEASE CYCLE
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Congestion
https://?ic.kr/p/bRgBZP
Ends at master Big bang Releases on
hold
Dog-fooding
becomes hard
Impact
For once a free lunch
Fork and Join solved most of our
problems.
12 FACTOR CONFLUENCE
Health check
200 app is alive. 500 app is unhealthy,
destroy the node
Stateless*
Run as many nodes as you need
Expose a port
Only access to the service
DECLARATIVE DEPLOYMENT
name: Confluence
description: Confluence Cloud
links:
binary:
type: docker
name: docker.atlassian.io/confluence
tag: latest
healthcheck:
uri: /wiki/internal/healthcheck
deepcheck:
uri: /wiki/internal/deepcheck
semanticCheck:
dockerImage:
+
RESOURCES
resources:
- type: sqs
name: default
attributes:
MaxReceiveCount: 20
VisibilityTimeout: 60
scaling:
instance: m3.xlarge
min: 7
Reduce build times Fix dog-fooding and
the feedback loop
Iterate
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Dog-fooding
https://?ic.kr/p/dxpLTm
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Feedback
https://?ic.kr/p/dxaAdk
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Soak time
https://?ic.kr/p/fmfG2q
Merge often
Feature flags
One processDecouple code and
features
Things to take home
Release
cadence
Ownership Test
automation
Complexity
Identify your problem areas
Early results
Focus on momentum
MomentumClear goal
Good intentions don¨t work.
BUILD MECHANISMS
? Big cool statistic
? 2,56
9
? Add-Ons in Marketplace
Sustainable system
https://?ic.kr/p/awVLLs
http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef015392a77686970b-pi
VINCENT KOK | ENGINEERING MANAGER, TRELLO | @VINCENTKOK
Thanks!

More Related Content

Dev opstalks 2018 releasing the monolith on a daily basis