際際滷

際際滷Share a Scribd company logo
2015	ROBERT	BIGOS
Cloud	capacity	planning	&	
monitoring	challenge	
Robert	Bigos	
bigosr@gmail.com	
+48	665-168-240
hCp://www.slideshare.net/RobertBigos
hCps://pl.linkedin.com/in/robertbigos
@bigosr
Blog:	bigosr.com
2015	ROBERT	BIGOS
2
CloudApps	
DevOps	culture
2015	ROBERT	BIGOS
Requirements	
	Func2onal	requirement:	A		want	make	sex	with	B
2015	ROBERT	BIGOS
Requirements
	Func2onal	requirement:	A	want	make	sex	with	B	
	Non-Func2onal:	one	per	day,	maybe	two,	not	to	short,	
not	to	long,	safe	but	not	to	safe,	24x7	with	99.9%	
monthly	availability	,	
Decisions:	price	?	cost	?	candles?		,	how	long	?
息	2015	ROBERT	BIGOS
Architecture
set	of	decisions	a鍖ec-ng	
func-onality		
	
documented	part	of	communica-on	
between	stakeholder	helping	
understand/balance:		Func-onal	
and	Non-Func-onal	requirements
2015	ROBERT	BIGOS
Con鍖ict	by	design
Dev
Ops
Owner
Divide	et	impera
2015	ROBERT	BIGOS
SoluPon	?
DevOps DevOpsDevOps
2015	ROBERT	BIGOS
Conway's	Law
Team	A
TEST
STAGE
kingdom
/silo	
team	B-
X
PROD
Organiza2ons	which	design	systems	are	constrained	to	produce	
designs	which	are	copies	of	the	communica2on	structures	of	these	
organiza2ons.
2015	ROBERT	BIGOS
2
WHY	?
2015	ROBERT	BIGOS
11
Why	is	so	important?
Capacity			=		fuel,		Performance		=		speed	and	al2tude	
Capacity	and	performance	management		helps	understand	
how	quickly	and	safely	you	can	transport	your	customers	to	planned	des2na2ons
Source:	http://s134.photobucket.com/user/charlesfrith/media/disaster.gif.html
2015	ROBERT	BIGOS
Queueing
From	operaPonal	perspecPve	Cloud	system	use	the	
same	formulas	to	manage	capacity	of	Queue	in	
Town	o鍖ce
Source:	Kanal	von	FerdinandLutz	"Stay	in	queue"	youtube.com
Cloud	it	is	all	about	cloud	scale		everything	is	
interconnected	and	instrumented
2015	ROBERT	BIGOS
Monitoring	faces
	dont	forget	about	Conways	Law	
Logs Metrics AlertsTrendsTresholdsEvents
2015	ROBERT	BIGOS
2
Backup	plan	?
try:	
CtrZ	
CmdZ	
 blue/green	deployment	gives	possibiliPes	not	availability	
 snapshot	is	not	a	backup
2015	ROBERT	BIGOS
2
There	is	no	magic	buCon	
Source:	http://make-everything-ok.com/
2015	ROBERT	BIGOS
You	will	fail	for	sure	!
Source:		presenter	studies	for	top	enterprises	in	Poland.
Source:	http://www.skybrary.aero/index.php/James_Reason_HF_Model Source:		J坦zef	Tischner	"The	Highlander's	History	of	Philosophy"
"the	truth,		
the	whole	truth	
	and		
the	bullshit	truth!
Postmortem	is	part	of	DevOps	culture.	
Recovery	has	to	be	part	of	design.
2015	ROBERT	BIGOS
WHAT	?
2015	ROBERT	BIGOS
IOT	/monitoring/	BigData
Monitoring	use	tools	to	collect/process/visualize	logs	and	metrics	for	
beCer	understanding	your	system	to	close	development	and	
operaPonal	feedback	loop.	It	is	not	kinder-garden	to	try	nice	tools.	
Today	everything	is	instrumented	and	interconnected	,	be	sure	you	
collect	the	right	data	in	the	right	scale	to	be	able	get	informaPon	from	
it.	Your	laptop	can	generate	BigData	volumes	
VOLUME VARIETY
VELOCITY
2015	ROBERT	BIGOS
32
Source:	ukasz	Piskorz	IBM	SWG	Lab
Death	or	lost	signal?
2015	ROBERT	BIGOS
32
Computer	vs	human	scale		
5	mins	=		5*60/10^-9/(60*60*24*365)	=	9512	years
few	objects,	few	variables,		
no	dependency,	no	relaPons
Peeping	through	the		
keyhole
2015	ROBERT	BIGOS
2
HOW	?
2015	ROBERT	BIGOS
Tools
.	you	need	tools	but	tool	is	not	a	soluKon
2015	ROBERT	BIGOS
32
Typical	Pme-centric	dashboard	
Source:	Demo	site	dashboard	grafana.org
2015	ROBERT	BIGOS
32
Threshold	violaPon	troubleshooPng
Source:	ukasz	Piskorz	IBM	SWG	Lab
2015	ROBERT	BIGOS
2
Big	
picture	?	
try	understand	,	go	deeper
2015	ROBERT	BIGOS
2
Know	unknowns	and	unknown	unknowns
Reports	 that	 say	 that	 something	 hasn't	 happened	 are	 always	
interes2ng	to	me,	because	as	we	know,	there	are	known	knowns;	
there	are	things	we	know	we	know.	We	also	know	there	are	known	
unknowns;	that	is	to	say	we	know	there	are	some	things	we	do	not	
know.	But	there	are	also	unknown	unknowns	--	the	ones	we	don't	
know	we	don't	know	
Donald Rumsfeld, February 12th, 2004 DOD News Briefing
	
Source:	http://www.defense.gov/transcripts/transcript.aspx?transcriptid=2636
2015	ROBERT	BIGOS
VisualisaPon	RealityGames
hhp://www.wearerealitygames.com/
2015	ROBERT	BIGOS
VisualisaPon	Cloud	Foundry
2015	ROBERT	BIGOS
2
Summary
2015	ROBERT	BIGOS
2
Lessons	learned	?
 suggested	approach	
 WHY	
 WHAT	
 HOW	
 most	popular	
 HOW	
 HOW	
 HOW	
.	try	understand	big	picture	,	go	deeper	,	focus	on	details
2015	ROBERT	BIGOS
2
Lessons	learned	?
 There	is	No	Single	Version	of	the	Truth		be	open	for	
communicaPon	
 There	is	NO	perfect	tools		we	can	build	beCer	teams	and	
beCer	communicate	to	close	feedback	loop	
 Cloud	scale	design	,	operaPonal	design	not	just	easy	to	
consume	
 There	is	no	GO@FASTER	opPon	in	the	Cloud
2015	ROBERT	BIGOS
2
Lessons	learned	?
 Keep	standards:		date	ISO_8601	example	
 Keep	what	you	need		not	just	everything	
 Keep	slim:	json	is	nice	but		
 Keep	informaPon	not	just	numbers/hashtags	
 AutomaPzaPon	requires	standardisaPon	
 Microservices	architecture	assume	:	team	=	silo/kingdom.		
 Dont	use	wrong	paCerns!	
 If	you	don't	understand	your	system	,	your	micro	service	will	
not	work	in	scale.	
Avoid:	designed	by	programmers	for	programmers,	keep	balance
2015	ROBERT	BIGOS
	never	deploy	at	
Friday	
!what	ever
息	2015	ROBERT	BIGOS
If	you	need	more
There	are	no	in鍖nite	resources.	There	are	no	perfect	resources.	Monitoring	is	
part	of	a	capacity	management	process	which	is	only	a	part	of	operaPons	
management.	Heading	to	simplicity	of	usage	we	accept	complexity	of	the	
soluPon	and	somePmes	blindness	hoping	that	when	the	Pme	comes	magic	
words	will	solve	all	the	problems.	Hope	always	dies	last	and	silence	ayer	
failure	gives	us	a	lesson	in	humility:	we	have	to	learn	how	to	interpret	
monitoring	data	as	this	is	engineering	not	magic.	Process	is	more	than	just	
tools	and	people	and	management	is	inspiraPon	and	determinaPon	to	
achieve	at	least	goals	that	were	de鍖ned.	More	failures,	less	Pme	to	learn	-	
this	is	part	of	:	
You	build	it,	you	run	it	
See	more	:
hCp://www.slideshare.net/RobertBigos
hCps://pl.linkedin.com/in/robertbigos
Blog:	bigosr.com

More Related Content

Microservices monitoring challange