際際滷

際際滷Share a Scribd company logo
Autoscaling for fun and pro鍖t
Alexander Demidko,
Software Engineer @ Metamarkets
API
Kafka Druid
0.5PB  于仂亟亳/亟亠仆 => $$$$
API
Kafka Druid
Batch  EC2 Spot :)
Real-time  EC2 On-Demand $-(
Autoscaling for fun and profit
Autoscaling for fun and profit
Autoscaling for fun and profit
 仂 亠舒亞亳仂于舒
 弍亳舒 舒亠亞亳:
A) +10 从仂仆亠亶仆亠仂于 (30 仄亳仆)=> 10 从仂仆亠亶仆亠仂于
B) +40 从仂仆亠亶仆亠仂于 (20 仄亳仆)=> 40 从仂仆亠亶仆亠仂于
 丕亳于舒 仗仂亞仆仂亰:
仂弍亠仄 于仂亟亳 亟舒仆仆 = f(于亠仄)
仗仂仗从仆舒 仗仂仂弍仆仂 = g(从仂仆亠亶仆亠仂于)
Monte Carlo Tree Search
仂亳仄仂 = 亢亠仍亠亰仂 + 仗亠仆舒仍亳 亰舒 亰舒亟亠亢从
亳仍仂 从仂仆亠亶仆亠仂于
penalty(s, l)(x) = (atan(x*s  l*s) + atan(l*s)) / (0.5* + atan(l*s))
 Selection - 于弍亳舒亠仄, 亞亟亠 仂亳 仆仂于亶 亰亠仍 (exploration VS exploitation tradeoff)
 Expansion - 仂亳仄 亠亞仂 :)
 Simulation - 于亳亳亠从亳 仂亠仆亳于舒亠仄, 于仂 从仂仍从仂 仂仆 仂弍仂亶亟亠
 Backpropagation - 仂弍仆仂于仍磳仄 仂亳仄仂亳 亠亠仆亳亶 于 亟亠亠于亠
Simulation
丕仄亠仆亠仆亳亠 舒亰仄亠仆仂亳 亟亠亠于舒
 亳从亠亳亰舒亳 亳仍舒 从仂仆亠亶仆亠仂于
 亳从亠亳亰舒亳 于亠仄亠仆亳
Guardian
job_A - 5 => 10 从仂仆亠亶仆亠仂于
job_B - 8 => 6 从仂仆亠亶仆亠仂于
job_C - 15 => 20 从仂仆亠亶仆亠仂于
job_A - 5 => 15 从仂仆亠亶仆亠仂于
job_D - 2 => 4 从仂仆亠亶仆亠仂于
Health monitoring
t_curr t_last
Kafka
仂仍亠弍舒仆亳: 仆亠 仄仂亢亠仄 仂仆仂 于舒于亳 亳亟亠舒仍仆仂亠 亳仍仂 从仂仆亠亶仆亠仂于
丼亳仍仂 Kafka 仗舒亳亳亶 % 丼亳仍仂 从仂仆亠亶仆亠仂于 must be(0)
avg = 35100 / (24*60) ~ 24
丕仄亠仆亳仍亳 仂弍亠仄 亢亠仍亠亰舒 仗仂亳 于 2 舒亰舒
舒亠仄舒亳从舒/亠舒仍亳亰舒亳: 20/80
弌仗舒亳弍仂!

More Related Content

Autoscaling for fun and profit