Nyc storm meetup_robdoherty
6 likes2,271 views
Presentation at NYC Storm Meetup #1 on the Kafka-Storm implementation used in production at Outbrain Engage to track thousands of web traffic pings per second.
Nyc storm meetup_robdoherty
- 1. Storm at Rob Doherty Senior Backend Engineer rdoherty@outbrain.com @robdoherty2
- 3. Before Storm Custom distributed processing system Python and ZMQ Advantages: Simple components Well-understood Disadvantages: Did not scale Batch-processing
- 4. Kafka + Storm Kafka: high-throughput distributed messaging Storm: distributed, real-time computation
- 5. Why Kafka? Need method to buffer clicks into stream Kafka + Storm common pattern for click tracking
- 6. Why Storm? Real time (15s latency requirements) Fault tolerance Easy to manage parallelism Stream grouping Active community Open-source project
- 7. Nginx Servers Kafka Cluster Storm Topology Elastic Load Balancer Customer Traffic AWS MongoDB Redis Algo API Architecture
- 8. Kafka Cluster 40 Producers (8 m1.large instances) Python brod 4 Brokers (4 m1.large instances) 10k Clicks per second (peak) 14B Clicks per month Kafka v0.7.2
- 9. Storm Topology 40 Supervisors (c1.xlarge instances) 35 Bolts, 1 Kafka spout 250+ Executors (worker threads) 160k+ tuples executed per second Storm v0.82 Leiningen v1.7
- 10. Customer Traffic Kafka Spout Aggregate 15s Aggregate 5m Position Customer Social Arrangement Front Page @Handle Storm Topology
- 11. Challenges Shell Bolts Anchor Bolts/Replaying Stream Acking Tuples Monitoring
- 12. Monitoring Scribe Logging Munin + Nagios JMX-JMXTrans + Ganglia Storm UI Thrift interface into Nimbus + D3
- 15. Monitoring Scribe Logging Munin + Nagios JMX-JMXTrans + Ganglia Storm UI Thrift interface into Nimbus + D3
- 18. Future Plans Load testing Break topology into smaller pieces Move from AWS to private data center
- 19. Thank you Rob Doherty Senior Backend Engineer rdoherty@outbrain.com @robdoherty2