Kafka Workshop

Oct 28, 20160 likes351 views

This document discusses Kafka and message buses. It describes synchronous vs asynchronous communication and how a message bus can be used for event aggregation, CQRS, microservices, and event sourcing. It then provides details on Apache Kafka, including its creation by LinkedIn, use by major companies, and basic workflow. Key components of the Kafka stack like Zookeeper, brokers, producers, consumers and topics are explained. The document ends with an example of launching a local Kafka cluster and producing and consuming messages.

Kafka & message bus
Robin GRAILLON & Alexandre ANDRÉ
12/10/2016

Synchronous
● Call
● Processing
○ Might be long
● Response

Asynchronous
● Call
● Response
● Post-Processing

Message Bus
● Events aggregation
● CQRS oriented
● µS oriented
● Event sourcing oriented
● Language agnostic
● Multiple implementations
○ RabbitMQ
○ Kafka
○ etc.

Apache Kafka
● LinkedIn creation
● Open sourced in 2011
● Filesystem oriented
● Wrote in Scala
● Highly scalable
● Used by big companies
○ LinkedIn
○ Netflix
○ Spotify
○ Meetic

LinkedIn statistics
● 800 billion m/day
○ 175 TB
● 13 million m/sec
○ 2.75 GB
● 1100 kafka instances
○ 60 clusters

Basic workflow
Producer 1
Producer 2
Producer 3
Producer X
Consumer 1
Consumer 2
Consumer 3
Consumer Y
Kafka Stack

Kafka stack
● Zookeeper : scalability, manager
● Kafka broker : kafka server instance
● Consumer : consumes events and do things
● Producer : produces events (like user as connected)
● topic : event name
● partition : way to split messages between brokers

Kafka partitions (1)
For one topic (let’s say “astronaut.connection”)

...but easily !
./console run
HELP
run:zk run zookeeper
kill:zk kill zookeeper
run:kk run kafka
kill:kk kill kafka
producer run producer on topic
consumer run consumer on topic
t:c create a topic
t:d delete a topic
t:l list topics
What do you want to do?

Step 1: launch Zookeeper!
What do you want to do? run:zk
Configuration file? [./zookeeper.properties]
Port? [2181]
./../bin/zookeeper-server-start.sh ./zookeeper.properties
What do you want to do?

Step 2: launch Kafka!
What do you want to do? run:kk

Step 3: list topics!
What do you want to do? t:l
Zookeeper host? [localhost:2181]
./../bin/kafka-topics.sh --list --zookeeper localhost:2181
What do you want to do?

Step 4: create a topic!
What do you want to do? t:c
Topic name? [test]
Zookeeper host? [localhost:2181]
Partitions? [1]
Replication factor? [1]
./../bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
--partitions 1 --replication-factor 1
Created topic "test".
What do you want to do?

Step 5: list topics AGAIIIIN!
What do you want to do? t:l
Zookeeper host? [localhost:2181]
./../bin/kafka-topics.sh --list --zookeeper localhost:2181
test
What do you want to do?

Step 6: run a consumer!
New terminal !
$ ./console run
What do you want to do? consumer
Topic name? [test]
Zookeeper host? [localhost:2181]
From beginning? [1]
./../bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
--from-beginning

Step 7: generate data!
What do you want to do? g
Message? [I'm an astronaut!]
How many times? [1] 10000000
File? [data_10000000.txt]
10000000/10000000 [============================] 100%

Step 8: produce events!
Back to the other terminal
What do you want to do? p
Topic name? [test]
brokers? [localhost:9092]
data_10000000.txt
Dataset to use? [data_10000000.txt]
./../bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test <
./data/data_10000000.txt

GroupId: experiment 1
● 2 groupId
● 2 partitions
● 2 consumers

GroupId: experiment 2
● 1 groupId
● 2 consumer

Documentation/resources
● Kafka quick setup
https://kafka.apache.org/quickstart
● Kafka at LinkedIn
https://engineering.linkedin.com/kafka/running-kafka-scale
● Why zookeeper ?
https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka

The document outlines exercises for learning the ELK stack using Docker. It introduces Elasticsearch for data storage, Logstash for collecting and parsing logs, and Kibana for visualization. The exercises demonstrate setting up the environment, sending a test message to ELK, analyzing Shakespeare works, customizing the Kibana dashboard, using Grok filters to parse logs, and filtering Apache logs with Logstash.

Docker and FluentdSATOSHI TAGOMORI

��

This document discusses using Docker and Fluentd together for logging container architectures. It describes different patterns for aggregating logs from containers using Fluentd, including 1-level and 2-level aggregation. A new Docker logging driver called "fluentd" is announced that will directly send container logs to Fluentd. The document also provides information on the official Fluentd Docker image.

How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecturevsoshnikov

��

A typical microservices architecture has five tiers: You need a web proxy like NGINX to handle slow clients and to serve static files You need an application web server like Apache or Django to run your application logic You need a caching layer like Memcached or Redis to cache frequently accessed data You need a database proxy to handle sharding and replication, abstracting that stuff away from the application You need a database management system to store your data Five tiers – isn't it too many? Yeah, that's what we think too. So we've come up with an idea to shrink the whole bunch of tiers down to just a couple. The web proxy and database proxy are combined into one tier, now just a proxy. The application server, the cache, and the DBMS are packed into the other tier. Fewer tiers mean less problems and more bonuses like data consistency and better performance. In this session, I'll explain how to do this magic and how it works under the hood.

Fluentd v1 and RoadmapTreasure Data, Inc.

��

Fluentd v1 will focus on providing a stable and pluggable architecture for log forwarding and processing. Key features include a new configuration format that allows embedding Ruby code, improved filtering and labeling of events, and an error stream for handling errors. Other potential improvements include multi-processing support, zero downtime restarts, an actor framework, and JRuby/Windows compatibility. td-agent2 will upgrade the core libraries and use the new v1 configuration by default.

How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà

��

Bol.com uses the Elastic (ELK) stack to make sense of logs from over 1,600 servers and 500-600 million events per day. Key aspects of their system include: 1. Shipping JSON-formatted log events from sources like Apache, databases, and applications to Redis queues to allow multiple Logstash instances to process events in real-time without data loss. 2. Enriching log events with information like request IDs to correlate requests across services, and IP-to-role mappings to identify client roles. 3. Using Elasticsearch aggregations and transformations to generate a directed graph of service dependencies based on logs, to help understand their distributed architecture.

Writing Rust Command Line ApplicationsAll Things Open

��

Collect distributed application logging using fluentd (EFK stack)Marco Pas

��

This document discusses using Fluentd to collect distributed application logging in a containerized environment. It provides an overview of Fluentd, including its pluggable architecture and configuration. It then demonstrates capturing logging from Docker containers and HTTP services and storing the logs in MongoDB and Elasticsearch as part of the ELK stack. It shows filtering and parsing logs from a Spring Boot application. Finally, it discusses setting up Fluentd for high availability.

Kong in 1.x TerritoryThibault Charbonnier

��

Kong 1.0 reached maturity with the addition of L4 proxying, service mesh capabilities using mTLS, and blue/green upgrade paths. Kong 1.1 introduced declarative configuration using a YAML file for DB-less deployments and importing configurations into databases. It also added tags for filtering entities. Upcoming versions will focus on performance, gRPC support, and Kong Enterprise features while continuing to consolidate the platform.

Event machinealmeidaricardo

��

Event Machine is a library for Ruby that provides event-driven I/O. It uses the reactor pattern to achieve high scalability, performance, and stability. Event Machine powers the backends of many large sites by enabling asynchronous programming and handling many concurrent connections efficiently. It can be used with tools like Rails, Thin, Goliath, MongoDB, and RabbitMQ to build highly scalable web applications. Benchmarks show that using Event Machine can process many more requests per second than a traditional Rails app due to its asynchronous and non-blocking nature.

Cortex: Prometheus as a Service, One Year OnKausal

��

Presented by Tom Wilkie at PromCon 2017 With Speaker Notes: https://goo.gl/V9qGva At PromCon 2016, I presented "Project Frankenstein: A multitenant, horizontally scalable Prometheus as a service". It's now one year later, and lots has changed - not least the name! This talk will discuss what we've learnt running a Prometheus service for the past year, the architectural changes we made from the original design, and the improvements we've made to the Cortex user experience.

Scaling an ELK stack at bol.comRenzo Tomà

��

A presentation about the deployment of an ELK stack at bol.com At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure. The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling. These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.

PharoDAYS 2015: Pharo Status - by Markus DenkerPharo

��

Pharo 4 will be released in Spring 2015 and contains many improvements and updates over previous versions. It has already seen over 1200 issues closed and is very stable. Small changes include improved refactorings, a smaller 6MB deployment image, and ifTrue: working on non-Booleans. Larger ongoing projects include first class variables, replacing the .changes system with Epicea, advanced reflection work, and VM improvements.

BSides London - Scapy WorkshopAdam Maxwell

��

This document provides an overview of a Scapy workshop that will cover topics including: - Installing and using Scapy to forge, decode, send, and receive packets - Creating basic packets like TCP, UDP, ICMP and modifying packet fields - Advanced packet crafting including HTTP requests and DNS queries - Using Scapy with Python for tasks like packet sniffing and parsing PCAP files - Replaying and filtering captured packets - The presenter's background and goals for the workshop

GeoDistributed datacenter: the DNS wayMoyd.co LTD

��

Alberto Zuin presents on moyd.co's geo-distributed infrastructure using OpenNebula. Moyd.co operates DNS servers across 4 datacenters in Europe and the US with anycast BGP routing. Each datacenter runs independent DNS and API servers with a shared MongoDB database. OpenNebula is used to federate the distributed infrastructure and configure DNS VMs through contextualization. Example use cases include distributing a news site, email servers, and log storage across multiple datacenters.

Ansible with AWSAllan Denot

��

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet

��

8 Ways Network Engineers use Snabb (RIPE 77)Igalia

��

Fluentd v1 and future at techtalkN Masahiro

��

Fluentd v1 provides major improvements over v0.12 including nanosecond event time resolution, multi-core support, Windows support, and new plugin APIs. The new plugin APIs provide well-controlled lifecycles and integrate all output plugins. v1 also introduces a server engine based supervisor, dynamic buffering capabilities, and various plugin helpers. While maintaining compatibility with v0.12 plugins, v1 focuses on ease of use, stability, performance and flexibility.

Docker Workshop - Orchestrating Docker ContainersHugo Henley

��

This document outlines an agenda for a Docker workshop on orchestrating Docker containers. It discusses various tools in the container ecosystem including Rancher, Mesos, Marathon, service discovery with Consul and Registrator, monitoring with Prometheus and Grafana, and continuous integration/deployment with Jenkins. Mesos is presented as a way to share resources across frameworks like Marathon. Consul and Registrator enable service discovery so applications can be accessed regardless of where they run. Prometheus and Grafana provide monitoring. The workshop emphasizes combining these tools - Docker, Mesos, Marathon, Rancher, Consul, Registrator, Consul Template, cAdvisor, Prometheus, Grafana, and Jenkins - for a complete container management solution.

AGES Presentation on Web, Python, Django and GeoServerNg'eno Victor

��

Beyond Fault Tolerance with Actor ProgrammingFabio Tiriticco

��

Actor Programming is a software building approach that lets you can go beyond fault tolerance and achieve Resilience, which is the capacity of a system to self-heal and spring back into a fresh shape. First I'll introduce the difference between Reactive Programming and Reactive Systems, and then we'll go over a couple of implementation examples using Scala and Akka. The coupled GitHub repository with the code is here: https://github.com/ticofab/ActorDemo

Woo: Writing a fast web server @ ELS2015fukamachi

��

Eitaro Fukamachi presented on writing the fast web server Woo in Common Lisp. He discussed three tactics for achieving speed: using a multithreaded event-driven architecture instead of prefork, developing a fast HTTP parser instead of regular expressions, and using the libev event library. Benchmarks showed Woo was 1.6 times faster than Node.js. The goal was to create the fastest HTTP server in Common Lisp.

Алексей Петров "PHP at Scale: Knowing enough to be dangerous!"Fwdays

��

PHP at Scale: Knowing enough to be dangerous! by Oleksii Petrov discusses how to scale PHP applications. It covers strategies like caching, queueing, read/write splitting, and sharding. It also discusses using load balancers and choosing the right database. The key is to improve system metrics without dramatically changing the system. Scaling is predefined by your stack and architecture. Performance comes from optimizations everywhere, not just PHP. Being distributed is very challenging.

Ruby eventmachine pres at rubybdxMathieu Elie

��

Sphinx && Perl Houston Perl Mongers - May 8th, 2014Brett Estrade

��

The document discusses using Sphinx to index and search data. Sphinx allows indexing data from MySQL, PostgreSQL, and XMLPipe2 formats. It supports searching via its native protocol or MySQL protocol. The document provides examples of indexing over 2 million rows from a MariaDB database, filtering search results using hashes, transforming data to the XMLPipe2 format, and dynamically configuring indexes when data partitions change over time using the Sphinx::Config::Builder module. Real-world use cases discussed include providing a search interface for auction data and indexing multiple data partitions that can change.

Developing high-performance network servers in LispVladimir Sedach

��

Pharo Status Fosdem 2015Marcus Denker

��

Pharo 4 was released in Spring 2015, containing many improvements and updates since Pharo 3 in April 2014. These include improved refactorings, a dark theme, GT tools replacing old tools, first class variables, advanced reflection capabilities, Epicea replacing .changes, and a new GC called Spur. Future plans include a 64-bit COG VM, an optimizer called Sista, updated windowing with SDL2, a Block redo in Morphic, 3D with Woden, and a virtual GPU. Pharo remains very active with ongoing development and a welcoming community.

Writing Better Haskellnkpart

��

- The Kit dependency manager was initially written in one weekend with little Haskell knowledge and resulted in bad code with IO everywhere and little usage of standard libraries. - Over time, the code was improved by removing IO usage, learning to leverage standard libraries and Hackage packages, cleaning up imports, and specifying invariants through QuickCheck properties and tighter types. - While an improvement, the code was still not considered "good" and could be further refined by fully removing IO from the monad stack and defining more precise invariants.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

��

Netflix Keystone—Cloud scale event processing pipelineMonal Daxini

��

More Related Content

What's hot (20)

Event machinealmeidaricardo

��

Cortex: Prometheus as a Service, One Year OnKausal

��

Scaling an ELK stack at bol.comRenzo Tomà

��

PharoDAYS 2015: Pharo Status - by Markus DenkerPharo

��

BSides London - Scapy WorkshopAdam Maxwell

��

GeoDistributed datacenter: the DNS wayMoyd.co LTD

��

Ansible with AWSAllan Denot

��

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet

��

8 Ways Network Engineers use Snabb (RIPE 77)Igalia

��

Fluentd v1 and future at techtalkN Masahiro

��

Docker Workshop - Orchestrating Docker ContainersHugo Henley

��

AGES Presentation on Web, Python, Django and GeoServerNg'eno Victor

��

Beyond Fault Tolerance with Actor ProgrammingFabio Tiriticco

��

Woo: Writing a fast web server @ ELS2015fukamachi

��

Алексей Петров "PHP at Scale: Knowing enough to be dangerous!"Fwdays

��

Ruby eventmachine pres at rubybdxMathieu Elie

��

Sphinx && Perl Houston Perl Mongers - May 8th, 2014Brett Estrade

��

Developing high-performance network servers in LispVladimir Sedach

��

Pharo Status Fosdem 2015Marcus Denker

��

Writing Better Haskellnkpart

��

Event machinealmeidaricardo

��

Cortex: Prometheus as a Service, One Year OnKausal

��

Scaling an ELK stack at bol.comRenzo Tomà

��

PharoDAYS 2015: Pharo Status - by Markus DenkerPharo

��

BSides London - Scapy WorkshopAdam Maxwell

��

GeoDistributed datacenter: the DNS wayMoyd.co LTD

��

Ansible with AWSAllan Denot

��

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet

��

8 Ways Network Engineers use Snabb (RIPE 77)Igalia

��

Fluentd v1 and future at techtalkN Masahiro

��

Docker Workshop - Orchestrating Docker ContainersHugo Henley

��

AGES Presentation on Web, Python, Django and GeoServerNg'eno Victor

��

Beyond Fault Tolerance with Actor ProgrammingFabio Tiriticco

��

Woo: Writing a fast web server @ ELS2015fukamachi

��

Алексей Петров "PHP at Scale: Knowing enough to be dangerous!"Fwdays

��

Ruby eventmachine pres at rubybdxMathieu Elie

��

Sphinx && Perl Houston Perl Mongers - May 8th, 2014Brett Estrade

��

Developing high-performance network servers in LispVladimir Sedach

��

Pharo Status Fosdem 2015Marcus Denker

��

Writing Better Haskellnkpart

��

Similar to Kafka Workshop (20)

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

��

Netflix Keystone—Cloud scale event processing pipelineMonal Daxini

��

Final_Report_new (1)Adarsh Burma

��

This document outlines a log analysis project that streams logs from Kafka into Spark for analysis. It includes configurations for setting up a multi-broker Kafka cluster, integrating Kafka with the Spark application, and code snippets for streaming the logs. The Spark application then analyzes the logs and outputs statistics to an interactive web page, including top endpoints, frequent IP addresses, and response code counts. Screenshots show the web output and Spark UI during job execution.

Introduction to apache kafkaSamuel Kerrien

��

Apache KafkaJoe Stein

��

Apache Kafka is a distributed publish-subscribe messaging system that was originally created by LinkedIn and contributed to the Apache Software Foundation. It is written in Scala and provides a multi-language API to publish and consume streams of records. Kafka is useful for both log aggregation and real-time messaging due to its high performance, scalability, and ability to serve as both a distributed messaging system and log storage system with a single unified architecture. To use Kafka, one runs Zookeeper for coordination, Kafka brokers to form a cluster, and then publishes and consumes messages with a producer API and consumer API.

Introduction to Apache KafkaShiao-An Yuan

��

This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.

Building Stream Processing as a ServiceSteven Wu

��

This document discusses building stream processing as a service (SPaaS) using Apache Flink. It introduces Flink's stream processing capabilities and describes how to build a SPaaS offering with different levels of complexity and ease of use. It also covers the Keystone router for simple stream routing, building custom Flink jobs, and techniques for recovering from failures using backfill from Hive or rewinding the Flink job.

Fullstack workshopAssaf Gannon

��

Scio - Moving to Google Cloud, A Spotify StoryNeville Li

��

Talk at Philly ETE Apr 28 2017 We will talk about Spotify’s story of migrating our big data infrastructure to Google Cloud. Over the past year or so we moved away from maintaining our own 2500+ node Hadoop cluster to managed services in the cloud. We replaced two key components in our data processing stack, Hive and Scalding, with BigQuery and Scio and are able to iterate at a much faster speed. We will focus the technical aspect of Scio, a Scala API for Apache Beam and Google Cloud Dataflow and how it changed the way we process data.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini

��

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

[KubeCon EU 2020] containerd Deep DiveAkihiro Suda

��

Ansiblefest 2018 Network automation journey at robloxDamien Garros

��

In December 2017, Roblox’s network was managed in a traditional way without automation. To sustained its growth, the team had to deploy 2 datacenters, a global network and multiple point of presence around the world in few months, the only solution to be able to achieve that was to automate everything. 6 months later, the team has made tremendous progress and many aspects of the network lifecycle has been automated from the routers, switches to the load balancers. Synopsis This talk is a retrospective of Roblox’s journey into Network automation: How we got started and how we automated an existing network. How we organized the project around Github and an DCIM/IPAM solution (netbox), How Docker helped us to package Ansible and create a consistent environment. How we managed many roles and variations of our design in single project How we have automated the provisioning of our F5 Load Balancers. For each point, we’ll cover what was successful, what was more challenging and what limitations we had to deal with.

04-Kafka.pptxMannMehta13

��

Kafka is a fast, scalable, and durable distributed commit log service that can handle millions of messages per second. It is used by many large companies as a messaging backbone and for real-time data processing. Kafka topics are partitioned and replicated for high throughput and fault tolerance. Producers write messages to topics that are consumed by consumer groups who track their progress using offsets.

04-Kafka.pptxAdityaGanguly12

��

10 Lessons Learned from using Kafka with 1000 microservices - java global summitNatan Silnitsky

��

Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices. We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place. In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include: * How to increase dev velocity of event driven style code. * How to optimize working with Kafka in polyglot setting * How to support growing amount of traffic and developers. * How to tackle multiple DCs environment.

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky

��

At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent

��

"As Apache Kafka gains widespread adoption, an increasing number of people face its pitfalls. Despite completing courses and reading documentation, many encounter hurdles navigating Kafka's subtle complexities. Join us for an enlightening session led by the customer support team of Conduktor, where we engage daily with users grappling with Kafka's subtleties. We've observed recurring themes in user queries: What happens when a consumer group rebalances? What is an advertised listener? Why aren't my records displayed in chronological order when I consume them? How does retention work? For all these questions, the answer is ""It depends"". In this talk, we aim to demystify these uncertainties by presenting nuanced scenarios for each query. That way you will be more confident on how your Kafka infrastructure works behind the scenes, and you'll be equipped to share this knowledge with your colleagues. By being aware of the most common misconceptions, you should be able to both speed up your own learning curve and also help others more effectively."

DockerCon 2022 - From legacy to Kubernetes, securely & quicklyEric Smalling

��

You’ve been developing software for years and now your team is ready to take the plunge into orchestrated containers and Kubernetes. You’ve learned about containers, images, and Dockerfiles, but standing up a Kubernetes cluster and actually running your app in it seems like a daunting task. In this session, we’ll go over the basics to get your app up and running in Kubernetes right on your own workstation using Docker Desktop. On the way, we’ll cover some of the security aspects you need to keep in mind and show you how to implement them in your Kubernetes manifests. We’ll go over: 1.) Kubernetes basics, including pods, deployments, and services 2.) Moving a legacy app into a container and running it in Kubernetes 3.) Some security best practices to watch out for — and what can happen if you don’t 4.) Implementing those best practices to defend against and limit the blast radius of an attack

Enabling Data Scientists to easily create and own Kafka ConsumersStefan Krawczyk

��

At Stitch Fix, we hire Full Stack Data Scientists (145+) and expect them to perform diverse functions: from conception to modeling to implementation to measurement. Since Kafka is the way we get event data, this inevitably means that a Data Scientist will need to write a Kafka consumer if they’re going to complete their implementation work. E.g. to transform some client data into features, or perform a model prediction, or allocate someone to an A/B test, etc. In this talk I’ll go over how we built an opinionated Kafka client to easily enable Data Scientists to deploy and own production Kafka consumers, by focusing on writing python functions rather than fighting pitfalls with Kafka.

Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...HostedbyConfluent

��

The document summarizes an expert talk given by Stefan Krawczyk on enabling data scientists at Stitch Fix to easily create and own Kafka consumers. It discusses Stitch Fix as a company, data science at Stitch Fix, their opinionated Kafka consumer code, and learnings. It describes how data scientists are responsible for full stack data science, from idea to production. It provides an example of a simple "hello world" Kafka consumer in Python and explains the code. It outlines the differences between platform concerns owned by engineers and data science concerns.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

��

Netflix Keystone—Cloud scale event processing pipelineMonal Daxini

��

Final_Report_new (1)Adarsh Burma

��

Introduction to apache kafkaSamuel Kerrien

��

Apache KafkaJoe Stein

��

Introduction to Apache KafkaShiao-An Yuan

��

Building Stream Processing as a ServiceSteven Wu

��

Fullstack workshopAssaf Gannon

��

Scio - Moving to Google Cloud, A Spotify StoryNeville Li

��

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini

��

[KubeCon EU 2020] containerd Deep DiveAkihiro Suda

��

Ansiblefest 2018 Network automation journey at robloxDamien Garros

��

04-Kafka.pptxMannMehta13

��

04-Kafka.pptxAdityaGanguly12

��

10 Lessons Learned from using Kafka with 1000 microservices - java global summitNatan Silnitsky

��

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky

��

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent

��

DockerCon 2022 - From legacy to Kubernetes, securely & quicklyEric Smalling

��

Enabling Data Scientists to easily create and own Kafka ConsumersStefan Krawczyk

��

Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...HostedbyConfluent

��

Recently uploaded (20)

Floating Offshore Wind in the Celtic Seapermagoveu

��

Floating offshore wind (FLOW) governance arrangements in the Celtic Sea case are changing and innovating in response to different drivers including domestic political priorities (e.g. net-zero, decarbonization, economic growth) and external shocks that emphasize the need for energy security (e.g. the war in Ukraine). To date, the rules of the game that guide floating wind in the UK have evolved organically rather than being designed with intent, which has created policy incoherence and fragmented governance arrangements. Despite this fragmentation, the UK has a well-established offshore wind sector and is positioning itself to become a global leader in floating wind. Marine governance arrangements are in a state of flux as new actors, resources, and rules of the game are being introduced to deliver on this aspiration. However, the absence of a clear roadmap to deliver on ambitious floating wind targets by 2030 creates uncertainty for investors, reduces the likelihood that a new floating wind sector will deliver economic and social value to the UK, and risks further misalignment between climate and nature goals.

Unit-03 Cams and Followers in Mechanisms of Machines.pptxKirankumar Jagtap

��

Urban Design and Planning Portfolio .pdfsonam254547

��

Introduction to Forensic Research Digital ForensicsSaanviMisar

��

Digital Forensics: Analyzing Cyber Crimes & Investigations This comprehensive guide on Digital Forensics covers key concepts, tools, and methodologies used in investigating cyber crimes. It explores forensic techniques, evidence collection, data recovery, malware analysis, and incident response with real-world applications. Topics Covered: Introduction to Digital Forensics Cybercrime Investigation Process Digital Evidence & Chain of Custody Popular Forensic Tools (Autopsy, EnCase, FTK) Memory & Network Forensics Challenges in Modern Cyber Investigations Ideal for students, cybersecurity professionals, and forensic analysts, this resource provides valuable insights into digital investigations.

PDF CH 1 Understansding Mechatronics.pdfKovidAggarwal

��

Electric Motors - DC Motors - Series, Shunt, Compound, Seprately Excited, PMD...VirajPasare

��

Protecting Secrets in Transparent SystemsLucaBarbaro3

��

AE- unit 5.ppt ELECTRIC AND HYBRID VEHICLESDr.PERIASAMY K

��

Energy Transition Factbook Bloomberg.pdfCarlosdelaFuenteMnde

��

Introduction to 3D Printing Technology.pptxpprakash21252

��

Welcome to our presentation on 3D printing technology, where we explore the transformative power of this innovative manufacturing process. Also known as additive manufacturing, 3D printing has been gaining momentum in recent years, and its potential to revolutionize industries is vast. In this presentation, we delve into the world of 3D printing, discussing its history, principles, and applications. We examine the various types of 3D printing technologies, including Fused Deposition Modeling (FDM), Stereolithography (SLA), and Selective Laser Sintering (SLS). We also explore the advantages and limitations of 3D printing, including its ability to create complex geometries, reduce material waste, and increase product customization. One of the most significant impacts of 3D printing is its potential to transform industries. We discuss the applications of 3D printing in various sectors, including aerospace, automotive, healthcare, and consumer products. We examine case studies of companies that have successfully implemented 3D printing, such as Boeing, BMW, and Procter & Gamble. The presentation also explores the future of 3D printing, including emerging trends and technologies. We discuss the potential of 3D printing to disrupt traditional supply chains and create new business models. We also examine the challenges and limitations of 3D printing, including the need for standardization, regulation, and education. This presentation is ideal for anyone interested in learning about 3D printing technology, including students, researchers, entrepreneurs, and industry professionals. Whether you are looking to gain a basic understanding of 3D printing or seeking to explore its applications in various industries, this presentation is a valuable resource.

wind energy types of turbines and advantagesMahmudHalef

��

International Journal of Advanced Information Technology (IJAIT)ijait

��

International journal of advanced Information technology (IJAIT) is a bi monthly open access peer-reviewed journal, will act as a major forum for the presentation of innovative ideas, approaches, developments, and research projects in the area advanced information technology applications and services. It will also serve to facilitate the exchange of information between researchers and industry professionals to discuss the latest issues and advancement in the area of advanced IT. Core areas of advanced IT and multi-disciplinary and its applications will be covered during the conferences.

Topic 3.NN and DL Hopfield Networks.pptxManjulaRavichandran5

��

ONLINE MOBILE PRICE PREDICTION USING MACHINE LEARNING (1).pptxbalajisshetty

��

Artificial-Intelligence-in-Cybersecurity.pptxVigneshwarar3

��

Ktor - Definizioni di Path, Integrazioni, Plugin e build fino al rilascioinfogdgmi

��

DEMU - Power distribution final of indian railwaysmishrasanjeev119

��

271094912XOULFHKBXRCVHBJKFG KMXCG HJKLMRTVBHNJMXRCVBHUINJQualityManager48

��

T32 AI Trends, Technologies, & Transformations 250323.pdfMoustafa M Elsayed

��

Unit-I-Water Technology.ppt (Chemistry for Electronics and Compter SystemsKrishnaveniKrishnara1

��

Introduction-types of water-hardness of water-expression of hardness-units of hardness-water quality parameters-estimation of hardness of water by EDTA method-determination of alkalinity, DO, BOD and COD-disadvantages of using hard water in industry. Scale, sludge and boiler corrosion-softening of water, Internal treatment process-carbonate and calgon conditioning-External treatment method-demineralization process and reverse osmosis.

Floating Offshore Wind in the Celtic Seapermagoveu

��

Unit-03 Cams and Followers in Mechanisms of Machines.pptxKirankumar Jagtap

��

Urban Design and Planning Portfolio .pdfsonam254547

��

Introduction to Forensic Research Digital ForensicsSaanviMisar

��

PDF CH 1 Understansding Mechatronics.pdfKovidAggarwal

��

Electric Motors - DC Motors - Series, Shunt, Compound, Seprately Excited, PMD...VirajPasare

��

Protecting Secrets in Transparent SystemsLucaBarbaro3

��

AE- unit 5.ppt ELECTRIC AND HYBRID VEHICLESDr.PERIASAMY K

��

Energy Transition Factbook Bloomberg.pdfCarlosdelaFuenteMnde

��

Introduction to 3D Printing Technology.pptxpprakash21252

��

wind energy types of turbines and advantagesMahmudHalef

��

International Journal of Advanced Information Technology (IJAIT)ijait

��

Topic 3.NN and DL Hopfield Networks.pptxManjulaRavichandran5

��

ONLINE MOBILE PRICE PREDICTION USING MACHINE LEARNING (1).pptxbalajisshetty

��

Artificial-Intelligence-in-Cybersecurity.pptxVigneshwarar3

��

Ktor - Definizioni di Path, Integrazioni, Plugin e build fino al rilascioinfogdgmi

��

DEMU - Power distribution final of indian railwaysmishrasanjeev119

��

271094912XOULFHKBXRCVHBJKFG KMXCG HJKLMRTVBHNJMXRCVBHUINJQualityManager48

��

T32 AI Trends, Technologies, & Transformations 250323.pdfMoustafa M Elsayed

��

Unit-I-Water Technology.ppt (Chemistry for Electronics and Compter SystemsKrishnaveniKrishnara1

��

Kafka Workshop

1. Kafka & message bus Robin GRAILLON & Alexandre ANDRÉ 12/10/2016

2. Synchronous ● Call ● Processing ○ Might be long ● Response

3. Asynchronous ● Call ● Response ● Post-Processing

4. Message Bus ● Events aggregation ● CQRS oriented ● µS oriented ● Event sourcing oriented ● Language agnostic ● Multiple implementations ○ RabbitMQ ○ Kafka ○ etc.

5. Apache Kafka ● LinkedIn creation ● Open sourced in 2011 ● Filesystem oriented ● Wrote in Scala ● Highly scalable ● Used by big companies ○ LinkedIn ○ Netflix ○ Spotify ○ Meetic

6. LinkedIn statistics ● 800 billion m/day ○ 175 TB ● 13 million m/sec ○ 2.75 GB ● 1100 kafka instances ○ 60 clusters

7. Basic workflow Producer 1 Producer 2 Producer 3 Producer X Consumer 1 Consumer 2 Consumer 3 Consumer Y Kafka Stack

8. Kafka stack ● Zookeeper : scalability, manager ● Kafka broker : kafka server instance ● Consumer : consumes events and do things ● Producer : produces events (like user as connected) ● topic : event name ● partition : way to split messages between brokers

9. Kafka partitions (1) For one topic (let’s say “astronaut.connection”)

10. Kafka stack & partitions

11. Let’s play...

12. ...but easily ! ./console run HELP run:zk run zookeeper kill:zk kill zookeeper run:kk run kafka kill:kk kill kafka producer run producer on topic consumer run consumer on topic t:c create a topic t:d delete a topic t:l list topics What do you want to do?

13. Step 1: launch Zookeeper! What do you want to do? run:zk Configuration file? [./zookeeper.properties] Port? [2181] ./../bin/zookeeper-server-start.sh ./zookeeper.properties What do you want to do?

14. Step 2: launch Kafka! What do you want to do? run:kk

15. Step 3: list topics! What do you want to do? t:l Zookeeper host? [localhost:2181] ./../bin/kafka-topics.sh --list --zookeeper localhost:2181 What do you want to do?

16. Step 4: create a topic! What do you want to do? t:c Topic name? [test] Zookeeper host? [localhost:2181] Partitions? [1] Replication factor? [1] ./../bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test --partitions 1 --replication-factor 1 Created topic "test". What do you want to do?

17. Step 5: list topics AGAIIIIN! What do you want to do? t:l Zookeeper host? [localhost:2181] ./../bin/kafka-topics.sh --list --zookeeper localhost:2181 test What do you want to do?

18. Step 6: run a consumer! New terminal ! $ ./console run What do you want to do? consumer Topic name? [test] Zookeeper host? [localhost:2181] From beginning? [1] ./../bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

19. Step 7: generate data! What do you want to do? g Message? [I'm an astronaut!] How many times? [1] 10000000 File? [data_10000000.txt] 10000000/10000000 [============================] 100%

20. Step 8: produce events! Back to the other terminal What do you want to do? p Topic name? [test] brokers? [localhost:9092] data_10000000.txt Dataset to use? [data_10000000.txt] ./../bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < ./data/data_10000000.txt

21. GroupId: experiment 1 ● 2 groupId ● 2 partitions ● 2 consumers

22. GroupId: experiment 2 ● 1 groupId ● 2 consumer

23. Documentation/resources ● Kafka quick setup https://kafka.apache.org/quickstart ● Kafka at LinkedIn https://engineering.linkedin.com/kafka/running-kafka-scale ● Why zookeeper ? https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka

24. Titre 1 Titre 2

�ݺ�ߣ

Kafka Workshop

Recommended

More Related Content

What's hot (20)

Similar to Kafka Workshop (20)

Recently uploaded (20)

Kafka Workshop