ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Quick reference for kafka

•Download as TXT, PDF•
•
Rajkumar Asohan, PMP
Rajkumar Asohan, PMP

This document provides instructions for setting up single and multi-broker Kafka clusters. For a single broker cluster, it describes how to start Zookeeper, the Kafka broker, create and view topics, and add partitions. For a multi-broker cluster, it explains how to configure additional brokers, create a replicated topic, kill a broker, and view that the topic remains accessible.

1 of 2
Downloaded 41 times
*** Single Broker Cluster *** Start Zookeeper [seperate terminal] --------------- bin/zookeeper-server-start.sh config/zookeeper.properties Start Kafka broker [seperate terminal] ------------------ bin/kafka-server-start.sh config/server.properties ________________________________________________________________________________ _____________________________________________________ Create Topic ------------ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testtopic List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Display Topic Information ------------------------- bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic testtopic Add Partitions to a Topic ------------------------- bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic testopic --partitions 2 ________________________________________________________________________________ _____________________________________________________ Start Producer [seperate terminal] -------------- bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopic Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from- beginning ________________________________________________________________________________ _____________________________________________________ Note: To understand the flow refer Kafka Tutorial.mp4 *** Multi Broker Cluster *** Create config file for 2 more Kafka Brokers ------------------------------------------- cp config/server.properties config/server-1.properties cp config/server.properties config/server-2.properties Edit config file 2 Kafka Brokers created above ---------------------------------------------- config/server-1.properties: broker.id=1 port=9093
log.dir=/tmp/kafka-logs-1 config/server-2.properties: broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2 Start 2 newly created Kafka Brokers [seperate terminal] ----------------------------------- JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties & JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties & Create Topic ------------ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Start Producer [seperate terminal] -------------- bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my- replicated-topic Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic Kill one of the Kafka Brokers ----------------------------- pkill -9 -f server-1.properties List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic

More Related Content

Quick reference for kafka

  • 1. *** Single Broker Cluster *** Start Zookeeper [seperate terminal] --------------- bin/zookeeper-server-start.sh config/zookeeper.properties Start Kafka broker [seperate terminal] ------------------ bin/kafka-server-start.sh config/server.properties ________________________________________________________________________________ _____________________________________________________ Create Topic ------------ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testtopic List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Display Topic Information ------------------------- bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic testtopic Add Partitions to a Topic ------------------------- bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic testopic --partitions 2 ________________________________________________________________________________ _____________________________________________________ Start Producer [seperate terminal] -------------- bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopic Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from- beginning ________________________________________________________________________________ _____________________________________________________ Note: To understand the flow refer Kafka Tutorial.mp4 *** Multi Broker Cluster *** Create config file for 2 more Kafka Brokers ------------------------------------------- cp config/server.properties config/server-1.properties cp config/server.properties config/server-2.properties Edit config file 2 Kafka Brokers created above ---------------------------------------------- config/server-1.properties: broker.id=1 port=9093
  • 2. log.dir=/tmp/kafka-logs-1 config/server-2.properties: broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2 Start 2 newly created Kafka Brokers [seperate terminal] ----------------------------------- JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties & JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties & Create Topic ------------ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Start Producer [seperate terminal] -------------- bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my- replicated-topic Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic Kill one of the Kafka Brokers ----------------------------- pkill -9 -f server-1.properties List Topic ---------- bin/kafka-topics.sh --list --zookeeper localhost:2181 Start Consumer [seperate terminal] -------------- bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic