» » » » » Apache-Kafka Tutorial for Beginners

Apache-Kafka Tutorial for Beginners

, , , Edit
                                                                        APACHE    KAFKA 

Introduction:

  • Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java.
  • It is a publish and subscribe mechanism.
  • The producer will push the message into brokers. and the consumer will consume the message from the broker.
Flow Diagram:

Flow

  • producer sends a message to the broker through the push mechanism.
  • The consumer reads data from the broker through the pull mechanism.
  • The broker is a very lightweight component that handles just TCP connections and writes data to the append-only log file.
  • Zookeeper acts as a coordinator between the broker and the consumer.
Software Requirements:
Installation:
  1. Step1: Before the install the Apache Kafka and zookeeper you have installed the java on your system otherwise Kafka will not running on because of Kafka is built on using Java and Scala.
  2. Step2: If you have to check java successfully installed or not your system using the: java -version it will show the version of java installed in your system.
  3. Step3: Before installation of Kafka zookeeper will install because Kafka will run on zookeeper instance. Zookeeper is a mediator between broker and consumer.
Zookeeper installation:
  1. Step1: Download the updated zookeeper setup and extract into your desired location: go the Config folder and rename the zoo_sample.cfg” to “zoo.cfg” 
  2. Step2: Open the file on the text editor and change the datadir to \zooleepr-3.4.7\data because of your data previously will store on temp directory so that we change the temp directory to data folder.
  3. Step3: By Default, zookeeper will run on 2182 port if you want to change the port change the port on client Port section.
  4. Step4: Set the zookeeper path class-path on your Environment variables.
  5. Step5: Open command prompt and type the Zkserver you will see the command prompt with some details like below. ZkServer6.  Step6: You will see the message like binding to port : localhost:9192 in my zookeeper install will run on 9192 port because I changed the client port, by default it will bind to 2181 port.
Kafka Installation:
  • Step1: Download the Apache Kafka and extract the folder and go to the Config folder edit the server.properties file change the directory log.dir = temp/Kafka-logs to “C:\kafka_2.11-0.9.0\kafka-logs”
  • Step2: Change the port in zookeeper. connect section if you had any change on default port in zookeeper client port, otherwise, it will bind to 2181 port.
  • Step3: Edit the zookeeper.properties file change the dataDir and clientPorts.
  • Step4: Before starting the Apache Kafka zookeeper instance will running up otherwise it will show some errors.
  • Step5: Go to Apache Kafka source folder open the command prompt and type the below command .\bin\windows\kafka-server-start.bat .\config\server.properties
  • Step6: If everything successfully installed your command prompt will look like below
Apache Kafka Server
  • Now your Kafka is up and running successfully, now you can create topics to store messages. Also, we can produce or consume data from the command prompt.
How To Create topics using the command line?
Now create a topic with the name “Janardhan” replication factor 1.
  • Step1: Go to C:/ApacheKafka/bin/windows folder and open the command prompt type the below command Kafka-topics.bat --create --zookeeper localhost:9192 --replication-factor 1 --partitions 1 --topic Janardhan
  • After successfully executed command you will get prompt like below
Creating Topics

 How to create Producer using the command line?
  • Step1: Open the Command prompt and go to C:/ApacheKafka/bin/windows and type the below command Kafka-console-producer.bat --broker-list localhost:9092 --topic Janardhan 
Creating Broker
How to create a consumer using the command line?
  • Step1: Open the command prompt and go the folder c:/apacheKafka/bin/windows then type the below command Kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Janardhan –from-beginning
  • Step2: After successfully run the producer and consumer, from producer side push messages into the broker and in consumer side pull the messages from the broker.
  • Step3: Now type anything in the producer command prompt and press Enter, and you should be able to see the message in the other consumer command prompt.
Producer and Consumer message Transformation
  • If you are able to push & see your messages in the consumer side.
Useful Commands:
List Topics : kafka-topics.bat --list --zookeeper localhost:9192
Read messages from beginning:kafka-console-consumer.bat --zookeeper localhost:2181 --topic [Topic Name] --from-beginning
Delete Topic :kafka-run-class.bat kafka.admin.TopicCommand --delete --topic [topic_to_delete] --zookeeper localhost:9192
Describe Topic :kafka-topics.bat --describe --zookeeper localhost:2181 --topic [Topic Name]

SUBSCRIBE TO OUR NEWSLETTER

janardhan Randhi

I’m the Founder of quickdevops.com. I am a Professional Blogger, Application developer, YouTuber. I’ve been blogging since 2015.I spend a lot of time learning new techniques and actively help other people learn web development through a variety of help groups and writing web development tutorials for my website and blog about advancements in web design and development.Besides programming I love spending time with friends and family and can often be found together going out catching the latest movie or planning a trip to someplace I've never been before.

3 Responses to "Apache-Kafka Tutorial for Beginners"

Subscribe to: Post Comments (Atom)