Install and setup Apache Kafka On Linux
Apache Kafka is a distributed event streaming platform that is open-source and developed by the Apache Software Foundation. It is written using Java and Scala programming languages, and can be installed on any platform that supports Java.
This tutorial provides a step-by-step guide on how to install Apache Kafka on a Linux system running Ubuntu 20.04 LTS. Additionally, the tutorial covers the creation of topics in Kafka, as well as the setup and running of producer and consumer nodes.
Step 1 — Installing Java
Apache Kafka can be run on all platforms supported by Java. In order to set up Kafka on the Ubuntu system, you need to install java first. As we know, Oracle java is now commercially available, So we are using its open-source version OpenJDK.
sudo apt update
sudo apt install default-jdk
java --version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
Step 2 — Download Latest Apache Kafka
Download the Apache Kafka binary files from its official download website. You can also select any nearby mirror to download.
wget https://downloads.apache.org/kafka/3.4.0/kafka_2.12-3.4.0.tgz
Then extract the archive file
tar xzf kafka_2.12-3.4.0.tgz
sudo mv kafka_2.12-3.4.0 /usr/local/kafka
Step 3 — Creating System Unit Files
Now, you need to create system unit files for the Zookeeper and Kafka services. Which will help you to start/stop the Kafka service in an easy way.
nano /etc/systemd/system/zookeeper.service
And add the following content:
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Save the file and close it.
Next, to create a system unit file for the Kafka service:
nano /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Reload the systemd daemon to apply new changes.
systemctl daemon-reload
Step 4 — Start Kafka and Zookeeper Service
First, you need to start the ZooKeeper service and then start Kafka. Use the systemctl command to start a single-node ZooKeeper instance.
sudo systemctl start zookeeper
Now start the Kafka server and view the running status:
sudo systemctl start kafka
sudo systemctl status kafka
All done. The Kafka installation has been successfully completed. The part of this tutorial will help you to work with the Kafka server.
Step 5 — Create a Topic in Kafka
Kafka provides multiple pre-built shell scripts to work on it. First, create a topic named “myTopic” with a single partition with a single replica:
cd /usr/local/kafka
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic myTopic
Created topic myTopic.
The replication factor describes how many copies of data will be created. As we are running with a single instance keep this value 1. Set the partition options as the number of brokers you want your data to be split between. As we are running with a single broker keep this value 1. You can create multiple topics by running the same command as above.
After that, you can see the created topics on Kafka by the running below command:
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Conclusion
This article helped you to install and configure the Apache Kafka service on an Ubuntu system.