Kafka Cluster

Server IP	Host Role
192.168.10.7	Kafka Node01
192.168.10.8	Kafka Node02
192.168.10.9	Kafka Node03

Kafka Node01

Download JDK installation package

Server with Internet Access
Server without Internet Access

wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

# JDK installation package download link, upload to the deployment server after downloading
https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

Extract the JDK to the installation directory

tar -zxvf OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz
mv jdk-21.0.8+9 /usr/local/openjdk-21

Configure Java symbolic link

ln -s /usr/local/openjdk-21/bin/java /bin/java

Download Kafka installation package

Server with Internet Access
Server without Internet Access

wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

# Kafka installation package download link, upload to the deployment server after downloading
https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

Extract Kafka to the installation directory

tar -zxvf kafka_2.13-3.9.1.tgz -C /usr/local
mv /usr/local/kafka_2.13-3.9.1/  /usr/local/kafka/

Create data directories

mkdir -p /data/kafka/zookeeper/ /data/kafka/kafka-logs/

Add Zookeeper myid
```
echo 1 > /data/kafka/zookeeper/myid
```
- Note: Each node has a different myid

Modify Zookeeper configuration file

cat > /usr/local/kafka/config/zookeeper.properties <<EOF
admin.enableServer=false
dataDir=/data/kafka/zookeeper/
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
server.1=192.168.10.7:2888:3888
server.2=192.168.10.8:2888:3888
server.3=192.168.10.9:2888:3888
EOF

server.1 corresponds to the server with myid content of 1
server.2 corresponds to the server with myid content of 2
server.3 corresponds to the server with myid content of 3

Modify Kafka memory limit to 4g

sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

Modify Kafka configuration file

cat > /usr/local/kafka/config/server.properties <<'EOF'
broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.10.7:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka-logs/
num.partitions=10
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.10.7:2181,192.168.10.8:2181,192.168.10.9:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
default.replication.factor=3
acks=all
min.insync.replicas=2
message.max.bytes=10485760
replica.fetch.max.bytes=10485760
EOF

Note: broker.id value is different for each node
Modify the listeners value to the actual local IP during deployment
Modify the zookeeper.connect value to the actual IPs of the three Zookeeper nodes during deployment

Create Kafka user
```
useradd -M -s /sbin/nologin kafka
```

Set permissions for Kafka related directories

chown -R kafka:kafka /usr/local/kafka /data/kafka

Configure systemd to manage Zookeeper

cat > /etc/systemd/system/zookeeper.service <<'EOF'
[Unit]
Description=Zookeeper
[Service]
User=kafka
Group=kafka
LimitNOFILE=102400
LimitNPROC=102400
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/kill  $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Configure systemd to manage Kafka

cat > /etc/systemd/system/kafka.service <<'EOF'
[Unit]
Description=Kafka
After=zookeeper.service
Requires=zookeeper.service
[Service]
User=kafka
Group=kafka
LimitNOFILE=102400
LimitNPROC=102400
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/kill  $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Start Zookeeper and Kafka and enable autostart on boot

systemctl start zookeeper
systemctl enable zookeeper
systemctl start kafka
systemctl enable kafka

Check service status

systemctl status zookeeper
systemctl status kafka

Kafka Node02

Download JDK installation package

Server with Internet Access
Server without Internet Access

wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

# JDK installation package download link, upload to the deployment server after downloading
https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

Extract the JDK to the installation directory

tar -zxvf OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz
mv jdk-21.0.8+9 /usr/local/openjdk-21

Configure Java symbolic link

ln -s /usr/local/openjdk-21/bin/java /bin/java

Download Kafka installation package

Server with Internet Access
Server without Internet Access

wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

# Kafka installation package download link, upload to the deployment server after downloading
https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

Extract Kafka to the installation directory

tar -zxvf kafka_2.13-3.9.1.tgz -C /usr/local
mv /usr/local/kafka_2.13-3.9.1/  /usr/local/kafka/

Create data directories

mkdir -p /data/kafka/zookeeper/ /data/kafka/kafka-logs/

Add Zookeeper myid
```
echo 2 > /data/kafka/zookeeper/myid
```
- Note: Each node has a different myid

Modify Zookeeper configuration file

cat > /usr/local/kafka/config/zookeeper.properties <<EOF
admin.enableServer=false
dataDir=/data/kafka/zookeeper/
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
server.1=192.168.10.7:2888:3888
server.2=192.168.10.8:2888:3888
server.3=192.168.10.9:2888:3888
EOF

server.1 corresponds to the server with myid content of 1
server.2 corresponds to the server with myid content of 2
server.3 corresponds to the server with myid content of 3

Modify Kafka memory limit to 4g

sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

Modify Kafka configuration file

cat > /usr/local/kafka/config/server.properties <<'EOF'
broker.id=1
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.10.8:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka-logs/
num.partitions=10
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.10.7:2181,192.168.10.8:2181,192.168.10.9:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
default.replication.factor=3
acks=all
min.insync.replicas=2
message.max.bytes=10485760
replica.fetch.max.bytes=10485760
EOF

Note: broker.id value is different for each node
Modify the listeners value to the actual local IP during deployment
Modify the zookeeper.connect value to the actual IPs of the three Zookeeper nodes during deployment

Create Kafka user
```
useradd -M -s /sbin/nologin kafka
```

Set permissions for Kafka related directories

chown -R kafka:kafka /usr/local/kafka /data/kafka

Configure systemd to manage Zookeeper

cat > /etc/systemd/system/zookeeper.service <<'EOF'
[Unit]
Description=Zookeeper
[Service]
User=kafka
Group=kafka
LimitNOFILE=102400
LimitNPROC=102400
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/kill  $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Configure systemd to manage Kafka

cat > /etc/systemd/system/kafka.service <<'EOF'
[Unit]
Description=Kafka
After=zookeeper.service
Requires=zookeeper.service
[Service]
User=kafka
Group=kafka
LimitNOFILE=102400
LimitNPROC=102400
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/kill  $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Start Zookeeper and Kafka and enable autostart on boot

systemctl start zookeeper
systemctl enable zookeeper
systemctl start kafka
systemctl enable kafka

Check service status

systemctl status zookeeper
systemctl status kafka

Kafka Node03

Download JDK installation package

Server with Internet Access
Server without Internet Access

wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

# JDK installation package download link, upload to the deployment server after downloading
https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

Extract the JDK to the installation directory

tar -zxvf OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz
mv jdk-21.0.8+9 /usr/local/openjdk-21

Configure Java symbolic link

ln -s /usr/local/openjdk-21/bin/java /bin/java

Download Kafka installation package

Server with Internet Access
Server without Internet Access

wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

# Kafka installation package download link, upload to the deployment server after downloading
https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

Extract Kafka to the installation directory

tar -zxvf kafka_2.13-3.9.1.tgz -C /usr/local
mv /usr/local/kafka_2.13-3.9.1/  /usr/local/kafka/

Create data directories

mkdir -p /data/kafka/zookeeper/ /data/kafka/kafka-logs/

Add Zookeeper myid
```
echo 3 > /data/kafka/zookeeper/myid
```
- Note: Each node has a different myid

Modify Zookeeper configuration file

cat > /usr/local/kafka/config/zookeeper.properties <<EOF
admin.enableServer=false
dataDir=/data/kafka/zookeeper/
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
server.1=192.168.10.7:2888:3888
server.2=192.168.10.8:2888:3888
server.3=192.168.10.9:2888:3888
EOF

server.1 corresponds to the server with myid content of 1
server.2 corresponds to the server with myid content of 2
server.3 corresponds to the server with myid content of 3

Modify Kafka memory limit to 4g

sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

Modify Kafka configuration file

cat > /usr/local/kafka/config/server.properties <<'EOF'
broker.id=2
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.10.9:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka-logs/
num.partitions=10
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.10.7:2181,192.168.10.8:2181,192.168.10.9:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
default.replication.factor=3
acks=all
min.insync.replicas=2
message.max.bytes=10485760
replica.fetch.max.bytes=10485760
EOF

Note: broker.id value is different for each node
Modify the listeners value to the actual local IP during deployment
Modify the zookeeper.connect value to the actual IPs of the three Zookeeper nodes during deployment

Create Kafka user
```
useradd -M -s /sbin/nologin kafka
```

Set permissions for Kafka related directories

chown -R kafka:kafka /usr/local/kafka /data/kafka

Configure systemd to manage Zookeeper

cat > /etc/systemd/system/zookeeper.service <<'EOF'
[Unit]
Description=Zookeeper
[Service]
User=kafka
Group=kafka
LimitNOFILE=102400
LimitNPROC=102400
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/kill  $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Configure systemd to manage Kafka

cat > /etc/systemd/system/kafka.service <<'EOF'
[Unit]
Description=Kafka
After=zookeeper.service
Requires=zookeeper.service
[Service]
User=kafka
Group=kafka
LimitNOFILE=102400
LimitNPROC=102400
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/kill  $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Start Zookeeper and Kafka and enable autostart on boot

systemctl start zookeeper
systemctl enable zookeeper
systemctl start kafka
systemctl enable kafka

Check service status

systemctl status zookeeper
systemctl status kafka

Kafka Cluster Verification

Start producer on Kafka Node01

/usr/local/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic test01

After starting, enter message content randomly in the terminal

Check messages on Kafka Node02/3

/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic test01

/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic test01

If the output includes message content from the producer, everything is functioning correctly.

Kafka Node01​

Kafka Node02​

Kafka Node03​

Kafka Cluster Verification​

Kafka Node01

Kafka Node02

Kafka Node03

Kafka Cluster Verification