Kafka Single Node

Server IP	Host Role
192.168.10.7	Kafka Server

Kafka Server

1. Download JDK installation package

   Server with Internet Access

   wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

   Server without Internet Access

   # JDK package download link. Download and upload to the deployment server.
   https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

2. Extract the JDK to the installation directory

   tar -zxvf OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz
   mv jdk-21.0.8+9 /usr/local/openjdk-21

3. Configure Java symbolic link

   ln -s /usr/local/openjdk-21/bin/java /bin/java

4. Download Kafka installation package

   Server with Internet Access

   wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

   Server without Internet Access

   # Kafka package download link. Download and upload to the deployment server.
   https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

5. Extract Kafka to the installation directory

   tar -zxvf kafka_2.13-3.9.1.tgz -C /usr/local
   mv /usr/local/kafka_2.13-3.9.1/ /usr/local/kafka/

6. Create data directories

   mkdir -p /data/kafka/zookeeper/ /data/kafka/kafka-logs/

7. Modify Zookeeper configuration file

   cat > /usr/local/kafka/config/zookeeper.properties <<EOF
   admin.enableServer=false
   dataDir=/data/kafka/zookeeper/
   clientPort=2181
   maxClientCnxns=0
   EOF

8. Modify Kafka memory limit to 4G

   sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
   sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

9. Modify Kafka configuration file

   cat > /usr/local/kafka/config/server.properties <<'EOF'
   broker.id=0
   listeners=PLAINTEXT://0.0.0.0:9092
   advertised.listeners=PLAINTEXT://192.168.10.7:9092
   num.network.threads=3
   num.io.threads=8
   socket.send.buffer.bytes=102400
   socket.receive.buffer.bytes=102400
   socket.request.max.bytes=104857600
   log.dirs=/data/kafka/kafka-logs/
   num.partitions=10
   num.recovery.threads.per.data.dir=1
   offsets.topic.replication.factor=1
   transaction.state.log.replication.factor=1
   transaction.state.log.min.isr=1
   log.retention.hours=168
   log.segment.bytes=1073741824
   log.retention.check.interval.ms=300000
   zookeeper.connect=127.0.0.1:2181
   zookeeper.connection.timeout.ms=6000
   group.initial.rebalance.delay.ms=0
   message.max.bytes=10485760
   replica.fetch.max.bytes=10485760
   EOF

   • Modify the value of listeners to the actual local IP during deployment.

10. Create Kafka user

    useradd -M -s /sbin/nologin kafka

11. Grant permissions to Kafka-related directories

    chown -R kafka:kafka /usr/local/kafka /data/kafka

12. Configure systemd to manage Zookeeper

    cat > /etc/systemd/system/zookeeper.service <<'EOF'
    [Unit]
    Description=Zookeeper
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
    ExecStop=/usr/bin/kill $MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

13. Configure systemd to manage Kafka

    cat > /etc/systemd/system/kafka.service <<'EOF'
    [Unit]
    Description=Kafka
    After=zookeeper.service
    Requires=zookeeper.service
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
    ExecStop=/usr/bin/kill $MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

14. Start Zookeeper and Kafka and enable startup on boot

    systemctl start zookeeper
    systemctl enable zookeeper
    systemctl start kafka
    systemctl enable kafka

15. Check service status

    systemctl status zookeeper
    systemctl status kafka