Skip to main content

Kafka Single Node

Server IPHost Role
192.168.10.7Kafka Server

Kafka Server​

  1. Download the JDK package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK8U-jdk_x64_linux_hotspot_8u292b10.tar.gz

  2. Extract JDK to the installation directory

    tar -zxvf OpenJDK8U-jdk_x64_linux_hotspot_8u292b10.tar.gz
    mv jdk8u292-b10/ /usr/local/openjdk-8

  3. Configure the Java symbolic link

    ln -s /usr/local/openjdk-8/bin/java /bin/java

  4. Download the Kafka package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

  5. Extract Kafka to the installation directory

    tar -zxvf kafka_2.13-3.9.1.tgz -C /usr/local
    mv /usr/local/kafka_2.13-3.9.1/ /usr/local/kafka/

  6. Create data directories

    mkdir -p /data/kafka/zookeeper/ /data/kafka/kafka-logs/

  7. Modify Zookeeper configuration file

    cat > /usr/local/kafka/config/zookeeper.properties <<EOF
    admin.enableServer=false
    dataDir=/data/kafka/zookeeper/
    clientPort=2181
    maxClientCnxns=0
    EOF

  8. Modify Kafka memory limit to 4G

    sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
    sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

  9. Modify Kafka configuration file

    cat > /usr/local/kafka/config/server.properties <<EOF
    broker.id=0
    listeners=PLAINTEXT://0.0.0.0:9092
    advertised.listeners=PLAINTEXT://192.168.10.7:9092
    num.network.threads=3
    num.io.threads=8
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    log.dirs=/data/kafka/kafka-logs/
    num.partitions=10
    num.recovery.threads.per.data.dir=1
    offsets.topic.replication.factor=1
    transaction.state.log.replication.factor=1
    transaction.state.log.min.isr=1
    log.retention.hours=168
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    zookeeper.connect=127.0.0.1:2181
    zookeeper.connection.timeout.ms=6000
    group.initial.rebalance.delay.ms=0
    message.max.bytes=10485760
    replica.fetch.max.bytes=10485760
    EOF
    • Modify the value of listeners to the actual local IP during deployment

  10. Create a Kafka user

    useradd -M -s /sbin/nologin kafka

  11. Grant permissions for Kafka related directories

    chown -R kafka:kafka /usr/local/kafka /data/kafka

  12. Configure systemd to manage Zookeeper

    cat > /etc/systemd/system/zookeeper.service <<EOF
    [Unit]
    Description=Zookeeper
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
    ExecStop=/usr/bin/kill  \$MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

  13. Configure systemd to manage Kafka

    cat > /etc/systemd/system/kafka.service <<EOF
    [Unit]
    Description=Kafka
    After=zookeeper.service
    Requires=zookeeper.service
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
    ExecStop=/usr/bin/kill  \$MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

  14. Start Zookeeper and Kafka and enable auto start on boot

    systemctl start zookeeper
    systemctl enable zookeeper
    systemctl start kafka
    systemctl enable kafka