Skip to main content

Kafka Single-Node Deployment

Server IPHost Role
192.168.10.7Kafka Broker and Controller

Kafka Server​

  1. Download the JDK installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

  2. Extract the JDK and configure the Java symbolic link

    tar -zxvf OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz -C /usr/local
    mv /usr/local/jdk-21.0.8+9 /usr/local/openjdk-21
    ln -s /usr/local/openjdk-21/bin/java /bin/java

  3. Download the Kafka installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-4.3.1.tgz

  4. Extract Kafka and create the data directory

    tar -zxvf kafka_2.13-4.3.1.tgz -C /usr/local
    mv /usr/local/kafka_2.13-4.3.1 /usr/local/kafka
    mkdir -p /data/kafka/kafka-logs

  5. Set the Kafka JVM heap size to 4 GB

    sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
    sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

  6. Configure Kafka

    cat > /usr/local/kafka/config/server.properties <<'EOF'
    process.roles=broker,controller
    node.id=1
    controller.quorum.bootstrap.servers=192.168.10.7:9093
    listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
    advertised.listeners=PLAINTEXT://192.168.10.7:9092
    listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
    inter.broker.listener.name=PLAINTEXT
    controller.listener.names=CONTROLLER
    log.dirs=/data/kafka/kafka-logs
    num.partitions=10
    offsets.topic.replication.factor=1
    transaction.state.log.replication.factor=1
    transaction.state.log.min.isr=1
    default.replication.factor=1
    min.insync.replicas=1
    log.retention.hours=168
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    group.initial.rebalance.delay.ms=3000
    message.max.bytes=10485760
    replica.fetch.max.bytes=10485760
    EOF

    Confirm the node IP address and ports for your environment, and update the following properties as needed:

    • controller.quorum.bootstrap.servers: Enter the local IP address and Controller port 9093 in the format LOCAL_IP:9093. Do not also configure the deprecated controller.quorum.voters property.
    • advertised.listeners: Enter the local IP address and Broker port 9092 that applications use to connect to Kafka.

  7. Generate the cluster ID and format the storage directory

    KAFKA_CLUSTER_ID=$(/usr/local/kafka/bin/kafka-storage.sh random-uuid)
    /usr/local/kafka/bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c /usr/local/kafka/config/server.properties
    • Run this command only when initializing an empty data directory for the first time. Do not format a node that already contains data.

  8. Create the Kafka user and grant directory permissions

    useradd -M -s /sbin/nologin kafka
    chown -R kafka:kafka /usr/local/kafka /data/kafka

  9. Configure systemd to manage Kafka

    cat > /etc/systemd/system/kafka.service <<'EOF'
    [Unit]
    Description=Kafka KRaft
    After=network.target
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
    ExecStop=/usr/bin/kill $MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

  10. Start Kafka and enable it at boot

    systemctl daemon-reload
    systemctl enable kafka
    systemctl start kafka
    systemctl status kafka

  11. Verify message production and consumption

    Create a test Topic

    /usr/local/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic test01 --partitions 1 --replication-factor 1

    Start a producer

    Run the following command in the current terminal. Enter a test message and press Enter to send it.

    /usr/local/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic test01

    Start a consumer

    Open another terminal and run the following command. If the consumer displays the message sent by the producer, Kafka can produce and consume messages correctly.

    /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test01 --from-beginning