跳到主要内容

Kafka 单节点

服务器IP主机角色
192.168.10.7Kafka Server

Kafka Server​

  1. 下载 jdk 安装包

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz

  2. 解压 jdk 到安装目录

    tar -zxvf OpenJDK21U-jdk_x64_linux_hotspot_21.0.8_9.tar.gz
    mv jdk-21.0.8+9 /usr/local/openjdk-21

  3. 配置 java 软链接

    ln -s /usr/local/openjdk-21/bin/java /bin/java

  4. 下载 kafka 安装包

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/kafka_2.13-3.9.1.tgz

  5. 解压 kafka 到安装目录

    tar -zxvf kafka_2.13-3.9.1.tgz -C /usr/local
    mv /usr/local/kafka_2.13-3.9.1/  /usr/local/kafka/

  6. 创建数据目录

    mkdir -p /data/kafka/zookeeper/ /data/kafka/kafka-logs/

  7. 修改 zookeeper 配置文件

    cat > /usr/local/kafka/config/zookeeper.properties <<EOF
    admin.enableServer=false
    dataDir=/data/kafka/zookeeper/
    clientPort=2181
    maxClientCnxns=0
    EOF

  8. 修改 kafka 内存限制为4g

    sed -i ':a;N;$!ba;s/Xm[xs]1G/Xmx4G/1' /usr/local/kafka/bin/kafka-server-start.sh
    sed -i ':a;N;$!ba;s/Xm[xs]1G/Xms4G/1' /usr/local/kafka/bin/kafka-server-start.sh

  9. 修改 kafka 配置文件

    cat > /usr/local/kafka/config/server.properties <<'EOF'
    broker.id=0
    listeners=PLAINTEXT://0.0.0.0:9092
    advertised.listeners=PLAINTEXT://192.168.10.7:9092
    num.network.threads=3
    num.io.threads=8
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    log.dirs=/data/kafka/kafka-logs/
    num.partitions=10
    num.recovery.threads.per.data.dir=1
    offsets.topic.replication.factor=1
    transaction.state.log.replication.factor=1
    transaction.state.log.min.isr=1
    log.retention.hours=168
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    zookeeper.connect=127.0.0.1:2181
    zookeeper.connection.timeout.ms=6000
    group.initial.rebalance.delay.ms=0
    message.max.bytes=10485760
    replica.fetch.max.bytes=10485760
    EOF
    • listeners 值在部署时修改为实际的本机IP

  10. 创建 kafka 用户

    useradd -M -s /sbin/nologin kafka

  11. 授权 kafka 相关目录权限

    chown -R kafka:kafka /usr/local/kafka /data/kafka

  12. 配置 systemd 管理 zookeeper

    cat > /etc/systemd/system/zookeeper.service <<'EOF'
    [Unit]
    Description=Zookeeper
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
    ExecStop=/usr/bin/kill  $MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

  13. 配置 systemd 管理 kafka

    cat > /etc/systemd/system/kafka.service <<'EOF'
    [Unit]
    Description=Kafka
    After=zookeeper.service
    Requires=zookeeper.service
    [Service]
    User=kafka
    Group=kafka
    LimitNOFILE=102400
    LimitNPROC=102400
    ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
    ExecStop=/usr/bin/kill  $MAINPID
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

  14. 启动 zookeeper 与 kafka 并加入开机自启动

    systemctl start zookeeper
    systemctl enable zookeeper
    systemctl start kafka
    systemctl enable kafka

  15. 服务状态检查

    systemctl status zookeeper
    systemctl status kafka