Skip to main content

etcd Cluster

Server IPHost Role
192.168.10.31etcd Node01
192.168.10.32etcd Node02
192.168.10.33etcd Node03

etcd Node01​

  1. Download the etcd installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/etcd-v3.6.11-linux-amd64.tar.gz

  2. Create the etcd runtime user

    useradd -M -s /sbin/nologin etcd

  3. Extract the etcd package and install the binaries

    mkdir -p /usr/local/etcd/bin
    tar -zxf etcd-v3.6.11-linux-amd64.tar.gz
    cp etcd-v3.6.11-linux-amd64/etcd /usr/local/etcd/bin/
    cp etcd-v3.6.11-linux-amd64/etcdctl /usr/local/etcd/bin/
    cp etcd-v3.6.11-linux-amd64/etcdutl /usr/local/etcd/bin/
    chmod +x /usr/local/etcd/bin/etcd*
    chown -R etcd:etcd /usr/local/etcd

  4. Create data and log directories

    mkdir -p /data/etcd
    mkdir -p /data/logs/etcd
    chown -R etcd:etcd /data/etcd /data/logs/etcd

  5. Modify the etcd configuration file

    cat > /usr/local/etcd/etcd.conf.yml <<'EOF'
    name: etcd-01
    data-dir: /data/etcd
    listen-client-urls: http://192.168.10.31:2379,http://127.0.0.1:2379
    advertise-client-urls: http://192.168.10.31:2379
    listen-peer-urls: http://192.168.10.31:2380
    initial-advertise-peer-urls: http://192.168.10.31:2380
    listen-metrics-urls: http://192.168.10.31:2381,http://127.0.0.1:2381
    initial-cluster: etcd-01=http://192.168.10.31:2380,etcd-02=http://192.168.10.32:2380,etcd-03=http://192.168.10.33:2380
    initial-cluster-token: hap-milvus-etcd
    initial-cluster-state: new
    auto-compaction-mode: periodic
    auto-compaction-retention: 10h
    quota-backend-bytes: 8589934592
    snapshot-count: 100000
    max-snapshots: 5
    max-wals: 5
    strict-reconfig-check: true
    pre-vote: true
    enable-v2: false
    metrics: basic
    logger: zap
    log-level: info
    log-outputs:
      - /data/logs/etcd/etcd.log
    enable-log-rotation: true
    log-rotation-config-json: '{"maxsize":100,"maxage":180,"maxbackups":0,"localtime":true,"compress":true}'
    EOF
    • Replace the IP addresses in listen-client-urls and advertise-client-urls with the actual deployment server IP.

  6. Configure systemd to manage etcd-01

    cat > /etc/systemd/system/etcd-01.service <<'EOF'
    [Unit]
    Description=etcd member etcd-01
    Documentation=https://etcd.io/docs/
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=simple
    User=etcd
    Group=etcd
    ExecStart=/usr/local/etcd/bin/etcd --config-file=/usr/local/etcd/etcd.conf.yml
    Restart=on-failure
    RestartSec=5s
    LimitNOFILE=65536
    LimitNPROC=65536
    OOMScoreAdjust=-999
    [Install]
    WantedBy=multi-user.target
    EOF

  7. Start etcd-01 and enable startup on boot

    systemctl daemon-reload
    systemctl start etcd-01
    systemctl enable etcd-01

  8. Check Service Status

    systemctl status etcd-01 --no-pager

  9. View Runtime Logs

    journalctl -u etcd-01 -n 100 -l --no-pager

etcd Node02​

  1. Download the etcd installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/etcd-v3.6.11-linux-amd64.tar.gz

  2. Create the etcd runtime user

    useradd -M -s /sbin/nologin etcd

  3. Extract the etcd package and install the binaries

    mkdir -p /usr/local/etcd/bin
    tar -zxf etcd-v3.6.11-linux-amd64.tar.gz
    cp etcd-v3.6.11-linux-amd64/etcd /usr/local/etcd/bin/
    cp etcd-v3.6.11-linux-amd64/etcdctl /usr/local/etcd/bin/
    cp etcd-v3.6.11-linux-amd64/etcdutl /usr/local/etcd/bin/
    chmod +x /usr/local/etcd/bin/etcd*
    chown -R etcd:etcd /usr/local/etcd
    • Installation verification
    /usr/local/etcd/bin/etcd --version
    /usr/local/etcd/bin/etcdctl version
    /usr/local/etcd/bin/etcdutl version

  4. Create data and log directories

    mkdir -p /data/etcd
    mkdir -p /data/logs/etcd
    chown -R etcd:etcd /data/etcd /data/logs/etcd

  5. Modify the etcd configuration file

    cat > /usr/local/etcd/etcd.conf.yml <<'EOF'
    name: etcd-02
    data-dir: /data/etcd
    listen-client-urls: http://192.168.10.32:2379,http://127.0.0.1:2379
    advertise-client-urls: http://192.168.10.32:2379
    listen-peer-urls: http://192.168.10.32:2380
    initial-advertise-peer-urls: http://192.168.10.32:2380
    listen-metrics-urls: http://192.168.10.32:2381,http://127.0.0.1:2381
    initial-cluster: etcd-01=http://192.168.10.31:2380,etcd-02=http://192.168.10.32:2380,etcd-03=http://192.168.10.33:2380
    initial-cluster-token: hap-milvus-etcd
    initial-cluster-state: new
    auto-compaction-mode: periodic
    auto-compaction-retention: 10h
    quota-backend-bytes: 8589934592
    snapshot-count: 100000
    max-snapshots: 5
    max-wals: 5
    strict-reconfig-check: true
    pre-vote: true
    enable-v2: false
    metrics: basic
    logger: zap
    log-level: info
    log-outputs:
      - /data/logs/etcd/etcd.log
    enable-log-rotation: true
    log-rotation-config-json: '{"maxsize":100,"maxage":180,"maxbackups":0,"localtime":true,"compress":true}'
    EOF
    • Replace the IP addresses in listen-client-urls and advertise-client-urls with the actual deployment server IP.

  6. Configure systemd to manage etcd-02

    cat > /etc/systemd/system/etcd-02.service <<'EOF'
    [Unit]
    Description=etcd member etcd-02
    Documentation=https://etcd.io/docs/
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=simple
    User=etcd
    Group=etcd
    ExecStart=/usr/local/etcd/bin/etcd --config-file=/usr/local/etcd/etcd.conf.yml
    Restart=on-failure
    RestartSec=5s
    LimitNOFILE=65536
    LimitNPROC=65536
    OOMScoreAdjust=-999
    [Install]
    WantedBy=multi-user.target
    EOF

  7. Start etcd-02 and enable startup on boot

    systemctl daemon-reload
    systemctl start etcd-02
    systemctl enable etcd-02

  8. Check Service Status

    systemctl status etcd-02 --no-pager

  9. View Runtime Logs

    journalctl -u etcd-02 -n 100 -l --no-pager

etcd Node03​

  1. Download the etcd installation package

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/etcd-v3.6.11-linux-amd64.tar.gz

  2. Create the etcd runtime user

    useradd -M -s /sbin/nologin etcd

  3. Extract the etcd package and install the binaries

    mkdir -p /usr/local/etcd/bin
    tar -zxf etcd-v3.6.11-linux-amd64.tar.gz
    cp etcd-v3.6.11-linux-amd64/etcd /usr/local/etcd/bin/
    cp etcd-v3.6.11-linux-amd64/etcdctl /usr/local/etcd/bin/
    cp etcd-v3.6.11-linux-amd64/etcdutl /usr/local/etcd/bin/
    chmod +x /usr/local/etcd/bin/etcd*
    chown -R etcd:etcd /usr/local/etcd
    • Installation verification
    /usr/local/etcd/bin/etcd --version
    /usr/local/etcd/bin/etcdctl version
    /usr/local/etcd/bin/etcdutl version

  4. Create data and log directories

    mkdir -p /data/etcd
    mkdir -p /data/logs/etcd
    chown -R etcd:etcd /data/etcd /data/logs/etcd

  5. Modify the etcd configuration file

    cat > /usr/local/etcd/etcd.conf.yml <<'EOF'
    name: etcd-03
    data-dir: /data/etcd
    listen-client-urls: http://192.168.10.33:2379,http://127.0.0.1:2379
    advertise-client-urls: http://192.168.10.33:2379
    listen-peer-urls: http://192.168.10.33:2380
    initial-advertise-peer-urls: http://192.168.10.33:2380
    listen-metrics-urls: http://192.168.10.33:2381,http://127.0.0.1:2381
    initial-cluster: etcd-01=http://192.168.10.31:2380,etcd-02=http://192.168.10.32:2380,etcd-03=http://192.168.10.33:2380
    initial-cluster-token: hap-milvus-etcd
    initial-cluster-state: new
    auto-compaction-mode: periodic
    auto-compaction-retention: 10h
    quota-backend-bytes: 8589934592
    snapshot-count: 100000
    max-snapshots: 5
    max-wals: 5
    strict-reconfig-check: true
    pre-vote: true
    enable-v2: false
    metrics: basic
    logger: zap
    log-level: info
    log-outputs:
      - /data/logs/etcd/etcd.log
    enable-log-rotation: true
    log-rotation-config-json: '{"maxsize":100,"maxage":180,"maxbackups":0,"localtime":true,"compress":true}'
    EOF
    • Replace the IP addresses in listen-client-urls and advertise-client-urls with the actual deployment server IP.

  6. Configure systemd to manage etcd-03

    cat > /etc/systemd/system/etcd-03.service <<'EOF'
    [Unit]
    Description=etcd member etcd-03
    Documentation=https://etcd.io/docs/
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=simple
    User=etcd
    Group=etcd
    ExecStart=/usr/local/etcd/bin/etcd --config-file=/usr/local/etcd/etcd.conf.yml
    Restart=on-failure
    RestartSec=5s
    LimitNOFILE=65536
    LimitNPROC=65536
    OOMScoreAdjust=-999
    [Install]
    WantedBy=multi-user.target
    EOF

  7. Start etcd-03 and enable startup on boot

    systemctl daemon-reload
    systemctl start etcd-03
    systemctl enable etcd-03

  8. Check Service Status

    systemctl status etcd-03 --no-pager

  9. View Runtime Logs

    journalctl -u etcd-03 -n 100 -l --no-pager

Verify the etcd Cluster​

  1. Set cluster endpoints

    export ETCD_ENDPOINTS=http://192.168.10.31:2379,http://192.168.10.32:2379,http://192.168.10.33:2379

  2. Check cluster health

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} endpoint health -w table

  3. Check cluster status

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} endpoint status -w table

  4. View the member list

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} member list -w table

  5. Run write, read, and delete tests

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} put /etcd-test/hello world
    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} get /etcd-test/hello
    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} del /etcd-test/hello

Enable Authentication​

Run the following commands on any node.

  1. Set variables

    export ETCD_ENDPOINTS=http://192.168.10.31:2379,http://192.168.10.32:2379,http://192.168.10.33:2379
    export ETCD_ROOT_PASSWORD='replace_with_strong_root_password'
    • ETCD_ENDPOINTS specifies the three access endpoints of the cluster. All subsequent authentication-related commands use it.
    • ETCD_ROOT_PASSWORD specifies the root user password. Replace it with a strong password in production deployments.

  2. Confirm that the root password variable is set

    test -n "$ETCD_ROOT_PASSWORD" && echo "ETCD_ROOT_PASSWORD is set" || echo "ETCD_ROOT_PASSWORD is not set"
    • Check whether the root password variable is set to avoid using an empty password in later commands.

  3. Create the root user

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} user add "root:${ETCD_ROOT_PASSWORD}"
    • user add creates the root user and sets its password.

  4. Grant the root role

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} user grant-role root root
    • user grant-role root root grants the root role to the root user so that it has administrator privileges.

  5. View root user information

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} user get root
    • user get root displays information about the root user and confirms that the user was created successfully.

  6. Enable authentication

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} auth enable
    • auth enable enables etcd authentication. After it is enabled, client access must include user credentials.

  7. Verify authentication

    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} --user="root:${ETCD_ROOT_PASSWORD}" auth status
    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} --user="root:${ETCD_ROOT_PASSWORD}" endpoint health -w table
    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} --user="root:${ETCD_ROOT_PASSWORD}" endpoint status -w table
    /usr/local/etcd/bin/etcdctl --endpoints=${ETCD_ENDPOINTS} --user="root:${ETCD_ROOT_PASSWORD}" member list -w table
    • auth status shows the current authentication status and confirms whether authentication is enabled.
    • endpoint health -w table checks endpoint health and confirms that the service remains accessible after authentication is enabled.
    • endpoint status -w table shows detailed endpoint status and confirms that the cluster is running normally.
    • member list -w table shows the member list and confirms that cluster member information is complete.