跳到主要内容

单Master集群部署

信息
  • 最低节点要求:1 或 2 台节点。
  • Master 节点一旦宕机,kubectl 等管理工具将无法操作集群,但运行中的业务容器不受影响。
  • 适用于精简版集群,通常在两台微服务节点的环境中使用。

本文档基于 CentOS 7.9 / Debian 12 操作系统进行 Kubernetes 集群部署

服务器IP主机角色
192.168.10.20Kubernetes 01(Master、Node)
192.168.10.21Kubernetes 02(Node)

服务器要求

  • 集群服务器之间网络策略无限制
  • 各节点主机名须唯一,且集群部署完成后不可更改
  • 主网卡 MAC 地址不能重复(执行 ip link 查看)
  • product_uuid 不能重复(执行 cat /sys/class/dmi/id/product_uuid 查看)
  • 6443 端口未被占用(执行 nc -vz 127.0.0.1 6443 验证)
  • 必须禁用 swap 内存(执行 swapoff -a 临时禁用,并注释 /etc/fstab 中 swap 分区的挂载项)

安装CRI容器运行环境

Kubernetes 集群各节点均需要操作

  1. 下载 Kubernetes 集群安装包

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/kubernetes-1.35.3/1.35-k8s-amd64-pkg.tar.gz
  2. 解压安装包

    tar xzvf 1.35-k8s-amd64-pkg.tar.gz
    • 检查安装包内文件

      ls -l 1.35-k8s-amd64-pkg
      输出结果示例
      -rw-r--r-- 1 root 197121 331360 Apr 14 16:38 calico.yaml
      -rw-r--r-- 1 root 197121 33899693 Apr 13 12:06 containerd-static-2.2.2-linux-amd64.tar.gz
      -rw-r--r-- 1 root 197121 19185064 Apr 13 13:18 crictl-v1.35.0-linux-amd64.tar.gz
      -rw-r--r-- 1 root 197121 28576212 Apr 13 13:22 istio-1.29.1-linux-amd64.tar.gz
      -rw-r--r-- 1 root 197121 72372408 Apr 13 13:20 kubeadm
      -rw-r--r-- 1 root 197121 58601656 Apr 13 13:18 kubectl
      -rw-r--r-- 1 root 197121 58110244 Apr 13 13:20 kubelet
      -rw-r--r-- 1 root 197121 11373900 Apr 13 13:17 nerdctl-2.2.2-linux-amd64.tar.gz
      -rw-r--r-- 1 root 197121 12741088 Apr 13 12:12 runc.amd64
  3. 安装 containerd

    需在安装包目录 1.35-k8s-amd64-pkg 下执行

    cd 1.35-k8s-amd64-pkg
    tar -zxvf containerd-static-2.2.2-linux-amd64.tar.gz
    mv -f bin/* /usr/local/bin/
  4. 创建 containerd 配置目录

    mkdir /etc/containerd
  5. 安装 runc

    需在安装包目录 1.35-k8s-amd64-pkg 下执行

    mv runc.amd64 /usr/local/bin/runc
    chmod +x /usr/local/bin/runc
    runc -v
  6. 生成 containerd 配置文件并修改相关参数

    containerd config default > /etc/containerd/config.toml

    sed -i \
    -e 's|SystemdCgroup =.*|SystemdCgroup = true|g' \
    -e 's|bin_dirs =.*|bin_dirs = ["/usr/local/kubernetes/cni/bin"]|' \
    -e 's|sandbox =.*|sandbox = "127.0.0.1:5000/pause:3.10.1"|' \
    -e 's|^root =.*|root = "/data/containerd"|' \
    /etc/containerd/config.toml
    • 验证配置是否生效

      grep "SystemdCgroup\|bin_dirs\|sandbox =\|^root =" /etc/containerd/config.toml
      输出结果示例
      root = "/data/containerd"
      sandbox_image = "127.0.0.1:5000/pause:3.10.1"
      bin_dir = "/usr/local/kubernetes/cni/bin"
      SystemdCgroup = true
  7. 配置 containerd systemd 服务文件

    cat > /etc/systemd/system/containerd.service <<EOF
    [Unit]
    Description=containerd
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=notify
    ExecStart=/usr/local/bin/containerd --config /etc/containerd/config.toml
    LimitNOFILE=1024000
    LimitNPROC=infinity
    LimitCORE=0
    TimeoutStartSec=0
    Delegate=yes
    KillMode=process
    Restart=on-failure
    StartLimitBurst=3
    StartLimitInterval=60s
    [Install]
    WantedBy=multi-user.target
    EOF
  8. 启动 containerd 并设置开机自启

    systemctl daemon-reload && systemctl restart containerd && systemctl enable containerd

安装 K8S 集群所需命令

安装 crictl / kubeadm / kubelet / kubectl,Kubernetes 集群各节点均需要操作

  1. 创建命令安装目录

    mkdir -p /usr/local/kubernetes/bin
  2. 将命令文件移至安装目录

    需在安装包目录 1.35-k8s-amd64-pkg 下执行

    tar -zxvf crictl-v1.35.0-linux-amd64.tar.gz -C /usr/local/kubernetes/bin

    cp ./{kubeadm,kubelet,kubectl} /usr/local/kubernetes/bin/
  3. 赋予命令文件可执行权限

    chmod +x /usr/local/kubernetes/bin/*
    chown $(whoami):$(groups) /usr/local/kubernetes/bin/*
  4. 配置 systemd 管理 kubelet

    cat > /etc/systemd/system/kubelet.service <<\EOF
    [Unit]
    Description=kubelet: The Kubernetes Node Agent
    Documentation=https://kubernetes.io/docs/home/
    Wants=network-online.target
    After=network-online.target

    [Service]
    ExecStart=/usr/local/kubernetes/bin/kubelet
    Restart=always
    StartLimitInterval=0
    RestartSec=10

    [Install]
    WantedBy=multi-user.target
    EOF
  5. 配置 kubelet 的 kubeadm 扩展参数

    mkdir -p /etc/systemd/system/kubelet.service.d

    cat > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf <<\EOF
    # Note: This dropin only works with kubeadm and kubelet v1.11+
    [Service]
    Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
    Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
    # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
    EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
    # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
    # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
    EnvironmentFile=-/etc/default/kubelet
    ExecStart=
    ExecStart=/usr/local/kubernetes/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
    EOF
  6. 启动 kubelet 并设置开机自启

    systemctl daemon-reload && systemctl restart kubelet && systemctl enable kubelet
    • kubelet 在 kubeadm init / kubeadm join 完成前处于等待状态,restart 后无需检查服务状态,集群初始化完成后会自动拉起
  7. 配置 K8S 命令路径至环境变量

    cat > /etc/profile.d/kubernetes.sh <<'EOF'
    export PATH=/usr/local/kubernetes/bin/:$PATH
    EOF
    source /etc/profile.d/kubernetes.sh
  8. 配置 crictl 运行时端点

    crictl config runtime-endpoint unix:///run/containerd/containerd.sock
    • 验证配置

      crictl config --get runtime-endpoint

安装 nerdctl 工具(可选)

nerdctl 是 containerd 的命令行工具,作用是用兼容 Docker 的语法,方便运维管理容器、镜像、等资源。

  1. 解压并安装 nerdctl

    需在安装包目录 1.35-k8s-amd64-pkg 下执行

    tar -zxvf nerdctl-2.2.2-linux-amd64.tar.gz
    rm -f containerd-rootless*.sh
    mv nerdctl /usr/local/kubernetes/bin/
  2. 配置命令别名并生效

    echo 'alias nerdctl="nerdctl -n k8s.io"' >> ~/.bashrc

    source ~/.bashrc
    • 执行 nerdctl -v,输出 nerdctl version 2.2.2 代表安装成功

安装环境依赖

Kubernetes 集群各节点均需要操作

  1. 安装 socat / conntrack

    # CentOS / RedHat
    yum install -y socat conntrack-tools

    # Debian / Ubuntu
    apt install -y socat conntrack
  2. 验证所有命令是否就绪

    crictl --version && containerd --version && runc -v|grep version && kubeadm version && kubelet --version && kubectl version --client=true && socat -V | grep 'socat version' && conntrack --version && echo ok || echo error
    • 最终输出 ok 代表正常,输出 error 则需根据错误信息补全缺失的命令

修改内核配置

Kubernetes 集群各节点均需要操作

  1. 持久化加载所需内核模块

    cat > /etc/modules-load.d/kubernetes.conf <<EOF
    overlay
    br_netfilter
    ip_vs
    ip_vs_rr
    ip_vs_wrr
    ip_vs_sh
    EOF
  2. 立即加载内核模块

    modprobe overlay
    modprobe br_netfilter
    modprobe ip_vs
    modprobe ip_vs_rr
    modprobe ip_vs_wrr
    modprobe ip_vs_sh
  3. 追加内核参数并生效

    cp -rfp /etc/sysctl.d/99-sysctl.conf /etc/sysctl.d/99-sysctl.conf.backup-$(date +%Y%m%d%H%M%S)
    cat >> /etc/sysctl.d/99-sysctl.conf <<EOF
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward = 1
    vm.max_map_count = 262144

    # MD Config
    net.nf_conntrack_max = 524288
    net.ipv4.tcp_max_tw_buckets = 5000
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_rmem = 8192 87380 16777216
    net.ipv4.tcp_wmem = 8192 65536 16777216
    net.ipv4.tcp_max_syn_backlog = 32768
    net.core.netdev_max_backlog = 32768
    net.core.netdev_budget = 600
    net.core.somaxconn = 32768
    net.core.wmem_default = 8388608
    net.core.rmem_default = 8388608
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_synack_retries = 2
    net.ipv4.tcp_syn_retries = 2
    net.ipv4.tcp_tw_recycle = 0
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_fin_timeout = 2
    net.ipv4.tcp_mem = 8388608 12582912 16777216
    net.ipv4.ip_local_port_range = 1024 65000
    net.ipv4.tcp_max_orphans = 16384
    net.ipv4.tcp_keepalive_intvl = 10
    net.ipv4.tcp_keepalive_probes = 3
    net.ipv4.tcp_keepalive_time = 600
    vm.max_map_count = 262144
    net.netfilter.nf_conntrack_tcp_be_liberal = 0
    net.netfilter.nf_conntrack_tcp_max_retrans = 3
    net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
    net.netfilter.nf_conntrack_tcp_timeout_established = 86400
    fs.inotify.max_user_watches=10485760
    fs.inotify.max_user_instances=10240
    EOF

    sysctl --system

K8S 环境镜像准备

Kubernetes 集群各节点均需要操作

  1. 下载并导入离线镜像

    wget https://pdpublic.mingdao.com/private-deployment/offline/common/kubernetes-1.35.3/kubeadm-1.35.3-images-amd64.tar.gz
    • 解压并导入镜像

      gunzip -d kubeadm-1.35.3-images-amd64.tar.gz

      ctr -n k8s.io image import kubeadm-1.35.3-images-amd64.tar
    • 导入完成后执行 crictl images 验证

      输出结果示例
      IMAGE TAG IMAGE ID SIZE
      127.0.0.1:5000/cni v3.31.4 c433a27dd94ce 72.2MB
      127.0.0.1:5000/coredns v1.13.1 aa5e3ebc0dfed 23.6MB
      127.0.0.1:5000/etcd 3.6.6-0 0a108f7189562 23.6MB
      127.0.0.1:5000/kube-apiserver v1.35.3 0f2b96c93465f 27.6MB
      127.0.0.1:5000/kube-controller-manager v1.35.3 0eb506280f9bc 23MB
      127.0.0.1:5000/kube-controllers v3.31.4 ff033cc89dab5 54MB
      127.0.0.1:5000/kube-proxy v1.35.3 53ed370019059 25.7MB
      127.0.0.1:5000/kube-scheduler v1.35.3 87c9b0e4f80d3 17.1MB
      127.0.0.1:5000/node v3.31.4 e6536b93706ed 160MB
      127.0.0.1:5000/pause 3.10.1 cd073f4c5f6a8 320kB
      127.0.0.1:5000/pilot 1.29.1 cd8219a164d79 73.4MB
      127.0.0.1:5000/proxyv2 1.29.1 599aea7eee05d 89.1MB

主节点配置

仅在 Kubernetes 01 节点操作

  1. 初始化 Master 节点

    kubeadm init --cri-socket unix:///var/run/containerd/containerd.sock -v 5 --kubernetes-version=1.35.3 --image-repository=127.0.0.1:5000 --pod-network-cidr=10.244.0.0/16
    • 初始化成功后会输出 kubeadm join 命令,请保存该输出,工作节点加入集群时需要用到
    • 若初始化失败,可执行 kubeadm reset -f 重置后重试
  2. 扩展 NodePort 可用端口范围

    sed -i '/- kube-apiserver/a\ \ \ \ - --service-node-port-range=1024-32767' /etc/kubernetes/manifests/kube-apiserver.yaml
  3. 配置主节点 kubectl 访问凭证

    echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >> /etc/profile.d/kubernetes.sh
    source /etc/profile.d/kubernetes.sh
  4. 调整 Master 节点 Pod 上限

    echo "maxPods: 300" >> /var/lib/kubelet/config.yaml
    systemctl restart kubelet
  5. 允许 Master 节点参与 Pod 调度

    kubectl taint node $(kubectl get node | grep control-plane | awk '{print $1}') node-role.kubernetes.io/control-plane:NoSchedule-
    • 初始化完成后,等待约 2 分钟再执行本命令

    • 预期输出:

      • 首次执行: 输出 <节点名> untainted,表示污点移除成功,节点已允许调度 Pod

      • 再次执行: 若提示 error: taint "node-role.kubernetes.io/control-plane:NoSchedule" not found,说明污点已不存在,无需重复操作

      • 注意: 若执行后输出不符,可能由于系统组件尚未完全就绪,请稍等片刻后重试

  6. 配置 kubectl 命令补全(可选)

    如服务器可以访问互联网或有内部 yum/apt 仓库,可选装 bash-completion,实现对 kubectl 子命令 tab 快捷键补全

    # 安装 bash-completion
    Debian 系:apt -y install bash-completion
    CentOS 系:yum -y install bash-completion

    # 添加 Bash 自动补全配置
    grep -q "kubectl completion" ~/.bashrc || echo "source <(kubectl completion bash)" >> ~/.bashrc && source ~/.bashrc
  7. 安装 Calico 网络插件

    1. 将 calico.yaml 移至安装目录

      需在安装包目录 1.35-k8s-amd64-pkg 下执行

      mv calico.yaml /usr/local/kubernetes/
    2. 修改配置文件

      • 批量替换镜像仓库地址,使用本地镜像

        sed -ri 's|image: quay.io/calico|image: 127.0.0.1:5000|g' /usr/local/kubernetes/calico.yaml

        grep image: /usr/local/kubernetes/calico.yaml
        输出结果示例
        image: 127.0.0.1:5000/cni:v3.31.4
        image: 127.0.0.1:5000/cni:v3.31.4
        image: 127.0.0.1:5000/node:v3.31.4
        image: 127.0.0.1:5000/node:v3.31.4
        image: 127.0.0.1:5000/kube-controllers:v3.31.4
      • 配置 Pod 网段

        sed -ri '/# - name: CALICO_IPV4POOL_CIDR/,/# value: ".*"/ {
        s/# - name: CALICO_IPV4POOL_CIDR/- name: CALICO_IPV4POOL_CIDR/
        s/# value: ".*"/ value: "10.244.0.0\/16"/
        }' /usr/local/kubernetes/calico.yaml

        grep -C 2 CALICO_IPV4POOL_CIDR /usr/local/kubernetes/calico.yaml
        输出结果示例
        # chosen from this range. Changing this value after installation will have
        # no effect. This should fall within `--cluster-cidr`.
        - name: CALICO_IPV4POOL_CIDR
        value: "10.244.0.0/16"
        # Disable file logging so `kubectl logs` works.
      • 配置 CNI 二进制路径

        sed -i '/- name: cni-bin-dir/,/type:/s|path: .*|path: /usr/local/kubernetes/cni/bin|' /usr/local/kubernetes/calico.yaml

        grep -C 2 cni-bin-dir /usr/local/kubernetes/calico.yaml
        输出结果示例
        name: host-local-net-dir
        - mountPath: /host/opt/cni/bin
        name: cni-bin-dir
        securityContext:
        privileged: true
        --
        volumeMounts:
        - mountPath: /host/opt/cni/bin
        name: cni-bin-dir
        - mountPath: /host/etc/cni/net.d
        name: cni-net-dir
        --
        path: /proc
        # Used to install CNI.
        - name: cni-bin-dir
        hostPath:
        path: /usr/local/kubernetes/cni/bin
    3. 部署 Calico

      kubectl apply -f /usr/local/kubernetes/calico.yaml
    4. 检查 Calico 服务状态

      kubectl get pod -n kube-system -l k8s-app=calico-node
      kubectl get pod -n kube-system -l k8s-app=calico-kube-controllers

      正常情况下,所有 Pod 的 READY 列为 1/1STATUS 列为 Running

      NAME READY STATUS RESTARTS AGE
      calico-node-4xbtk 1/1 Running 0 2m
      calico-node-9fzwp 1/1 Running 0 2m
      calico-kube-controllers-7d9d9b7b9c-x2kqt 1/1 Running 0 2m
      • STATUS 长时间为 InitPending,可执行 kubectl describe pod <pod名> -n kube-system 查看详细事件排查原因

工作节点配置

仅在 Kubernetes 02 节点操作

  1. 加入 Kubernetes 集群

    kubeadm join 192.168.10.20:6443 --token 3nwjzw.pdod3r27lnqqhi0x \
    --discovery-token-ca-cert-hash sha256:a84445303a0f8249e7eae3059cb99d46038dc275b2dc2043a022de187a1175a2
    • 上方命令为示例,请使用主节点 kubeadm init 时实际输出的 join 命令
    • 若已遗忘,可在主节点执行 kubeadm token create --print-join-command 重新获取
  2. 调整工作节点 Pod 上限

    echo "maxPods: 300" >> /var/lib/kubelet/config.yaml
    systemctl restart kubelet

集群状态检查

  1. 检查节点与 Pod 状态

    kubectl get pod -n kube-system # READY 列需要是 "1/1"
    kubectl get node # STATUS 列需要是 "Ready"
  2. 下载与导入基础镜像(K8s 集群各节点均需要操作)

    镜像下载链接

    https://pdpublic.mingdao.com/private-deployment/offline/common/centos7.9.2009.tar.gz

    将该镜像文件 centos7.9.2009.tar.gz 分发至 K8s 集群各节点,并执行镜像导入操作:

    gunzip -d centos7.9.2009.tar.gz
    ctr -n k8s.io image import centos7.9.2009.tar
  3. 在主节点部署测试应用

    cat > /usr/local/kubernetes/test.yaml <<'EOF'
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: test
    namespace: default
    spec:
    replicas: 3
    selector:
    matchLabels:
    app: test
    template:
    metadata:
    labels:
    app: test
    annotations:
    md-update: '20200517104741'
    spec:
    containers:
    - name: test
    image: centos:7.9.2009
    command:
    - sh
    - -c
    - |
    echo $(hostname) > hostname.txt
    python -m SimpleHTTPServer
    resources:
    limits:
    memory: 512Mi
    cpu: 1
    requests:
    memory: 64Mi
    cpu: 0.01
    volumeMounts:
    - name: tz-config
    mountPath: /etc/localtime
    volumes:
    - name: tz-config
    hostPath:
    path: /usr/share/zoneinfo/Etc/GMT-8

    ---

    apiVersion: v1
    kind: Service
    metadata:
    name: test
    namespace: default
    spec:
    selector:
    app: test
    ports:
    - name: external-test
    port: 8000
    targetPort: 8000
    nodePort: 8000
    type: NodePort
    EOF
  4. 启动服务

    kubectl apply -f /usr/local/kubernetes/test.yaml
  5. 检查 Pod 运行状态

    kubectl get pod -o wide
  6. 验证服务访问

    curl 127.0.0.1:8000/hostname.txt
    • 多次执行应返回不同 Pod 的 hostname,说明负载均衡正常
  7. 测试没问题后,可以删除此服务

    kubectl delete -f /usr/local/kubernetes/test.yaml