Kubernetes FAQs

`crictl images` Command Execution Failed

Applicable Version: Kubernetes 1.35.3, kernel version < 4.11

Error Message:

FATA[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService 

Solution:

Disable the enable_unprivileged_ports and enable_unprivileged_icmp settings in containerd:

# Check the current configuration
grep -nE 'unprivileged_(icmp|port)' /etc/containerd/config.toml

# Set both options to false
sed -i \
    -e 's/^\(\s*enable_unprivileged_ports\s*=\s*\)true/\1false/' \
    -e 's/^\(\s*enable_unprivileged_icmp\s*=\s*\)true/\1false/' \
/etc/containerd/config.toml

# Verify the changes
grep -nE 'unprivileged_(icmp|port)' /etc/containerd/config.toml

systemctl restart containerd

After restarting, run the following command to verify plugin status. All plugins showing ok indicates normal operation:

ctr plugin ls | grep -E 'cri|io.containerd.grpc.v1.cri'

Expected output

io.containerd.cri.v1                      images                   -              ok        
io.containerd.cri.v1                      runtime                  linux/amd64    ok        
io.containerd.grpc.v1                     cri                      -              ok

Calico Installation Failed

Applicable Version: Kubernetes 1.35.3

Error Message:

kubectl -n kube-system logs calico-node-xxx -c upgrade-ipam

This program can only be run on AMD64 processors with v2 microarchitecture support.

Cause: The CPU does not support the x86-64-v2 architecture. Starting from Calico v3.31.1, only x86-64-v2 and above architectures are supported. Reference: Calico GitHub Issue #11352

Solution: Switch to Calico 3.30.7

Download and extract the installation package, then import the images

wget https://pdpublic.mingdao.com/private-deployment/offline/common/kubernetes-1.35.3/calico-3.30.7-yaml-images-amd64.tar.gz

tar xzvf calico-3.30.7-yaml-images-amd64.tar.gz
cd calico-3.30.7-yaml-images-amd64
gunzip -d calico-3.30.7-images-amd64.tar.gz
ctr -n k8s.io image import calico-3.30.7-images-amd64.tar

Move calico.yaml to the installation directory

Run this command from within the calico-3.30.7-yaml-images-amd64 directory
```
mv calico.yaml /usr/local/kubernetes/
```

Modify the configuration file

Replace the image registry address

sed -ri 's|image: docker.io/calico|image: 127.0.0.1:5000|g' /usr/local/kubernetes/calico.yaml

grep image: /usr/local/kubernetes/calico.yaml

Example output

    image: 127.0.0.1:5000/cni:v3.30.7
    image: 127.0.0.1:5000/cni:v3.30.7
    image: 127.0.0.1:5000/node:v3.30.7
    image: 127.0.0.1:5000/node:v3.30.7
    image: 127.0.0.1:5000/kube-controllers:v3.30.7

Configure the Pod CIDR

sed -ri '/# - name: CALICO_IPV4POOL_CIDR/,/#   value: ".*"/ {
  s/# - name: CALICO_IPV4POOL_CIDR/- name: CALICO_IPV4POOL_CIDR/
  s/#   value: ".*"/  value: "10.244.0.0\/16"/
}' /usr/local/kubernetes/calico.yaml

grep -C 2 CALICO_IPV4POOL_CIDR /usr/local/kubernetes/calico.yaml

Example output

      # chosen from this range. Changing this value after installation will have
      # no effect. This should fall within `--cluster-cidr`.
      - name: CALICO_IPV4POOL_CIDR
        value: "10.244.0.0/16"
      # Disable file logging so `kubectl logs` works.

Configure the CNI binary path

sed -i '/- name: cni-bin-dir/,/type:/s|path: .*|path: /usr/local/kubernetes/cni/bin|' /usr/local/kubernetes/calico.yaml

grep -C 2 cni-bin-dir /usr/local/kubernetes/calico.yaml

Example output

        name: host-local-net-dir
      - mountPath: /host/opt/cni/bin
        name: cni-bin-dir
    securityContext:
      privileged: true
--
    volumeMounts:
      - mountPath: /host/opt/cni/bin
        name: cni-bin-dir
      - mountPath: /host/etc/cni/net.d
        name: cni-net-dir
--
      path: /proc
  # Used to install CNI.
  - name: cni-bin-dir
    hostPath:
      path: /usr/local/kubernetes/cni/bin

Deploy Calico

kubectl apply -f /usr/local/kubernetes/calico.yaml

calico-node Pod Health Check Failed

Applicable Version: Kubernetes 1.35.3

Error Message:

Warning  Unhealthy  115s  kubelet  Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "...": OCI runtime exec failed: exec failed: unable to start container process: error adding pid 23804 to cgroups: Unknown method 'AttachProcessesToUnit' or interface 'org.freedesktop.systemd1.Manager'

Cause: The system does not support the systemd cgroup driver; the cgroup drivers of containerd and kubelet are incompatible.

Solution:

Switch the containerd cgroup driver from systemd to cgroupfs and reinitialize the cluster:

Modify the containerd configuration

sed -i 's/SystemdCgroup = true/SystemdCgroup = false/g' /etc/containerd/config.toml

grep -n "SystemdCgroup" /etc/containerd/config.toml
systemctl restart containerd

Append the kubelet cgroup configuration to kubeadm-config.yaml, then re-run kubeadm init to reinitialize

cd /usr/local/kubernetes/
cat >> kubeadm-config.yaml <<'EOF'

---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: cgroupfs
EOF

Kubernetes 1.35.3 Installation Fails Because `cgroups v1` Is Incompatible

Applicable Scenario: During the installation of Kubernetes 1.35.3, the host still uses cgroup v1, so kubeadm init fails during preflight checks.

Error Message:

[WARNING SystemVerification]: kernel release 3.10.0-1160.el7.x86_64 is unsupported. Supported LTS versions from the 5.x series are 5.4, 5.10 and 5.15. Any 6.x version is also supported. For cgroups v2 support, the recommended version is 5.10 or newer
[WARNING Hostname]: hostname "k8s-master-01" could not be reached
[WARNING Hostname]: hostname "k8s-master-01": lookup k8s-master-01 on 172.17.30.241:53: server misbehaving
[preflight] Some fatal errors occurred:
        [ERROR SystemVerification]: cgroups v1 support is deprecated and will be removed in a future release. Please migrate to cgroups v2.
error: error execution phase preflight: preflight checks failed

Cause:
- During the installation of Kubernetes 1.35.3, support for cgroup v1 is already limited in the preflight stage.
- The host is currently using cgroup v1.
- If the environment cannot be switched to cgroup v2, you need to switch to Kubernetes 1.25.4 and reinstall.
Solution:

If the current environment must keep cgroup v1, reset the Kubernetes cluster environment first, then switch the installation version from 1.35.3 to 1.25.4 and install again.

If you need to continue installing Kubernetes 1.35.3 or a later version, switch the host to cgroup v2 first, then run initialization again.

How to Verify:

Check the cgroup version according to the current system.

Check whether cgroup v2 or cgroup v1 is enabled

if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
  echo "cgroup v2"
else
  echo "cgroup v1"
fi

Example output:

cgroup v2
cgroup v1

cgroup v2

cgroup v1

Check the current filesystem type
```
stat -fc %T /sys/fs/cgroup
```
Example output:
- cgroup v2
- cgroup v1
cgroup2fs
tmpfs

Optionally check the current process cgroup information

cat /proc/1/cgroup

Example output:

cgroup v2
cgroup v1

0::/init.scope

cpuacct,cpu:/
perf_event:/
devices:/
blkio:/
hugetlb:/
cpuset:/
memory:/
freezer:/
net_prio,net_cls:/
pids:/
name=systemd:/

crictl images Command Execution Failed​

Calico Installation Failed​

calico-node Pod Health Check Failed​

Kubernetes 1.35.3 Installation Fails Because cgroups v1 Is Incompatible​

`crictl images` Command Execution Failed

Calico Installation Failed

calico-node Pod Health Check Failed

Kubernetes 1.35.3 Installation Fails Because `cgroups v1` Is Incompatible