Kubernetes has been phasing out cgroup v1 for a while, but the timeline is now concrete. Starting with Kubernetes v1.35, the kubelet does not start on a cgroup v1 node unless you change a default. If any Linux node in your cluster still runs cgroup v1, this is the change to plan for before your next upgrade.
What changed in Kubernetes
Kubernetes uses cgroups (control groups), a Linux kernel feature, to allocate CPU and memory to containers so that one workload cannot starve another. The kernel offers two generations of this interface. cgroup v2 provides a single unified hierarchy, a more consistent interface and a stronger base for resource isolation. Kubernetes has moved users toward it in stages:
- With Kubernetes v1.31, support for cgroup v1 management moved into maintenance mode.
- Support for cgroup v2 management has been stable since Kubernetes v1.25.
- Starting with Kubernetes v1.35, the
failCgroupV1setting defaults totrue, so the kubelet does not start on a cgroup v1 node by default.
Removal of cgroup v1 support will follow the Kubernetes deprecation policy. The remaining removal work is tracked in KEP-5573.
Why the default matters for upgrades
The risk is that the failure appears during the upgrade, not before it. Under the default configuration, a cgroup v1 node fails during kubelet startup. kubeadm catches the problem earlier. The SystemVerification preflight check returns an error during kubeadm init, kubeadm join and kubeadm upgrade when it detects cgroup v1 and the kubelet is v1.35 or later. With an older kubelet, the same check only prints a warning, and warnings are easy to miss on a cluster that is still on older releases.
Check every Linux node
Check each node itself rather than relying on an inventory file. On a node, this command prints cgroup2fs when cgroup v2 is active:
stat -fc %T /sys/fs/cgroup/To check several nodes from an admin host over SSH, run a loop like this:
for node in worker-01 worker-02 worker-03; do
echo -n "$node: "
ssh "$node" 'stat -fc %T /sys/fs/cgroup/'
doneRead the output carefully. Only cgroup2fs confirms cgroup v2. An SSH error or a blank line means that node is unverified, not passing. Resolve those nodes before you upgrade.
Requirements for cgroup v2 nodes
The Kubernetes post lists these prerequisites:
- At least one Linux node. cgroup is a Linux-only concept.
- The OS must run with cgroup v2 enabled.
- A kernel of version 5.8 or later. Version 5.9 or later is recommended if you use memory QoS.
- A container runtime that supports cgroup v2: containerd v1.4 or later (v2.0 or later for automatic cgroup-driver discovery), or CRI-O v1.20 or later.
- The kubelet and the container runtime both configured to use the correct cgroup driver.
The article does not explain how to enable cgroup v2 on each distribution. Use your distribution’s documentation and the Kubernetes migration guide for that step, and test on a non-production node first.
Setting the cgroup driver and the temporary override
If your runtime does not support automatic driver detection, set the driver in the kubelet configuration file:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemdIf a cgroup v1 node must keep running after you upgrade to v1.35 or later, the article allows a temporary override in the same file:
failCgroupV1: falseTreat this as a stopgap. Cgroup v1 support will be removed under the deprecation policy, so any node that depends on the override will need migrating eventually. Record which nodes use it and why.
How TechProvidence can help
Moving nodes from cgroup v1 to v2 touches the operating system, the container runtime and the kubelet at the same time, so it is best done one node at a time with a rollback plan. TechProvidence provides Linux server administration, DevOps and cloud support, including cluster upgrades and fleet audits before a version change. If you want your nodes checked before the next upgrade, contact us.
Source: Kubernetes Blog
