Fix broken k3s on Raspbian Kernel 5.4

Aug 4, 2020 · 2 min read

After upgrading to kernel 5.x using sudo apt -y full-upgrade on my Raspberry Pi k3s cluster I found that it suddenly didn’t work! I found myself getting an error from any kubectl command:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

Inspecting further I found that k3s was failing to start up, inspecting further with a bit of journalctl -xe, this was due to:

failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry 
Pi)

But I have setup the /boot/cmdline.txt file correctly, it is automated using Ansible!

Running cat /proc/cgroups confirmed that indeed, the memory cgroup was not loaded:

#subsys_name	hierarchy	num_cgroups	enabled
cpuset	5	1	1
cpu	4	1	1
cpuacct	4	1	1
devices	2	25	1
freezer	7	1	1
net_cls	3	1	1
pids	6	31	1

Turns out this is a bug in the kernel which has now been fixed. The only thing I need to do is to update the kernel, this can be done by running rpi-update. This is really annoying to do manually so I automated it using Ansible and added it to my initial cluster deployment scripts. It is achieved using this super simple task:

- name: Update kernel
  shell: "SKIP_WARNING=1 /usr/bin/rpi-update"
  notify:
    - reboot host

Now my k3s cluster is functioning again.

Fix broken k3s on Raspbian Kernel 5.4

Comments