after apt upgrade to kernel 6.6.31 pi still boots into 6.6.20

[mgrouch]

Describe the bug

I did

sudo apt update
sudo apt upgrade

it installed kernel 6.6.31

however old kernel 6.6.20 is still not removed and takes lots of space

Worse, pi5 still boots into old 6.6.20 kernel

Steps to reproduce the behaviour

sudo apt update
sudo apt upgrade

it installed kernel 6.6.31

however old kernel 6.6.20 is still not removed and takes lots of space

Worse, pi5 still boots into old 6.6.20 kernel

Device (s)

Raspberry Pi 5

System

cat /etc/rpi-issue
Raspberry Pi reference 2024-03-15
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, f19ee211ddafcae300827f953d143de92a5c6624, stage2

vcgencmd version
2024/04/20 11:53:30 
Copyright (c) 2012 Broadcom
version d1744d21 (release) (embedded)

uname -a
Linux lysmarine 6.6.20+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux

Logs

No response

Additional context

No response

[mgrouch]

Unfortunately same doesn't work for me.

I got some warning during install about MD subsystem is not loaded. Might be relevant

Setting up linux-image-6.6.31+rpt-rpi-2712 (1:6.6.31-1+rpt1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.6.31+rpt-rpi-2712
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
'/boot/initrd.img-6.6.31+rpt-rpi-2712' -> '/boot/firmware/initramfs_2712'
/etc/kernel/postinst.d/z50-raspi-firmware:
'/boot/vmlinuz-6.6.31+rpt-rpi-2712' -> '/boot/firmware/kernel_2712.img'

Processing triggers for shared-mime-info (2.2-1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.6.31+rpt-rpi-v8
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
'/boot/initrd.img-6.6.31+rpt-rpi-v8' -> '/boot/firmware/initramfs8'
update-initramfs: Generating /boot/initrd.img-6.6.31+rpt-rpi-2712
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
'/boot/initrd.img-6.6.31+rpt-rpi-2712' -> '/boot/firmware/initramfs_2712'
update-initramfs: Generating /boot/initrd.img-6.6.20+rpt-rpi-v8
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
update-initramfs: Generating /boot/initrd.img-6.6.20+rpt-rpi-2712
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for libc-bin (2.36-9+rpt2+deb12u7) ...
Processing triggers for man-db (2.11.2-2) ...

Did old kernels got deleted for you or they are still installed?

[mgrouch]

cat  /etc/apt/apt.conf.d/01autoremove 
APT
{
  NeverAutoRemove
  {
	"^firmware-linux.*";
	"^linux-firmware$";
	"^linux-image-[a-z0-9]*$";
	"^linux-image-[a-z0-9]*-[a-z0-9]*$";
  };

  VersionedKernelPackages
  {
	# kernels
	"linux-.*";
	"kfreebsd-.*";
	"gnumach-.*";
	# (out-of-tree) modules
	".*-modules";
	".*-kernel";
  };

  Never-MarkAuto-Sections
  {
	"metapackages";
	"tasks";
  };

  Move-Autobit-Sections
  {
	"oldlibs";
  };
};

so old kernels are not removed by

apt-get autoremove -y --purge

What would be the way to properly remove them?

Thanks

[mgrouch]

sudo apt reinstall raspberrypi-kernel

installed even older kernel 6.1.21+

but on re-boot pi is booted into 6.6.20 kernel.

[pelwell]

Have you attempted to update from Bullseye to Bookworm, or similar? Nobody else is (or should be) having these problems.

[XECDesign]

sudo apt reinstall raspberrypi-kernel

That's definitely the wrong thing to do.

Your logs earlier shows the right version getting installed. If it's not booting the right one, then your partition mountpoints may be set up incorrectly.

Apt will keep the currently running kernel and the previous kernel available just in case something goes wrong and you need to copy the old one.

There's a proposal to add an option to raspi-config to remove unused kernels here,

[mgrouch]

I’m building an image on CircleCi.
I’m downloading 2024-03-15 lite image,
I mount all partitions and run apt update/upgrade.
Then unmount it and compress with xz.
Apt installs new kernels, doesn’t remove old one.
And on newly created image it boots with old kernel instead of new one.

[XECDesign]

I mount all partitions

Can you show how you do that?

[mgrouch]

I mount all partitions

Can you show how you do that?

Via these scripts:

https://github.com/bareboat-necessities/lysmarine_gen/blob/bookworm/cross-build-release/raspbian.sh
https://github.com/bareboat-necessities/lysmarine_gen/blob/bookworm/cross-build-release/lib.sh

And whole circleci starting point is here:

https://github.com/bareboat-necessities/lysmarine_gen/blob/bookworm/.circleci/config.yml

[XECDesign]

I think this should be /boot/firmware rather than just /boot.

[pelwell]

(with a corresponding change to the line above as well)