Dear Maintainer, the Installed-Size of the package has occasionally grown up to 375 MB, which is about 30% larger than several minor releases before. A kindful anonymous person has collected some more information here: https://www.linux.org.ru/forum/general/16628666?cid=16628797 , and found out that virtually every module has been grown in size. So this is likely related to compilation options, rather than to some added modules as I suspected before. Please investigate the actual reason and report if this can/would be fixed in further packages, thanks.
Hi Going to close tis bugreport, as the image will steadily increase. Regards, Salvatore
Let's reopen this for adding an explanation. Regards, Salvatore
[...]
The linked thread mentioned floppy.ko as an exmaple. Comparing the
versions I have installed here:
~$ ls -l /lib/modules/*/kernel/drivers/block/floppy.ko
-rw-r--r-- 1 root root 182555 Aug 3 07:50 /lib/modules/5.10.0-8-amd64/kernel/drivers/block/floppy.ko
-rw-r--r-- 1 root root 196947 Nov 26 06:33 /lib/modules/5.15.0-2-amd64/kernel/drivers/block/floppy.ko
there's about a 14 KiB increase from 5.10 to 5.15.
The code and static data sizes are roughly the same, actually slightly
smaller:
~$ size /lib/modules/*/kernel/drivers/block/floppy.ko
text data bss dec hex filename
64213 4893 14660 83766 14736 /lib/modules/5.10.0-8-amd64/kernel/drivers/block/floppy.ko
63619 4836 16516 84971 14beb /lib/modules/5.15.0-2-amd64/kernel/drivers/block/floppy.ko
The bss can be ignored as it doesn't take up disk space.
Listing the sections with "objdump -h", I see a new section in 5.15:
Idx Name Size VMA LMA File off Algn
[...]
26 .BTF 00002b58 0000000000000000 0000000000000000 00010c40 2**0
CONTENTS, READONLY
That's a size of about 11 KiB, so most of the increase.
After that I compared *all* the modules installed by these versions:
~$ du --bytes --summ /lib/modules/5.10.0-8-amd64/kernel
294650546 /lib/modules/5.10.0-8-amd64/kernel
~$ du --bytes --summ /lib/modules/5.15.0-2-amd64/kernel
371262312 /lib/modules/5.15.0-2-amd64/kernel
About a 73 MiB increase.
I calculated the total size of .BTF sections:
$ find /lib/modules/5.15.0-2-amd64/ -name '*.ko' | xargs objdump -h -j .BTF | awk 'BEGIN { total = 0 } $2 == ".BTF" { total = total + strtonum("0x" $3) } END { print total }'
objdump: Warning: Separate debug info file /usr/lib/modules/5.15.0-2-amd64/kernel/sound/usb/usx2y/snd-usb-us122l.ko found, but CRC does not match - ignoring
objdump: Warning: Separate debug info file /usr/lib/modules/5.15.0-2-amd64/kernel/drivers/leds/leds-gpio.ko found, but CRC does not match - ignoring
objdump: Warning: Separate debug info file /usr/lib/modules/5.15.0-2-amd64/kernel/fs/nls/nls_cp862.ko found, but CRC does not match - ignoring
61693267
About 59 MiB, so again most of the increase.
It appears that BTF in modules was enabled in Linux 5.11 by
<https://git.kernel.org/linus/5f9ae91f7c0dbbc4195e2a6c8eedcaeb5b9e4cbb>
I also compared the total sizes of code and static data:
$ find /lib/modules/5.10.0-8-amd64/ -name '*.ko' | xargs objdump -h | awk 'BEGIN { total = 0 } $2 ~ /^\..*(text|data)/ { total = total + strtonum("0x" $3) } END { print total }'
objdump: Warning: Separate debug info file /usr/lib/modules/5.10.0-8-amd64/kernel/sound/usb/usx2y/snd-usb-us122l.ko found, but CRC does not match - ignoring
objdump: Warning: Separate debug info file /usr/lib/modules/5.10.0-8-amd64/kernel/drivers/hid/hid-macally.ko found, but CRC does not match - ignoring
objdump: Warning: Separate debug info file /usr/lib/modules/5.10.0-8-amd64/kernel/fs/fat/vfat.ko found, but CRC does not match - ignoring
88844481
$ find /lib/modules/5.15.0-2-amd64/ -name '*.ko' | xargs objdump -h | awk 'BEGIN { total = 0 } $2 ~ /^\..*(text|data)/ { total = total + strtonum("0x" $3) } END { print total }'
objdump: Warning: Separate debug info file /usr/lib/modules/5.15.0-2-amd64/kernel/sound/usb/usx2y/snd-usb-us122l.ko found, but CRC does not match - ignoring
objdump: Warning: Separate debug info file /usr/lib/modules/5.15.0-2-amd64/kernel/drivers/leds/trigger/ledtrig-default-on.ko found, but CRC does not match - ignoring
objdump: Warning: Separate debug info file /usr/lib/modules/5.15.0-2-amd64/kernel/fs/nls/mac-roman.ko found, but CRC does not match - ignoring
93202503
About 4 MiB increase. This is probably a combination of changes in code
generation between gcc 10 and 11, increases in complexity of existing
code modules, and a few new drivers and features being enabled. Without
doing some full rebuilds it's not possible to separate these.
That leaves about 10 MiB of the increase in installed module size not
yet explained.
Ben.
So the patch that was supposed to drastically reduce the size of modules by extracting the debugging info to a separate deduplicated file, actually increased it even more? Sounds pretty ironic. Or did the deduplication finally make BTFs reasonably small to be included by default, as they were much larger before and thus were disabled?
possibly interesting in that context (I asked/posted the link in #debian-kernel a few days ago as well) - these BTF sections now actually reference the BTF info in the kernel image itself (as part of the deduplication of shared information), which makes the latter part of the ABI, and AFAICT this is not (yet?) tracked in Debian.. https://lore.kernel.org/all/1637926692.uyvrkty41j.astroid@nora.none/ an otherwise ABI compatible kernel upgrade thus has the potential to break module loading altogether, and I'd recommend disabling the split BTF feature for the time being unless you plan on bumping ABI for every kernel update anyway.
It is confusing, yes. I *think* that the commit message is actually saying that it makes the module BTF section smaller than in an earlier version that was not applied. Ben.
On Tue, 2021-11-30 at 11:01 +0100, Fabian Grünbichler wrote: [...] Yes, that is interesting/concerning. If we continue to not bump the ABI number on every update, then I think: 1. In-tree modules should not be loadable between an upgrade and a reboot. (This can happen already for specific modules, due to symbol version changes that we think don't affect out-of-tree modules.) Alternatively, they could still be loadable but then their BTF info should be completely discarded. 2. Out-of-tree modules should be built without BTF deduplication, or without BTF info. The main reason for not bumping the ABI number every time is to avoid forcing an unnecessary rebuild of out-of-tree modules. We could try switching to something like RHEL's "weak-update" mechanism where ABI- compatible out-of-tree modules are automatically linked into a new version's modules directory without rebuilding them. In that case we would still need to implement item (2) above. Ben.
booted kernel: Linux host 5.15.0-2-amd64 #1 SMP Debian 5.15.5-1 (2021-11-26) x86_64 GNU/Linux installed kernel: ii linux-image-5.15.0-2-amd64 5.15.5-2 amd64 Linux 5.15 for 64-bit PCs (signed) attempting to (auto-)load any module not already loaded before the upgrade: Dec 26 17:18:48 host mtp-probe[319902]: checking bus 4, device 3: "/sys/devices/pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:08.0/0000:05:00.3/usb4/4-4" Dec 26 17:18:48 host mtp-probe[319902]: bus: 4, device: 3 was not an MTP device Dec 26 17:18:49 host kernel: scsi 3:0:0:0: Direct-Access Multi-Reader -0 1.00 PQ: 0 ANSI: 6 Dec 26 17:18:49 host kernel: scsi 3:0:0:1: Direct-Access Multi-Reader -1 1.00 PQ: 0 ANSI: 6 Dec 26 17:18:49 host kernel: scsi 3:0:0:2: Direct-Access Multi-Reader -2 1.00 PQ: 0 ANSI: 6 Dec 26 17:18:49 host kernel: scsi 3:0:0:3: Direct-Access Multi-Reader -3 1.00 PQ: 0 ANSI: 6 Dec 26 17:18:49 host kernel: scsi 3:0:0:0: Attached scsi generic sg0 type 0 Dec 26 17:18:49 host kernel: scsi 3:0:0:1: Attached scsi generic sg1 type 0 Dec 26 17:18:49 host kernel: scsi 3:0:0:2: Attached scsi generic sg2 type 0 Dec 26 17:18:49 host kernel: scsi 3:0:0:3: Attached scsi generic sg3 type 0 Dec 26 17:18:49 host kernel: BPF:[86226] ENUM T_CONDITION_MET Dec 26 17:18:49 host kernel: BPF:size=4 vlen=11 Dec 26 17:18:49 host kernel: BPF: Dec 26 17:18:49 host kernel: BPF:Invalid name Dec 26 17:18:49 host kernel: BPF: Dec 26 17:18:49 host kernel: failed to validate module [sd_mod] BTF: -22 module loading fails until booted and on-disk kernel images match again - either by downgrading the latter (to 5.15.5-1 in this case), or by rebooting. note that just disabling the relevant KConfig doesn't work in my experience, since it will be automatically enabled again by the presence of a split-BTF capable pahole version in the build environment. patching the default value to 'n' does work though[0]. 0: https://git.proxmox.com/?p=pve-kernel.git;a=commitdiff;h=bc1d1913898940cabcea142f75a2a4759790a503
I also encountered this issue just now. Basically "tomb open" was not working because of this, until I rebooted. It gave an error message about a module not being loaded, and from there to dmesg I found this issue. The information contained in this electronic mail is confidential information intended only for the use of the individual(s) or entity(s) named. If the reader of the message is not the addressee (or authorized to receive for the addressee), you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by reply e-mail and/or by telephone and destroy the original message.