#1021918 debian-installer: Kernel module blacklisting inconsistent

Package:
rootskel
Source:
rootskel
Submitter:
Olaf Meeuwissen
Date:
2023-01-08 09:57:03 UTC
Severity:
normal
Tags:
#1021918#5
Date:
2022-10-17 11:13:32 UTC
From:
To:
Dear Maintainer,

I recently tried this version with hardware that triggers loading of the
mt7921e kernel module.  Loading the module fails due to a firmware file
load error but the installer starts okay.  However, the installer later
crashes when probing for network hardware (when it tries to rmmod the
kernel module).

Since the hardware has 2 additional wired NICs that work fine (and I
wanted to do an air-gap install anyway!), I decided to blacklist the
module.

The first issue I ran into was that the documented[1] way to blacklist
kernel modules is no longer correct

 [1]: https://www.debian.org/releases/testing/amd64/ch05s03.en.html#module-blacklist

Instead of

  mt7921e.blacklist=yes

I had to use

  modprobe.blacklist=mt7921e

to prevent the kernel from loading the module (between the boot menu and
starting the installer proper).  With the latter appended to the `linux`
line of the boot menu entry, the installation appeared to have completed
without problems.

However, upon booting I saw a pile of ATA bus and I/O errors that made
me suspicious.  The disk is brand new and a smartmontools extended test
reports no errors.

I found a /etc/modprobe.d/blacklist.local.conf file with

  blacklist modprobe

Removing that file made the ATA bus and I/O errors go away.

For completeness' sake, the /etc/default/grub file included

  GRUB_CMDLINE_LINUX="modprobe.blacklist=mt7921e"

so the offending module is not loaded upon reboot after installation.

Seeing that the kernel boot argument is added correctly to the GRUB
configuration, there is no need to create a file in /etc/modprobe.d/.
In addition, the installation manual needs to be updated to use the
correct syntax.

Hope this helps,

#1021918#10
Date:
2022-10-18 23:50:01 UTC
From:
To:
How does the installer crash exactly ? Kernel panic ? Freeze ? Error ?

/lib/debian-installer-startup.d/S02module-params has the following comment:

# Before udev is started, parse kernel command word for module params of
# the form module.param=value and register them so they will be used when
# modules are loaded. Also check for modules to be blacklisted.

But udev is actually started earlier, so the first method does not work
with modules included in initrd.gz (e.g. storage drivers).

However it should work with network driver modules which are installed
much later.

This is a minor bug in /lib/debian-installer-startup.d/S02module-params
which can be easily fixed. However, it should not have any actual impact
as "modprobe" does not match any kernel module name or alias.

As expected.

If I understand correctly, this is how things work:

The kernel runs /init.
/init runs /lib/debian-installer/start-udev which starts udevd.
udevd gets hotplug events and calls modprobe to load matching modules
included in initrd.gz.
Then /init exec's /bin/busybox init.

busybox init reads /etc/inittab and runs /sbin/debian-installer-startup.
debian-installer-startup runs
/lib/debian-installer-startup.d/S02module-params.
/lib/debian-installer-startup.d/S02module-params calls
/bin/register-module for each module parameter or blacklist in the
kernel command line.
register-module writes module blacklists in
/etc/modprobe.d/blacklist.local.conf and module parameters in
/etc/modprobe.d/local.conf.

Later, network driver modules are installed and loaded.

Later, /usr/bin/grub-installer calls /bin/user-params which retrieves
kernel parameters after "--" or "---" in the command line and writes
them to grub2/linux_cmdline in debconf and GRUB_CMDLINE_LINUX in
/etc/default/grub.

So indeed kernel module parameters and blacklists may be duplicated in
/etc/modprobe.d and GRUB_CMDLINE_LINUX. To avoid duplication they should
be inserted before "---" in the installer command line.

#1021918#15
Date:
2022-10-22 01:39:44 UTC
From:
To:
Pascal Hambourg <pascal@plouf.fr.eu.org> writes:

Freeze.  Even after ten minutes the network hardware probe does not
complete.  FWIW, I have seen an error log as well but that may have
been with Devuan's preview installer for daedalus.

You can still switch VTs but trying to `poweroff` or `reboot` after
starting a shell on VT2 or VT3 will also freeze.  The last bit of the
error can be seen on VT4.

I've attached the installer's syslog.  /dev/sdc is the installer ISO.
The other disks, /dev/sda, /dev/sdb and /dev/nmve0n1 are the machine's
internal disks.  I just ran the installer after the machine was already
installed with the workaround I mentioned in the original bug report.

The error starts at

  Oct 19 23:06:13 check-missing-firmware: removing and loading kernel module mt7921e

and looks like what I've seen on VT4.

In that case, shouldn't that be mentioned in the installation manual?
Actually, a single method that works for *all* modules, whether in the
initrd.gz or installed later is much preferred.

You may want to double check how the kernel command parse results are
used then.

Or maybe the mt7921e module is in the initrd.gz?
Just checked, it is not.

Strange, because removing it made those ATA bus and I/O errors go away,
reproducibly at that.

I have since removed that blacklist statement and installed
firmware-misc-nonfree.  That makes the firmware load without any
trouble.

Since there is no mt7921e module in the initrd.gz, in fact no modules
matching mt7, it should not have been loaded at this point according to
your understanding.

That appears to work but blacklisting syntax is *not* as documented in
the installation manual.

The blacklist.local.conf file is created as documented but using the
alternative syntax I had to use leads to the oxymoronic

  blacklist modprobe

entry, trying to tell modprobe to blacklist itself :-)

You mentioned above that's a minor bug and easily fixed.  If so, then
please fix it.

Seeing that the module is not in initrd.gz, this is where it would be
loaded according to your understanding.  Does this step happen *before*
the installer screen appears?  If yes, that is in agreement with what I
have seen.  If no, and network driver modules are installed and loaded
at the network hardware probe step of the installer then that does *not*
correspond to what I have seen.  That is to say, unless I blacklist the
module with modprobe.blacklist=mt7921e, I see piles or firmware loading
error fly by before the installer screen appears, asking me to select a
language.

I put it after the "--" and it did end up in GRUB_CMDLINE_LINUX.

That should be documented then as well, including where the kernel
module parameters inserted before the "--" end up, in a modprobe.d file
or the GRUB_CMDLINE_LINUX variable.

Actually, reading the installation manual info, I found myself wondering
whether I should put it before or after.  After was easier to navigate
to in the editor so that's what I went with.

Hope this helps,
--
Olaf Meeuwissen

#1021918#20
Date:
2022-11-01 09:00:40 UTC
From:
To:
I could not reproduce this after tricking the installer into unloading
and reloading the mt7921e module. The module unloads and reloads
cleanly. But I do not have any hardware matching this module.
(...)

It looks like a kernel bug when unloading this module. Can you trigger
the bug in an installed system ? If yes it means that it not specific to
the installer.

I did, and <module>.blacklist works as expected with NIC modules
matching my hardware (iwlwifi and e1000e).

Indeed, it is in the package nic-wireless-modules-<kernel-version>-di.

Yes, really strange. I cannot explain nor reproduce it.

It avoids cluttering the kernel command line with module parameters.

As I wrote, "modprobe" is not a module name so this should be a no-op.

I am not a Debian developer. The best I can do is submit a patch.

No, it should happen at the "Detect network hardware" (or so) step.

Which installation image did you use ?

#1021918#25
Date:
2022-11-01 09:22:05 UTC
From:
To:
Here are two proposed patches for rootskel.

1) The first patch adds "modprobe" to the ignore list when parsing
module parameters in the kernel command line to skip modprobe.blacklist
parameters which are intended to modprobe, not to a kernel module.

2) Module parameters passed in the kernel command line after --- or --
are duplicated in both /etc/modprobe.d/local.conf by rootskel and
/etc/default/grub by grub-installer. The second patch skips parsing of
module parameters after --- or -- to avoid the useless duplicate.

I seek advice from d-i developpers about a third issue in rootskel.

The installer parses "<module>.blacklist=yes" kernel parameters and
creates "blacklist <module>" entries in
/etc/modprobe.d/blacklist.local.conf. However it happens after udev is
started so it is uneffective with modules included in initrd.gz.

Note: "modprobe.blacklist=<module>" kernel parameters are parsed by
modprobe so do not have this flaw.

1. Should this flaw be fixed ?
2. If yes, how ?
3. Should the <module>.blacklist=yes method be deprecated in favour of
modprobe.blacklist=<module> which is handled directly by modprobe and
does not require any special processing by rootskel ?

I tried to delay the start of udev until after blacklist.local.conf has
been created, but then /dev/tty* device files are created late and the
selected console is /dev/console instead of /dev/tty0|1. As a result,
the installer does not seem to use the framebuffer and does not offer
languages other than English.

#1021918#36
Date:
2022-11-07 11:07:19 UTC
From:
To:
Sorry for the belated follow-up.

Pascal Hambourg <pascal@plouf.fr.eu.org> writes:

The freeze probably only occurs when an attempt to loadh the firmware
file is made.  Unlikely that will happen if the hardware is not found.
Anyway, this issue is not with this particular kernel module but with
the installer's inconsistent module blacklisting behaviour.

FWIW, I've since installed firmware-misc-nonfree and removed all the
blacklisting bits for the module.  WiFi works fine.

Machine's at the office, will see if I can test tomorrow or the day after.

I meant double check by reading the source code, not by trying with some
rather commonly used modules.

It may be a no-op for modprobe but it's not a no-op for my brain ;-)
Actually, it's more like a WTF for my brain.

By all means, please do.  Thanks in advance.

The netinst one, according to the syslog I attached earlier.  Looking at
that log, I don't see any proof of the firmware loading error fly by
before the installer starts.  Weird.  I'll see if I can reproduce it.
Maybe it is a default vs. advanced install difference.  I do remember
using both while trying to get the blacklisting to work but I think I
captured that syslog using the default install.

Hope this helps,
--
Olaf Meeuwissen

#1021918#41
Date:
2022-11-08 11:35:05 UTC
From:
To:
Hi again,

Olaf Meeuwissen <paddy-hack@member.fsf.org> writes:

Checked this morning.  With the firmware file loaded, rmmod succeeds
without problems.  With the firmware file not loaded, trying to rmmod it
echoes "Killed" on the command-line and created a longish call trace in
/var/log/kern.log.  Not sure if it was the same, though.

I cannot reproduce the firmware file load failures before the installer
starts.  Tried graphical and text installs as well as the expert text
install.  No idea why I believed they did.  Some for the confusion.

Hope this helps,
--
Olaf Meeuwissen

#1021918#46
Date:
2022-11-08 23:12:18 UTC
From:
To:
I don't think so. I think it happens when unloading the module.

The freeze is related with this kernel module, not the installer.

As expected.

I did.

The blacklist handling is not related to a specific module, so any
module matching present hardware will do.

Already done in another mail with attached patches sent to the bug.

Indeed, the log clearly shows that the module is loaded at the
"detecting networking hardware" step as expected.
Consistent with the previous log.

#1021918#51
Date:
2023-01-07 19:02:57 UTC
From:
To:
Hi.  I just did an install of bullseye on an amd64 machine with a
troublesome wifi interface - suspected hardware fault[1].  We did
achieve a successful install, blacklisting the module for the bad
hardware, but we did encounter a version of this bug.

We edited the vmlinuz line in the installer image boot menu (we were
using the non-graphical installer) to add
  modprobe.blacklist=rtw88_8723de

The installer was able to find the USB wifi dongle we'd added, didn't
touch the cursed hardware, and ran to completion.  But, as described
in this bug report, it had created a file
  /etc/modprobe.d/blacklist.local.conf
containing just
  blacklist modprobe

I deleted this using the installer shell before rebooting into the
installed system.  The installer had *not* included anything in
GRUB_CMDLINE_LINUX.

So the troublesome hardware *wasn't* blacklisted on our reboot.
Happily nothing tried to use it.  I manually created a file
  /etc/modprobe.d/rtw88.conf
containing
  blacklist rtw88_8723de
  blacklist rtw88_8723d
and that has resulted in a working setup.

Thanks,
Ian.

[1] The fault results in periodic wifi breakage and strange symtoms
with an older Debian release.  With the bullseye installer, it causes
the installer environment to reliably hang (crashed with no response
even to capslock led).

#1021918#60
Date:
2023-01-08 09:45:34 UTC
From:
To:
Hello,

On 07/01/2023, Ian Jackson wrote :
(...)
previously, only parameters located after "---" are added to the target
system /etc/default/grub GRUB_CMDLINE_LINUX by grub-installer.
Parameters located before "---" are intended for the installer only, not
the target system.

Note: modprobe.blacklist=<module> or "blacklist <module>" only prevent
modprobe from loading the module by alias (mostly when called by udev).
It does not prevent modprobe from loading the module by name or
dependency of another module nor loading the module by pathname with
insmod. Since v4.8, the kernel supports the parameter
module_blacklist=<module>,<module>... which prevents from loading a list
of modules in any way.