Package: hw-detect
Severity: important
X-Debbugs-Cc: debian-arm@lists.debian.org

Hi,

[debian-arm@ in copy for the question regarding concatenateable
images near the end.]

I'm filing this as important for the time being, but this can be made
serious if required. Reminder: doing so might or might not block the
12.0 release (it could be usertagged can-defer).


Firmware support has long been a very difficult topic to navigate, for
mere users but also for more advanced users who might try and include
firmware files and/or packages onto external media, then wonder what
they did wrong because that still doesn't work.

With the 2022 General Resolution about non-free firmware, and with
non-free-firmware packages being allowed on official installation
images, Steve and I were hoping[1] to limit ourselves to a much more
streamlined process: use whatever's available on the official image, and
stop pretending to support loading firmware material from elsewhere.

 1. https://lists.debian.org/debian-boot/2022/10/msg00044.html

Since then, the received feedback and my own testing triggered mixed
feelings:
 - In that thread, some reason was given suggesting to retain the
   ability to load firmware packages from external media;
 - On IRC, while I was rubberducking firmare support in hw-detect,
   the same person mentioned that the current implementation doesn't
   suit their needs anyway, due to the way the search for firmware is
   performed.
 - My own tests show that the “Detect network hardware” step can be
   stuck for a long time (e.g. 15+ seconds without anything on the
   screen but a title, probably depends on the storage). While I initially
   suspected a problem with my ongoing work, it turned out that both
   “mountmedia” and “mountmedia driver” calls were responsible for most of
   that time's being wasted.

The first call (“mountmedia”):
 - tries to find “loose firmware files” under /media as mounted by
   mountmedia, to copy them in the installer's context;
 - lets those files be propagated to /target later on.

The second call (“mountmedia driver”):
 - uses the same check_firmware function that's already called on
   /firmware and /cdrom/firmware (where official installation images for
   Bookworm will find non-free-firmware packages); this means packages
   are unpacked in the installer environment, and queued for later
   installation into /target;
 - looks under both /media and /media/firmware (with /media having been
   mounted by “mountmedia driver”).


Some use cases:
 - Users might require some firmware packages that aren't available in
   Debian yet;
 - Users might want to stash a higher version of a firmware package that
   is available in Debian already;
 - <yours goes here.>

Some problems:
 - Delays for users that have everything on their official installation
   images.
 - Insufficient lookup (something about stopping at the first partition
   that's found or something, but I don't know the specifics, and I'm
   not diving into what “mountmedia driver” does or should be doing).
 - What happens in the “higher version” case? Currently we do not have
   anything to detect/process multiple versions of a package, so we might
   end up deploying a firmware package found on the installation image,
   then deploying a different version of the same firmware package found
   on external media; but then both are queued and which one gets
   installed first and which one second depends on the order in which
   their filenames come up in the for loop around `in-target dpkg -i`…
   If we wanted to support that, we would need more code to only keep the
   higher version.


I'm also not sure what our plan for e.g. ARM images (concatenateable
images) would be; I don't think we'll pull any firmware packages during a
d-i build, so they wouldn't end up under [2], but maybe it'd be feasible
to just concatenate some file/image containing non-free-firmware packages
(along with the associated metadata if relevant) as published by debian-cd
or something? Or would those concatenateable images need to load firmware
from external media anyway?

 2. https://deb.debian.org/debian/dists/bookworm/main/installer-arm64/current/images/netboot/SD-card-images/

In any case, we could probably try and accommodate architecture-specific
issues… by wrapping required calls inside proper conditions, so that
special needs (on release archs or non-release archs) don't negatively
impact regular users.


In the meanwhile, I'll disable both mountmedia calls in the upcoming
upload:
 - to reduce delays when official installation images are just
   self-sufficient (that's why we had that GR in the first place!);
 - to gather feedback/complaints from users for which those calls are
   actually important, so that we can better assess the needs, and the
   possible shortcomings in the historical implementation.


Cheers,

Cyril Brulebois <kibi@debian.org> (2023-01-24):

I thought about this a few hours/days ago but forgot to mention it: we
have a list of requested files, and a list of requesting modules. We
could maintain a list of modules for which a matching package has been
found and installed, and decide to skip mountmedia call(s) if a package
has been installed for each and every requesting module.

This wouldn't help in the “what about a possible higher version of an
existing package” use case, but it would be better for users of
exotic¹ hardware than axing the mountmedia call(s) altogether.

  [¹] as in: not supported by non-free-firmware packages.


Cheers,

Hi,

Cyril Brulebois <kibi@debian.org> (2023-01-24):

I've thought about this a little more, with the following in mind:
netboot in general and netboot SD card images in particular.


# netboot

Netboot images are built by debian-installer, and d-i builds will
certainly not start pulling things from outside main; that being said,
it's been well known that CPIO archives can be concatenated (see
https://wiki.debian.org/DebianInstaller/NetbootFirmware), so it's
possible to take the firmware directory produced by a debian-cd build
(as found on netinst images), and append a CPIO archive of that
directory to a netboot or netboot-gtk initrd. Tested by using direct
kernel loading in virt-manager, stitching things together this way:

    # pick firmware/ as prepared by debian-cd (e.g. tmp/sid/CD1):
    pax -L -x sv4cpio -w firmware | pigz -9c >firmware.cpio.gz

    # append to d-i's initrd.gz:
    cat initrd.gz firmware.cpio.gz > initrd+firmware.gz

Then boot with:
 - Kernel path: netboot/gtk/debian-installer/amd64/linux
 - Initrd path: netboot/gtk/debian-installer/amd64/initrd+firmware.gz
 - Kernel args: vga=791 (to ensure the framebuffer gets enabled)

Beware, the initial initrd.gz is 83M, the firmware.cpio.gz is 186M! That
means I needed to bump my very usual VM's RAM rom 1G to 1.5G, maybe even
2G (depending on which d-i features get used). Things should be less crazy
once we limit debian-cd builds to picking firmware packages from one
section (non-free-firmware only, instead of contrib + non-free + n-f-f),
but that's still an important size bump.

All that ensures a /firmware is found by hw-detect, and firmware can be
detected/deployed/installed as needed, just like in the netinst case.

It shouldn't be a problem to provide a pre-built firmware.cpio.gz
archive with the relevant metadata (produced by debian-cd) in the dep11
subdirectory, for use by hw-detect. We should probably start publishing
them in a less hidden location than the existing:
https://cdimage.debian.org/cdimage/unofficial/non-free/firmware/bookworm/20230123/

Maybe adding a “non-free-firmware” directory directly under this
location?
https://cdimage.debian.org/cdimage/release/current/

After all, each arch-specific directory will have non-free-firmware bits
included, so having an extra directory in the release directory shouldn't
be an issue?

And we might clean things up in:
https://wiki.debian.org/DebianInstaller/NetbootFirmware

Maybe strongly recommend starting from the upcoming new firmware.cpio.gz
file, and only keep the relevant bits (this and that firmware package,
and relevant metadata), to keep the overhead low?


# SD card images (might also be applicable to the upcoming ChromeOS images)

Looking at how those images are built, it seems we have a predefined,
per-arch image size (150M or 200M), and gen-hd-image that's called in two
modes: firmware or partition. The former takes care of the board-specific
part (e.g. firmware.a64-olinuxino.img.gz) while the latter takes care of
the board-independent part (partition.img.gz). Both can be concatenated
with the following command as documented in the README:
https://deb.debian.org/debian/dists/bullseye/main/installer-arm64/current/images/netboot/SD-card-images/README.concatenateable_images

    zcat firmware.<board_name>.img.gz partition.img.gz > complete_image.img

And that works, generating an image with a single fat16, fat32, or ext2
partition with contents merged together, because of the pre-defined image
size.

We could think about bumping the image size, but given the size of the
firmware directory (as in containing the firmware-* packages and metadata,
not the board-specific firmware files), this would seem like a waste of
space for all those not needing firmware files (probably requiring twice
the current size to make sure everything fits)…

I'm not sure how often boards installed via SD card images require
firmware packages, and how many of them, maybe it would be feasible to
include instructions or even a script that only extracts this and that
firmware package (and metadata) from the upcoming new firmware.cpio.gz, so
that it can be appended after partition.img.gz, provided there's enough
space within the defined image size?

But I think it would require at least detecting the underlying partition
type, determining the right numbers, and calling either mcopy or genext2fs
with the right parameters…
https://sources.debian.org/src/debian-installer/20210731%2Bdeb11u7/build/util/gen-hd-image/#L348-L367

All of this assuming that the end results can be appended as the third
part of the <board firmware> + <partition> + <firmware> combination!

(Anticipated caveat: we would likely need to tweak filenames for some FAT
variants, I haven't restricted myself to the good old 8.3 format.)


# Summary

For regular netboot images, existing firmware addition should keep
working, even if we should produce a slightly different firmware CPIO
archive (adding metadata), publish it into a more discoverable place, and
update instructions accordingly. For those *not* adding firmware stuff, it
might make sense to keep mountmedia calls; an easy way to find out when
those might make sense would be to check if a (/cdrom)/firmware directory
exists at all (it would on netinst, wouldn't on netboot, would on netboot
augmented with firmware stuff).

For netboot SD card images, we are space-constrained, so *maybe* adding a
few selected firmware packages (and metadata for automated use by
hw-detected) could work, but I suppose the general case might end up users
don't add anything. Especially since that's not just about concatenating
compressed initrds together, we would need some more elaborate glue…
All this reinforces my impression regarding adding (/cdrom)/firmware
detection. Since we're talking about an installation image with a single
partition, maybe those mountmedia calls work more reliably when there's
external storage plugged in anyway? Again, I don't know much about how
mountmedia works, or about problems/limitations users might have
encountered.

I might prepare the upcoming D-I release without mountmedia at all though,
that's probably the best way to get feedback about it.


Cheers,

This sounds like a nice solution for the SD card images.
I myself always use a wired connection, but I saw yesterday that someone tried
to use d-i for RockPro64 but didn't get any output on screen and then tried
again using their TV ... but then it needed to use a wireless connection and I
actually had no idea how that could/should be provided during install.

Giving ppl the option to concat a firmware archive if they need it, sounds like
an elegant solution to me :-)

Please ignore me. Apparently I didn't fully and/or properly understand.
The ('wireless') firmware archive is likely to be very involved to construct,
let alone other issues (like size) it may add.

...

Well, empty space does compress quite well, at least!

Though a larger image would take a bit longer and burn a few extra write
cycles for those who did not need to actually use the space.

I do not know that we are all that size-constrained; it is hard to come
by USB sticks or SD cards that are less that 4GB these days. So even
bumping the image size to 500MB or even 1GB does not seem unreasonable
to me... I would rather err on the side of slightly larger than
currently needed; maybe there are use-cases I am not considering?

Another option is to concatenate the firmware part directly to the
initrd image manually, though it is a bit of a cumbersome
process... soemthing like:

  gunzip partition.img
  mount -o loop partition.img /mnt
  cat kernelfirmware.cpio.gz >> /mnt/initrd.gz
  umount /mnt
  zcat firmware.BOARD.gz partition.img > complete.img

Though I realize that is quite a few extra steps for an already slightly
complicated manual two-part SD card image process... but it could
probably be scripted; it is not absurdly complicated. (presuming you
have the kernel-firmware.cpio.gz creation sorted out)

This technically only requires changing the size of the partition.img to
make more room for the kernel firmware image, but otherwise would be
identical to the current process for those who did not need to add the
firmware.

Another option would be to use a growable filesystem, and then grow it
to whatever size needed? Are FAT filesystems reasonably growable?

Thanks for all your work on this!

live well,
  vagrant

Hi Vagrant,

Thanks for your feedback!

Vagrant Cascadian <vagrant@debian.org> (2023-01-25):

:D

I have no idea how much people care how much time is spent copying things
around, but I suppose that at least for those repeatedly copying things
around (e.g. for debugging purposes), more writes could mean accelerated
wear-and-tear?

(The only things in that area I'm familiar with is Pi images that contain
full-blown systems, which are in the 1-2G ball park already.)

Yeah, I've seen you notice that you were aiming to allow for a little more
room, when you bumped sizes, rather than increasing them minimally. I'm
just wondering whether it makes sense (how often is that really useful?).

That looks much easier than anything we could come up with if we were
trying to concatenate 3 parts (the third one would need to be created
depending on the first one — fat or ext2). Requiring mount could be
considered a tad dangerous, but big red warnings and all that…

I'm working on it right now, trying to also reduce code duplication in
debian-cd for easier long-term maintenance. I should have some proof of
concept by tomorrow, but we need to find a better location that under
unrelease/, so it won't be deployed right away.

Yes!

Just to be sure, are we talking about something like the following?
 - cat both parts as usual to get complete_image.img (via README);
 - fallocate to make it bigger for those who want more stuff;
 - fdisk to extend the partition to the new end of disk (making sure
   not to change the beginning, firmwares are picky as you know; not
   lose the signature, etc);
 - fatresize or resize2fs to extend the filesystem;
 - mount, copy, umount;
 - hope that none of this changed tricky parts that would break the
   board-specific firmare/bootloader/config/etc.

I've used fatresize on FAT32 partitions in the past (for USB storage
on Intel-based machines, but nothing firmware/bootloader-related) quite
successfully. A quick grep through its source report this claim:

    The FAT16/FAT32 non-destructive resizer.

All of this (if it works) seems reasonable enough, or at least I don't
think should be a big issue for experienced users.

But I suspect less advanced users wanting to toy with an ARM board would
probably just follow the README to prepare the installer image, and rely
on an extra USB stick to hold the firmware part (and rely on mountmedia).


But again, I have no idea how common it is for the boards we support in
the installer to require loading firmware files during the installation
process…


Cheers,

Nor I, really... so yes, just some extra writes that might be needless
in most cases.

It is mostly useful with new kernel versions where the module
names/dependencies/indirect-dependencies have changed between kernel
versions, and I haven't yet figured out which kernel modules go in which
.udeb files.  Similar for new boards, just trying to figure out which
modules are needed at all.

The main approach I use is appending all modules (or more selective
modules) from /lib/modules/VERSION/ in a cpio archive that is appended
to the debian-installer initrd.gz... then compare the lsmod results from
a booted d-i... it is not the simplest process with many iteration
cycles; any reduction in required steps is a huge benefit for that!

It does seem similar to the use-case of loading extra firmware in the
installer... appending selective firmwares or all the firmwares to the
initrd as a way to make sure it is loadable early in debian-installer.


Another use-case for more-free-space-by-default are things like
raspberry pi boot firmware, which need to be loaded from the 1st FAT
partition on the microSD. Do not recall off the top of my head how large
rpi boot firmware is. I have manually copied it onto debian-installer SD
card images in the past, possibly doing a partition/filesystem resize
dance...

Ok, good!

Yeah, figured that was in-process :)

Yeah, that is basically what I was thinking.

The biggest pain point is some boot firmware likes to be on particular
raw offsets. Some offsets are incompatible with GPT (allwinner)... some
load data from offsets that edge into 4MB (ti?) or even 16MB
(rockchip!)... I think most partitioning tools default to creating with
a 2MB offset for the first partition. Some models of raspberry pi
require the boot firmare to be on the 1st fat partition.

As long as you leave those first 16MB clear and don't use GPT
partitioning... dare I say it ... I'm not aware of other issues that one
would get tripped up on!

Somehow I missed the existance of this, thanks!

Yeah, that would likely cover most use-cases.

Some rarer boards might be very limited in boot or installation media
selections... "You need a USB port *and* a microSD slot? well, it
technically has both... warm up your soldering iron... and get to it!"

I have certainly biased my selection of boards over the years towards
ones that did not need it... so cannot offer much help there.

Quite a few boards only have wifi (or bluetooth networking?), which
usually requires firmware (if supported at all), usually worked around
with USB ethernet or USB wifi with free firmware... or USB-gadget
ethernet... or a few other exotic options. :)


live well,
  vagrant

Hello,

I'm actually affected by the call to mountmedia in
check-missing-firmware. Especially by:

"
  - Insufficient lookup (something about stopping at the first partition
    that's found or something, but I don't know the specifics, and I'm
    not diving into what “mountmedia driver” does or should be doing).
"

Indeed on my EFI system, the first partition is the ESP that is FAT type
(/dev/sda1) while the firmwares are on an USB stick that is on /dev/sdc1
  (but could also be on /dev/sdd3 or wathever). Since mountmedia will
return true for the first partition that it can mount, in my case it
will only mount /dev/sda1 to /media and stop searching on the other
partitions, even if /dev/sda1 doesn't have the required firmware.

This probably explains why many people are not able to install their
firmware this way.

In my case, the required firmware is for "b43" driver. The problem is
that no package contains the firmware, so it is not available on
bookworm installer medium. Actually Debian has a package that will
download, extract and install the firmware: firmware-b43-installer.

Although the package is available on the installation medium, it is
useless because without b43, no wifi, without wifi there is no internet,
and without internet the installer package is of no use.

I could prepare the b43 firmware files by installing the package another
system and copying them to the USB stick into /firmware/b43 for use on a
system that connects through wifi, but with the bugs of
"check-missing-firmware" script it is not possible to access them. This
leads to some frustration.

Would it be possible to fix the check-missing-firmware script so that it
scans all devices/partitions until it finds the appropriate firmware files?

I also noticed some issues in what is displayed to the user in the
installer (graphical). For example the UI asks for each firmware file if
we want to search for it. Maybe it should ask once for a set of files,
instead of asking for each single file.

My issue is reported also here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1088295

The process to load external firmware is already rather complex. I'm not
so in favor of having to complexify it more by creating a cpio archive
to append to init.rd. By the way, one would have then to inject that
into the ISO image?

Regards
Fab

Long known issue.

IIUC license issue prevents Debian from shipping this firmware.
I built my own .deb package containing b43 firmware files because the
search process for firmware packages is less broken than the one for
"loose" firmware files (it stops at the first filesystem containing .deb
or .udeb packages instead of the first mountable one).

Not sure missing firmware files are searched in /firmware
subdirectories. Did you try to put the firmware directly at the root or
in /firmware ?

Maybe it is time I submit my patches for mountmedia and hw-detect again?

They are searched in /firmware. I put the files into it on the drive that is mounted first and they were taken from that location.

If you have a patch it would be nice that it were considered for merging into the installer.

Regards