Hi,

I followed the instructions given on
https://www.raspberrypi.org/forums/viewtopic.php?t=282839
to install Debian on a Raspberry Pi using the UEFI firmware
available for the raspi.

This involves unpacking the official Debian 11 image into an ESP
partition on an USB medium and using the Rasberry Pi UEFI firmware
to boot the installer via UEFI => grub => Linux. Once the
installe has started, this feels 100 % like a standard Debian
installation.

I experienced the following issues:

"Detecting Hardware" complains about missing firmware. I guess
that's Wifi related, since the Wifi Interface doesn't work in the
installed system and the 1000Base-TX interface works fine without.

Requesting German keyboard layout does not work in the beginning
of installation, domain name and user name configuration still happens
with the default US keyboard layout. Later, when configuring the
network mirror, the keyboard layout has been changed to the requested
german layout. I guess this happens during base system installation.

For very strange reasons, the installed system ends up without
/sbin/start-stop-daemon despite the dpkg package being in state "ii".
This is reproducible. In the installer, I see a temporary diversion of
/sbin/start-stop-daemon, but in the installed system, dpkg-divert --list
doesn't show that.

/var/lib/dpkg/diversions-old has those three lines:
/sbin/start-stop-daemon
/sbin/start-stop-daemon.REAL
:

There is no /sbin/start-stop-daemon.REAL though.

And, the installed system does neither have /var/log/installer nor
/var/log/debian-installer. This is also reproducible :-(   Where would I
find the files that get copied to /target/var/log/installer in the
still running installer?

Greetings
Marc

Hallo Marc,

Marc Haber <mh+debian-packages@zugschlus.de> (2021-06-08):

Fun!

AFAICR (I'll get back to that soon enough) you should get a message
indicating which files were expected, so that you don't have to guess.

Was that with the text or graphical installer?
 - debootstrap
 - debian-installer-utils
 - apt-setup (code apparently copied from d-i-utils until some clean-up
   happens, which never did)

Simply /var/log/syslog for the main log. Other files are mentioned
there:
https://salsa.debian.org/installer-team/installation-report/-/blob/master/debian/installation-report.postinst


Cheers,

Yes!

The Raspi 4 has both Wifi and Ethernet interfaces named brcmsomething,
so it's still kind of guessing, but I have verified in the mean time
that it's actually really the Wifi firmware.

Text installer ("expert"). I have used the graphical installer just a
handful of times in two decades.

Since my default domain types differently with US and DE keyboard, I
guess this might be an actual regression, I would have noticed this in
earlier versions of the installer.

Is there anything you need from me to go on?

Greetings
Marc

Marc Haber <mh+debian-bugs@zugschlus.de> (2021-06-08):

Alright, thanks for confirming.

OK, good guess then, but checking never hurts.

I guess someone could try various images outside the Pi 4 setup and see
whether/when that regressed. Given the following issue (ssd), I wouldn't
rule out some Pi-specific, but such a trivial thing seems a little
strange… That being said, I test the graphical installer most of the
time, and I don't know the specifics of the text version by heart.

If you were to retry the installation, saving /var/log/syslog manually
before rebooting into the installed system would be useful. Without it,
I fear much guessing would be needed to try and figure out what happened
on that system specifically. Unless someone else has better ideas of
course.


Cheers,

Hi,

Cyril Brulebois <kibi@debian.org> wrote (Tue, 8 Jun 2021 17:07:52 +0200):

I checked with rc1 and the *brandnew* rc2 netinst images on amd64 qemu
VM machine (text installer), and I cannot reproduce such issue with wrong
keyboard layout; German is correctly available immediately after the
"Configure keyboard" step.

So, I guess that a Pi-specific thingy somehow.


Holger

On the Pi, keyboard-configuration, console-data etc only seem to get
installed at the end of the "installing the base system" step, after
which the keyboard setting works.

The more pressing issue IMO is the deletion of /sbin/start-stop-daemon.

Greetings
Marc

Repeating this:

If you were to retry the installation, saving /var/log/syslog manually
before rebooting into the installed system would be useful. Without it,
I fear much guessing would be needed to try and figure out what happened
on that system specifically. Unless someone else has better ideas of
course.


Cheers,
--
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

I apologize for waiting for delivery of a new usb stick so that the
new installation would not be a complete waste of time.

I have attached the installer's syslog of a new installation. All issues
that I reported were observed during this installation.

Let me know if I can be of additional help. I appreciate the swiftness
of your reactions, this is very helpful.

Greetings
Marc

Marc Haber <mh+debian-bugs@zugschlus.de> (2021-06-09):

Thanks, this sticks out:

    May 25 11:15:32 localechooser: info: System locale (debian-installer/locale) = 'en_US.UTF-8'
    May 25 11:15:32 localechooser: info: Set debian-installer/language = 'en_US:en'
    May 25 11:15:32 debconf: Setting debconf/language to en_US:en
    May 25 11:15:32 main-menu[261]: INFO: Falling back to the package description for brltty-udeb
    May 25 11:15:32 main-menu[261]: INFO: Falling back to the package description for brltty-udeb
    May 25 11:15:34 main-menu[261]: INFO: Falling back to the package description for brltty-udeb
    May 25 11:15:34 main-menu[261]: INFO: Menu item 'console-setup-udeb' selected
    May 25 11:15:37 main-menu[261]: (process:1290): setupcon: gzip is not accessible. Will not save cached keyboard map.
    May 25 11:15:37 main-menu[261]: (process:1290): setupcon: gzip is not accessible. Will not save cached keyboard map.
    May 25 11:15:37 main-menu[261]: (process:1290): ckbcomp-mini: /usr/share/console-setup/pc105.ekmap.gz does not exist
    May 25 11:15:37 main-menu[261]: (process:1290): setupcon: gzip is not accessible. Will not save cached keyboard map.
    May 25 11:15:37 main-menu[261]: (process:1290): setupcon: gzip is not accessible. Will not save cached keyboard map.
    May 25 11:15:37 main-menu[261]: (process:1290): /bin/setupcon: line 722: can't create /etc/console-setup/cached_setup_keyboard.sh: nonexistent directory
    May 25 11:15:37 main-menu[261]: (process:1290): /bin/setupcon: line 730: can't create /etc/console-setup/cached_setup_font.sh: nonexistent directory
    May 25 11:15:37 main-menu[261]: (process:1290): /bin/setupcon: line 748: can't create /etc/console-setup/cached_setup_terminal.sh: nonexistent directory
    May 25 11:15:37 main-menu[261]: (process:1290): chmod: /etc/console-setup/cached_setup_keyboard.sh: No such file or directory
    May 25 11:15:37 main-menu[261]: (process:1290): chmod: /etc/console-setup/cached_setup_font.sh: No such file or directory
    May 25 11:15:37 main-menu[261]: (process:1290): chmod: /etc/console-setup/cached_setup_terminal.sh: No such file or directory
    May 25 11:15:37 main-menu[261]: (process:1290): ckbcomp-mini: /usr/share/console-setup/pc105.ekmap.gz does not exist

and might explain why your keyboard settings don't work right after
selecting the layout.

This seems also strange (even if I must confess I've never grepped for
s-s-d lines before):

    Jun  9 16:20:40 base-installer: Using CD-ROM mount point /media/cdrom/
    Jun  9 16:20:40 base-installer: Identifying...
    Jun  9 16:20:40 base-installer: [27dc08d5c910ea7df2dee6e03e127e52-2]
    Jun  9 16:20:40 base-installer: Scanning disc for index files...
    Jun  9 16:20:41 base-installer: Found 1 package indexes, 0 source indexes, 1 translation indexes and 0 signatures
    Jun  9 16:20:41 base-installer: Found label 'Debian GNU/Linux testing _Bullseye_ - Official Snapshot arm64 NETINST 20210607-08:56'
    Jun  9 16:20:41 base-installer: This disc is called:
    Jun  9 16:20:41 base-installer: 'Debian GNU/Linux testing _Bullseye_ - Official Snapshot arm64 NETINST 20210607-08:56'
    Jun  9 16:20:41 base-installer: Copying package lists...
    Jun  9 16:20:41 base-installer: ^MReading Package Indexes... 0%^M
    Jun  9 16:20:41 base-installer: ^MReading Package Indexes... 0%^M
    Jun  9 16:20:41 base-installer: ^MReading Package Indexes... Done^M
    Jun  9 16:20:41 base-installer: ^MReading Translation Indexes... 0%^M
    Jun  9 16:20:41 base-installer: ^MReading Translation Indexes... Done^M
    Jun  9 16:20:41 base-installer: Writing new source list
    Jun  9 16:20:41 base-installer: Source list entries for this disc are:
    Jun  9 16:20:41 base-installer: deb cdrom:[Debian GNU/Linux testing _Bullseye_ - Official Snapshot arm64 NETINST 20210607-08:56]/ bullseye main
    Jun  9 16:20:41 base-installer: Repeat this process for the rest of the CDs in your set.
    Jun  9 16:20:41 base-installer: Ign:1 cdrom://[Debian GNU/Linux testing _Bullseye_ - Official Snapshot arm64 NETINST 20210607-08:56] bullseye InRelease
    Jun  9 16:20:41 base-installer: Err:2 cdrom://[Debian GNU/Linux testing _Bullseye_ - Official Snapshot arm64 NETINST 20210607-08:56] bullseye Release
    Jun  9 16:20:41 base-installer:   Please use apt-cdrom to make this CD-ROM recognized by APT. apt-get update cannot be used to add new CD-ROMs
    Jun  9 16:20:41 base-installer: Reading package lists...
    Jun  9 16:20:41 base-installer:
    Jun  9 16:20:41 base-installer: E: The repository 'cdrom://[Debian GNU/Linux testing _Bullseye_ - Official Snapshot arm64 NETINST 20210607-08:56] bullseye Release' does not have a Release file.
    Jun  9 16:20:41 base-installer: warning: apt update failed: 100
    Jun  9 16:20:41 base-installer: dpkg-divert: warning: diverting file '/sbin/start-stop-daemon' from an Essential package with rename is dangerous, use --no-rename

And that last line pops up frequently in the rest of the log.

I'm not exactly sure what led to this, but there's definitely something
fishy in this installation process. The fact some directories seem to be
missing early isn't reassuring, to be honest. Could this be an artifact
of the say creative way to start the installer?


Cheers,

Hi Cyril,

thanks for the analysis.

Where would the Installer look for gzip? I can check at this stage
whether gzip is there.

Does the Installer divert start-stop-daemon to prevent services from
being started?

The way to start the Installer is only creative for a Raspberry Pi. It's
remarkably similiar to what an amd64 EFI box experiences: The UEFI
firmware is started (on a PC it's the first thing that runs, on the
Raspi it is called from the binary firmware blob that is read from the
boot medium and runs on the GPU), the UEFI firmware fires up grub which
in turn invokes the installer.

I am open to suggestsions for other ways to start the installer.

Greetings
Marc

That's a sensible strategy :)

Could either of you (Cyril, Diederik) recommend where I should ask
(and/or clone this bug) to follow up on the firmware filename issue,
given that the filename(s) seem to be generated from the kernel
module?

(as a recap: the brcmfmac module attempts to load a file of the format
"brcmfmac43455-sdio.Raspberry Pi Foundation-Raspberry Pi 4 Model
B.txt" instead of "brcmfmac43455-sdio.txt" -- I saw the same thing
during my install, with string adjustments for brcmfmac43456 and Pi
400)

I think that that's likely to be the cause of the firmware-not-loaded
problems in installation-reports #989593 and #1035392 (that second
report is from me, earlier today).  Even with the 'Contents-firmware'
file-to-package mapping, we won't find the relevant firmware file if
the name is wrong.

Thank you; yep, I've followed _most_ of that (and arrived back here
again).  I will admit that most of what I've cognitively loaded from
it is "this script could use refactoring post-bookworm", and have not
processed the complete details.

Regards,
James

Hi James,

I've just finished a very long debugging session[1], so the following is
really best effort and probably not 100%.

 1. https://lore.kernel.org/all/20230428223500.23337-1-jim2101024@gmail.com/T/#m846b07884ef687d3669654c782aa3b61216f15b7

James Addison <jay@jp-hosting.net> (2023-05-02):

It looks to me someone should understand the Linux kernel code, and
possibly where inputs/variables come from (there might be stuff coming
from the DTB, some bootloader thing, etc.), and why spaces show up in
there. Then decide whether the “external” source (if any!) or the kernel
should be adjusted to use more straightforward names.

Yes; my point was really to say “sorry it's _so_ not perfect yet”, but
seeing this kind of things get uncovered is definitely *not* unexpected…

Let's take a step back and check summarize where we come from, so that
the picture is not only clear in my own head. :)


During the bullseye release cycle, I've tried to deal with GPU problems
by adding modalias support (no specific modules in d-i means no messages
in dmesg, meaning we needed something else), to avoid abysmal experience
once reaching the post-install reboot. (And being able to work on that
only late in the release cycle had some impact on the release date…
which is definitely *not* a feeling I wish anyone else to experience!)

During the bookworm release cycle, getting official support for firmware
packages has been basically a solo effort in each and every component
(even if I got some help from firmware package maintainers and members
of some infrastructure teams, you know who you are <3). While Pascal has
been submitting a bunch of patches for hw-detect already, there are
still some gaps to fill. But again, I'll take *not* supporting _some_
use cases right away over risking losing support for _most_ use cases;
that's a no-brainer, especially just days away from the next RC(s), and
from the actual release.

But enough meta-talk now! :)


Cheers,

Mystery may be (partially) solved.  Responses inline below.

Agreed.  The 'compatible' string is what seems intended for inclusion
into the fw filename request - by the driver authors,
linux-firmware.git, ourselves, and the RPi operating system.

After editing and rebuilding the Device Tree (DTS) files, and
deploying those changes to the system, I can confirm that adjusting
the 'model' field value in there has no effect on the requested fw
filename.

The system's dmesg includes this line:

  DMI: Raspberry Pi Foundation Raspberry Pi 400/Raspberry Pi 400, BIOS
UEFI Firmware v1.34 1 2/16/2022

As Cyril said though.. this can't (shouldn't) be genuine DMI.  So
what's going on?

It seems that the cause may be this:

The default settings within the EDK2 UEFI, under "Device Manager" ->
"Raspberry Pi Configuration" -> "Advanced Configuration" contained a
key labeled "System Table Selection" that was set to "ACPI".  Changing
that value to "Devicetree" and then booting caused the correct,
expected fw filename to be requested:

  firmware: failed to load brcm/brcmfmac43456-sdio.raspberrypi,400.bin (-2)

Agreed here too - we can (and should) only make adjustments that are
acceptable and compatible for adjacent components.  I think we're
aligned with upstream here, it's simply that some other component --
and at the moment, that appears to me to be the EDK2 UEFI -- is
producing an unusual effect.

Cheers,
James

Hi,

James Addison <jay@jp-hosting.net> (2023-05-03):

Did those modifications stay in place once you switched to Device Tree
in your bootloader configuration? Just wondering whether you tested two
cumulative changes (DTS tweaks + switch from ACPI), or independent ones
(DTS tweaks, then switch from ACPI but using pristine DTB files).

I know nothing about hardware, it only seemed like a red flag to me,
that's why I chose not to dig deeper at the time.

I don't have any Pi 400 and haven't been following what the stock
configuration is (and sorry I didn't read the whole backstory)… if EDF2
UEFI comes by default, or is recommended, or fixes/works around bugs,
and in the end is expected to be relevant and widely used, and if ACPI
is indeed some kind of default setup, it would be best if we were to
support that.

I suppose that would mean either having the relevant files/symlinks in
the firmware package *and* d-i support for it (hw-detect limitations…);
or have some on-the-fly conversion in the Linux module so that it ends
up requesting files that are actually in the firmware package, and that
d-i can work with, without requiring any changes?


Cheers,

That was tested cumulatively, yep - applied the DTS tweaks, found no
change, and then updated the UEFI setting from ACPI to Devicetree,
resulting in the expected fw requests.

Since then I've reverted the DTS tweaks, and the correct behaviour continues.

After that I reverted the UEFI setting back to ACPI briefly.  I forget
what I was checking, but the problem reappeared, and now I'm back to
the expected behaviour (no spaces in the fw filenames) with the
Devicetree setting.

I'd defer to people with more familiarity of the ecosystem on these
kind of questions, although am also keen to learn more.

Some thoughts:

  * Maybe there's a bug to report in the upstream Linux brcmfmac
driver here; are there better alternatives than DMI that it could use
to determine fallback filenames?  (again, I'm not sure, but can follow
up on that)
  * Perhaps Devicetree is a better default in EDK2 for ARM systems?
(that wouldn't solve the root cause, though)
  * Alternatively, if EDK2/RPI4-UEFI became Debian-packaged (excluding
the brcm firmware, because that's already in raspi-firmware), _and_
could be installed on the ESP partition by d-i (it's beyond my
experience to say whether that's a good idea...), then perhaps it
could be preconfigured (or d-i could request) Devicetree at
install-time.

If & when attempting this again - it could be a while, this took some
energy - then I would probably experiment with u-boot as a comparison.

At the moment, I think that fixing this in the brcmfmac driver would
resolve the problem in a bootloader-agnostic way, so that seems worth
exploring.

Symlinks seem like they'd be a reasonable short-term workaround in our
packaging - but likely maintainer discretion on that one, as usual (if
that were me, it would help to know that it would be a temporary
measure while other problems were being resolved).

(sorry for what I think may be an overly-large reply list here - I'd
prefer that than to have anyone miss some hopefully-relevant details)

Hello all and sorry for pinging in late into this conversation.

First of all, as someone who noticed the original issue but never got
around to properly report or investigate it, a big thanks go to the
people here who have been devoting time and effort trying to get to the
bottom of it, and offering good clues as to what's causing the issue.

Note that, in case you think there may be something that we can improve
in the SMBIOS data reported by the UEFI firmware (which is currently
generated from the source code at [1], with the full output from a
Raspberry Pi 4, from UEFI Shell's smbiosview command at [2]) we can look
into updating the UEFI firmware to alter the data we output.

Please note that the reason why the Raspberry Pi UEFI firmware defaults
to ACPI is so that this ARM device follows the relatively new ARM SBBR
standard [3], which we (hopefully) expect more and more ARM64 based
device to follow.

Obviously, with the idea of not having ARM based device that are
constrained to a single OS (be it Windows, Linux, BSD or something
else), and considering that Windows and Device Tree don't work together,
you want to go with a mode of operation that isn't Linux specific,
which, even if ACPI has its drawbacks, pretty much forces the use of
ACPI over Device Tree. Else, you have Linux going into the exact
Microsoft strong-arm tactics that it should strive to avoid...

Well, if you package the UEFI firmware with Debian, I guess you can do
whatever you want. You may however run into (a very limited number of)
folks who might try to dual boot between Debian and Windows on Pi, just
like you run into folks who may want to dual boot between Debian and
Windows on x86 based PC... though I have to concede that, considering
that, with Windows on Pi becoming a headache ever since Microsoft made
Windows 11 22H2 incompatible with the ARM revision used in the Pi 4, and
most of the dual boot crowd expected to be using separate devices, each
with their own UEFI firmware (since the UEFI firmware is not actually
written into SPI but resides on a removable device), it's probably
something that's unlikely to be a big deal.

So I guess I can only mention my *preference* of having Debian striving
to fix the issue with ACPI, rather than work around it by switching to
Device Tree, knowing, once again, that if it boils down to something
that we can alter in the SMBIOS data (and that passes the SMBIOS
compliance tests, which are themselves part of SBBR compliance), we
should be able to work out something in the UEFI firmware if needed.

Regards,

/Pete

[1]
https://github.com/tianocore/edk2-platforms/blob/20e07099d8f11889d101dd710ca85001be20e179/Platform/RaspberryPi/Drivers/PlatformSmbiosDxe/PlatformSmbiosDxe.c
[2] https://pastebin.com/3vLr1ZVJ
[3]
https://community.arm.com/arm-community-blogs/b/internet-of-things-blog/posts/arm-server-standards-part-2-sbbr-specification-released