#1105000 installation-reports: Install media device disappears during install, reappears unmounted, failing install

#1105000#5
Date:
2025-05-09 17:06:43 UTC
From:
To:
Boot method: USB flash drive (tried two different ones)
Image version: Trixie Alpha 1 netinst amd64
    [https://cdimage.debian.org/cdimage/trixie_di_alpha1/amd64/iso-cd/debian-trixie-DI-alpha1-amd64-netinst.iso]
    Also tried weekly snapshot amd64 netinst image as of 2025-05-01
Date: 2025-05-07

Machine: Supermicro H13SSL-N motherboard with AMD EPYC 9115 processor


Base System Installation Checklist:
[O] = OK, [E] = Error (please elaborate below), [ ] = didn't try it

Initial boot:           [O]
Detect network card:    [E]
Configure network:      [O]
Detect media:           [O]
Load installer modules: [O]
Clock/timezone setup:   [O]
User/password setup:    [O]
Detect hard drives:     [O]
Partition hard drives:  [E]
Install base system:    [O]
Install tasks:          [O]
Install boot loader:    [O]
Overall install:        [O]


Comments/Problems:

I'm excited for the upcoming release of Trixie, so, as requested, here's an
installation report of how it went when trying out the current state. I hope
it's useful.

Overall, I was able to successfully install the system -- but I did
encounter one severe snag along the way that I consider to be a major
problem; at least for my hardware configuration. I can't speak to how
widespread it is or whether anyone else is likely to encounter it.

There are also a couple of smaller rough edges that I'll list at the end.


Issue #1 - Installation media device (/dev/sda) disappears, failing install

Upon reaching the stage of the installer where one sets up disk partitions,
I got the following error message:

   [!!] Partition disks

   There was a problem reading data. Please make sure you have inserted the
   installation media correctly. If retrying does not work, you should check
   the integrity of your installation media (there is an associated entry in
   the main menu for that).

   Failed to copy file from installation media. Retry?
   <Yes>  <No>

Retrying did not help. Selecting "No" simply moved on to the disk
partitioning step, with no immediately apparent ill effects... however,
after setting up partitions, the installer would fail completely.

Backing out to the main menu and checking the integrity of the media
predictably also fails with an error ("No valid Debian installation media").

Dropping into a shell reveals the odd state of the system:

  # mount
  [...snip...]
  /dev/sda1 on /cdrom type iso9660 (ro,relatime,nojoliet,[...snip...]

  # ls /cdrom
  [no output]

  # ls /dev/sda*
  ls: /dev/sda*: No such file or directory

  # ls /dev/sd*
  /dev/sdb /dev/sdb1 /dev/sdb2

Fortunately this also suggests a path to recovery; namely mounting the
newly-appeared /dev/sdb1 to /cdrom where the installer ought to be:

  # umount /cdrom
  # mount /dev/sdb1 /cdrom
  # ls /cdrom
  EFI   README.txt   doc   md5sum.txt   [...snip...]

From here, one can return to the menu and continue the install successfully.

Unfortunately I have no idea what could be triggering the
disappearance/reappearance of the drive. That it happens at the partitioning
step suggests it's perhaps somehow related to probing for disks/filesystems, or
perhaps it has to do with the firmware error I also got...


Issue #2 - Error about non-free Renesas firmware upon detecting network hardware

Early in the install process, when detecting network hardware, I get an error
about missing non-free firmware ("Some of your hardware needs non-free firmware
files to operate [...]"). The missing firmware file is 'renesas_usb_fw.mem'.

However, that firmware file doesn't seem to be available in any package?
https://packages.debian.org/search?suite=trixie&arch=any&searchon=contents&keywords=renesas_usb_fw.mem

This is the only reference to it I can find:
https://sources.debian.org/src/linux/6.12.27-1/drivers/usb/host/xhci-pci-renesas.c/?hl=53#L53

Ignoring the error and continuing without loading firmware from removable media
appears to be OK, and I haven't yet noticed any ill effects. I have to admit I
don't know what this particular firmware is supposed to do, so I don't know what
to check for.

I'm calling it out for a couple of reasons:
  - It's maybe weird that it's showing up at the stage of detecting network
    hardware, since it seems to be a USB controller?
  - Could it possibly be related to the disk error above?
  - Should the firmware be present in some package, e.g. firmware-misc-nonfree?
  - If it's actually not an issue, should the error message be appearing?


The remaining issues are much more trivial, arguably just subjective aesthetics.


Issue #3 - After successful media integrity test, prompted to insert media

If one goes into the top-level installer and runs an installation media
integrity check, it appears that one is prompted to insert that same media even
if the check is successful. If the check succeeds, surely nothing needs to be
inserted? Just a trivial little oddity, no big deal.


Issue #4 - Toggling the "bootable" flag on partitions does nothing/doesn't work

When editing partitions, there's a line for toggling the "Bootable" flag.
However, attempting to toggle it doesn't actually change its state. Presumably
this is because the EFI system partition is present and is used for booting? In
that case, this option that isn't expected to be used and which doesn't work
should perhaps not even be present?


Issue #5 - Apt prompts to modernize sources instead of already being modernized

I like the new sources format, and it's nice that Apt can auto-modernize for me.
But shouldn't it already come in the "correct" new format out-of-the-box, and
only use this fallback for other sources that the user may have added?


Issue #6 - 'systemctl status' says system is tainted in default configuration

It seems that systemd wants /usr/bin/ and /usr/sbin/ to be the same directory.
That's fine; I wouldn't mind that change... but it's not the actual current
default state of a new freshly installed Debian system. In my humble opinion, a
major component of Debian should not be claiming that Debian's default setup is
misconfigured.

It appears this was previously reported as #1085370, but the maintainer closed
it as "wontfix". I agree with the original reporter of that bug that there ought
to be a Debian patch to suppress this particular warning in this prominent area.


Thank you again for the exciting progress on Trixie!

#1105000#10
Date:
2025-05-09 20:06:27 UTC
From:
To:
Hi,

Am 9. Mai 2025 19:06:43 MESZ schrieb Felix Crux <felixc@felixcrux.com>:: installation-reports

The alpha1 image has gotten a bit outdated.
Maybe you could retry with the netinst.iso from
https://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/

... just to make sure we are not hunting a bug which is no longer there ...


Thanks
Holger

#1105000#15
Date:
2025-05-10 00:09:04 UTC
From:
To:
Hi Felix,

Felix Crux <felixc@felixcrux.com> (2025-05-09):

This is really weird to me.
https://salsa.debian.org/kernel-team/linux/-/merge_requests/675

Unfortunately (to the best of my knowledge at the time, and I didn't
check if that changed recently) the firmware file is not distributable
and cannot be shipped in any packages. We could and probably should
filter out this specific filename in d-i, like iwl-debug-yoyo.bin
(that'd be in hw-detect's check-missing-firmware.sh). But if it's
actually required to get the installation through, it'd be best to keep
the warning, and get a pointer added somewhere in the installation
guide. I suppose this depends on whether this has any relationship with
issue #1 (sda disappearing midflight).

It's spotted by the firmware check running at the network detection
step, but is only likely to matter (in that context/at that step) if
network devices are connected through some USB lines controlled by USB
hosts like µPD720201 or µPD720202 (see https://bugs.debian.org/1032671
for more context).

Are you trying to win some “best bug report”/“spot on” challenge?

Yes weird but explainable (see above). I realize how this can be
confusing, and I'm sorry for that.

Maaaybe. Not enough braincells/time to dig into this though, sorry.

I don't think it can (see above).

If it turns out to be only “log noise” (see beginning of my answer), it
should be silenced. Otherwise, we would need to figure out what counter
measures can be implemented when people encounter this, and document
them.

Feeling responsible for the renesas thing, I've taken the time to reply
until this point and I'll (almost) stop here, leaving others to comment
on the rest…

[…]

We have a merge request for that, and checking/merging it is on my list
of things before the next release.

[…]

Hats off to you for an exciting and thoughtful installation report!


Cheers,

#1105000#20
Date:
2025-05-10 00:12:59 UTC
From:
To:
Nothing like hitting the Send button to have another idea pop up.

Cyril Brulebois <kibi@debian.org> (2025-05-10):

Possible hint for people trying to understand this bug: forgetting
possible firmware fun for a second, could this be some kind of hardware
fault, like some power surge, some bad cable, some bad storage, etc.? Of
course, looking into the installer's dmesg (made persistent in
/var/log/installer/syslog alongside the rest) is likely to be a good
place to start instead of relying on a (very) (wild) hunch of mine…


Cheers,

#1105000#25
Date:
2025-05-10 06:05:28 UTC
From:
To:
That was my first thought. But the fact that it happens after the
missing firmware detection may not be just a coincidence. Could it
happen when modules are unloaded and reloaded ?

Yes, we really need the installer syslog.

#1105000#30
Date:
2025-05-11 02:18:09 UTC
From:
To:
Thank you for all the thoughts so far -- I'm glad it's at least an
interesting case!

I can confirm that the issue still appears when using the latest daily amd64
netinst image as of today, May 10th (from https://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/)

I have attached the syslog from that latest installation attempt.

Hopefully this also helps narrow it down: I can confirm that the device has
disappeared by the time I get the error message about missing firmware while
detecting network hardware. That is the earliest in the install process that
I can switch to another terminal to verify it; and by then it's gone. That
is to say, it's not happening later, e.g. at the partitioning step.

With regards to the possibility of a hardware glitch: I of course can't rule
it out, but I can confirm I've tried with two different USB drives, and in
different USB ports on the motherboard. The drives are always connected
directly, without any cable or hub in between. Of course, that doesn't mean
there can't be something happening on the motherboard itself...

A couple more notes inline below...

Thanks for the context! I tracked down the firmware on Renesys' site (it
requires registration!), and tried to have it ready on a second USB drive
during the installation process. I was hoping to confirm whether the missing
firmware could have anything to do with the disappearing drive, but
unfortunately the result is inconclusive.

As I noted above the first chance I get to switch to another terminal is
after I have already gotten the error about missing firmware, and by then
the drive has already disappeared. At that point I was able to copy the
firmware from the second USB drive and clear the error, but that doesn't
tell us anything useful. I do not know of a way to have the firmware present
before that step happens.

Incidentally -- and I apologize for bundling yet another unrelated issue
into this; I can file a separate bug if that's better -- I wasn't able to
get the installer to automatically load the firmware from removable media
the way it suggests in the error message. I tried putting the file on the
root of a FAT32-formatted drive, and also in a directory named "firmware",
but in both cases the installer couldn't find it and load it. I had to mount
the drive and copy it myself from /mnt to /lib/firmware, at which time the
installer did recognize that it was present and the error cleared.

Nice, thanks!

:)

Thank you all again for the help!

#1105000#35
Date:
2025-05-11 08:35:30 UTC
From:
To:
The log confirms that the issue is caused by check-missing-firmware when
unloading and reloading the xhci-pci-renesas module.

I was able to reproduce it on my laptop by unloading and reloading
ehci-pci. I naively assumed that a mounted partition on the USB drive
would mark all drivers in the path as used and prohibit unloading them,
but it seems I was wrong.

The Renesas USB controller seems to work fine without it, and according
to the kernel log, a default firmware is loaded from non-volatile memory:

xhci-pci-renesas 0000:a3:00.0: failed to load firmware
renesas_usb_fw.mem, *fallback to ROM*

So I think this is the way to go. The fix is trivial, do you wish to
take care of it or shall I prepare a MR ?

You could also just mount the partition containing the firmware file to
/media and re-run the firmware detection.

This is a known issue, the loose firmware file search is unreliable.
Firmware package search is much better. See #740503 and #1029962. I
proposed a fix but the required change was rather invasive and the time
was not ideal (before bookworm release), then it kind of stalled...

#1105000#40
Date:
2025-05-11 14:53:48 UTC
From:
To:
Hi,

Pascal Hambourg <pascal@plouf.fr.eu.org> (2025-05-11):

Thanks for looking/confirming.

Lots of fun can be had with USB… :(

That might leave other users on the side of the road, since the ROM
might not be available or might not have been equipped with firmware
data.

Seeing how that particular system would (presumably) work fine without
the check+reload plus how bad the side effects are, it might be prudent
to just ignore renesas_usb_fw.mem / xhci-pci-renesas as a first step.
I'm not sure if the iwl-debug-yoyo.bin code is the best way to do that,
but feel free to look into proposing an MR (based or not on that code
path, as you see fit).

For users that would actually miss the firmware (file and/or ROM) and
need it, meaning they would be missing devices, it's important that we
are able to put the finger on the issue. As long as we have the “failed
to load […]” log line on the kernel, and/or whatever “let's ignore this
particular case” in check-missing-firmware, I would consider this topic
covered.


Cheers,

#1105000#45
Date:
2025-05-11 18:28:25 UTC
From:
To:
You are right, the driver source code confirms it.

What about adding an extra check ?
- "failed to load firmware renesas_usb_fw.mem, fallback to ROM" in
kernel messages;
- or some indication in /sys that the driver actually works (in my
experience, just being bound to a device is not reliable enough; maybe
the existence of child devices)
<https://salsa.debian.org/installer-team/hw-detect/-/merge_requests/13>

Felix might want to test the mini-ISO:
<https://salsa.debian.org/pham/hw-detect/-/jobs/7568796/artifacts/file/debian/output/debian-202501XX+salsaci+20250511+10-amd64-gtkmini.iso>

It does not mount the installation media but you can test if it does not
ask for renesas_usb_fw.mem nor unload+reload xhci-pci-renesas. No need
to go further "detect network devices".

I agree.