Dear Maintainer,

After the upgrade to Debian Trixie, my MDADM RAID1 and ZFS mirrors
failed reporting one of the two Intel P4510 SSDs in the respective
mirrors to be offline.

~~~
This is an automatically generated mail message.
DegradedArray event detected on md device /dev/md/1
The /proc/mdstat file currently contains the following:

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 nvme1n1p2[0]
      379868160 blocks super 1.2 [2/1] [U_]
      bitmap: 3/3 pages [12KB], 65536KB chunk

[...]
~~~

The NVMe device appears to not be recognized at all, although lspci
still lists the device to be present (this is taken from the “working”
state but the failing state looked similar):

~~~
10000:01:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
10000:02:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
~~~

After upgrading the UEFI firmware of the workstation and the problem
still persisting, I managed to track it down to the difference in Linux
Kernel versions.

Currently, I am back to using the Linux 6.1-series kernel from Bookworm
which correctly recognizes both of the SSDs.

The following dmesg (as recorded by `journalctl -k`) is observed on the
6.12 kernel (mirror fails, only one drive recognized)

~~~
Aug 24 21:55:18: [drm] Found VCN firmware Version ENC: 1.24 DEC: 8 VEP: 0 Revision: 3
Aug 24 21:55:18: nvme nvme0: pci function 10000:01:00.0
Aug 24 21:55:18: nvme 10000:01:00.0: enabling device (0000 -> 0002)
Aug 24 21:55:18: pcieport 10000:00:02.0: can't derive routing for PCI INT A
Aug 24 21:55:18: nvme 10000:01:00.0: PCI INT A: no GSI
Aug 24 21:55:18: nvme nvme1: pci function 10000:02:00.0
Aug 24 21:55:18: pcieport 10000:00:03.0: can't derive routing for PCI INT A
Aug 24 21:55:18: nvme 10000:02:00.0: PCI INT A: no GSI
Aug 24 21:55:18: sr 6:0:0:0: [sr0] scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Aug 24 21:55:18: cdrom: Uniform CD-ROM driver Revision: 3.20
Aug 24 21:55:18: nvme nvme1: D3 entry latency set to 15 seconds
Aug 24 21:55:18: nvme nvme1: 32/0/0 default/read/poll queues
Aug 24 21:55:18: nvme nvme1: Ignoring bogus Namespace Identifiers
Aug 24 21:55:18:  nvme1n1: p1 p2 p3 p4
Aug 24 21:55:18: amdgpu 0000:67:00.0: amdgpu: reserve 0x900000 from 0x81fd000000 for PSP TMR
~~~

Compare 6.1 (mirror works, both drives recognized)

~~~
Aug 27 18:18:16: sd 1:0:0:0: [sdb] Attached SCSI disk
Aug 27 18:18:16: nvme nvme0: pci function 10000:01:00.0
Aug 27 18:18:16: pcieport 10000:00:02.0: can't derive routing for PCI INT A
Aug 27 18:18:16: nvme 10000:01:00.0: PCI INT A: no GSI
Aug 27 18:18:16: nvme nvme1: pci function 10000:02:00.0
Aug 27 18:18:16: pcieport 10000:00:03.0: can't derive routing for PCI INT A
Aug 27 18:18:16: nvme 10000:02:00.0: PCI INT A: no GSI
Aug 27 18:18:16: nvme nvme1: Shutdown timeout set to 15 seconds
Aug 27 18:18:16: nvme nvme1: 32/0/0 default/read/poll queues
Aug 27 18:18:16: nvme nvme1: Ignoring bogus Namespace Identifiers
Aug 27 18:18:16:  nvme1n1: p1 p2 p3 p4
Aug 27 18:18:16: ixgbe 0000:b3:00.0: Intel(R) 10 Gigabit Network Connection

[...some unrelated hardware messages skipped...]

Aug 27 18:18:16: hid-generic 0003:0738:1703.0008: input,hidraw7: USB HID v1.00 Mouse [Madcatz Mad Catz R.A.T.3 Mouse] on usb-0000:00:14.0-7.1.4/input0
Aug 27 18:18:16: nvme nvme0: Shutdown timeout set to 15 seconds
Aug 27 18:18:16: nvme nvme0: 32/0/0 default/read/poll queues
Aug 27 18:18:16: nvme nvme0: Ignoring bogus Namespace Identifiers
Aug 27 18:18:16:  nvme0n1: p1 p2 p3 p4
~~~

On 6.12, there is no “second” block where the nvme0n1 would be
recognized.

I currently use the following two out-of-tree modules:

 * ZFS on Linux (zfs-dkms)
 * Merging Ravenna ALSA Linux Kernel Module (third-party module)

I attempted to test with the linux-image-6.16-amd64-unsigned from
experimental but since it would fail to build either modules, I could
not complete that particular test.

After some searching, I also found the following bug report in Ubuntu
(UBUNTU-2111521) which could be the same issue and might contain a
patch: <https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2111521>.

To summarize, I answer the questions from reportbug:

Upgrade from Debian 12 Bookworm to Debian 13 Trixie

Upgraded UEFI firmware to no success - same error.
Checked the cabling in the PC - all looks OK.
Booted old Linux kernel - it works for now!

Upgrade to Linux 6.12 no longer recognizes 2nd NVMe SSD.

Expected drives and mirrors to continue working normally.

Thanks in Advance and Kind Regards,
Linux-Fan

I have observed that ZFS 2.3.4 is newly available in sid and supports Linux
Kernel 6.16 hence I was able to check if the issue also appeared on the
newer kernel and indeed, the issue is still present on Linux 6.16 from
Debian sid.

Linux [...] 6.16.3+deb14-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.16.3-1 (2025-08-24) x86_64 GNU/Linux

Excerpt from dmesg (journactl -k)

~~~
Aug 31 13:33:20: nvme nvme0: pci function 10000:02:00.0
Aug 31 13:33:20: pcieport 10000:00:03.0: can't derive routing for PCI INT A
Aug 31 13:33:20: nvme 10000:02:00.0: PCI INT A: no GSI
Aug 31 13:33:20: nvme nvme1: pci function 10000:01:00.0
Aug 31 13:33:20: nvme 10000:01:00.0: enabling device (0000 -> 0002)
Aug 31 13:33:20: pcieport 10000:00:02.0: can't derive routing for PCI INT A
Aug 31 13:33:20: nvme 10000:01:00.0: PCI INT A: no GSI
Aug 31 13:33:20: nvme nvme0: D3 entry latency set to 15 seconds
Aug 31 13:33:20: nvme nvme0: 32/0/0 default/read/poll queues
Aug 31 13:33:20: nvme nvme0: Ignoring bogus Namespace Identifiers
Aug 31 13:33:20:  nvme0n1: p1 p2 p3 p4
Aug 31 13:33:20: sr 6:0:0:0: [sr0] scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
[...]
Aug 31 13:33:26: Process accounting resumed
Aug 31 13:33:26: nvme nvme0: using unchecked data buffer
Aug 31 13:33:26: nf_hook_func first message received
~~~

Like before, only one of the two NVMe drives is recognized...

Hello,

after a long time of no update, I had asked at the upstream aobut how this
might be fixed, but so far, progress on this issue seems to have stalled? [1]

Is there a chance that Debian could include a patch (maybe the same as is
included in Ubuntu [2]) as a workaround? Right now I am still using my
old Linux 6.1 kernel but of course I would like to update...

[1] https://lore.kernel.org/linux-pci/cone.1771011420.835087.30022.1000@masysma-18/

[2] https://lore.kernel.org/linux-pci/cone.1771011420.835087.30022.1000@masysma-18/3-ubuntu.patch

Thanks in Advance and Kind Regards,
Linux-Fan

Hi,

[Bjorn, for information, this only includes the Debian downstream bug,
not the full list of recipients in the original upstream thread]

We need an upstreamed patch and so beeing sustainable with the
frequent rebases. So we won't follow the Ubuntu approach of applying a
non-acked patch upstream.

I think Canonical should be equally intersted to not ship their patch
indefintively. Bjorn Helgaas said to try to come up with some ideas on
how to tackle the problem. Bjorn do have any idea on how to move
forward here on this issue?

Is there anything else people affected, like Linux-Fan, which could
help further?

Regards,
Salvatore