last kernel version (6.1.0-51) was working fine. after an update last
week, one of our servers, booting via fc crashed to the rescue shell
after failing to find its boot devices via fiberchannel. Turns out they
cause a kernel trace.
As I needed the machine back online asap I cloudn't do much debugging so
far. I stopped all other reboots. Maybe I can replicate this on another
server to get some more information.

Hi Christian,

Thanks for the report. Unfortunately with the available infomation
aobove we cannot really act on the bug. If you can reproduce the issue
provide please the caused backtrace.

You might want as well to attach a netconsole so we get more
information from the kernel during boot.

Additionally if you reliably can reproduce the issue by upgrading to
6.12.101 from 6.12.100 it would be helpful if you have a server which
you can take out of production and then do a bisect between 6.12.100
and 6.12.101 to identify a breaking change.

Regards,
Salvatore

Hello Salvatore,

I was able to do a quick record this morning via the servers kvm but
still with the bookworm kernel. For the 6.12 Kernel I need to free
another system.

Some info about the fc hardware:

02:00.0 PCI bridge: Emulex Corporation x1 PCIe Gen2 Bridge[Pilot4]
(prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0, IRQ 17, NUMA node 0,
IOMMU group 41
         Memory at 9c900000 (64-bit, non-prefetchable) [size=4K]
         Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
         I/O behind bridge: [disabled] [32-bit]
         Memory behind bridge: 9b000000-9c8fffff [size=25M] [32-bit]
         Capabilities: [40] Power Management version 3
         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
         Capabilities: [60] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [150] Device Serial Number 00-00-00-00-00-00-00-00

5e:00.0 Fibre Channel: Emulex Corporation LPe31000/LPe32000 Series
16Gb/32Gb Fibre Channel Adapter (rev 01)
         Subsystem: Emulex Corporation LPe32002-M2 2-Port 32Gb Fibre
Channel Adapter
         Flags: bus master, fast devsel, latency 0, IRQ 261, NUMA node
0, IOMMU group 1
         Memory at b8800000 (64-bit, prefetchable) [size=32K]
         Memory at b8910000 (64-bit, prefetchable) [size=4K]
         Expansion ROM at c5e80000 [disabled] [size=512K]
         Capabilities: [54] Vendor Specific Information: Len=0c <?>
         Capabilities: [40] Power Management version 3
         Capabilities: [60] MSI: Enable- Count=1/32 Maskable+ 64bit+
         Capabilities: [78] MSI-X: Enable+ Count=512 Masked-
         Capabilities: [84] Vendor Specific Information: Len=0c <?>
         Capabilities: [94] Express Endpoint, MSI 00
         Capabilities: [f8] Vital Product Data
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [12c] Power Budgeting <?>
         Capabilities: [13c] Single Root I/O Virtualization (SR-IOV)
         Capabilities: [228] Vendor Specific Information: ID=ba5d Rev=1
Len=020 <?>
         Capabilities: [1d0] Alternative Routing-ID Interpretation (ARI)
         Capabilities: [1e8] Vendor Specific Information: ID=c511 Rev=1
Len=010 <?>
         Capabilities: [200] Transaction Processing Hints
         Capabilities: [20c] Secondary PCI Express
         Kernel driver in use: lpfc
         Kernel modules: lpfc

5e:00.1 Fibre Channel: Emulex Corporation LPe31000/LPe32000 Series
16Gb/32Gb Fibre Channel Adapter (rev 01)
         Subsystem: Emulex Corporation LPe32002-M2 2-Port 32Gb Fibre
Channel Adapter
         Flags: bus master, fast devsel, latency 0, IRQ 278, NUMA node
0, IOMMU group 1
         Memory at b8888000 (64-bit, prefetchable) [size=32K]
         Expansion ROM at c5e00000 [disabled] [size=512K]
         Capabilities: [54] Vendor Specific Information: Len=0c <?>
         Capabilities: [40] Power Management version 3
         Capabilities: [60] MSI: Enable- Count=1/32 Maskable+ 64bit+
         Capabilities: [78] MSI-X: Enable+ Count=512 Masked-
         Capabilities: [94] Express Endpoint, MSI 00
         Capabilities: [f8] Vital Product Data
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [12c] Power Budgeting <?>
         Capabilities: [13c] Single Root I/O Virtualization (SR-IOV)
         Capabilities: [228] Vendor Specific Information: ID=ba5d Rev=1
Len=020 <?>
         Capabilities: [1d0] Alternative Routing-ID Interpretation (ARI)
         Capabilities: [200] Transaction Processing Hints
         Kernel driver in use: lpfc
         Kernel modules: lpfc

Hi Christian,
finger memory. While it defintively would be good to know if 6.12.y
exposes the problem as well, we are interested here more on the 6.1.y
specific regression. I will do some more research based on the capture
you were able to do.

Were you able to take for testing off one such system? If so it would
be awesome if you could do the following, so we know 6.1.177-1 is good
and 6.1.180-1 is bad. And e need to have bisect, the procedure is as
follows:

    git clone --single-branch -b linux-6.1.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
    cd linux-stable
    git checkout v6.1.177
    cp /boot/config-$(uname -r) .config
    yes '' | make localmodconfig
    make savedefconfig
    mv defconfig arch/x86/configs/my_defconfig

    # test 6.1.177 to ensure this is "good"
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm it successfully boots

    # test 6.1.180 to ensure this is "bad"
    git checkout v6.1.180
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm it fails to boot

With that confirmed, the bisection can start:

    git bisect start
    git bisect good v6.1.177
    git bisect bad v6.1.180

In each bisection step git checks out a state between the oldest
known-bad and the newest known-good commit. In each step test using:

    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install, try to boot

and if the problem is hit run:

    git bisect bad

and if the problem doesn't trigger run:

    git bisect good

. Please pay attention to always select the just built kernel for
booting, it won't always be the default kernel picked up by grub.

Iterate until git announces to have identified the first bad commit.

Then provide the output of

    git bisect log

In the course of the bisection you might have to uninstall previous
kernels again to not exhaust the disk space in /boot. Also in the end
uninstall all self-built kernels again.

The expected number of compilations will be:

Bisecting: 516 revisions left to test after this (roughly 9 steps)

With the procedure you will build a targeted kernel, so the rounds
should be relatively fast.

Regards,
Salvatore