Dear Maintainer,
At Wikimedia Foundation we are running Trixie debian-installer on Dell r450
hardware with an mpt3sas (HBA355i with id 1000:00e6) controller and SSD
attached. While debian-installer finished successfully, grub was then unable to
boot the installed system.

Partman is instructed to assemble a raid10 over four devices with LVM on top.
Upon inspection the LVM PV is created with ~4GB metadata area which tricks grub
into allocating the same amount of memory during LVM detection. While
grub-install taking ~4GB of RAM "works" during installation, albeit
grub-install being quite slow, it obviously fails when booting.

I tracked down the problem to md0 reporting optimal_io_size of ~4GB, and LVM
defaults to align metadata with said size, resulting in abnormally large
PV metadata area.

The large md0 optimal_io_size seems to come from component devices reporting
16MB optimal_io_size as shown below.

This host was working fine with Bookworm, which makes me think something has
changed in mpt3sas.

My understanding is that the controller queries devices via block limits VPD
page for these values, and I'm attaching the output below. The original task
which spawned this work is https://phabricator.wikimedia.org/T407586

I'm happy to conduct further testing for bug fixes and/or investigation.

best,
Filippo

====

# uname -a
Linux cloudcontrol2010-dev 6.12.57+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.57-1 (2025-11-05) x86_64 GNU/Linux

# lsblk -t
NAME           ALIGNMENT MIN-IO     OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE      RA WSAME
sda                    0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sda1                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sda2                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
`-sda3                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
  `-md0                0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-swap         0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-root         0 524288 4293918720    4096     512    0                     4192256    0B
    `-vg0-srv          0 524288 4293918720    4096     512    0                     4192256    0B
sdb                    0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sdb1                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sdb2                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
`-sdb3                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
  `-md0                0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-swap         0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-root         0 524288 4293918720    4096     512    0                     4192256    0B
    `-vg0-srv          0 524288 4293918720    4096     512    0                     4192256    0B
sdc                    0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sdc1                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sdc2                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
`-sdc3                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
  `-md0                0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-swap         0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-root         0 524288 4293918720    4096     512    0                     4192256    0B
    `-vg0-srv          0 524288 4293918720    4096     512    0                     4192256    0B
sdd                    0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sdd1                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
|-sdd2                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
`-sdd3                 0   4096   16773120    4096     512    0 mq-deadline     256   32760    0B
  `-md0                0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-swap         0 524288 4293918720    4096     512    0                     4192256    0B
    |-vg0-root         0 524288 4293918720    4096     512    0                     4192256    0B
    `-vg0-srv          0 524288 4293918720    4096     512    0                     4192256    0B


# pvck --dump headers /dev/md0
  label_header at 512
  label_header.id LABELONE
  label_header.sector 1
  label_header.crc 0xbdf3a961
  label_header.offset 32
  label_header.type LVM2 001
  pv_header at 544
  pv_header.pv_uuid KDkSuWsrIico15Y0PenxiLzT8Ad2dGLa
  pv_header.device_size 1919546294272
  pv_header.disk_locn[0] at 584 # location of data area
  pv_header.disk_locn[0].offset 4293918720
  pv_header.disk_locn[0].size 0
  pv_header.disk_locn[1] at 600 # location list end
  pv_header.disk_locn[1].offset 0
  pv_header.disk_locn[1].size 0
  pv_header.disk_locn[2] at 616 # location of metadata area
  pv_header.disk_locn[2].offset 4096
  pv_header.disk_locn[2].size 4293914624
  pv_header.disk_locn[3] at 632 # location list end
  pv_header.disk_locn[3].offset 0
  pv_header.disk_locn[3].size 0
  pv_header_extension at 648
  pv_header_extension.version 2
  pv_header_extension.flags 1
  pv_header_extension.disk_locn[0] at 656 # location list end
  pv_header_extension.disk_locn[0].offset 0
  pv_header_extension.disk_locn[0].size 0
  mda_header_1 at 4096 # metadata area
  mda_header_1.checksum 0x84d8039
  mda_header_1.magic 0x204c564d3220785b35412572304e2a3e
  mda_header_1.version 1
  mda_header_1.start 4096
  mda_header_1.size 4293914624
  mda_header_1.raw_locn[0] at 4136 # commit
  mda_header_1.raw_locn[0].offset 4608
  mda_header_1.raw_locn[0].size 1724
  mda_header_1.raw_locn[0].checksum 0xdd78f68b
  mda_header_1.raw_locn[0].flags 0x0
  mda_header_1.raw_locn[1] at 4160 # precommit
  mda_header_1.raw_locn[1].offset 0
  mda_header_1.raw_locn[1].size 0
  mda_header_1.raw_locn[1].checksum 0x0
  mda_header_1.raw_locn[1].flags 0x0
  metadata text at 8704 crc 0xdd78f68b # vgname vg0 seqno 4

# Devices are all reporting the same information

# sg_vpd -p bl /dev/sda
Block limits VPD page (SBC)
  Write same non-zero (WSNZ): 1
  Maximum compare and write length: 0 blocks [command not implemented]
  Optimal transfer length granularity: 0 blocks [not reported]
  Maximum transfer length: 0 blocks [not reported]
  Optimal transfer length: 0 blocks [not reported]
  Maximum prefetch length: 0 blocks [not reported]
  Maximum unmap LBA count: 0x3ffff
  Maximum unmap block descriptor count: 0x20
  Optimal unmap granularity: 0x1
  Unmap granularity alignment valid: false
  Maximum write same length: 0xffff
  Maximum atomic transfer length: 0 blocks [not reported]
  Atomic alignment: 0 blocks [unaligned atomic writes permitted]
  Atomic transfer length granularity: 0 blocks [no granularity requirement]
  Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
  Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]

# sg_vpd -p ai /dev/sda
ATA information VPD page:
  SAT Vendor identification: LSI
  SAT Product identification: LSI SATL
  SAT Product revision level: 0008
  Device signature indicates SATA transport
  Command code: 0xec
  ATA command IDENTIFY DEVICE response summary:
    model: MTFDDAK960TGA-1BC1ZABDA
    serial number:         XXX
    firmware revision:  D4DK003

c3:00.0 Serial Attached SCSI controller: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx
        Subsystem: Dell HBA355i Front
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at e6800000 (64-bit, prefetchable) [size=1M]
        Memory at e6900000 (64-bit, prefetchable) [size=1M]
        Memory at e6a00000 (32-bit, non-prefetchable) [size=1M]
        I/O ports at e000 [size=256]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, IntMsgNum 0
        Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Power Budgeting <?>
        Capabilities: [158] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Physical Layer 16.0 GT/s <?>
        Capabilities: [1b0] Lane Margining at the Receiver
        Capabilities: [218] Dynamic Power Allocation <?>
        Capabilities: [248] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
        Capabilities: [348] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Capabilities: [380] Data Link Feature <?>
        Kernel driver in use: mpt3sas

Hi Filippo,
reproduce without the hardware if it comes to be specific to the
controller type and driver.

I would like to ask: Do you have the possibility to make an OS
instalaltion such that you can freely experiment with various kernels
and then under them assemble the arrays? If so that would be great
that you could start bisecting the changes to find where find changes.

I.e. install the OS independtly on the controller, find by bisecting
Debian versions manually the kernels between bookworm and trixie
(6.1.y -> 6.12.y to narrow down the upsream range).

Then bisect the ustream changes to find the offending commits. Let me
know if you need more specific instructions on the idea.

Additionally it would be interesting to know if the issue persist in
6.17.8 or even 6.18~rc6-1~exp1 to be able to clearly indicate upstream
that the issue persist in upper kernels.

Idealy actually this goes asap to upstream once we are more confident
ont the subsystem to where to report the issue. If we are reasonably
confident it it mpt3sas specific already then I would say to go
already to:

./scripts/get_maintainer.pl ./drivers/scsi/mpt3sas
Sathya Prakash <sathya.prakash@broadcom.com> (maintainer:LSILOGIC MPT FUSION DRIVERS (FC/SAS/SPI))
Sreekanth Reddy <sreekanth.reddy@broadcom.com> (maintainer:LSILOGIC MPT FUSION DRIVERS (FC/SAS/SPI))
Suganath Prabu Subramani <suganath-prabu.subramani@broadcom.com> (maintainer:LSILOGIC MPT FUSION DRIVERS (FC/SAS/SPI))
"James E.J. Bottomley" <James.Bottomley@HansenPartnership.com> (maintainer:SCSI SUBSYSTEM)
"Martin K. Petersen" <martin.petersen@oracle.com> (maintainer:SCSI SUBSYSTEM)
MPT-FusionLinux.pdl@broadcom.com (open list:LSILOGIC MPT FUSION DRIVERS (FC/SAS/SPI))
linux-scsi@vger.kernel.org (open list:LSILOGIC MPT FUSION DRIVERS (FC/SAS/SPI))
linux-kernel@vger.kernel.org (open list)

Do you concur?

Regards,
Salvatore

Hello Salvatore,
Thank you for the quick reply.

On Wed, Nov 19, 2025 at 05:59:48PM +0100, Salvatore Bonaccorso wrote:
[...]
hadn't thought of testing. Namely if there's another angle to this: does any
block device with the same block I/O hints exhibit the same problem? The answer is
actually "yes".

I used qemu 'scsi-hd' device to set the same values to be able to test locally.
On an already-installed VM I added the following to present four new devices:

Yes I'm able to perform testing on this host, in fact I worked around the
problem for now by disabling LVM's md alignment auto detection and thus we have
an installed system.
For reference that's "devices { data_alignment_detection = 0 }" in lvm's
config.

Having pointers on how the recommended way to build Debian kernels would be of
great help, thank you!
raid10 as per above, and mpt3sas presenting 0xFFF000 as optimal_io_size. While
the latter might be suspicious maybe it is not wrong per-se though?

best,
Filippo

part of a mdraid raid10 the array's optimal_io_size results in 0xFFF000.

When an LVM PV is created on the array its metadata area by default is aligned
with its optimal_io_size, resulting in an abnormally-large size of ~4GB. During
GRUB's LVM detection an allocation is made based on the metadata area size
which results in an unbootable system. This problem shows up only for
newly-created PVs and thus systems with existing PVs are not affected in my
testing.

I was able to reproduce the problem on qemu using scsi-hd devices as shown
below and on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121006. The bug
is present both on Debian' stable kernel and Linux 6.18, though I haven't yet
determined when the change was introduced in mpt3sas.

I'm wondering where the problem is in this case and what could be done to fix
it?

thank you,
Filippo

Hi,

在 2025/11/21 19:01, Filippo Giunchedi 写道:

Re: [PATCH 1/2] block: ignore underlying non-stack devices io_opt - Yu Kuai <https://lore.kernel.org/all/a3a98a81-16e9-2f3c-b6e5-c83a0055c784@huaweicloud.com/>

Hi Yu,

[apologies for the maybe overlong list of recipients, reason see below]

Thanks for pointing that out, I will leave context further down as
well intact.

mpt3sas folks, block-layer experts the above thread seems to have been
stalled recently, do you have any input on the way forward or if
indeed mpt3sas driver is behaving as expected here as well?

Filippo recently reported an issue while setting up a system at
wikimedia, https://bugs.debian.org/1121006 and
https://phabricator.wikimedia.org/T407586 for full context.

Regards,
Salvatore