Somewhere between 6.18 and 6.19, a bug was introduced that causes one of my servers to somewhat randomly throw DMA errors on all SATA HDDs. Teej has kindly been helping me troubleshoot this issue and here is the summary: * 6.18.1-1~exp1 works. * 6.19.1~exp1 throws random I/O errors. * 6.19.1~exp1 works when booted with "iommu=off". This issue is still present in the 7.0 trixie-backports kernel, and the "iommu=off" workaround works on it as well. Sample error: Both reads and writes are affected. Note the 6.18 log has btrfs scrub errors. These are unrelated in the sense that it's not a 6.18 issue, but related in the sense that the on-disk errors were caused by this issue in an earlier test of a kernel exhibiting this bug. Attached are kernel logs from each run as well as "lspci -nnk" output.
I've worked with Chris in Matrix Debian room over the last two days to
diagnose this issue. Initial report was I/O errors after kernel upgrade
from v6.12.90 to v7.0.10 with:
[22396.952764] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[22396.952791] ata2.00: failed command: WRITE DMA EXT
[22396.952799] ata2.00: cmd 35/00:b8:02:30:0b/00:1d:00:00:00/e0 tag 26 dma 3895296 out
res 50/00:00:01:20:0b/00:00:00:00:00/e0 Emask 0x40 (internal error)
[22396.952822] ata2.00: status: { DRDY }
We initially focused on the C220 series chipset and the SATA links and determined this HP
ProLiant ML10 v2 (J10) server's specific chipset is C222 - this means that of the 6 SATA
ports 2 are 6Gbps and 4 3Gbps.
After checking sysfs attributes for the links and Chris trying alternate
ports with no improvement we next considered the several BIOS firmware
bugs reported by the kernel but discounted those since they exist in the
good kernel versions too. BIOS was upgraded to latest with no
difference.
I found b.k.o #220693 "SATA bus goes offline after a while" [0] that has similar
symptoms and has several intermingled issues that seem to exhibit very
similar symptoms. It discusses several possible workarounds:
1. ATA LPM ( libata.force=nolpm )
2. maximum transfer size ( libata.force=maxsec1024 )
3. IOMMU ( iommu=off )
The first two didn't show improvements on the failing kernel versions
but disabling IOMMU did. Comment 44 [1] of the bug report considers that may
indicate an issue with readahead code.
Using this as a clue and looking through v6.12..v7.0 commits I found a
group of related iomap commits introduced at the start of the v6.19
cycle.
That spurred the test of v6.18 that doesn't exhibit the issue, and
v6.19 that does.
I've built and shared v6.18.37 with Chris since this does not contain
the suspect iomap commits and he'll report back on that later.
Bisecting the iomap commits in v6.19 might be a challenge due to how
there are several later changes based on the series, and they might be a
red herring.
So this needs more eyeballs to consider alternative triggers of the bug
and once we have the v6.18.37 result forwarding upstream.
Tj.
[0] https://bugzilla.kernel.org/show_bug.cgi?id=220693
[1] https://bugzilla.kernel.org/show_bug.cgi?id=220693#c44
I forgot to mention that this may be related to the following kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=220693 Neither libata.force=maxsec1024 nor libata.force=nolpm mitigate the issue.------------------------------------------------------------------------ IMPORTANT INFORMATION/DISCLAIMER This document should be read only by those persons to whom it is addressed. If you have received this message it was obviously addressed to you and therefore you can read it. Additionally, by sending an email to ANY of my addresses or to ANY mailing lists to which I am subscribed, whether intentionally or accidentally, you are agreeing that I am "the intended recipient," and that I may do whatever I wish with the contents of any message received from you, unless a pre-existing agreement prohibits me from so doing. This overrides any disclaimer or statement of confidentiality that may be included on your message.
I could not get the 6.18.37 kernel TJ sent me to exhibit this issue after letting btrfs scrub for awhile, so it does not seem this kernel version is affected.
Bisection has taken some time and false started but we've now confirmed:
v7.2-rc2 @ dd3210c47e8d3ac6 is broken:
[ 83.224361] ata4.00: exception Emask 0x0 SAct 0x2000 SErr 0x0 action 0x0
[ 83.224432] ata4.00: failed command: READ FPDMA QUEUED
[ 83.224457] ata4.00: cmd 60/00:68:00:99:5d/04:00:02:00:00/40 tag 13 ncq dma 524288 in
res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x40 (internal error)
[ 83.224525] ata4.00: status: { DRDY }
[ 83.333448] ata4.00: configured for UDMA/133
[ 83.333481] ata4: EH complete
Bisection shows the regression is in the IOMMU Merge tag 'iommu-updates-v6.19' ce5cfb0fa20dc.
So far attempting to bisect inside that merge has resulted in unrelated failures that prevent the ability to initiate the tests that provoke the exception.
Since the mainline merge is itself a merge on the iommu tree of several branches at 0d081b16946ef449fc:
Merge branches 'arm/smmu/updates', 'arm/smmu/bindings', 'mediatek', 'nvidia/tegra', 'intel/vt-d', 'amd/amd-vi' and 'core' into next
I tried to isolate the likely candidate branches intel/vt-d and core but as those branches have moved on I've not been able to accurately identify the collection of patches that come from each branch at that time.
However, I suspect d373449d8e97891434 "iommu/vt-d: Use the generic iommu page table" that removes the original intel_iommu PT code is the most likely culprit.
Continuing to try to find usable bisect points in that merge series.
that Debian does some changes to the default IOMMU configuration. So applying the same config to an upstream kernel will need explicit setting the wanted iommu state. Bastian
Yes, we're testing with the related Debian config (now without any Debian patches though) # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set CONFIG_INTEL_IOMMU_DEFAULT_ON_INTGPU_OFF=y # CONFIG_INTEL_IOMMU_DEFAULT_OFF is not set and intel_iommu=on Originally we suspected an interaction with the Debian IOMMU intgpu patches but were able to rule them out.
Dear Maintainer,
I confirm this issue when booting on Debian testing with kernel 7.2.11+deb14-amd64.
Errors in kernel logs with a SATA device (SSD disk used for system):
[Thu Oct 22 12:47:58 2026] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[Thu Oct 22 12:47:58 2026] ata1.00: configured for UDMA/100
[Thu Oct 22 12:47:58 2026] ata1: EH complete
[Thu Oct 22 12:47:58 2026] ata1.00: exception Emask 0x60 SAct 0x0 SErr 0x40800 action 0x6 frozen
[Thu Oct 22 12:47:58 2026] ata1.00: irq_stat 0x20000000, host bus error
[Thu Oct 22 12:47:58 2026] ata1: SError: { HostInt CommWake }
[Thu Oct 22 12:47:58 2026] ata1.00: failed command: WRITE DMA EXT
[Thu Oct 22 12:47:58 2026] ata1.00: cmd 35/00:00:00:08:fc/00:20:05:00:00/e0 tag 29 dma 4194304 out
res 50/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x60 (host bus error)
[Thu Oct 22 12:47:58 2026] ata1.00: status: { DRDY }
[Thu Oct 22 12:47:58 2026] ata1: hard resetting link
[Thu Oct 22 12:47:59 2026] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[Thu Oct 22 12:47:59 2026] ata1.00: configured for UDMA/100
[Thu Oct 22 12:47:59 2026] sd 0:0:0:0: [sdc] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
[Thu Oct 22 12:47:59 2026] sd 0:0:0:0: [sdc] tag#29 Sense Key : Aborted Command [current]
[Thu Oct 22 12:47:59 2026] sd 0:0:0:0: [sdc] tag#29 Add. Sense: No additional sense information
[Thu Oct 22 12:47:59 2026] sd 0:0:0:0: [sdc] tag#29 CDB: Write(10) 2a 00 05 fc 08 00 00 20 00 00
[Thu Oct 22 12:47:59 2026] I/O error, dev sdc, sector 100403200 op 0x1:(WRITE) flags 0x0 phys_seg 64 prio class 2
[Thu Oct 22 12:47:59 2026] EXT4-fs warning (device sdc5): ext4_end_bio:368: I/O error 10 writing to inode 261272 starting block 12550400)
[Thu Oct 22 12:47:59 2026] ata1: EH complete
[Thu Oct 22 12:47:59 2026] EXT4-fs (sdc5): failed to convert unwritten extents to written extents -- potential data loss! (inode 261272, error -5)
[Thu Oct 22 12:47:59 2026] ata1.00: exception Emask 0x60 SAct 0x0 SErr 0x40800 action 0x6 frozen
[Thu Oct 22 12:47:59 2026] ata1.00: irq_stat 0x20000000, host bus error
[Thu Oct 22 12:47:59 2026] ata1: SError: { HostInt CommWake }
[Thu Oct 22 12:47:59 2026] ata1.00: failed command: WRITE DMA EXT
[Thu Oct 22 12:47:59 2026] ata1.00: cmd 35/00:00:00:68:13/00:20:05:00:00/e0 tag 31 dma 4194304 out
res 50/00:00:4f:e1:e1/00:00:00:00:00/e0 Emask 0x60 (host bus error)
[Thu Oct 22 12:47:59 2026] ata1.00: status: { DRDY }
[Thu Oct 22 12:47:59 2026] ata1: hard resetting link
(...)
$ lspci -nn |grep -i sata
00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
$ sudo lspci -s 00:1f.2 -vvv
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] (prog-if 01 [AHCI 1.0])
Subsystem: ASUSTeK Computer Inc. Device 8534
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupts: pin C disabled, MSI(X) routed to IRQ 36
Region 0: I/O ports at f0d0 [size=8]
Region 1: I/O ports at f0c0 [size=4]
Region 2: I/O ports at f0b0 [size=8]
Region 3: I/O ports at f0a0 [size=4]
Region 4: I/O ports at f060 [size=32]
Region 5: Memory at df839000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee00338 Data: 0000
Capabilities: [70] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
Kernel driver in use: ahci
Kernel modules: ahci
Issue temporary solved when booting with 'intel_iommu=off'
regards, Laurent