Dear Maintainer, I have four four terabyte hard drives. Each has one partition on it. The four partitions comprise a RAID 5 array using mdadm. On top of that, LUKS encryption, then LVM with ext4 logical volumes. On one LVM partition I have a number of backup files, tarred, bzipped, and sha256 and sha512 summed. I have a script which will find checksum files, and execute the appropriate program to test the archives. It puts each program into the background, parallising any number of checksum tests. Starting early in May, the script found an error in one or more files out of several. Results are inconsistent: one pass may find an error in a given file, the next pass not find any errors in it. Running checksums manually, one at a time, does not turn up an error. Running "tar tvf" finds no error in a suspect tarball. Running "bunzip2 -t" also turns up no error. Only running the script turns up any errors. I create two checksum files when I create the backups, for sha256 and sha512. After this problem surfaced, I then made two new checksum files of a suspect file. The two checksum file pairs (e.g. both sha512sum files) show the same checksums. The script now tests using both the old and new checksum files. Sometime only one pair of checksum files fail the suspect file. In addition to all of that, I also get the occasional "bad message" error. I have no idea what that means, but an fsck seems to deal with it. To be thorough, I have run extended SMART tests on the hard drives, kicked mdadm into testing the RAID array, and fscked the LVM partitions on the RAID array. Only fsck turned up issues, and that has not stopped. SMART showed problems with one hard drive. I replaced the drive. That did not solve this problem. However, while the drive was "failed" and before the replacement installed, the error count went down noticeably. I also back some of this up to offsite USB drives. I ran the script on one of those, using this computer, and a different computer. No errors reported. I believe a successful work-around is to install the backports kernel, linux-image-7.0.10+deb13-amd64 et seq. This problem went away, although another similar problem has occurred with that kernel. There is extensive discussion of this problem on the Debian user list, under the subject "Schrödinger's hash", starting at https://lists.debian.org/debian-user/2026/05/msg00377.html
"It puts each program into the background, parallising any number of checksum tests" + "Running checksums manually, one at a time, does not turn up an error" This is possibly a classic system of a RAM module with sticky bits. In addition it suggests the module with the problem is at the higher physical addresses since the impact is only noticed when the host memory usage is high (due to parallel tasks). A long memtest86+ is strongly recommended.
Hello Charles, You got a reply to your report, but you might not have noticed it unless you subscribed to the bug. So in case you missed it: Tj suggested to run memtest86+ on your system to rule out (or confirm) RAM issues. You don't happen to have overclocked your system? Best regards Uwe
I don't seem to have it, so thank you for the followup. Already done; no issues found. I also substituted a single stick of RAM (8 gig) for the two I usually run. That resulted in fewer errors. Tj said, Tj, please define long? I seem to recall letting it run for about 6 hours and seeing no errors. root@hawk:~# lspci -vs 4:0 04:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller (prog-if 01 [AHCI 1.0]) Subsystem: JMicron Technology Corp. Device 0000 Flags: bus master, fast devsel, latency 0, IRQ 31 I/O ports at d200 [size=128] I/O ports at d180 [size=128] I/O ports at d100 [size=128] I/O ports at d080 [size=128] I/O ports at d000 [size=128] Memory at f7c10000 (32-bit, non-prefetchable) [size=8K] Expansion ROM at f7c00000 [disabled] [size=64K] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=1/8 Maskable- 64bit+ Capabilities: [c0] Express Legacy Endpoint, IntMsgNum 0 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [160] Power Budgeting <?> Capabilities: [1b8] Latency Tolerance Reporting Capabilities: [300] Secondary PCI Express Capabilities: [900] L1 PM Substates Kernel driver in use: ahci Kernel modules: ahci root@hawk:~# That suggests a test to try when I get a chance: power down the DVD drive, move the boot SSD to that card and the one HD currently on that card to the motherboard. It may be several days before I get to it, though.
Well, I may have a diagnosis. In any case, I have other problems which will delay further work on this. I made the above changes, except I left the DVD drive powered and on the above SATA card. On booting, I added "fsck.mode=force" to the linux line. fack failed the root partition. That is on the SSD, now on the SATA card. manually fscking that several times produced two to four errors of the form "directory has no checksum" and "directory corrupted". On each pass the errors showed up in different inode numbers. Also, for the halibut, I fscked the home partition, which should have been completely untouched by the boot process. Similar errors obtained. I then powered down, removed power from the DVD drive, and moved the SSD to the other SATA port on the card. This time both home and root had similar issues. I then shut the system down, and hooked the SSD up to another computer via a USB-SATA adapter (Bus 001 Device 008: ID 152d:2338 JMicron Technology Corp. / JMicron USA Technology Corp. JM20337 Hi-Speed USB to SATA & PATA Combo Bridge). I may have cooked the SSD in the process. I couldn't read the device on the other computer. Also, the power connector was warm when I took it off the SSD. I then set the computer under discussion back up as it had been originally. It refused even to power on. I unhooked the SSD from data and power. The computer now boots. I have booted finnix (a rescue Linux), and will fsck the hard drive while I contemplate my next move. But I suspect a replacement SSD and SATA card are in order. It will take me a few days to get replacements, during which time I can run memory tests.
On July 6, my main drive (an SSD) died on me. I have been rebuilding the system since. While waiting for the replacement drive, I ran memtest86. Four days of that showed no problems. Current status: The old dead SSD was a SATA interface. The replacement is an m.2 SATA SSD. This allowed me to remove all hard drives from the added PCI SATA card. That now has only the DVD drive. The four RAID hard drives are now attached directly to the motherboard. I have installed trixie, kernel 6.12.96+deb13-amd64. This is not exactly the kernel I was using when the problems first appeared (6.12.94-1), but close. Testing as above shows no errors. Fscking the RAID logical volumes shows no errors. No-one else has chimed in observing the same thing. I will continue testing but I am inclined to think that the problem was my hardware, not the kernel. Unless I hear otherwise in a few days I will close this bug per https://www.debian.org/Bugs/Developer#closing. Thank you for your responses.
now the my I agree that this looks more like a hardware failure than a software bug. Thank you for keeping us updated. Ben.
Closing per message 34 above. This appears to be a hardware issue, not software. Thank you.