Dear Maintainer, after upgrading to trixie, the HP Microserver Gen8 started to crash randomly, hard, to system error (blinking red light in the front, not recoverable with reset but only removing power, IPMI/ILO still usable though). No dmesg/stacktrace visible. IML (Integrated Management Log from ILO) says: Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible (no additional information can be found) This can happen within half an hour after boot or 12 hours later, so debugging might be a little hard. I rolled back to the bookworm kernel and it has been running stable for 20 hours now. I found somewhat related bugs (#898336, #995773) with NMI stacktraces mentioning the wdt module - this isn't loaded in 6.1, haven't booted 6.12 again to check. Also haven't tried fiddling with boot options, since it takes ages to reproduce. I'm running iLO 4 2.82 since 2023 - which is the latest firmware to my knowledge.
I had the same issue. Doing this, I have no more problems: #/etc/modprobe.d/blacklist-hp.conf blacklist hpwdt
I had the same issue. Doing this, I have no more problems: #/etc/modprobe.d/blacklist-hp.conf blacklist hpwdt
I ran some more tests with a fresh trixie install, and can tell that those measurements previously taken don't help here: - disabling intel_iommu(=off in cmdline) - blacklisting hpwdt (which was not loaded in bookworm, but is in trixie kernel, was the same with a bookworm base installation) It takes a variable amount of hours, up to 2-3 days, and then my (now idle) cube blinks red again. IML: "ID","Severity","Class","Last Update","Initial Update","Count","Description", "96","Critical","OS","08/31/2025 04:01","08/31/2025 04:01","1","User Initiated NMI Switch", "95","Critical","System Error","08/31/2025 04:01","08/31/2025 04:01","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "94","Critical","OS","08/29/2025 17:04","08/29/2025 17:04","1","User Initiated NMI Switch", "93","Critical","System Error","08/29/2025 17:04","08/29/2025 17:04","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "92","Critical","OS","08/29/2025 14:48","08/29/2025 14:48","1","User Initiated NMI Switch", "91","Critical","System Error","08/29/2025 14:48","08/29/2025 14:48","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "90","Critical","OS","08/26/2025 09:47","08/26/2025 09:47","1","User Initiated NMI Switch", "89","Critical","System Error","08/26/2025 09:47","08/26/2025 09:47","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "88","Caution","POST Message","08/25/2025 20:46","08/25/2025 20:46","1","POST Error: 1785-Slot X Drive Array Not Configured", "87","Informational","POST Message","08/31/2025 07:10","08/25/2025 20:45","12","POST Information: Processor 1, DIMM 2 could not be authenticated as genuine HP SmartMemory. Enhanced and extended HP SmartMemory features will not be active.", "86","Informational","Maintenance","[NOT SET] ","[NOT SET] ","1","IML Cleared (iLO 4 user:me)", No dmesg entries (lured using tmux, dmesg -Tw from another machine) on freeze. Now at 6.12.41+deb13-amd64 Any other ideas?
While leaving it running until it crashes; do dmesg; done; I found a log entry yesterday. You can also see how it might take five days to crash. The issue came up in older kernels aswell, it is probably some incompatible configuration. [So Aug 31 09:11:30 2025] tg3 0000:03:00.0 eno1: Link is up at 1000 Mbps, full duplex [So Aug 31 09:11:30 2025] tg3 0000:03:00.0 eno1: Flow control is on for TX and on for RX [So Aug 31 09:11:30 2025] tg3 0000:03:00.0 eno1: EEE is enabled [Fr Sep 5 21:12:34 2025] NMI: IOCK error (debug interrupt?) for reason 71 on CPU 0. [Fr Sep 5 21:12:34 2025] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.41+deb13-amd64 #1 Debian 6.12.41-1 [Fr Sep 5 21:12:34 2025] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 06/06/2014 [Fr Sep 5 21:12:34 2025] RIP: 0010:intel_idle+0x55/0x90 [Fr Sep 5 21:12:34 2025] Code: 2d 1c 8d 40 00 eb 47 cc cc cc 48 89 f0 0f 1f 00 31 c9 48 89 ca 0f 01 c8 48 8b 06 a8 08 75 0b b9 01 00 00 00 4c 89 c0 0f 01 c9 <f0> 80 66 02 df f0 83 44 24 fc 00 48 8b 06 a8 08 74 0b 65 81 25 26 [Fr Sep 5 21:12:34 2025] RSP: 0018:ffffffff88403e38 EFLAGS: 00000046 [Fr Sep 5 21:12:34 2025] RAX: 0000000000000020 RBX: 0000000000000004 RCX: 0000000000000001 [Fr Sep 5 21:12:34 2025] RDX: 0000000000000000 RSI: ffffffff88410940 RDI: 0000000000000004 [Fr Sep 5 21:12:34 2025] RBP: ffff90e4816400e0 R08: 0000000000000020 R09: 000000000000ca80 [Fr Sep 5 21:12:34 2025] R10: ffff90e481634764 R11: 071c71c71c71c71c R12: ffffffff885b7580 [Fr Sep 5 21:12:34 2025] R13: ffffffff885b7738 R14: 0000000000000004 R15: 0000000000000000 [Fr Sep 5 21:12:34 2025] FS: 0000000000000000(0000) GS:ffff90e481600000(0000) knlGS:0000000000000000 [Fr Sep 5 21:12:34 2025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fr Sep 5 21:12:34 2025] CR2: 00007fb5a978d8c0 CR3: 0000000076622002 CR4: 00000000001726f0 [Fr Sep 5 21:12:34 2025] Call Trace: [Fr Sep 5 21:12:34 2025] <TASK> [Fr Sep 5 21:12:34 2025] cpuidle_enter_state+0x81/0x420 [Fr Sep 5 21:12:34 2025] cpuidle_enter+0x2d/0x40 [Fr Sep 5 21:12:34 2025] do_idle+0x1e5/0x240 [Fr Sep 5 21:12:34 2025] cpu_startup_entry+0x29/0x30 [Fr Sep 5 21:12:34 2025] rest_init+0xcc/0xd0 [Fr Sep 5 21:12:34 2025] start_kernel+0x74c/0x750 [Fr Sep 5 21:12:34 2025] x86_64_start_reservations+0x24/0x30 [Fr Sep 5 21:12:34 2025] x86_64_start_kernel+0x95/0xa0 [Fr Sep 5 21:12:34 2025] common_startup_64+0x13e/0x141 [Fr Sep 5 21:12:34 2025] </TASK> [Fr Sep 5 21:12:36 2025] NMI: IOCK error (debug interrupt?) for reason 61 on CPU 0. [Fr Sep 5 21:12:36 2025] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.41+deb13-amd64 #1 Debian 6.12.41-1 [Fr Sep 5 21:12:36 2025] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 06/06/2014 [Fr Sep 5 21:12:36 2025] RIP: 0010:intel_idle+0x55/0x90 [Fr Sep 5 21:12:36 2025] Code: 2d 1c 8d 40 00 eb 47 cc cc cc 48 89 f0 0f 1f 00 31 c9 48 89 ca 0f 01 c8 48 8b 06 a8 08 75 0b b9 01 00 00 00 4c 89 c0 0f 01 c9 <f0> 80 66 02 df f0 83 44 24 fc 00 48 8b 06 a8 08 74 0b 65 81 25 26 [Fr Sep 5 21:12:36 2025] RSP: 0018:ffffffff88403e38 EFLAGS: 00000046 [Fr Sep 5 21:12:36 2025] RAX: 0000000000000020 RBX: 0000000000000004 RCX: 0000000000000001 [Fr Sep 5 21:12:36 2025] RDX: 0000000000000000 RSI: ffffffff88410940 RDI: 0000000000000004 [Fr Sep 5 21:12:36 2025] RBP: ffff90e4816400e0 R08: 0000000000000020 R09: 000000000000ca80 [Fr Sep 5 21:12:36 2025] R10: ffff90e481634764 R11: 071c71c71c71c71c R12: ffffffff885b7580 [Fr Sep 5 21:12:36 2025] R13: ffffffff885b7738 R14: 0000000000000004 R15: 0000000000000000 [Fr Sep 5 21:12:36 2025] FS: 0000000000000000(0000) GS:ffff90e481600000(0000) knlGS:0000000000000000 [Fr Sep 5 21:12:36 2025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fr Sep 5 21:12:36 2025] CR2: 00007fb5a978d8c0 CR3: 0000000076622002 CR4: 00000000001726f0 [Fr Sep 5 21:12:36 2025] Call Trace: [Fr Sep 5 21:12:36 2025] <TASK> [Fr Sep 5 21:12:36 2025] cpuidle_enter_state+0x81/0x420 [Fr Sep 5 21:12:36 2025] cpuidle_enter+0x2d/0x40 [Fr Sep 5 21:12:36 2025] do_idle+0x1e5/0x240 [Fr Sep 5 21:12:36 2025] cpu_startup_entry+0x29/0x30 [Fr Sep 5 21:12:36 2025] rest_init+0xcc/0xd0 [Fr Sep 5 21:12:36 2025] start_kernel+0x74c/0x750 [Fr Sep 5 21:12:36 2025] x86_64_start_reservations+0x24/0x30 [Fr Sep 5 21:12:36 2025] x86_64_start_kernel+0x95/0xa0 [Fr Sep 5 21:12:36 2025] common_startup_64+0x13e/0x141 [Fr Sep 5 21:12:36 2025] </TASK> [Fr Sep 5 21:12:38 2025] clocksource: Long readout interval, skipping watchdog check: cs_nsec: 4660595886 wd_nsec: 4660594828 Read from remote host thecube: Connection reset by peer Connection to thecube closed. client_loop: send disconnect: Broken pipe
Hate to be like this, but I was wrong again. The box is not entirely dead, but blinks read (for the first time it behaves like this, I haven't physically looked at the box until now, today). Timestamp in IML entry roughly matches dmesg time (off by 20 minutes after a week looks normal to me). So it _is_ the cause (?). "ID","Severity","Class","Last Update","Initial Update","Count","Description", "100","Critical","OS","09/05/2025 20:57","09/05/2025 20:57","1","User Initiated NMI Switch", "99","Critical","System Error","09/05/2025 20:57","09/05/2025 20:57","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "98","Critical","OS","09/05/2025 19:12","09/05/2025 19:12","1","User Initiated NMI Switch", "97","Critical","System Error","09/05/2025 19:12","09/05/2025 19:12","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "96","Critical","OS","08/31/2025 04:01","08/31/2025 04:01","1","User Initiated NMI Switch", "95","Critical","System Error","08/31/2025 04:01","08/31/2025 04:01","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "94","Critical","OS","08/29/2025 17:04","08/29/2025 17:04","1","User Initiated NMI Switch", "93","Critical","System Error","08/29/2025 17:04","08/29/2025 17:04","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "92","Critical","OS","08/29/2025 14:48","08/29/2025 14:48","1","User Initiated NMI Switch", "91","Critical","System Error","08/29/2025 14:48","08/29/2025 14:48","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "90","Critical","OS","08/26/2025 09:47","08/26/2025 09:47","1","User Initiated NMI Switch", "89","Critical","System Error","08/26/2025 09:47","08/26/2025 09:47","1","Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible", "88","Caution","POST Message","08/25/2025 20:46","08/25/2025 20:46","1","POST Error: 1785-Slot X Drive Array Not Configured", "87","Informational","POST Message","09/05/2025 19:21","08/25/2025 20:45","13","POST Information: Processor 1, DIMM 2 could not be authenticated as genuine HP SmartMemory. Enhanced and extended HP SmartMemory features will not be active.", "86","Informational","Maintenance","[NOT SET] ","[NOT SET] ","1","IML Cleared (iLO 4 user:me)",
Hi, While the system is old itself, there seem to be a newer BIOS version available for the system. Please update first the BIOS to the newes available version (TTBOMK, this the newest one is 2019.04.04). Can you then still trigger the problem an provide fresh logs? Regards, Salvatore
Hi Thorsten, Thank you for providing this additional followup. I will include the bugreport again, so we can followup on it in the team. Regards, Salvatore
Hello Rafael, Thorsten (on Cc:) reported a bug against the 6.12.38 Debian kernel running on an HP Microserver Gen8 after upgrading from an older Debian release (which uses a kernel based on 6.1.y). The problem is that the machine sometimes just hangs, requiring to unplug the PSU to revive it. A key hint seems to me a kernel message: [Fr Sep 5 21:12:34 2025] NMI: IOCK error (debug interrupt?) for reason 71 on CPU 0. (sometimes also "reason 61") that happens just before the hang. See https://bugs.debian.org/1111027 for all the details. I asked Thomas on irc if that rings a bell for him and he forwarded me to you. Do you have an idea for a reason for this message and the corresponding(?) machine hang? The bug reporter up to now was very cooperative for debug measures (e.g. a BIOS update). I would expect him to be open to support you if questions should arise or a patch should be tested. A complication however is that the trigger for the problem is unknown and it takes up to several days to happen. Thanks in advance for your feedback, Uwe
Hello Thorsten, after writing the mail to Rafael Thomas provided the idea on irc to test using intel_idle.max_cstate=2 (or =1 or =0) on the kernel command line given that the splat hints at this happening for C3. Can you please add this (with =2) to your bootloader and if the problem happens again, tell us here and reduce the number further? Best regards Uwe