Hi linux maintainers, Today I restarted the s390x host of ci.d.n because I lost access. I inspected the journal and noticed there were a lot kernel messages (several tens to hundreds per day) like "User process fault: interruption code 003b ilc:3 in my_kmcdump[2aa07980000+f000]". I attached the first block I found in the journal after the reboot. Please let me know if you need more information. Paul PS: I checked with carnil before filing this report, he thought it was worth reporting.
Hi all, finally found the culprit. Several days ago I configured the host to do: # panic kernel on OOM vm.panic_on_oom=1 # reboot after 10 sec on panic kernel.panic=10 An idea I got from h01ger last year when discussing issues on arm64 and that has been working well there [1]. (See also https://www.debuntu.org/how-to-reboot-on-oom/) However, that doesn't seem to work on our s390x host as it seems to freeze instead. Is this something known? Something I'm doing wrong (E.g. these options behaving differently on s390x)? Is this a s390x kernel bug? Paul PS: the package that triggers this is hisat2. If you look at it's history [2] you see that the test was always Terminated (ignoring the run from 2024-07-17), I now assume by the OOM killer. I filed bug 1076524 against hisat2 to tell them they are using an insane amount of memory on s390x. [1] See e.g. the period around February/March 2024 on https://ci.debian.net/munin/ci-worker-arm64-11/ci-worker-arm64-11/uptime.html where a lot of reboots happened automatically. [2] https://ci.debian.net/packages/h/hisat2/testing/s390x/
Hi I see "003b" and "0007" as interruption codes. All those are page faults (and are reported to the user space process via SIGSEGV). Aka the process tries to access data that is not available in the page table. Why s390 decides to always dump them to the kernel log if the signal is unhandled is currently over me. There is similar code on other architectures, and also enabled by default, but I frankly have never seen it. Right now I have no idea what this could tell. Bastian
This now points to a kernel bug. Which requires a new kernel first for further debugging. What does "freeze" mean"? Also no sysrq? See https://www.kernel.org/doc/html/v5.3/s390/debugging390.html#sysrq, but you need to enable that before. Bastian
Hi waldi, With freeze I mean I have no connection anymore. And when I reboot, there's nothing in the journal since I lost connection. I don't know yet what sysrq means, so I'll look into that. Maybe tomorrow. Paul