I am maintaining an Ampere Altra system for Freexian. Since version
6.12.96, we are experiencing system hangs. Thus far I have observed
three hangs of kernel 6.12.100 and one hang of 6.12.96. After the first
hang of 6.12.100, the hardware was replaced by the hosting provider.
After the third hang, we downgraded the kernel to 6.12.96 and that hung
as well. At this point a hardware defect seems unlikely yet possible.
We're now trying 101 and seeing whether it also exhibits such behavior
(none observed so far).

For the last hang on 6.12.96, we were able to capture a screen. Here is
a machine transcription with three errors corrected (may contain more):

==========
 polyval_generic md_mod ghash_ce xhci_hcd gf128mul sha2_ce nvme sha256_arm64 usbcore igb nvme_core sha1_ce sbsa_gwdt usb_common
i2c_algo_bit nvme_auth i2c_designware_platform efivarfs
[333998.664627] CPU: 45 UID: 101 PID: 591946 Comm: sshd-auth Tainted: G             L        6.12.96+deb13-arm64 #1 Debian 6.12.96
-1
[333998.664631] Tainted: [L]=SOFTLOCKUP
[333998.664632] Hardware name: GIGABYTE MP32-AR1-SW-HZ-001/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022
[333998.664634] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[333998.664636] pc : smp_call_function_many_cond+0x40c/0x580
[333998.664642] lr : smp_call_function_many_cond+0x3c0/0x580
[333998.664645] sp : ffff8000a1e23940
[333998.664646] x29: ffff8000a1e23940 x28: 0000000000000050 x27: ffff083e5fb72800
[333998.664650] x26: ffffd4d77e84d320 x25: ffffd4d77e9031f8 x24: 0000000000000050
[333998.664653] x23: ffffd4d77e8fee00 x22: 0000000000000000 x21: ffff07ff81b36db0
[333998.664656] x20: ffffd4d77ca86c68 x19: ffffd4d77e9031f8 x18: 0000000000000000
[333998.664659] x17: 0000000000000000 x16: 0000000000000000 x15: 000000d201000015
[333998.664662] x14: ffffd4d77c913404 x13: 0000000000000005 x12: 0000000000000001
[333998.664665] x11: 0000000000000000 x10: 0000000000000007 x9 : ffffd4d77d0462b8
[333998.664669] x8 : ffff07ff81b36dc0 x7 : 0000000000000000 x6 : ffffd4d77e8fe000
[333998.664672] x5 : 0000000000000008 x4 : 0000000000000043 x3 : ffff083e5fe63328
[333998.664675] x2 : 0000000000000000 x1 : 0000000000000011 x0 : 0000000000000043
[333998.664678] Call trace:
[333998.664679]     smp_call_function_many_cond+0x40c/0x580
[333998.664683]     kick_all_cpus_sync+0x38/0x50
[333998.664686]     __text_poke+0xd0/0x140
[333998.664691]     aarch64_insn_copy+0x2c/0x40
[333998.664696]     bpf_arch_text_copy+0x20/0x48
[333998.664699]     bpf_jit_binary_pack_finalize+0x2c/0x88
[333998.664704]     bpf_int_jit_compile+0x35c/0x600
[333998.664707]     bpf_prog_select_runtime+0x12c/0x1c8
[333998.664711]     bpf_prepare_filter+0x42c/0x5b0
[333998.664715]     bpf_prog_create_from_user+0xc4/0x168
[333998.664718]     do_seccomp+0x3e8/0xc48
[333998.664720]     prctl_set_seccomp+0x3c/0x80
[333998.664723]     __arm64_sys_prctl+0x678/0x9c8
[333998.664728]     invoke_syscall+0x60/0xe0
[333998.664732]     el0_svc_common.constprop.0+0x48/0xf0
[333998.664736]     do_el0_svc+0x24/0x38
[333998.664741]     el0_svc+0x38/0x148
[333998.664744]     el0t_64_sync_handler+0xc8/0xe0
[333998.664749]     el0t_64_sync+0x190/0x198
[334002.352581] usb 1-2: new high-speed USB device number 3 using xhci-pci-renesas
[334003.084452] ast 0004:02:00.0: firmware: failed to load edid/1280x1024.bin (-2)
[334003.092344] ast 0004:02:00.0: firmware: failed to load edid/1280x1024.bin (-2)
[334003.100082] ast 0004:02:00.0: firmware: failed to load edid/1280x1024.bin (-2)
[334003.107777] ast 0004:02:00.0: Direct firmware load for edid/1280x1024.bin failed with error -2
[334003.116874] ast 0004:02:00.0: [drm] *ERROR* [CONNECTOR:38:VGA-1] Requesting EDID firmware "edid/1280x1024.bin" failed (err=-
2)
==========
Please find the original screen capture attached.

I cannot match this output to the kernel messages recorded in the
journal. The output there is:

==========
Aug 09 14:51:40 demeter kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug 09 14:51:40 demeter kernel: rcu:         30-....: (5249 ticks this GP) idle=d62c/1/0x4000000000000000 softirq=1424610/1424610 fqs=2620
Aug 09 14:51:40 demeter kernel: rcu:         (t=5254 jiffies g=9983525 q=3633 ncpus=80)
Aug 09 14:51:40 demeter kernel: CPU: 30 UID: 0 PID: 590 Comm: kswapd0 Not tainted 6.12.96+deb13-arm64 #1  Debian 6.12.96-1
Aug 09 14:51:40 demeter kernel: Hardware name: GIGABYTE MP32-AR1-SW-HZ-001/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022
Aug 09 14:51:40 demeter kernel: pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 09 14:51:40 demeter kernel: pc : queued_spin_lock_slowpath+0x78/0x400
Aug 09 14:51:40 demeter kernel: lr : get_swap_pages+0x2d8/0x418
Aug 09 14:51:40 demeter kernel: sp : ffff800083c235e0
Aug 09 14:51:40 demeter kernel: x29: ffff800083c235e0 x28: ffffd4d77ed05450 x27: ffff07ff8e086400
Aug 09 14:51:40 demeter kernel: x26: ffff07ff8f0d1800 x25: ffff07ff8e0866a8 x24: 0000000000000000
Aug 09 14:51:40 demeter kernel: x23: fffffffffffffd40 x22: ffff07ff8e086650 x21: 00000000000002a8
Aug 09 14:51:40 demeter kernel: x20: ffff07ff8e0866c0 x19: ffff07ff8e086650 x18: 0000000000000014
Aug 09 14:51:40 demeter kernel: x17: 0000000084030980 x16: ffffd4d77cf2a588 x15: ffff800083c22ef0
Aug 09 14:51:40 demeter kernel: x14: ffff07ff8c96b000 x13: 0000000000000000 x12: 0000000000000000
Aug 09 14:51:40 demeter kernel: x11: 000000000000001d x10: ffffd4d77e84c280 x9 : ffffd4d77cca47e8
Aug 09 14:51:40 demeter kernel: x8 : 0000000000000001 x7 : ffff800083c2335e x6 : ffffd4d77ed05000
Aug 09 14:51:40 demeter kernel: x5 : ffff07ff8f0d1ac0 x4 : ffff07ff8f698f20 x3 : 0000000000000002
Aug 09 14:51:40 demeter kernel: x2 : 0000000000000003 x1 : 0000000000000000 x0 : 0000000000000001
Aug 09 14:51:40 demeter kernel: Call trace:
Aug 09 14:51:40 demeter kernel:  queued_spin_lock_slowpath+0x78/0x400
Aug 09 14:51:40 demeter kernel:  get_swap_pages+0x2d8/0x418
Aug 09 14:51:40 demeter kernel:  folio_alloc_swap+0x16c/0x260
Aug 09 14:51:40 demeter kernel:  add_to_swap+0x20/0x90
Aug 09 14:51:40 demeter kernel:  shrink_folio_list+0xa88/0xe38
Aug 09 14:51:40 demeter kernel:  evict_folios+0x274/0x738
Aug 09 14:51:40 demeter kernel:  try_to_shrink_lruvec+0x210/0x288
Aug 09 14:51:40 demeter kernel:  shrink_one+0x78/0x220
Aug 09 14:51:40 demeter kernel:  shrink_node+0xbb0/0xda0
Aug 09 14:51:40 demeter kernel:  balance_pgdat+0x428/0x990
Aug 09 14:51:40 demeter kernel:  kswapd+0x220/0x470
Aug 09 14:51:40 demeter kernel:  kthread+0xe8/0xf8
Aug 09 14:51:40 demeter kernel:  ret_from_fork+0x10/0x20
Aug 09 14:51:49 demeter kernel: watchdog: Watchdog detected hard LOCKUP on cpu 20
Aug 09 14:51:49 demeter kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun binfmt_misc nls_ascii nls_cp437 vfat fat dummy bridge stp llc acpi_ipmi arm_spe_pmu ipmi_ssif ast cfg80211 drm_shmem_helper drm_kms_helper ipmi_devintf arm_cmn ipmi_msghandler xgene_hwmon rfkill cppc_cpufreq arm_dsu_pmu acpi_tad evdev nft_masq nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables efi_pstore drm dm_mod configfs nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c crc32c_generic onboard_usb_dev raid0 raid1 aes_ce_blk aes_ce_cipher crct10dif_ce xhci_pci_renesas polyval_ce xhci_pci polyval_generic md_mod ghash_ce xhci_hcd gf128mul sha2_ce nvme sha256_arm64 usbcore igb nvme_core sha1_ce sbsa_gwdt usb_common i2c_algo_bit nvme_auth i2c_designware_platform efivarfs
Aug 09 14:51:49 demeter kernel: Sending NMI from CPU 19 to CPUs 20:
Aug 09 14:51:49 demeter kernel: watchdog: Watchdog detected hard LOCKUP on cpu 67
Aug 09 14:51:49 demeter kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun binfmt_misc nls_ascii nls_cp437 vfat fat dummy bridge stp llc acpi_ipmi arm_spe_pmu ipmi_ssif ast cfg80211 drm_shmem_helper drm_kms_helper ipmi_devintf arm_cmn ipmi_msghandler xgene_hwmon rfkill cppc_cpufreq arm_dsu_pmu acpi_tad evdev nft_masq nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables efi_pstore drm dm_mod configfs nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c crc32c_generic onboard_usb_dev raid0 raid1 aes_ce_blk aes_ce_cipher crct10dif_ce xhci_pci_renesas polyval_ce xhci_pci polyval_generic md_mod ghash_ce xhci_hcd gf128mul sha2_ce nvme sha256_arm64 usbcore igb nvme_core sha1_ce sbsa_gwdt usb_common i2c_algo_bit nvme_auth i2c_designware_platform efivarfs
Aug 09 14:51:53 demeter kernel: watchdog: BUG: soft lockup - CPU#45 stuck for 23s! [sshd-auth:591946]
Aug 09 14:51:53 demeter kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun binfmt_misc nls_ascii nls_cp437 vfat fat dummy bridge stp llc acpi_ipmi arm_spe_pmu ipmi_ssif ast cfg80211 drm_shmem_helper drm_kms_helper ipmi_devintf arm_cmn ipmi_msghandler xgene_hwmon rfkill cppc_cpufreq arm_dsu_pmu acpi_tad evdev nft_masq nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables efi_pstore drm dm_mod configfs nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c crc32c_generic onboard_usb_dev raid0 raid1 aes_ce_blk aes_ce_cipher crct10dif_ce xhci_pci_renesas polyval_ce xhci_pci polyval_generic md_mod ghash_ce xhci_hcd gf128mul sha2_ce nvme sha256_arm64 usbcore igb nvme_core sha1_ce sbsa_gwdt usb_common i2c_algo_bit nvme_auth i2c_designware_platform efivarfs
Aug 09 14:51:53 demeter kernel: CPU: 45 UID: 101 PID: 591946 Comm: sshd-auth Not tainted 6.12.96+deb13-arm64 #1  Debian 6.12.96-1
Aug 09 14:51:53 demeter kernel: Hardware name: GIGABYTE MP32-AR1-SW-HZ-001/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022
Aug 09 14:51:53 demeter kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 09 14:51:53 demeter kernel: pc : smp_call_function_many_cond+0x40c/0x580
Aug 09 14:51:53 demeter kernel: lr : smp_call_function_many_cond+0x3c0/0x580
Aug 09 14:51:53 demeter kernel: sp : ffff8000a1e23940
Aug 09 14:51:53 demeter kernel: x29: ffff8000a1e23940 x28: 0000000000000050 x27: ffff083e5fb72800
Aug 09 14:51:53 demeter kernel: x26: ffffd4d77e84d320 x25: ffffd4d77e9031f8 x24: 0000000000000050
Aug 09 14:51:53 demeter kernel: x23: ffffd4d77e8fee00 x22: 0000000000000000 x21: ffff07ff81b36db0
Aug 09 14:51:53 demeter kernel: x20: ffffd4d77ca86c68 x19: ffffd4d77e9031f8 x18: 0000000000000000
Aug 09 14:51:53 demeter kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 000000d201000015
Aug 09 14:51:53 demeter kernel: x14: ffffd4d77c913404 x13: 0000000000000005 x12: 0000000000000001
Aug 09 14:51:53 demeter kernel: x11: 0000000000000000 x10: 0000000000000007 x9 : ffffd4d77d0462b8
Aug 09 14:51:53 demeter kernel: x8 : ffff07ff81b36dc0 x7 : 0000000000000000 x6 : ffffd4d77e8fe000
Aug 09 14:51:53 demeter kernel: x5 : 0000000000000008 x4 : 0000000000000043 x3 : ffff083e5fe63328
Aug 09 14:51:53 demeter kernel: x2 : 0000000000000000 x1 : 0000000000000011 x0 : 0000000000000043
Aug 09 14:51:53 demeter kernel: Call trace:
Aug 09 14:51:53 demeter kernel:  smp_call_function_many_cond+0x40c/0x580
Aug 09 14:51:53 demeter kernel:  kick_all_cpus_sync+0x38/0x50
Aug 09 14:51:53 demeter kernel:  __text_poke+0xd0/0x140
Aug 09 14:51:53 demeter kernel:  aarch64_insn_copy+0x2c/0x40
Aug 09 14:51:53 demeter kernel:  bpf_arch_text_copy+0x20/0x48
Aug 09 14:51:53 demeter kernel:  bpf_jit_binary_pack_finalize+0x2c/0x88
Aug 09 14:51:53 demeter kernel:  bpf_int_jit_compile+0x35c/0x600
Aug 09 14:51:53 demeter kernel:  bpf_prog_select_runtime+0x12c/0x1c8
Aug 09 14:51:53 demeter kernel:  bpf_prepare_filter+0x42c/0x5b0
Aug 09 14:51:53 demeter kernel:  bpf_prog_create_from_user+0xc4/0x168
Aug 09 14:51:53 demeter kernel:  do_seccomp+0x3e8/0xc48
Aug 09 14:51:53 demeter kernel:  prctl_set_seccomp+0x3c/0x80
Aug 09 14:51:53 demeter kernel:  __arm64_sys_prctl+0x678/0x9c8
Aug 09 14:51:53 demeter kernel:  invoke_syscall+0x60/0xe0
Aug 09 14:51:53 demeter kernel:  el0_svc_common.constprop.0+0x48/0xf0
Aug 09 14:51:53 demeter kernel:  do_el0_svc+0x24/0x38
Aug 09 14:51:53 demeter kernel:  el0_svc+0x38/0x148
Aug 09 14:51:53 demeter kernel:  el0t_64_sync_handler+0xc8/0xe0
Aug 09 14:51:53 demeter kernel:  el0t_64_sync+0x190/0x198
Aug 09 14:52:05 demeter kernel: watchdog: BUG: soft lockup - CPU#30 stuck for 45s! [kswapd0:590]
Aug 09 14:52:05 demeter kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun binfmt_misc nls_ascii nls_cp437 vfat fat dummy bridge stp llc acpi_ipmi arm_spe_pmu ipmi_ssif ast cfg80211 drm_shmem_helper drm_kms_helper ipmi_devintf arm_cmn ipmi_msghandler xgene_hwmon rfkill cppc_cpufreq arm_dsu_pmu acpi_tad evdev nft_masq nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables efi_pstore drm dm_mod configfs nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c crc32c_generic onboard_usb_dev raid0 raid1 aes_ce_blk aes_ce_cipher crct10dif_ce xhci_pci_renesas polyval_ce xhci_pci polyval_generic md_mod ghash_ce xhci_hcd gf128mul sha2_ce nvme sha256_arm64 usbcore igb nvme_core sha1_ce sbsa_gwdt usb_common i2c_algo_bit nvme_auth i2c_designware_platform efivarfs
Aug 09 14:52:05 demeter kernel: CPU: 30 UID: 0 PID: 590 Comm: kswapd0 Tainted: G             L     6.12.96+deb13-arm64 #1  Debian 6.12.96-1
Aug 09 14:52:05 demeter kernel: Tainted: [L]=SOFTLOCKUP
Aug 09 14:52:05 demeter kernel: Hardware name: GIGABYTE MP32-AR1-SW-HZ-001/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022
Aug 09 14:52:05 demeter kernel: pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 09 14:52:05 demeter kernel: pc : queued_spin_lock_slowpath+0x78/0x400
Aug 09 14:52:05 demeter kernel: lr : get_swap_pages+0x2d8/0x418
Aug 09 14:52:05 demeter kernel: sp : ffff800083c235e0
Aug 09 14:52:05 demeter kernel: x29: ffff800083c235e0 x28: ffffd4d77ed05450 x27: ffff07ff8e086400
Aug 09 14:52:05 demeter kernel: x26: ffff07ff8f0d1800 x25: ffff07ff8e0866a8 x24: 0000000000000000
Aug 09 14:52:05 demeter kernel: x23: fffffffffffffd40 x22: ffff07ff8e086650 x21: 00000000000002a8
Aug 09 14:52:05 demeter kernel: x20: ffff07ff8e0866c0 x19: ffff07ff8e086650 x18: 0000000000000014
Aug 09 14:52:05 demeter kernel: x17: 0000000084030980 x16: ffffd4d77cf2a588 x15: ffff800083c22ef0
Aug 09 14:52:05 demeter kernel: x14: ffff07ff8c96b000 x13: 0000000000000000 x12: 0000000000000000
Aug 09 14:52:05 demeter kernel: x11: 000000000000001d x10: ffffd4d77e84c280 x9 : ffffd4d77cca47e8
Aug 09 14:52:05 demeter kernel: x8 : 0000000000000001 x7 : ffff800083c2335e x6 : ffffd4d77ed05000
Aug 09 14:52:05 demeter kernel: x5 : ffff07ff8f0d1ac0 x4 : ffff07ff8f698f20 x3 : 0000000000000002
Aug 09 14:52:05 demeter kernel: x2 : 0000000000000003 x1 : 0000000000000000 x0 : 0000000000000001
Aug 09 14:52:05 demeter kernel: Call trace:
Aug 09 14:52:05 demeter kernel:  queued_spin_lock_slowpath+0x78/0x400
Aug 09 14:52:05 demeter kernel:  get_swap_pages+0x2d8/0x418
Aug 09 14:52:05 demeter kernel:  folio_alloc_swap+0x16c/0x260
Aug 09 14:52:05 demeter kernel:  add_to_swap+0x20/0x90
Aug 09 14:52:05 demeter kernel:  shrink_folio_list+0xa88/0xe38
Aug 09 14:52:05 demeter kernel:  evict_folios+0x274/0x738
Aug 09 14:52:05 demeter kernel:  try_to_shrink_lruvec+0x210/0x288
Aug 09 14:52:05 demeter kernel:  shrink_one+0x78/0x220
Aug 09 14:52:05 demeter kernel:  shrink_node+0xbb0/0xda0
Aug 09 14:52:05 demeter kernel:  balance_pgdat+0x428/0x990
Aug 09 14:52:05 demeter kernel:  kswapd+0x220/0x470
Aug 09 14:52:05 demeter kernel:  kthread+0xe8/0xf8
Aug 09 14:52:05 demeter kernel:  ret_from_fork+0x10/0x20
Aug 09 14:52:17 demeter kernel: watchdog: Watchdog detected hard LOCKUP on cpu 26
Aug 09 14:52:17 demeter kernel: Modules linked in: vhost_net vhost vhost_iotlb tap tun binfmt_misc nls_ascii nls_cp437 vfat fat dummy bridge stp llc acpi_ipmi arm_spe_pmu ipmi_ssif ast cfg80211 drm_shmem_helper drm_kms_helper ipmi_devintf arm_cmn ipmi_msghandler xgene_hwmon rfkill cppc_cpufreq arm_dsu_pmu acpi_tad evdev nft_masq nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables efi_pstore drm dm_mod configfs nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c crc32c_generic onboard_usb_dev raid0 raid1 aes_ce_blk aes_ce_cipher crct10dif_ce xhci_pci_renesas polyval_ce xhci_pci polyval_generic md_mod ghash_ce xhci_hcd gf128mul sha2_ce nvme sha256_arm64 usbcore igb nvme_core sha1_ce sbsa_gwdt usb_common i2c_algo_bit nvme_auth i2c_designware_platform efivarfs
Aug 09 14:52:17 demeter kernel: Sending NMI from CPU 25 to CPUs 26:
Aug 09 14:52:17 demeter kernel: watchdog: BUG: soft lockup - CPU#45 stuck for 45s! [sshd-auth:591946]
-- Boot 4e5bdaa2a28e4c4dac0fb7739caa5757 --
==========

Such a diagnostic was not recorded in the journal for any other hang
than the 6.12.96 one.

The machine is running several virtual machines operating as
debusine.debian.net workers. Memory is occasionally full and swapping,
so we also use ksmtuned to reduce the memory pressure.

I am filing this bug report, because TJ asked me to do so. I don't think
it is actionable on its own. It can be used for correlating with other
reports maybe.

If this bug report becomes silent for a month or more and you run into
it, please close it.

Helmut

The traces related to potential memory pressure are affecting kswapd.
That does seem to implicate the ksmtuned settings. Is
KSM_MAX_KERNEL_PAGES set manually in ksmtuned.conf ? I'd recommend
setting DEBUG=1 there too so its actions and values are logged and could
be correlated with future hangs.

My feeling right now - with not much to go on - is ksm is trying too
hard during memory pressure and there's a vicious cycle between it and
kswapd.

There are 3 commits related to ksm for the v6.12 stable series. The most
pertinent in terms of the explanation in its commit message seems to be
commit 6673d1d818f04ff2f  "mm/ksm: fix exec/fork inheritance support for
prctl" (introduced in v6.12.64)

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.12.y&id=6673d1d818f04ff2f397590e5229a7449b0edf77

However, comparing mainline to the stable series there is a series of
commits in mainline that revert and replace the stable commit
74f78421c925b6d176955 "ksm: use range-walk function to jump over holes
in scan_get_next_rmap_item" (introduced in v6.12.59)

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.12.y&id=74f78421c925b6d17695566f0c5941de57fd44b3

Mainline since v6.19 has added revert and rework of that commit due to
causing CPU hangs

05c3fa9c9fa63 2025-11-16 17:28:28 -0800 N Pedro Demarchi Gomes ksm:
replace function unmerge_ksm_pages with break_ksm
5d4939fc2258e 2025-11-16 17:28:28 -0800 N Pedro Demarchi Gomes ksm:
perform a range-walk in break_ksm
912aa825957f5 2025-11-16 17:28:28 -0800 N Pedro Demarchi Gomes Revert
"mm/ksm: convert break_ksm() from walk_page_range_vma() to folio_walk"

Of those the last has an explanation that could match what you're seeing
here:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/mm/ksm.c?id=912aa825957f556a29d781c8f4cb4f4dfd938a9d

It would help to know the earliest kernel version where the hangs do not
occur. If the hardware has always suffered this behaviour since first
commissioning then the first kernel version used.

I've done an initial general sweep of commits in the mm/ sub-system for
the v6.12 cycle. There are quite a few so having a version range to
focus on may well help.

v6.12.3..v6.12.4 has 4 ./mm/ commits
v6.12.4..v6.12.5 has 8 ./mm/ commits
v6.12.6..v6.12.7 has 5 ./mm/ commits
v6.12.8..v6.12.9 has 8 ./mm/ commits
v6.12.10..v6.12.11 has 3 ./mm/ commits
v6.12.11..v6.12.12 has 3 ./mm/ commits
v6.12.13..v6.12.14 has 6 ./mm/ commits
v6.12.16..v6.12.17 has 1 ./mm/ commits
v6.12.18..v6.12.19 has 13 ./mm/ commits
v6.12.19..v6.12.20 has 2 ./mm/ commits
v6.12.20..v6.12.21 has 4 ./mm/ commits
v6.12.22..v6.12.23 has 3 ./mm/ commits
v6.12.23..v6.12.24 has 7 ./mm/ commits
v6.12.24..v6.12.25 has 5 ./mm/ commits
v6.12.27..v6.12.28 has 2 ./mm/ commits
v6.12.28..v6.12.29 has 5 ./mm/ commits
v6.12.29..v6.12.30 has 1 ./mm/ commits
v6.12.30..v6.12.31 has 3 ./mm/ commits
v6.12.34..v6.12.35 has 3 ./mm/ commits
v6.12.35..v6.12.36 has 2 ./mm/ commits
v6.12.36..v6.12.37 has 3 ./mm/ commits
v6.12.38..v6.12.39 has 1 ./mm/ commits
v6.12.40..v6.12.41 has 4 ./mm/ commits
v6.12.41..v6.12.42 has 2 ./mm/ commits
v6.12.42..v6.12.43 has 5 ./mm/ commits
v6.12.43..v6.12.44 has 3 ./mm/ commits
v6.12.45..v6.12.46 has 8 ./mm/ commits
v6.12.47..v6.12.48 has 10 ./mm/ commits
v6.12.48..v6.12.49 has 3 ./mm/ commits
v6.12.49..v6.12.50 has 3 ./mm/ commits
v6.12.53..v6.12.54 has 7 ./mm/ commits
v6.12.55..v6.12.56 has 2 ./mm/ commits
v6.12.58..v6.12.59 has 9 ./mm/ commits
v6.12.59..v6.12.60 has 1 ./mm/ commits
v6.12.60..v6.12.61 has 1 ./mm/ commits
v6.12.63..v6.12.64 has 23 ./mm/ commits
v6.12.66..v6.12.67 has 11 ./mm/ commits
v6.12.67..v6.12.68 has 1 ./mm/ commits
v6.12.68..v6.12.69 has 4 ./mm/ commits
v6.12.69..v6.12.70 has 1 ./mm/ commits
v6.12.73..v6.12.74 has 2 ./mm/ commits
v6.12.74..v6.12.75 has 4 ./mm/ commits
v6.12.77..v6.12.78 has 8 ./mm/ commits
v6.12.80..v6.12.81 has 1 ./mm/ commits
v6.12.82..v6.12.83 has 1 ./mm/ commits
v6.12.83..v6.12.84 has 1 ./mm/ commits
v6.12.85..v6.12.86 has 6 ./mm/ commits
v6.12.87..v6.12.88 has 1 ./mm/ commits
v6.12.89..v6.12.90 has 4 ./mm/ commits
v6.12.91..v6.12.92 has 1 ./mm/ commits
v6.12.92..v6.12.93 has 5 ./mm/ commits
v6.12.93..v6.12.94 has 5 ./mm/ commits
v6.12.95..v6.12.96 has 9 ./mm/ commits
v6.12.96..v6.12.97 has 8 ./mm/ commits
v6.12.100..v6.12.101 has 6 ./mm/ commits
v6.12.102..v6.12.103 has 10 ./mm/ commits