Dear Maintainer,

Thanks for your hard work in maintaining Debian.  Two days in a row I have found
that the computer had hung when I tried to interact with it after a period of
hours.  I had enabled pstore collection, and studying the captured logs find
that a null pointer dereference is occuring during timer cancellation or
deletion.  These panics are NOT captured in the log automatically included
below.  Here is an example:

<1>[ 6941.130333] BUG: kernel NULL pointer dereference, address: 0000000000000000
<1>[ 6941.130348] #PF: supervisor write access in kernel mode
<1>[ 6941.130350] #PF: error_code(0x0002) - not-present page
<6>[ 6941.130353] PGD 0 P4D 0
<4>[ 6941.130356] Oops: Oops: 0002 [#1] PREEMPT SMP NOPTI
<4>[ 6941.130362] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.101+deb13-amd64 #1  Debian 6.12.101-1
<4>[ 6941.130367] Hardware name: System76 Thelio Mira/Thelio Mira, BIOS 3.11.SP01 12/05/2024
<4>[ 6941.130369] RIP: 0010:_raw_spin_lock_irqsave+0x27/0x50
<4>[ 6941.130377] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 53 9c 58 0f 1f 40 00 48 89 c3 fa 0f 1f 44 00 00 65 ff 05 b8 3e 52 6a 31 c0 ba 01 00 00 00 <f0> 0f b1 17 75 09 48 89 d8 5b c3 cc cc cc cc 89 c6 e8 a3 08 00 00
<4>[ 6941.130379] RSP: 0018:ffffcff600247e68 EFLAGS: 00010046
<4>[ 6941.130381] RAX: 0000000000000000 RBX: 0000000000000087 RCX: 000000010019557e
<4>[ 6941.130383] RDX: 0000000000000001 RSI: 0000000015d60000 RDI: 0000000000000000
<4>[ 6941.130385] RBP: ffff8d8e9f3a5e00 R08: 0000000000000008 R09: 0000000000000000
<4>[ 6941.130385] R10: 0000000000000000 R11: 000000000000ffff R12: ffff8d8e9f3a6418
<4>[ 6941.130386] R13: ffff8d80167ce400 R14: 0000000000000003 R15: 0000000000000000
<4>[ 6941.130388] FS:  0000000000000000(0000) GS:ffff8d8e9f380000(0000) knlGS:0000000000000000
<4>[ 6941.130389] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Panic#2 Part2
<4>[ 6941.130391] CR2: 0000000000000000 CR3: 0000000c14a22000 CR4: 0000000000f50ef0
<4>[ 6941.130392] PKRU: 55555554
<4>[ 6941.130393] Call Trace:
<4>[ 6941.130396]  <TASK>
<4>[ 6941.130402]  hrtimer_try_to_cancel.part.0+0x24/0xe0
<4>[ 6941.130409]  hrtimer_cancel+0x21/0x40
<4>[ 6941.130411]  tick_nohz_restart_sched_tick+0x36/0xa0
<4>[ 6941.130415]  tick_nohz_idle_exit+0x81/0xe0
<4>[ 6941.130417]  do_idle+0x161/0x2a0
<4>[ 6941.130420]  cpu_startup_entry+0x29/0x30
<4>[ 6941.130422]  start_secondary+0x11e/0x140
<4>[ 6941.130427]  common_startup_64+0x13e/0x141
<4>[ 6941.130432]  </TASK>

and another

<1>[122473.510807] BUG: kernel NULL pointer dereference, address: 0000000000000000
<1>[122473.510813] #PF: supervisor write access in kernel mode
<1>[122473.510815] #PF: error_code(0x0002) - not-present page
<6>[122473.510816] PGD 0 P4D 0
<4>[122473.510818] Oops: Oops: 0002 [#1] PREEMPT SMP NOPTI
<4>[122473.510819] CPU: 14 UID: 0 PID: 551 Comm: jbd2/nvme0n1p2- Not tainted 6.12.101+deb13-amd64 #1  Debian 6.12.101-1
<4>[122473.510822] Hardware name: System76 Thelio Mira/Thelio Mira, BIOS 3.11.SP01 12/05/2024
<4>[122473.510822] RIP: 0010:jbd2_journal_try_remove_checkpoint+0x17/0x50 [jbd2]
<4>[122473.510832] Code: 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 53 48 89 fb 48 8b 3f 48 83 7b 28 00 75 22 <f0> 48 0f ba 2f 02 72 1a 48 8b 07 a8 02 75 0e e8 c5 0a 4a c9 48 89
Oops#1 Part2
<4>[122473.510833] RSP: 0018:ffffd06981127c20 EFLAGS: 00010246
<4>[122473.510834] RAX: 0000000000004000 RBX: ffff8afd0f488c30 RCX: ffff8afd0f488c30
<4>[122473.510835] RDX: ffff8afd0f488c30 RSI: fffffa88c93d2200 RDI: 0000000000000000
<4>[122473.510836] RBP: 0000000000000002 R08: ffff8afd0f488690 R09: 0000000080220014
<4>[122473.510836] R10: 0000000080220014 R11: ffff8afbc9086800 R12: 0000000000000001
<4>[122473.510836] R13: ffffd06981127c6f R14: ffff8afd0f488690 R15: ffff8afd0f488618
<4>[122473.510837] FS:  0000000000000000(0000) GS:ffff8b0a5f900000(0000) knlGS:0000000000000000
<4>[122473.510838] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[122473.510838] CR2: 0000000000000000 CR3: 000000019e46e000 CR4: 0000000000f50ef0
<4>[122473.510839] PKRU: 55555554
<4>[122473.510840] Call Trace:
<4>[122473.510842]  <TASK>
<4>[122473.510844]  journal_shrink_one_cp_list+0x77/0xd0 [jbd2]
<4>[122473.510847]  __jbd2_journal_clean_checkpoint_list+0x79/0xb0 [jbd2]
<4>[122473.510850]  jbd2_journal_commit_transaction+0x340/0x1cb0 [jbd2]
<4>[122473.510853]  ? sched_balance_update_blocked_averages+0x5f1/0x760
<4>[122473.510857]  ? update_curr+0x1ac/0x210
<4>[122473.510859]  ? update_entity_lag+0x1b/0x90
<4>[122473.510861]  ? psi_task_switch+0x113/0x290
<4>[122473.510863]  ? __timer_delete_sync+0x7b/0xc0
<4>[122473.510866]  kjournald2+0xaa/0x250 [jbd2]
<4>[122473.510872]  ? __pfx_autoremove_wake_function+0x10/0x10
<4>[122473.510873]  ? __pfx_kjournald2+0x10/0x10 [jbd2]
<4>[122473.510877]  kthread+0xcf/0x100
<4>[122473.510879]  ? __pfx_kthread+0x10/0x10
Oops#1 Part1
<4>[122473.510880]  ret_from_fork+0x31/0x50
<4>[122473.510883]  ? __pfx_kthread+0x10/0x10
<4>[122473.510884]  ret_from_fork_asm+0x1a/0x30
<4>[122473.510886]  </TASK>

The attached files are pretty redundant.   I'm not why there are so many
of them.   I hope that the additional output is helpful.

Thanks,
Alison

---
Alison Chaiken                   alison@she-devel.com
https://she-devel.com
"Trying to look good limits me." -- Stefan Sagmeister

Hi Alison,
collection. The captured oopses are exactly what's needed to reason
about this.

Having gone through the three pstore dumps, I do not think this is a
kernel bug. The evidence points quite strongly at hardware memory
corruption (bad or marginal RAM, or an unstable DDR5 memory overclock
such as an EXPO/XMP profile). Here is why.

There are two completely different crash signatures on the same machine:

1) pstore-1787614683, uptime ~122473 s (~34 h), the jbd2 kjournald2 thread:

   BUG: kernel NULL pointer dereference, address: 0000000000000000
   #PF: supervisor write access in kernel mode
   CPU: 14 PID: 551 Comm: jbd2/nvme0n1p2-  Not tainted 6.12.101+deb13-amd64
   RIP: jbd2_journal_try_remove_checkpoint+0x17 [jbd2]
    journal_shrink_one_cp_list+0x77 [jbd2]
    __jbd2_journal_clean_checkpoint_list+0x79 [jbd2]
    jbd2_journal_commit_transaction+0x340 [jbd2]
    kjournald2+0xaa [jbd2]

   RDI is 0x0, and the faulting instruction bytes (48 8b 3f =
   "mov (%rdi),%rdi") are reading jh->b_transaction from a NULL
   struct journal_head. In other words the checkpoint list that
   journal_shrink_one_cp_list() was walking contained a NULL/garbage
   entry.

2) pstore-1787638256/...257, uptime ~6941 s (~1.9 h), on the idle task:

   BUG: kernel NULL pointer dereference, address: 0000000000000000
   #PF: supervisor write access in kernel mode
   CPU: 3 PID: 0 Comm: swapper/3  Not tainted 6.12.101+deb13-amd64
   RIP: _raw_spin_lock_irqsave+0x27
    hrtimer_try_to_cancel.part.0+0x24
    hrtimer_cancel+0x21
    tick_nohz_restart_sched_tick+0x36
    tick_nohz_idle_exit+0x81
    do_idle+0x161
   Kernel panic - not syncing: Attempted to kill the idle task!

   Here RDI (the spinlock pointer) is 0x0. In hrtimer_try_to_cancel that
   lock is &timer->base->cpu_base->lock, so the hrtimer's base/cpu_base
   pointer was corrupted to zero. Because this happened on the idle task,
   the secondary "Attempted to kill the idle task!" panic followed.

Both are NULL-pointer dereferences in very old, extremely well-exercised
core code paths (ext4/jbd2 checkpointing and the hrtimer/tick machinery),
and in both cases a pointer that is never legitimately NULL there had been
zeroed. A software defect normally reproduces in one specific path; it is
very unusual for two unrelated, mature subsystems to independently trip
over zeroed pointers hours apart. The kernel is "Not tainted" in every
capture, so there is no out-of-tree module involved, and journald also
reported the system journal as "corrupted or uncleanly shut down",
consistent with the earlier unclean crashes.

For completeness: the one plausible software candidate here, the upstream
fix "jbd2: check 'jh->b_transaction' before removing it from checkpoint",
has been in the tree since long before 6.12, so it is already present in
6.12.101 and is not the cause.

Your machine is an AMD Granite Ridge / Raphael desktop with DDR5 (two
SPD5118 DIMMs visible in the log). Rare, multi-hour, random-location
corruption like this is a very common symptom of a marginal DDR5 EXPO/XMP
profile or a failing DIMM. Could you please try the following?

  1. Run a memory test: boot MemTest86 / MemTest86+ and let it run several
     full passes, ideally overnight. stressapptest or "stress-ng --vm"
     under load is also useful. This kind of corruption can take a while
     to surface.

  2. Disable any memory overclock in the BIOS (Thelio Mira, AMI 3.11.SP01):
     turn off EXPO/XMP and run the RAM at JEDEC default speed/timings. If
     the crashes stop, the DIMMs or the EXPO profile were the cause.
     Updating to the latest System76 BIOS is also worthwhile for the
     AGESA/memory-training fixes.

  3. If MemTest reports errors, re-test the DIMMs one at a time and in
     different slots to isolate a bad module.

  4. Keep an eye on temperatures/power if the box is under load when it
     hangs.

If, and only if, MemTest is completely clean across many passes with
EXPO/XMP disabled and the crashes still recur, then it is worth treating
as a possible kernel issue. In that case:

  - Please retest with a newer kernel (the trixie / trixie-security 6.12
    series, or the newer upstream stable currently in unstable/experimental);
    a later stable point release may help.
  - Boot with "slub_debug=FZP page_poison=1" (and optionally "nohz=off" to
    see whether the tick path stops crashing) to catch stray writes closer
    to their source.

I'll tag this moreinfo/unreproducible for now. Please let us know how the
memory test and JEDEC-default run go, and we can take it from there.

Thanks again,
Yunseong