Upgrading my bookworm ppc64el machine lead to unsuccessfull boot:
[    4.376878] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[    4.376927] CPU: 5 PID: 797 Comm: nvidia-persiste Tainted: P           O       6.1.0-28-powerpc64le #1  Debian 6.1.119-1
[    4.376972] Hardware name: T2P9D01 REV 1.00 POWER9 0x4e1203 opal:skiboot-9858186 PowerNV
[    4.377006] Call Trace:
[    4.377037] [c00020000fa3af10] [c000000000d7a2ec] dump_stack_lvl+0x70/0xa0 (unreliable)
[    4.377083] [c00020000fa3af40] [c000000000137c68] panic+0x16c/0x444
[    4.377131] [c00020000fa3afd0] [c000000000daf00c] __schedule+0xbfc/0xc00
[    4.377179] [c00020000fa3b0b0] [c000000000daf084] schedule+0x74/0x140
[    4.377212] [c00020000fa3b120] [c000000000db7e34] schedule_timeout+0xb4/0x1c0
[    4.377271] [c00020000fa3b1f0] [c008000015544b50] os_delay+0x168/0x3e0 [nvidia]
[    4.377768] [c00020000fa3b2a0] [c0080000164294e0] _nv038220rm+0x20/0x50 [nvidia]
[    4.378313] [c00020000fa3b2d0] [c0080000155ec704] _nv038438rm+0x264/0x530 [nvidia]
[    4.378821] [c00020000fa3b3c0] [c00800001643da08] _nv000655rm+0x13a8/0x2170 [nvidia]
[    4.379309] [c00020000fa3b550] [c008000016432ff4] rm_init_adapter+0x114/0x130 [nvidia]
[    4.379799] [c00020000fa3b640] [c0080000155308a4] nv_open_device+0x60c/0xa40 [nvidia]
[    4.380099] [c00020000fa3b6f0] [c0080000155320e0] nvidia_open+0x2e8/0x570 [nvidia]
[    4.380389] [c00020000fa3b7a0] [c00800001554cde8] nvidia_frontend_open+0xa0/0x120 [nvidia]
[    4.380705] [c00020000fa3b7f0] [c000000000530ac8] chrdev_open+0x178/0x3b0
[    4.380734] [c00020000fa3b860] [c000000000521130] do_dentry_open+0x290/0x560
[    4.380784] [c00020000fa3b8b0] [c000000000544460] path_openat+0xb90/0x15a0
[    4.380803] [c00020000fa3b9a0] [c000000000545b98] do_filp_open+0xc8/0x1a0
[    4.380821] [c00020000fa3bad0] [c000000000523e40] do_sys_openat2+0x110/0x230
[    4.380862] [c00020000fa3bb40] [c000000000524378] sys_openat+0x88/0xe0
[    4.380887] [c00020000fa3bba0] [c00000000002b038] system_call_exception+0x138/0x260
[    4.380900] [c00020000fa3be10] [c00000000000c0f0] system_call_vectored_common+0xf0/0x280
[    4.380938] --- interrupt: 3000 at 0x7fff92b3a948
[    4.380963] NIP:  00007fff92b3a948 LR: 00007fff92b3a948 CTR: 0000000000000000
[    4.380980] REGS: c00020000fa3be80 TRAP: 3000   Tainted: P           O        (6.1.0-28-powerpc64le Debian 6.1.119-1)
[    4.381020] MSR:  900000000280f033 <SF,HV,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 44004448  XER: 00000000
[    4.381091] IRQMASK: 0
[    4.381091] GPR00: 000000000000011e 00007ffffd5dc360 00007fff92c56f00 ffffffffffffff9c
[    4.381091] GPR04: 00007ffffd5dc400 0000000000080002 0000000000000000 0000000000000030
[    4.381091] GPR08: 00007fff928e01b8 0000000000000000 0000000000000000 0000000000000000
[    4.381091] GPR12: 0000000000000000 00007fff92f4dc40 0000000000000000 0000000000000000
[    4.381091] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    4.381091] GPR20: 0000000000000000 0000000000000000 0000000000000000 000000000000ffff
[    4.381091] GPR24: 00007fff92903398 0000000000000000 00007fff92902a48 00007fff92902a48
[    4.381091] GPR28: 00007ffffd5dc53c 0000000000000000 0000000000080002 00007fff92903398
[    4.381305] NIP [00007fff92b3a948] 0x7fff92b3a948
[    4.381334] LR [00007fff92b3a948] 0x7fff92b3a948
[    4.381352] --- interrupt: 3000
[    4.415728] bridge: filt

This happen with both 6.1.0-28-powerpc64le and 6.1.0-21-powerpc64le.
Removing /lib/modules/6.1.0-28-powerpc64le/updates/dkms/* was needed to recover.

System was working perfectly with previous 525.147.05-7~deb12u1

Regards

Control: tag -1 upstream moreinfo

Have you tried 535.216.01-1~deb12u1 from bookworm?

You could also try 535.216.03-2 from unstable.


Andreas

Sorry, I forgot to update this bug.
It works with:
linux-image-6.1.0-30-powerpc64le and nvidia-kernel-dkms 535.216.01-1~deb12u1

Thanks
Regards

reopen

It refail the same way with 6.1.0-31-powerpc64le and 6.1.0-32-powerpc64le

I tried unstable package but still fail:
    4.469944] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[    4.469980] CPU: 19 PID: 785 Comm: nvidia-persiste Tainted: P        W  O       6.1.0-32-powerpc64le #1  Debian 6.1.129-1
[    4.470009] Hardware name: T2P9D01 REV 1.00 POWER9 0x4e1203 opal:skiboot-9858186 PowerNV
[    4.470044] Call Trace:
[    4.470050] [c00020000c6aad60] [c000000000d7c70c] dump_stack_lvl+0x70/0xa0 (unreliable)
[    4.470084] [c00020000c6aad90] [c000000000137cf8] panic+0x16c/0x444
[    4.470121] [c00020000c6aae20] [c000000000db137c] __schedule+0xbfc/0xc00
[    4.470156] [c00020000c6aaf00] [c000000000db13f4] schedule+0x74/0x140
[    4.470192] [c00020000c6aaf70] [c000000000dba1a4] schedule_timeout+0xb4/0x1c0
[    4.470219] [c00020000c6ab040] [c008000015674890] os_delay+0x168/0x3e0 [nvidia]
[    4.470498] [c00020000c6ab0f0] [c00800001655afe0] _nv038241rm+0x20/0x50 [nvidia]
[    4.470788] [c00020000c6ab120] [c00800001571c4a4] _nv038459rm+0x264/0x530 [nvidia]
[    4.471081] [c00020000c6ab210] [c00800001656f538] _nv000655rm+0x13a8/0x2170 [nvidia]
[    4.471357] [c00020000c6ab3a0] [c008000016564af4] rm_init_adapter+0x114/0x130 [nvidia]
[    4.471638] [c00020000c6ab490] [c0080000156608a4] nv_open_device+0x60c/0xa40 [nvidia]
[    4.471878] [c00020000c6ab540] [c0080000156620e0] nvidia_open+0x2e8/0x570 [nvidia]
[    4.472101] [c00020000c6ab5f0] [c00800001567ca88] nvidia_frontend_open+0xa0/0x120 [nvidia]
[    4.472337] [c00020000c6ab640] [c000000000531258] chrdev_open+0x178/0x3b0
[    4.472352] [c00020000c6ab6b0] [c0000000005218c0] do_dentry_open+0x290/0x560
[    4.472389] [c00020000c6ab700] [c000000000544c50] path_openat+0xb90/0x15a0
[    4.472403] [c00020000c6ab7f0] [c000000000546388] do_filp_open+0xc8/0x1a0
[    4.472456] [c00020000c6ab920] [c0000000005245d0] do_sys_openat2+0x110/0x230
[    4.472503] [c00020000c6ab990] [c000000000524b08] sys_openat+0x88/0xe0
[    4.472550] [c00020000c6ab9f0] [c00000000002b038] system_call_exception+0x138/0x260
[    4.472599] [c00020000c6abe10] [c00000000000c0f0] system_call_vectored_common+0xf0/0x280
[    4.472637] --- interrupt: 3000 at 0x7fffa8b3a948
[    4.472680] NIP:  00007fffa8b3a948 LR: 00007fffa8b3a948 CTR: 0000000000000000
[    4.472726] REGS: c00020000c6abe80 TRAP: 3000   Tainted: P        W  O        (6.1.0-32-powerpc64le Debian 6.1.129-1)
[    4.472764] MSR:  900000000280f033 <SF,HV,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 44004448  XER: 00000000
[    4.472793] IRQMASK: 0
[    4.472793] GPR00: 000000000000011e 00007fffe0ccd900 00007fffa8c56f00 ffffffffffffff9c
[    4.472793] GPR04: 00007fffe0ccd9a0 0000000000080002 0000000000000000 0000000000000030
[    4.472793] GPR08: 00007fffa87f01f8 0000000000000000 0000000000000000 0000000000000000
[    4.472793] GPR12: 0000000000000000 00007fffa8e5dc40 0000000000000000 0000000000000000
[    4.472793] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    4.472793] GPR20: 0000000000000000 0000000000000000 0000000000000000 000000000000ffff
[    4.472793] GPR24: 00007fffa8813398 0000000000000000 00007fffa8812a48 00007fffa8812a48
[    4.472793] GPR28: 00007fffe0ccdadc 0000000000000000 0000000000080002 00007fffa8813398
[    4.472991] NIP [00007fffa8b3a948] 0x7fffa8b3a948
[    4.473012] LR [00007fffa8b3a948] 0x7fffa8b3a948
[    4.473044] --- interrupt: 3000
[    4.488316] ------------[ cut here ]------------
[    4.488321] WARNING: CPU: 2 PID: 982 at kernel/sched/core.c:3147 set_task_cpu+0x240/0x250
[    4.488334] Modules linked in: nfs(+) fscache netfs overlay i2c_dev evdev nvidia_drm(PO) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia_modeset(PO) binfmt_misc hid_generic ofpart xts ecb powernv_flash ipmi_powernv ctr ipmi_devintf vmx_crypto gf128mul at24(+) mtd ipmi_msghandler opal_prd regmap_i2c sg usbhid hid nvidia(PO) nfsd auth_rpcgss nfs_acl lockd grace drm loop fuse drm_panel_orientation_quirks sunrpc configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod ses enclosure sd_mod scsi_transport_sas xhci_pci xhci_hcd nvme crc32c_vpmsum nvme_core t10_pi tg3 crc64_rocksoft crc64 usbcore crc_t10dif crct10dif_generic crct10dif_common libphy ahci aacraid usb_common libahci
[    4.488454] CPU: 2 PID: 982 Comm: netstat Tainted: P        W  O       6.1.0-32-powerpc64le #1  Debian 6.1.129-1
[    4.488460] Hardware name: T2P9D01 REV 1.00 POWER9 0x4e1203 opal:skiboot-9858186 PowerNV
[    4.488463] NIP:  c00000000018b8f0 LR: c00000000018dfd4 CTR: c00000000019a590
[    4.488466] REGS: c000000014a7b640 TRAP: 0700   Tainted: P        W  O        (6.1.0-32-powerpc64le Debian 6.1.129-1)
[    4.488471] MSR:  9000000

Given that Nvidia has discontinued ppc64el support also for the 535 LTSB
series (which still has a year of support left), I have not much hope
for getting this fixed.

The last version you could try is src:nvidia-graphics-drivers-tesla-550
550.54.15-1 from sid.

I have now removed ppc64el support from src:nvidia-graphics-drivers from
sid (535.*) and experimental (550.*), removal from bookworm (535.*) will
happen with the next point release in a week.
Removal from src:nvidia-graphics-driver-tesla-535 in sid will happen in
the next days.
Removal of the ppc64el cuda stack will happen after the trixie release.

The last drivers still providing ppc64el support are
- src:nvidia-graphics-driver-tesla-550 in sid
- src:nvidia-graphics-driver-tesla-535 in bookworm (will be added with
the next point release)
These won't see any newer upstream versions.


Andreas