Dear Maintainer,
After the ext4 regression in 6.1.192-1 (#1154101, #1154151), I upgraded
to 6.1.192-2 (linux-image-6.1.0-56-amd64). The ext4 mballoc crash is gone
(the create files / drop_caches / delete reproducer passes), but the system
still locked up about 9 minutes after boot, with a different oops in the
page cache path.
* Hardware / setup
- Intel DN2800MT (Atom N2800), 2 GB RAM, single SATA HDD
- ext4 root filesystem, swap partition
- Docker (overlay2) workload: MariaDB, PostgreSQL, Apache/PHP,
reverse proxy, file sync service, WireGuard
* Kernel history on this machine
- 6.1.0-54-amd64 (6.1.190-1): stable, 8+ hours with the same
workload, currently used as workaround
- 6.1.0-55-amd64 (6.1.192-1): lockups from a few minutes to ~4.5 hours
after boot (consistent with #1154101)
- 6.1.0-56-amd64 (6.1.192-2): oops below ~9 minutes after boot
* Symptoms on 6.1.192-2
- Oops in xas_load called from filemap_map_pages, in a page fault from
an apache2 process
- The task dies inside an RCU read-side critical section
("Voluntary context switch within RCU read-side critical section!")
and then blocks in exit_mmap on the mmap rwsem
- Followed by repeated RCU expedited stalls on that task, every minute
- All disk writes then hang, SSH login hangs right after authentication,
the machine still answers ping; a hard reset is required
My guess (not verified) is another NULL vs ERR_PTR mismatch introduced by
the folio backports, similar to the ext4 one, this time in the page cache
fault-around path.
* Oops (hostname removed)
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 5fd4067 P4D 5fd4067 PUD 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 2537 Comm: apache2 Not tainted 6.1.0-56-amd64 #1 Debian
6.1.192-2
Hardware name: /DN2800MT, BIOS MTCDT10N.86A.0158.2012.0808.1446 08/08/2012
RIP: 0010:xas_load+0x26/0x40
Code: 00 00 00 00 e8 3b ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d
00 10 00 00 77 05 c3 cc cc cc cc 0f b6 4f 10 48 8d 70 fe <38> 48 fe 72 ee
e8 90 fd ff ff 80 3e 00 75 d0 c3 cc cc cc cc 66 0f
RSP: 0000:ffffd2d0c133bce8 EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffffd2d0c133bd40 RCX: 0000000000000048
RDX: 0000000000000003 RSI: 0000000000000000 RDI: fffff68d806f1f80
RBP: ffff8cc9cac91778 R08: 000000004056f000 R09: 000000004856efff
R10: 00000000000008c0 R11: 00000000000008b1 R12: 00000000000008c0
R13: fffff68d806f1fb4 R14: ffff8cc9cac91778 R15: ffff8cc9c5fd4008
FS: 00007ff390e38c00(0000) GS:ffff8cca3c100000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000000612c000 CR4: 00000000000006e0
Call Trace:
<TASK>
? filemap_map_pages+0xb0/0x720
? do_fault+0x1a4/0x410
? __handle_mm_fault+0x660/0xfa0
? handle_mm_fault+0xdb/0x2d0
? do_user_addr_fault+0x191/0x570
? exc_page_fault+0x70/0x170
? asm_exc_page_fault+0x22/0x30
</TASK>
Modules linked in: wireguard libchacha20poly1305 chacha_x86_64
poly1305_x86_64 curve25519_x86_64 libcurve25519_generic libchacha
ip6_udp_tunnel udp_tunnel nf_conntrack_netlink xt_nat xt_tcpudp veth
xt_conntrack xt_MASQUERADE bridge stp llc xfrm_user xfrm_algo xt_set ip_set
nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype
ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nf_tables nfnetlink
overlay cfg80211 rfkill isofs cdrom binfmt_misc intel_powerclamp
drm_kms_helper coretemp evdev sha512_ssse3 sha512_generic ppdev iTCO_wdt
sha256_ssse3 sha1_ssse3 i2c_algo_bit at24 intel_pmc_bxt iTCO_vendor_support
parport_pc watchdog parport sg button acpi_cpufreq drm dm_mod efi_pstore
fuse loop configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs
blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1
raid0 multipath linear md_mod sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif
crct10dif_generic crct10dif_common ahci libahci libata scsi_mod e1000e
uhci_hcd ehci_pci ehci_hcd usbcore scsi_common i2c_i801 i2c_smbus lpc_ich
usb_common video wmi
CR2: 0000000000000000
---[ end trace 0000000000000000 ]---
------------[ cut here ]------------
Voluntary context switch within RCU read-side critical section!
WARNING: CPU: 2 PID: 2537 at kernel/rcu/tree_plugin.h:320
rcu_note_context_switch+0x4ee/0x650
CPU: 2 PID: 2537 Comm: apache2 Tainted: G D 6.1.0-56-amd64
#1 Debian 6.1.192-2
Hardware name: /DN2800MT, BIOS MTCDT10N.86A.0158.2012.0808.1446 08/08/2012
RIP: 0010:rcu_note_context_switch+0x4ee/0x650
Call Trace:
<TASK>
__schedule+0xae/0xa00
schedule+0x5a/0xd0
rwsem_down_write_slowpath+0x315/0x6f0
exit_mmap+0xf6/0x310
__mmput+0x3e/0x130
do_exit+0x326/0xb10
? do_user_addr_fault+0x191/0x570
make_task_dead+0x8d/0x90
rewind_stack_and_make_dead+0x17/0x20
</TASK>
Then, every minute until the hard reset:
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { P2537 }
85122 jiffies s: 1505 root: 0x0/T
So far this happened once on 6.1.192-2. I can provide the full kernel log
of that boot or test patches/kernels if needed.
Thanks for your work.