Kernel oops following NULL pointer dereference in aa_file_perm() when
running containers with podman + crun under Incus, triggered during
UNIX socket file-descriptor passing (SCM_RIGHTS).
This appears to be an AppArmor regression somewhere around 6.17, and
seems likely related to AppArmor AF_UNIX mediation and refactoring ?
https://git.kernel.org/pub/scm/linux/kernel/git/jj/linux-apparmor.git/commit/?h=apparmor-next&id=c05e705812d179f4b85aeacc34a555a42bc4f9ac
Confirmed not working: 6.17.8+deb13, 6.18.4 (non debian kernel)
Confirmed working: 6.16.12+deb13
Steps I used to reproduce:
- Starting with a clean Debian 13/Trixie install (VM or bare metal)
running kernel version >= ~6.17
- Install Incus (latest 6.20 for reference)
- Create a non-privileged debian 13 container under incus with
'security.nesting=true' enabled
- Install podman in to the incus container (from debian distribution
v5.4.2 / apt get podman)
- Attempt to run a rootful woodpecker-ci pod:
# podman run --rm -v /run/podman/podman.sock:/var/run/docker.sock -e
'WOODPECKER_SERVER=xxxxx' -e 'WOODPECKER_AGENT_SECRET=xxxx'-p
3000:3000 docker.io/woodpeckerci/woodpecker-agent:v3
Key points that trigger the issue:
- Podman is running nested inside a non-privileged container
- The podman container bind mounts the /run/podman/podman.sock UNIX
socket (this is within the incus container)
- Accessing the podman UNIX socket from within the nested podman
container is what triggers the oops
What does work:
- Podman on its own without nesting works fine
- Using crun instead of runc (I understand crun makes more use of FD
passing which is what appears to trigger the issue)
- Kernels earlier than 6.17
Full trace below
Jan 16 11:06:59 incus-podman kernel: BUG: kernel NULL pointer
dereference, address: 0000000000000018
Jan 16 11:06:59 incus-podman kernel: #PF: supervisor read access in kernel mode
Jan 16 11:06:59 incus-podman kernel: #PF: error_code(0x0000) - not-present page
Jan 16 11:06:59 incus-podman kernel: PGD 0 P4D 0
Jan 16 11:06:59 incus-podman kernel: Oops: Oops: 0000 [#1] SMP PTI
Jan 16 11:06:59 incus-podman kernel: CPU: 1 UID: 1000000 PID: 981
Comm: crun Not tainted 6.18.4-zabbly+ #debian13 PREEMPT(voluntary)
Jan 16 11:06:59 incus-podman kernel: Hardware name: QEMU Standard PC
(Q35 + ICH9, 2009)/Incus, BIOS unknown 02/02/2022
Jan 16 11:06:59 incus-podman kernel: RIP: 0010:aa_file_perm+0xc0/0x5d0
Jan 16 11:06:59 incus-podman kernel: Code: 45 31 c9 c3 cc cc cc cc 49
8b 46 20 41 8b 57 10 0f b7 00 66 25 00 f0 66 3d 00 c0 75 1c 41 f7 c4
46 00 10 00
75 13 49 8b 46 18 <48> 8b 40 18 66 83 78 10 01 0f 84 d9 02 00 00 89 d0
f7 d0 44 21 e0
Jan 16 11:06:59 incus-podman kernel: RSP: 0018:ffffcc4900efb5f0 EFLAGS: 00010246
Jan 16 11:06:59 incus-podman kernel: RAX: 0000000000000000 RBX:
ffff898294ff8180 RCX: ffff898283610b40
Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI:
ffff898282ae13c0 RDI: ffffffffa88e8430
Jan 16 11:06:59 incus-podman kernel: RBP: ffffcc4900efb6a0 R08:
0000000000000000 R09: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11:
0000000000000000 R12: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R13: ffff898294ff8180 R14:
ffff898283610b40 R15: ffff898282e6d3d0
Jan 16 11:06:59 incus-podman kernel: FS: 00007f3616418840(0000)
GS:ffff898340c3c000(0000) knlGS:0000000000000000
Jan 16 11:06:59 incus-podman kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Jan 16 11:06:59 incus-podman kernel: CR2: 0000000000000018 CR3:
0000000103626002 CR4: 0000000000372ef0
Jan 16 11:06:59 incus-podman kernel: Call Trace:
Jan 16 11:06:59 incus-podman kernel: <TASK>
Jan 16 11:06:59 incus-podman kernel: ? __slab_free+0xdf/0x2c0
Jan 16 11:06:59 incus-podman kernel: common_file_perm+0x69/0x1b0
Jan 16 11:06:59 incus-podman kernel: apparmor_file_receive+0x42/0x80
Jan 16 11:06:59 incus-podman kernel: security_file_receive+0x4a/0x120
Jan 16 11:06:59 incus-podman kernel: receive_fd+0x1d/0xf0
Jan 16 11:06:59 incus-podman kernel: scm_detach_fds+0xad/0x1c0
Jan 16 11:06:59 incus-podman kernel: __scm_recv_common.isra.0+0x66/0x180
Jan 16 11:06:59 incus-podman kernel: scm_recv_unix+0x30/0x130
Jan 16 11:06:59 incus-podman kernel: ? unix_destroy_fpl+0x3a/0xa0
Jan 16 11:06:59 incus-podman kernel: __unix_dgram_recvmsg+0x2ac/0x450
Jan 16 11:06:59 incus-podman kernel: unix_seqpacket_recvmsg+0x43/0x70
Jan 16 11:06:59 incus-podman kernel: sock_recvmsg+0xe1/0xf0
Jan 16 11:06:59 incus-podman kernel: ____sys_recvmsg+0xa0/0x230
Jan 16 11:06:59 incus-podman kernel: ___sys_recvmsg+0xc7/0xf0
Jan 16 11:06:59 incus-podman kernel: __sys_recvmsg+0x89/0x100
Jan 16 11:06:59 incus-podman kernel: __x64_sys_recvmsg+0x1d/0x30
Jan 16 11:06:59 incus-podman kernel: x64_sys_call+0x840/0x2350
Jan 16 11:06:59 incus-podman kernel: do_syscall_64+0x80/0x590
Jan 16 11:06:59 incus-podman kernel: ? ___sys_recvmsg+0xd2/0xf0
Jan 16 11:06:59 incus-podman kernel: ? ____sys_recvmsg+0x10e/0x230
Jan 16 11:06:59 incus-podman kernel: ? __sys_recvmsg+0x89/0x100
Jan 16 11:06:59 incus-podman kernel: ? __x64_sys_recvmsg+0x1d/0x30
Jan 16 11:06:59 incus-podman kernel: ? x64_sys_call+0x840/0x2350
Jan 16 11:06:59 incus-podman kernel: ? do_syscall_64+0xb8/0x590
Jan 16 11:06:59 incus-podman kernel: ? __sys_recvmsg+0x89/0x100
Jan 16 11:06:59 incus-podman kernel: ? __x64_sys_recvmsg+0x1d/0x30
Jan 16 11:06:59 incus-podman kernel: ? x64_sys_call+0x840/0x2350
Jan 16 11:06:59 incus-podman kernel: ? do_syscall_64+0xb8/0x590
Jan 16 11:06:59 incus-podman kernel: ? irqentry_exit_to_user_mode+0x2e/0x2a0
Jan 16 11:06:59 incus-podman kernel: ? irqentry_exit+0x43/0x50
Jan 16 11:06:59 incus-podman kernel: ? clear_bhb_loop+0x50/0xa0
Jan 16 11:06:59 incus-podman kernel: ? clear_bhb_loop+0x50/0xa0
Jan 16 11:06:59 incus-podman kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jan 16 11:06:59 incus-podman kernel: RIP: 0033:0x7f361659c687
Jan 16 11:06:59 incus-podman kernel: Code: 48 89 fa 4c 89 df e8 58 b3
00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00
00 00 00 48
8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de
e8 23 ff ff ff
Jan 16 11:06:59 incus-podman kernel: RSP: 002b:00007fff89de51f0
EFLAGS: 00000202 ORIG_RAX: 000000000000002f
Jan 16 11:06:59 incus-podman kernel: RAX: ffffffffffffffda RBX:
00007f3616418840 RCX: 00007f361659c687
Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI:
00007fff89de5240 RDI: 0000000000000009
Jan 16 11:06:59 incus-podman kernel: RBP: 00007fff89de5240 R08:
0000000000000000 R09: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11:
0000000000000202 R12: 00007fff89de58c0
Jan 16 11:06:59 incus-podman kernel: R13: 0000000000000007 R14:
00007fff89de58c0 R15: 000000000000000c
Jan 16 11:06:59 incus-podman kernel: </TASK>
Jan 16 11:06:59 incus-podman kernel: Modules linked in: nft_nat nft_ct
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib overlay veth nft_masq
nft_chain_nat nf_nat nf_conntrack n
f_defrag_ipv6 nf_defrag_ipv4 bridge stp llc nf_tables vhost_vsock
vhost vhost_iotlb binfmt_misc nls_iso8859_1 intel_rapl_msr
intel_rapl_common intel_uncore_frequency_common in
tel_pmc_core pmt_telemetry pmt_discovery pmt_class
intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm irqbypass
polyval_clmulni ghash_clmulni_intel aesni_intel virtio_snd rapl
snd_pcsp virtio_gpu snd_pcm virtio_dma_buf drm_shmem_helper
vmw_vsock_virtio_transport drm_client_lib 9p snd_timer
vmw_vsock_virtio_transport_common 9pnet_virtio drm_kms_help
er 9pnet snd netfs vsock virtio_input soundcore vmgenid joydev
input_leds mac_hid cfg80211 sch_fq_codel efi_pstore drm nfnetlink
dmi_sysfs qemu_fw_cfg virtio_rng ip_tables x_t
ables autofs4 iTCO_wdt intel_pmc_bxt iTCO_vendor_support psmouse
i2c_i801 i2c_mux serio_raw i2c_smbus ahci libahci lpc_ich
Jan 16 11:06:59 incus-podman kernel: CR2: 0000000000000018
Jan 16 11:06:59 incus-podman kernel: ---[ end trace 0000000000000000 ]---
Jan 16 11:06:59 incus-podman kernel: RIP: 0010:aa_file_perm+0xc0/0x5d0
Jan 16 11:06:59 incus-podman kernel: Code: 45 31 c9 c3 cc cc cc cc 49
8b 46 20 41 8b 57 10 0f b7 00 66 25 00 f0 66 3d 00 c0 75 1c 41 f7 c4
46 00 10 00 75 13 49 8b 46 18 <48> 8
b 40 18 66 83 78 10 01 0f 84 d9 02 00 00 89 d0 f7 d0 44 21 e0
Jan 16 11:06:59 incus-podman kernel: RSP: 0018:ffffcc4900efb5f0 EFLAGS: 00010246
Jan 16 11:06:59 incus-podman kernel: RAX: 0000000000000000 RBX:
ffff898294ff8180 RCX: ffff898283610b40
Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI:
ffff898282ae13c0 RDI: ffffffffa88e8430
Jan 16 11:06:59 incus-podman kernel: RBP: ffffcc4900efb6a0 R08:
0000000000000000 R09: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11:
0000000000000000 R12: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R13: ffff898294ff8180 R14:
ffff898283610b40 R15: ffff898282e6d3d0
Jan 16 11:06:59 incus-podman kernel: FS: 00007f3616418840(0000)
GS:ffff898340c3c000(0000) knlGS:0000000000000000
Jan 16 11:06:59 incus-podman kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Jan 16 11:06:59 incus-podman kernel: CR2: 0000000000000018 CR3:
0000000103626002 CR4: 0000000000372ef0
Jan 16 11:06:59 incus-podman kernel: note: crun[981] exited with irqs disabled
Jan 16 11:06:59 incus-podman kernel: ------------[ cut here ]------------
Jan 16 11:06:59 incus-podman kernel: Voluntary context switch within
RCU read-side critical section!
Jan 16 11:06:59 incus-podman kernel: WARNING: CPU: 0 PID: 981 at
kernel/rcu/tree_plugin.h:332 rcu_note_context_switch+0x523/0x590
Jan 16 11:06:59 incus-podman kernel: Modules linked in: nft_nat nft_ct
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib overlay veth nft_masq
nft_chain_nat nf_nat nf_conntrack n
f_defrag_ipv6 nf_defrag_ipv4 bridge stp llc nf_tables vhost_vsock
vhost vhost_iotlb binfmt_misc nls_iso8859_1 intel_rapl_msr
intel_rapl_common intel_uncore_frequency_common in
tel_pmc_core pmt_telemetry pmt_discovery pmt_class
intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm irqbypass
polyval_clmulni ghash_clmulni_intel aesni_intel virtio_snd rapl
snd_pcsp virtio_gpu snd_pcm virtio_dma_buf drm_shmem_helper
vmw_vsock_virtio_transport drm_client_lib 9p snd_timer
vmw_vsock_virtio_transport_common 9pnet_virtio drm_kms_help
er 9pnet snd netfs vsock virtio_input soundcore vmgenid joydev
input_leds mac_hid cfg80211 sch_fq_codel efi_pstore drm nfnetlink
dmi_sysfs qemu_fw_cfg virtio_rng ip_tables x_t
ables autofs4 iTCO_wdt intel_pmc_bxt iTCO_vendor_support psmouse
i2c_i801 i2c_mux serio_raw i2c_smbus ahci libahci lpc_ich
Jan 16 11:06:59 incus-podman kernel: CPU: 0 UID: 1000000 PID: 981
Comm: crun Tainted: G D 6.18.4-zabbly+ #debian13
PREEMPT(voluntary)
Jan 16 11:06:59 incus-podman kernel: Tainted: [D]=DIE
Jan 16 11:06:59 incus-podman kernel: Hardware name: QEMU Standard PC
(Q35 + ICH9, 2009)/Incus, BIOS unknown 02/02/2022
Jan 16 11:06:59 incus-podman kernel: RIP:
0010:rcu_note_context_switch+0x523/0x590
Jan 16 11:06:59 incus-podman kernel: Code: ff 49 89 96 a8 00 00 00 e9
35 fd ff ff 45 85 ff 75 ef e9 2b fd ff ff 48 c7 c7 f0 db 7f a8 c6 05
25 4a 2c 02 01 e8 9d 36 f2 ff <0f> 0
b e9 23 fb ff ff 4d 8b 74 24 20 4c 89 f7 e8 09 97 fa 00 41 c6
Jan 16 11:06:59 incus-podman kernel: RSP: 0018:ffffcc4900efbc50 EFLAGS: 00010046
Jan 16 11:06:59 incus-podman kernel: RAX: 0000000000000000 RBX:
ffff898295dc9b80 RCX: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI:
0000000000000000 RDI: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: RBP: ffffcc4900efbc78 R08:
0000000000000000 R09: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11:
0000000000000000 R12: ffff8982ea633600
Jan 16 11:06:59 incus-podman kernel: R13: 0000000000000000 R14:
ffffcc4900efbe00 R15: ffff898295dca980
Jan 16 11:06:59 incus-podman kernel: FS: 0000000000000000(0000)
GS:ffff898340bfc000(0000) knlGS:0000000000000000
Jan 16 11:06:59 incus-podman kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Jan 16 11:06:59 incus-podman kernel: CR2: 00007f2eb92ec4a8 CR3:
000000005e234006 CR4: 0000000000372ef0
Jan 16 11:06:59 incus-podman kernel: Call Trace:
Jan 16 11:06:59 incus-podman kernel: <TASK>
Jan 16 11:06:59 incus-podman kernel: __schedule+0xc6/0x1310
Jan 16 11:06:59 incus-podman kernel: ? try_to_wake_up+0x392/0x8a0
Jan 16 11:06:59 incus-podman kernel: ? kthread_insert_work+0xb8/0xe0
Jan 16 11:06:59 incus-podman kernel: schedule+0x27/0xf0
Jan 16 11:06:59 incus-podman kernel: synchronize_rcu_expedited+0x1c2/0x220
Jan 16 11:06:59 incus-podman kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Jan 16 11:06:59 incus-podman kernel: ? __pfx_wait_rcu_exp_gp+0x10/0x10
Jan 16 11:06:59 incus-podman kernel: namespace_unlock+0x295/0x380
Jan 16 11:06:59 incus-podman kernel: put_mnt_ns+0x79/0xb0
Jan 16 11:06:59 incus-podman kernel: free_nsproxy+0x16/0x190
Jan 16 11:06:59 incus-podman kernel: switch_task_namespaces+0x74/0xa0
Jan 16 11:06:59 incus-podman kernel: exit_task_namespaces+0x10/0x20
Jan 16 11:06:59 incus-podman kernel: do_exit+0x2a5/0xa20
Jan 16 11:06:59 incus-podman kernel: make_task_dead+0x93/0xa0
Jan 16 11:06:59 incus-podman kernel: rewind_stack_and_make_dead+0x16/0x20
Jan 16 11:06:59 incus-podman kernel: RIP: 0033:0x7f361659c687
Jan 16 11:06:59 incus-podman kernel: Code: Unable to access opcode
bytes at 0x7f361659c65d.
Jan 16 11:06:59 incus-podman kernel: RSP: 002b:00007fff89de51f0
EFLAGS: 00000202 ORIG_RAX: 000000000000002f
Jan 16 11:06:59 incus-podman kernel: RAX: ffffffffffffffda RBX:
00007f3616418840 RCX: 00007f361659c687
Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI:
00007fff89de5240 RDI: 0000000000000009
Jan 16 11:06:59 incus-podman kernel: RBP: 00007fff89de5240 R08:
0000000000000000 R09: 0000000000000000
Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11:
0000000000000202 R12: 00007fff89de58c0
Jan 16 11:06:59 incus-podman kernel: R13: 0000000000000007 R14:
00007fff89de58c0 R15: 000000000000000c
Jan 16 11:06:59 incus-podman kernel: </TASK>
Jan 16 11:06:59 incus-podman kernel: ---[ end trace 0000000000000000 ]---
Jan 16 11:07:58 incus-podman kernel: rcu: INFO: rcu_preempt detected
stalls on CPUs/tasks:
Jan 16 11:07:58 incus-podman kernel: rcu: Tasks blocked on
level-0 rcu_node (CPUs 0-7): P981/1:b..l
Jan 16 11:07:58 incus-podman kernel: rcu: (detected by 0,
t=60013 jiffies, g=8493, q=965 ncpus=2)
Jan 16 11:07:58 incus-podman kernel: task:crun state:D
stack:0 pid:981 tgid:981 ppid:980 task_flags:0x40014c
flags:0x00080001
Jan 16 11:07:58 incus-podman kernel: Call Trace:
Jan 16 11:07:58 incus-podman kernel: <TASK>
Jan 16 11:07:58 incus-podman kernel: __schedule+0x468/0x1310
Jan 16 11:07:58 incus-podman kernel: ? try_to_wake_up+0x392/0x8a0
Jan 16 11:07:58 incus-podman kernel: schedule+0x27/0xf0
Jan 16 11:07:58 incus-podman kernel: synchronize_rcu_expedited+0x1c2/0x220
Jan 16 11:07:58 incus-podman kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Jan 16 11:07:58 incus-podman kernel: ? __pfx_wait_rcu_exp_gp+0x10/0x10
Jan 16 11:07:58 incus-podman kernel: namespace_unlock+0x295/0x380
Jan 16 11:07:58 incus-podman kernel: put_mnt_ns+0x79/0xb0
Jan 16 11:07:58 incus-podman kernel: free_nsproxy+0x16/0x190
Jan 16 11:07:58 incus-podman kernel: switch_task_namespaces+0x74/0xa0
Jan 16 11:07:58 incus-podman kernel: exit_task_namespaces+0x10/0x20
Jan 16 11:07:58 incus-podman kernel: do_exit+0x2a5/0xa20
Jan 16 11:07:58 incus-podman kernel: make_task_dead+0x93/0xa0
Jan 16 11:07:58 incus-podman kernel: rewind_stack_and_make_dead+0x16/0x20
Jan 16 11:07:58 incus-podman kernel: RIP: 0033:0x7f361659c687
Jan 16 11:07:58 incus-podman kernel: RSP: 002b:00007fff89de51f0
EFLAGS: 00000202 ORIG_RAX: 000000000000002f
Jan 16 11:07:58 incus-podman kernel: RAX: ffffffffffffffda RBX:
00007f3616418840 RCX: 00007f361659c687
Jan 16 11:07:58 incus-podman kernel: RDX: 0000000000000000 RSI:
00007fff89de5240 RDI: 0000000000000009
Jan 16 11:07:58 incus-podman kernel: RBP: 00007fff89de5240 R08:
0000000000000000 R09: 0000000000000000
Jan 16 11:07:58 incus-podman kernel: R10: 0000000000000000 R11:
0000000000000202 R12: 00007fff89de58c0
Jan 16 11:07:58 incus-podman kernel: R13: 0000000000000007 R14:
00007fff89de58c0 R15: 000000000000000c
Jan 16 11:07:58 incus-podman kernel: </TASK>
Jan 16 11:08:00 incus-podman kernel: rcu: INFO: rcu_preempt detected
expedited stalls on CPUs/tasks: { P981 } 61834 jiffies s: 873 root:
0x0/T
Jan 16 11:08:00 incus-podman kernel: rcu: blocking rcu_node structures
(internal RCU debug):