Kernel BPF JIT free path deadlock (bpf_prog_pack_free) wedges worker nodes. Customer reports a deadlock, no repro case is available and this is a production environment so we are not able to do any debugging. Customer has shared kernel output showing hung tasks. We can request more info from the customer if required. Customer have since confirmed the issue also occurs on: kernel 6.12.90 kernel 6.12.95 Signature (all consistent with the bpf_prog_pack_free / events-workqueue deadlock): loadavg 1038, but only 6 of 1841 threads runnable 1027 processes in uninterruptible D-state, of which ~1024 are kworker/[0-3]:N+events — i.e. the kernel "events" workqueue is fully stalled and the kernel keeps spawning event kworkers that immediately block (proc-Dstate-comm-breakdown.txt, proc-Dstate-processes-with-stacks.txt). CPU essentially idle (~15%) the entire time load was pinned at ~1030 — confirming the threads are blocked, not busy. Not memory exhaustion: 1.6 GB free; SUnreclaim elevated vs a healthy node in the same cluster (393 MB vs 238 MB). buddyinfo shows high-order block depletion vs the healthy baseline === ALL HUNG_TASK + TRACES (boot -1) === Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task cilium-agent:2877 blocked for more than 120 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:cilium-agent state:D stack:0 pid:2877 tgid:2675 ppid:1618 flags:0x00004000 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_timeout+0x12f/0x160 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: wait_for_completion+0x8a/0x160 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __flush_work+0x269/0x350 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_wq_barrier_func+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task test_ebpf:3306652 blocked for more than 120 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:test_ebpf state:D stack:0 pid:3306652 tgid:3306652 ppid:3302464 flags:0x00000006 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:[2:INIT]:3306799 blocked for more than 120 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc:[2:INIT] state:D stack:0 pid:3306799 tgid:3306799 ppid:3306768 flags:0x00004006 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:[2:INIT]:3306818 blocked for more than 120 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc:[2:INIT] state:D stack:0 pid:3306818 tgid:3306818 ppid:3306802 flags:0x00004006 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:3306853 blocked for more than 120 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc state:D stack:0 pid:3306853 tgid:3306853 ppid:3306848 flags:0x00000002 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:3306874 blocked for more than 121 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc state:D stack:0 pid:3306874 tgid:3306862 ppid:3306847 flags:0x00004002 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:3306909 blocked for more than 121 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc state:D stack:0 pid:3306909 tgid:3306909 ppid:3306896 flags:0x00000002 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:3306944 blocked for more than 121 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc state:D stack:0 pid:3306944 tgid:3306944 ppid:3306933 flags:0x00004002 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:3307022 blocked for more than 121 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc state:D stack:0 pid:3307022 tgid:3307022 ppid:3306997 flags:0x00000002 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 -- Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: INFO: task runc:3307023 blocked for more than 121 seconds. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Tainted: G X 6.12.86+deb13-amd64 #1 Debian 6.12.86-1 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: task:runc state:D stack:0 pid:3307023 tgid:3307023 ppid:3307018 flags:0x00000002 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: Call Trace: Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: <TASK> Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __schedule+0x505/0xc00 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule+0x27/0xf0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: schedule_preempt_disabled+0x15/0x30 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: __mutex_lock.constprop.0+0x3d0/0x6d0 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 Jun 08 10:01:08 oobit-monolith-k8s-38itqh kernel: ? __pfx_jit_fill_hole+0x10/0x10 === KWORKER/EVENTS WORKQUEUE FREE-SIDE ===