Dear Maintainer,
slurmctld can be crashed (SIGSEGV) by a single "scontrol show assoc_mgr"
RPC whenever the association or QOS caches are NULL. This is the case
with AccountingStorageType=accounting_storage/none, and also during the
window after slurmctld has started but before it has connected to
slurmdbd. Any user able to reach the controller can trigger it, so it is
effectively a remotely-triggerable denial of service against the
controller (feel free to reclassify as grave/security if you agree).
I have verified the affected code by reading src/common/assoc_mgr.c on
every maintained upstream stable branch (22.05.11, 23.02.8, 23.11.11,
24.05.8, 24.11.7, 25.05.9, 25.11.8): all carry the same defect. Only the
upstream master branch (26.11) is fixed, via a large refactor
(commit 710f909b0c) that is too invasive to backport. A minimal,
targeted fix is attached below and is what I intend to forward upstream.
Steps to reproduce
------------------
1. Configure a controller with
AccountingStorageType=accounting_storage/none
(or start slurmctld while slurmdbd is unreachable).
2. Run: scontrol show assoc_mgr
3. slurmctld dies with SIGSEGV.
With no arguments, scontrol sets
req.flags = ASSOC_MGR_INFO_FLAG_ASSOC | ASSOC_MGR_INFO_FLAG_USERS |
ASSOC_MGR_INFO_FLAG_QOS (src/scontrol/info_assoc_mgr.c), so the
unguarded ASSOC path is always taken.
Backtrace (23.11, -DNDEBUG build)
---------------------------------
#0 __pthread_rwlock_wrlock (rwlock=0x30)
#2 list_iterator_create (l=0x30) at src/common/list.c
#3 assoc_mgr_info_get_pack_msg (...) at src/common/assoc_mgr.c:3614
#4 _slurm_rpc_assoc_mgr_info (...) at src/slurmctld/proc_req.c
#5 slurmctld_req (...) at src/slurmctld/proc_req.c
#6 _service_connection (...) at src/slurmctld/controller.c
Root cause
----------
assoc_mgr_assoc_list is NULL at init and only assigned a real list once
_get_assoc_mgr_assoc_list() succeeds (or via an assoc update RPC). In
assoc_mgr_info_get_pack_msg() the three sections are inconsistent:
ASSOC : if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC)) goto no_assocs;
-> no NULL check, then list_iterator_create(assoc_mgr_assoc_list)
QOS : if (!(flags & ASSOC_MGR_INFO_FLAG_QOS)) { ... }
-> no NULL check
USERS : if (!(flags & ASSOC_MGR_INFO_FLAG_USERS) || !assoc_mgr_user_list)
-> already guarded
In a release build, xassert(l != NULL) inside list_iterator_create() is
compiled out (xassert.h: "#ifdef NDEBUG #define xassert(expr) ((void)(0))"),
so control reaches slurm_rwlock_wrlock(&l->mutex) with l == NULL. The
mutex field is at offset 0x30 in struct xlist, which is exactly the
rwlock=0x30 fault address in the backtrace.
The fix simply extends the existing USERS-style NULL guard to the ASSOC
and QOS sections. When the list is NULL the section is packed empty,
which is the same behaviour the USERS path already has.
Patch
-----
The following also carries a DEP-3 header and can be dropped straight
into debian/patches/ (remember to add it to debian/patches/series). It
applies cleanly to the 23.11.4 source and, unchanged, to every other
affected stable branch.
--- BEGIN PATCH: fix-slurmctld-segfault-null-assoc-qos-list.patch ---
Description: Fix slurmctld SIGSEGV in assoc_mgr_info_get_pack_msg() on NULL lists
assoc_mgr_info_get_pack_msg() unconditionally iterates the association
and QOS caches when the corresponding request flag is set, but those
global lists are NULL until slurmdbd data has been loaded (e.g. with
accounting_storage/none, or before slurmdbd is connected). In a release
build the xassert() in list_iterator_create() is compiled out, so a NULL
list leads to a write-lock on address 0x30 and a segfault. Any client
issuing "scontrol show assoc_mgr" can crash the controller.
.
The USERS section already guards against a NULL list; this extends the
same guard to the ASSOC and QOS sections, so an empty section is packed
instead of crashing.
Author: Shengqi Chen <harry@debian.org>
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=NNNNNN
Forwarded: no
Last-Update: 2026-08-14
--- a/src/common/assoc_mgr.c
+++ b/src/common/assoc_mgr.c
@@ -3608,7 +3608,7 @@
assoc_mgr_lock(&locks);
- if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC))
+ if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC) || !assoc_mgr_assoc_list)
goto no_assocs;
itr = list_iterator_create(assoc_mgr_assoc_list);
@@ -3687,7 +3687,7 @@
list_iterator_destroy(itr);
list_flush(ret_list);
- if (!(flags & ASSOC_MGR_INFO_FLAG_QOS)) {
+ if (!(flags & ASSOC_MGR_INFO_FLAG_QOS) || !assoc_mgr_qos_list) {
tmp_list = ret_list;
goto no_qos;
}
--- END PATCH ---
Affected versions
-----------------
Confirmed present in upstream stable branches 22.05, 23.02, 23.11,
24.05, 24.11, 25.05 and 25.11. Fixed in upstream master (26.11) by
commit 710f909b0c. The minimal patch above is suitable for all stable
branches and for the Debian package.