Dear Maintainer,
Please consider applying the attached patch to the Debian slurm-wlm
package. It adds a new SlurmctldParameters option,
periodic_check_interval=#, that makes the slurmctld periodic background
loop interval configurable. The default is the existing PERIODIC_TIMEOUT
value (30s), so behavior is unchanged unless the option is set.
Motivation
----------
The slurmctld background thread uses a hard-coded PERIODIC_TIMEOUT (30s)
for the periodic timelimit / reservation / node-timer checks. After a
suspended or powered-down node resumes and registers, the queued job
waits for the next periodic pass before transitioning to RUNNING. The
delay is therefore up to one interval (0..30s depending on timing).
For on-prem clusters that suspend idle nodes and resume them via
wake-on-LAN, the wake/boot/register path is already fast; the remaining
delay is purely this controller-side poll. Making it tunable lets such
sites bring post-registration scheduling latency down to match the fast
wake.
Live A/B validation
-------------------
Tested on a 2-node on-prem Debian 13 cluster with a locally rebuilt
package (24.11.5-4+periodiccheck1), same node and workflow:
periodic_check_interval=2 registration-to-RUNNING = 1s
periodic_check_interval=30 registration-to-RUNNING = 14s
(14s rather than 30s because the loop is not anchored to registration;
the penalty is time-until-next-pass, bounded by the interval.)
Controller log excerpt for the 30s case:
23:20:48 Node nyc1 now responding
23:21:03 job_time_limit: Configuration for JobId=125 complete
These figures isolate the Slurm controller overhead only; HW+OS
wake/boot time is separate and unaffected by this option.
Patch details
-------------
- New helper get_periodic_check_interval() in src/slurmctld/controller.c
(caches parsed value until slurm.conf last_update changes; rejects 0
and INFINITE, falls back to default).
- Three call sites switched from the PERIODIC_TIMEOUT literal:
_slurmctld_background() in controller.c, job_time_limit() and
send_job_warn_signal() in job_mgr.c.
- Docs: doc/man/man5/slurm.conf.5, doc/html/power_save.shtml.
- Test: testsuite/python/tests/test_141_1.py (new test exercising the
reduced-interval cloud-node resume path).
The attached patch is a quilt-format patch with a DEP-3 header; it
applies cleanly to the slurm-wlm 24.11.5-4 source and I have built and
installed the resulting packages on Debian 13.
Upstream
--------
Also submitted upstream to SchedMD:
https://support.schedmd.com/show_bug.cgi?id=25294