This is an odd one, that is very hard to trigger. I am acutely aware of what makes a good bug report and fully understand that this is not one of them, so please let me know what I can do to help create more clarity here.

I've observed on several systems that run both prometheus-node-exporter's systemd collector there is an eventual failure mode in which systemd calls never come back. I encountered it primarily due to using pacemaker's systemd OCF type, which also communicates with systemd over a dbus socket. The failure that would occur would be after a period of 60-90 days of uptime, dbus messages to systemd would begin to timeout with an error like:

prometheus-node-exporter[4315]: time="2022-11-28T12:25:30-05:00" level=error msg="ERROR: systemd collector failed after 0.082480s: couldn't get units: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)" source="collector.go:132"

This would continue until systemd would be incidentally restarted by some other operation, or until the node rebooted. In some cases, systemd could not be recovered and a reset would need to occur.

Unfortnately I have never been able to find firm evidence of the underlying defect, and was only turned on to the possibility of the issue through conversations with systemd contributors at conferences who firmly advised me NOT to use the dbus-systemd connector to talk to the daemon, as they were aware of some irregular buggy behavior that could cause socket failure.

I was able to reasonably prove to myself that the collector was at fault in some measure by modifying it run at a high frequency and observing that the aforementioned failure triggered more rapidly (from 60-90 days to 5-10 days). The issue seems to increase in likelihood (decrease in reproduction time) the more communication there is over that socket.

I'd recommend that Debian revert its patch that enables the systemd-collector by default, unless it can be demonstrated that this bug does not recur in bookworm+. I have seen reproductions it on Buster and Bullseye.

I'd like to propose the following patch:


diff --git a/debian/patches/0001-Debian-defaults.patch
b/debian/patches/0001-Debian-defaults.patch
index f8d93053..9f3fd310 100644
--- a/debian/patches/0001-Debian-defaults.patch
+++ b/debian/patches/0001-Debian-defaults.patch
@@ -1,6 +1,6 @@
 From: Martina Ferrari <tina@debian.org>
 Date: Mon, 13 Nov 2023 13:49:27 +0100
-Subject: Set defaults that make sense for Debian systems
+Subject: Remove deprecated options and set sane defaults for Debian

 Forwarded: not-needed

@@ -9,7 +9,6 @@ We remove the deprecated options completely because they
were introduced in
 the default value setting as it then makes the code consider it a usage and
 errors out when the legacy option is used.

This really ought to be reported upstream, since it is unlikely to be a
Debian-specific bug. The fact that the vanilla release options do not
enable the systemd collector by default is really just wallpapering over
the problem, and disabling the collector again is not really a
satisfactory solution.