Dear Maintainer,
On a machine with no NVMe hardware at all, prometheus-node-exporter-nvme.timer
is armed and prometheus-node-exporter-nvme.service runs every 15 minutes. It
produces a nvme.prom that carries no device metric whatsoever, only the
collector's own version gauge:
$ grep -v '^#' /var/lib/prometheus/node-exporter/nvme.prom
nvme_nvmecli{version="2.13"} 1.0
The cause is the glob in prometheus-node-exporter-nvme.timer:
ConditionFileIsExecutable=/usr/sbin/nvme
ConditionPathExistsGlob=/dev/nvme*
/dev/nvme-fabrics matches that glob, but it is not an NVMe device. It is the
NVMe-over-Fabrics control node, a misc character device created when the
nvme_fabrics module is loaded:
$ ls -l /dev/nvme-fabrics
crw------- 1 root root 10, 261 Aug 26 08:53 /dev/nvme-fabrics
Character device, major 10 (misc). Actual NVMe controllers and namespaces are
named /dev/nvme0, /dev/nvme0n1 and so on: they always carry a digit right
after "nvme".
WHY THE NODE IS PRESENT ON A MACHINE WITH NO NVMe
The chain is self-sustaining, and installing this package is enough to close
it:
1. prometheus-node-exporter-collectors recommends nvme-cli, and apt installs
Recommends by default;
2. nvme-cli ships /etc/nvme/discovery.conf -- a fully commented example file,
no active directive -- and nvmf-autoconnect.service, whose triggering
condition tests exactly that path:
ConditionPathExists=|/etc/nvme/config.json
ConditionPathExists=|/etc/nvme/discovery.conf
3. the condition is therefore satisfied on every boot, the service runs, and
it pulls in modprobe@nvme_fabrics.service via Wants=;
4. the module is loaded and creates /dev/nvme-fabrics;
5. the timer's glob matches.
Observed on this machine (Debian 13 trixie, arm64, Raspberry Pi 4, two USB
SATA disks, no PCIe and therefore no NVMe capability at all):
$ lsblk -o NAME,TRAN
NAME TRAN
sda usb
sdb usb
mmcblk0 mmc
$ lsmod | grep -i nvme
nvme_fabrics 36864 0
nvme_keyring 20480 1 nvme_fabrics
nvme_core 192512 1 nvme_fabrics
nvme_auth 24576 1 nvme_core
$ systemctl status nvmf-autoconnect.service
Active: inactive (dead) ... Main PID: 445941 (code=exited, status=0/SUCCESS)
$ journalctl -b -u modprobe@nvme_fabrics.service
Starting modprobe@nvme_fabrics.service - Load Kernel Module nvme_fabrics...
Finished modprobe@nvme_fabrics.service - Load Kernel Module nvme_fabrics.
Note that ConditionFileIsExecutable=/usr/sbin/nvme cannot help here: nvme-cli
is a Recommends of this very package, so on a default installation the binary
is always present. The glob is the only filter that can actually discriminate.
By contrast, prometheus-node-exporter-smartmon.timer behaves correctly on a
machine with no SATA/SAS disk, because its globs name device families that only
exist when hardware does:
ConditionPathExistsGlob=|/dev/sd*
ConditionPathExistsGlob=|/dev/hd*
SUGGESTED FIX
In debian/prometheus-node-exporter-nvme.timer:
-ConditionPathExistsGlob=/dev/nvme*
+ConditionPathExistsGlob=/dev/nvme[0-9]*
Every real controller and namespace carries a digit after "nvme", so nothing is
lost, while the fabrics control node no longer matches.
IMPACT
Small but not nil, and it applies to every Debian machine that has nvme-cli
installed and no NVMe hardware: one python3 process plus sponge four times an
hour, a nvme.prom with no useful content permanently exposed by node_exporter,
and roughly 300 journal lines a day.
Thanks for maintaining this package.