#1146547 prometheus-node-exporter-collectors: nvme timer fires on systems with no NVMe hardware (/dev/nvme-fabrics matches the glob)

#1146547#5
Date:
2026-09-02 21:25:30 UTC
From:
To:
Dear Maintainer,

On a machine with no NVMe hardware at all, prometheus-node-exporter-nvme.timer
is armed and prometheus-node-exporter-nvme.service runs every 15 minutes. It
produces a nvme.prom that carries no device metric whatsoever, only the
collector's own version gauge:

    $ grep -v '^#' /var/lib/prometheus/node-exporter/nvme.prom
    nvme_nvmecli{version="2.13"} 1.0

The cause is the glob in prometheus-node-exporter-nvme.timer:

    ConditionFileIsExecutable=/usr/sbin/nvme
    ConditionPathExistsGlob=/dev/nvme*

/dev/nvme-fabrics matches that glob, but it is not an NVMe device. It is the
NVMe-over-Fabrics control node, a misc character device created when the
nvme_fabrics module is loaded:

    $ ls -l /dev/nvme-fabrics
    crw------- 1 root root 10, 261 Aug 26 08:53 /dev/nvme-fabrics

Character device, major 10 (misc). Actual NVMe controllers and namespaces are
named /dev/nvme0, /dev/nvme0n1 and so on: they always carry a digit right
after "nvme".

WHY THE NODE IS PRESENT ON A MACHINE WITH NO NVMe

The chain is self-sustaining, and installing this package is enough to close
it:

  1. prometheus-node-exporter-collectors recommends nvme-cli, and apt installs
     Recommends by default;
  2. nvme-cli ships /etc/nvme/discovery.conf -- a fully commented example file,
     no active directive -- and nvmf-autoconnect.service, whose triggering
     condition tests exactly that path:

         ConditionPathExists=|/etc/nvme/config.json
         ConditionPathExists=|/etc/nvme/discovery.conf

  3. the condition is therefore satisfied on every boot, the service runs, and
     it pulls in modprobe@nvme_fabrics.service via Wants=;
  4. the module is loaded and creates /dev/nvme-fabrics;
  5. the timer's glob matches.

Observed on this machine (Debian 13 trixie, arm64, Raspberry Pi 4, two USB
SATA disks, no PCIe and therefore no NVMe capability at all):

    $ lsblk -o NAME,TRAN
    NAME    TRAN
    sda     usb
    sdb     usb
    mmcblk0 mmc

    $ lsmod | grep -i nvme
    nvme_fabrics   36864  0
    nvme_keyring   20480  1 nvme_fabrics
    nvme_core     192512  1 nvme_fabrics
    nvme_auth      24576  1 nvme_core

    $ systemctl status nvmf-autoconnect.service
    Active: inactive (dead) ... Main PID: 445941 (code=exited, status=0/SUCCESS)

    $ journalctl -b -u modprobe@nvme_fabrics.service
    Starting modprobe@nvme_fabrics.service - Load Kernel Module nvme_fabrics...
    Finished modprobe@nvme_fabrics.service - Load Kernel Module nvme_fabrics.

Note that ConditionFileIsExecutable=/usr/sbin/nvme cannot help here: nvme-cli
is a Recommends of this very package, so on a default installation the binary
is always present. The glob is the only filter that can actually discriminate.

By contrast, prometheus-node-exporter-smartmon.timer behaves correctly on a
machine with no SATA/SAS disk, because its globs name device families that only
exist when hardware does:

    ConditionPathExistsGlob=|/dev/sd*
    ConditionPathExistsGlob=|/dev/hd*

SUGGESTED FIX

In debian/prometheus-node-exporter-nvme.timer:

    -ConditionPathExistsGlob=/dev/nvme*
    +ConditionPathExistsGlob=/dev/nvme[0-9]*

Every real controller and namespace carries a digit after "nvme", so nothing is
lost, while the fabrics control node no longer matches.

IMPACT

Small but not nil, and it applies to every Debian machine that has nvme-cli
installed and no NVMe hardware: one python3 process plus sponge four times an
hour, a nvme.prom with no useful content permanently exposed by node_exporter,
and roughly 300 journal lines a day.

Thanks for maintaining this package.