Dear Maintainer,


   * What led up to the situation?

        During a power cut long enough to bring my UPS batteries to a
        critical level, I noticed that my server was not shut down by
        NUT as I expected.

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

        An UPS is connected to the server and managed by NUT.
        (daemon /sbin/upsmon = nut-monitor)
        NUT is configured to switch off the server in "low battery"
        state.
        The UPS regularly performs a battery self-test, which is
        recognized by UPSMON as a calibration.
        A few days later, there was a power cut.

   * What was the outcome of this action?

        During a power cut long enough to bring the battery level to a
        critical threshold, UPSMON didn't initiate the shutdown
        procedure because it thought the UPS was still in calibration,
        which was wrong, and the server was finally abruptly power down when
        the battery was drained.

   * What outcome did you expect instead?

        When the battery charge level reached a threshold defined as "low
        battery", UPSMON should have initiated the server shutdown procedure.


This bug is known on the github of networkupstools here :
https://github.com/networkupstools/nut/issues/2168
Bug fix here :
https://github.com/networkupstools/nut/pull/2169/commits/b606d757660b85ba9123757609e6ac70a1684cbf


Further Informations :

UPSMON detects a battery self-test as a calibration and records this
state with a flag.

However, it will never clear this flag.

During a power failure, when the battery charge level reaches "low
battery" state, the is_ups_critical function is called.

This detects the "on battery" and "low battery" states, but since the
"calibration" flag is not cleared, the function will block the rest of
the expected procedure. A log entry is created:
is_ups_critical: seems that UPS [Onduleur] is OB+LB now, but it is also
calibrating - not declaring a critical state

This behavior is easy to reproduce :

First test : (Shows that NUT is correctly configured)
- After starting the nut-monitor daemon, simulate a power failure
- The station is powered by the UPS battery
- Battery charge level becomes low
- UPSMON detects this and correctly shuts down the server

Second test : (Show the bug)
- After starting the nut-monitor daemon, simulate or wait for a battery
  self-test
- Wait for the test to be successful, so the station is powered by the
  mains
- Simulate a power failure
- The station is powered by the UPS battery
- Battery charge level becomes low
- UPSMON detects this but thinks the UPS is in calibration. A log
  entry is created, but the station will not be shut down

Third test : (Get around the problem)
- After starting the nut-monitor daemon, simulate or wait for a
  battery self-test.
- Wait for the test to be successful, so the station is powered
  by the mains.
- Restart the nut-monitor daemon (only this one)
- Simulate a power failure
- The station is powered by the UPS battery
- Battery charge level becomes low
- UPSMON detects this and correctly switches off the station


To solve the problem quickly:

Using the NUT v2.8.0 sources provided by Debian Bookworm
(apt source nut),
edit the file ./nut-2.8.0/clients/upsmon.c

Add this just after line 1678:
  if (!strstr(status, "CAL"))
        clearflag(&ups->status, ST_CAL);

see
https://github.com/networkupstools/nut/pull/2169/commits/b606d757660b85ba9123757609e6ac70a1684cbf

Thank you very much
Fabien

Dear Maintainer,

I have hit this bug too. Our server (hypervisor with several virtual
machines) did not shut down properly during power outage due to this
bug. Looking at other related reports, it does not matter what type of
UPS is used.

This bug makes the upsmon quite useless.
Please backport this fix to Debian stable.
Thank you in advance.

This is what I got in syslog during power outage:

May 17 07:43:16 xxx nut-monitor[2380]: UPS xxx@localhost on battery
May 17 08:51:26 xxx nut-monitor[2380]: UPS xxx@localhost battery is low
May 17 08:51:31 xxx nut-monitor[2380]: is_ups_critical: seems that UPS
[xxx] is OB+LB now, but it is also calibrating - not declaring a
critical state
... repeated many times for the following 5-6 minutes, until the server
died.

I ran into the exact same bug (with the same error messages Vladislav noted) on my Raspberry Pis running the latest version of Pi OS Bookworm (downstream from Debian).

I had triggered a self test just before doing an end-to-end test of the UPS configuration, and my Ubuntu servers running 20.04 all shut down, but my Pis running Debian 12 did not, due to this bug!

Dear Maintainer,

The harmful effect of this bug is greater than the original
description. It not only causes the failure to poweroff during
a fault, but also causes unexpected poweroffs when there's no
fault.

If nut-monitor is used, and the UPS has undergone a calibration
cycle, with the "CAL" flag active, the "CAL" status would remain
active permanently. Later, if nut-monitor disconnects from the
server (e.g. due to a server reboot), nut-monitor would determine
incorrectly that the UPS went offline in the middle of a
calibration, interpreting as a failed UPS, thus shutting down the
client.

As a result, any server reboots may cause unexpected poweroffs on
the client. All downstream distributions are affected, an example
on Debian 13.5 is provided.

    nut-monitor[1644]: Poll UPS [huawei@10.10.10.101] failed - Server disconnected
    nut-monitor[1644]: Communications with UPS huawei@10.10.10.101 lost
    nut-monitor[1644]: UPS [huawei@10.10.10.101] was last known to be calibrating and currently is not communicating, assuming dead
    nut-monitor[1644]: FSD set on UPS huawei@10.10.10.101 failed: Driver not connected
    nut-monitor[1644]: Executing automatic power-fail shutdown
    nut-monitor[1644]: Auto logout and shutdown proceeding
    nut-monitor[2622026]: Network UPS Tools upsmon 2.8.1

This problem has already been fixed on NUT 2.8.2. But since Debian
Stable will continue to be based on NUT 2.8.1 for over a year, please
consider cherry-picking the patch. Namely, the ups_is_notcal() logic
from NUT 2.8.2 to NUT 2.8.1.

For more information, see:

* Nut shutdown assuming dead ups #2794
https://github.com/networkupstools/nut/issues/2794

* Commit c174a8f: clients/upsmon.{c,h} et al: add notification support for NOTCAL state change [#2169]
https://github.com/networkupstools/nut/commit/c174a8f387ce0337aea9ef611831be5ca2b828d3

Thanks,
Tom Li