I'm running Debian bullseye with systemd. I also have installed mdadm and nut-server. Looking in /lib/systemd/system-shutdown, I see two drop-in files with contents as follows: mdadm.shutdown: #!/bin/sh # We need to ensure all md arrays with external metadata # (e.g. IMSM, DDF) are clean before completing the shutdown. /sbin/mdadm --wait-clean --scan nutshutdown: #!/bin/sh /sbin/upsmon -K >/dev/null 2>&1 && /sbin/upsdrvctl shutdown The man page for systemd-poweroff.service explains that: All executables in this directory are executed in parallel This is the problem. mdadm.shutdown is trying to delay the shutdown action until data integrity can be guaranteed (for arrays with external metadata). Meanwhile, nutshutdown will immediately turn off the power. These are obviously incompatible. If execution were changed from parallel to sequential with a deterministic ordering, this race condition could be avoided. There may also be other ways to address the problem, such as relocating the mdadm stuff to some other spot in the systemd world. This is speculation on my part, as I'm no expert on systemd. I should note that I do not personally have any md arrays with external metadata. I do, however, have a UPS powering my Debian box and I can verify that orderly shutdown causes it to de-energize immediately at this point. I think it's self-evident that waiting for anything is doomed to fail when run in parallel with powering off the host. So, to be clear, this is a design defect I noticed, not a problem I'm experiencing. Still, it looks like a doozy, and I feel compelled to report it. I see this as a structural problem. It's clearly documented as the intended behavior. It doesn't depend on specific contents of my mdadm or nut configuration files, nor on my specific kernel. Nevertheless: # cat /etc/debian_version 11.7 # uname -a Linux vault 5.10.178-alex #8 SMP Fri May 19 15:44:12 PDT 2023 x86_64 GNU/Linux
Control: retitle -1 conflict between mdadm.shutdown and nutshutdown Control: reassign -1 nut-server 2.8.0-7 mdadm are relocating external mdadm This is nothing to do with src:systemd, both of these are shipped by the respective packages, if they do not interact well then they need to be fixed there. Reassigning to nutserver as cutting power during shutdown seems the slightly worst of the two, but will let the two maintainers figure out if that's the right destination.
It may not have been clear from my initial report, but nut is doing precisely the correct thing at exactly the right time by shutting down the power after the system is quiet and remaining mounts are read-only. Keep in mind that it's only doing this during the process of orderly shutdown due to power failure. Otherwise, /sbin/upsmon -K will return an error. It may be possible to relocate the mdadm logic. I don't know, but I'm pretty sure this is not a nut bug.
If, in /lib/systemd/system-shutdown there were just one file with the following contents: #!/bin/sh # We need to ensure all md arrays with external metadata # (e.g. IMSM, DDF) are clean before completing the shutdown. /sbin/mdadm --wait-clean --scan # If this is a shutdown due to low battery, turn off the UPS /sbin/upsmon -K >/dev/null 2>&1 && /sbin/upsdrvctl shutdown Then the race condition would be gone. There would be no parallelism to worry about. But, this mostly flies in the face of the modern approach of per-package drop-in files, and may require package installers to do file editing. Not a happy prospect. Uncharacteristically, systemd doesn't provide any other means of managing dependencies or order for scripts in this directory. To me, a far more elegant solution would be for systemd-shutdown to provide the needed determinism, by running scripts serially. The drop-in files could then be: 50-mdadm.shutdown 99-nutshutdown
On Mon, 22 May 2023 12:24:15 -0700 Alex Meyer <debbug523@zezax.com> wrote: Hello, > I'm running Debian bullseye with systemd. I also have installed mdadm > and nut-server. > > Looking in /lib/systemd/system-shutdown, I see two drop-in files with > contents as follows: > > mdadm.shutdown: > > #!/bin/sh > # We need to ensure all md arrays with external metadata > # (e.g. IMSM, DDF) are clean before completing the shutdown. > /sbin/mdadm --wait-clean --scan > > nutshutdown: > > #!/bin/sh > /sbin/upsmon -K >/dev/null 2>&1 && /sbin/upsdrvctl shutdown > > The man page for systemd-poweroff.service explains that: > > All executables in this directory are executed in parallel > > This is the problem. The systemd-shutdown(8) manpage is explicitly saying: > Shortly before executing the actual system power-off/halt/reboot/kexec, systemd-shutdown will run all executables in /usr/lib/systemd/system-shutdown/. Those executables are called with one argument: either "poweroff", "halt", "reboot", or "kexec", depending on the chosen action. All executables in this directory are executed in parallel, *and execution of the action is not continued before all executables finished*. That means that, during normal shutdown, even if nutshutdown script exits, mdadm will have time to finish its job of flushing the metadata. Only in case of a shutdown triggered by the fact that the power of the UPS is low might indeed cause the power to be cut before mdadm has finished. This could be mitigated by using POWEROFF_WAIT to pause the execution of nutshutdown for several seconds, I think. There was a problem that could cause the power to be cut if the MODE was set to none in /etc/nut/nut.conf, that has been fixed in 2.8.3-1 Kind regards, Laurent Bigonville