#1120743 apt.systemd.daily update fails because it runs before network interfaces are ready

Package:
apt
Source:
apt
Description:
commandline package manager
Submitter:
Petter H
Date:
2025-11-15 19:37:02 UTC
Severity:
normal
#1120743#5
Date:
2025-11-15 18:20:07 UTC
From:
To:
(Likely affects multiple versions)

unattended-upgrades was not working (I let it try and fail for over a
month while there were known updates available, it never worked).  The
problem was with /usr/lib/apt/apt.systemd.daily.

'apt.systemd.daily update' was invoked, by systemd, after system wake
from suspend, before the network was available.  Even though it was
unsuccessful, it still touched the timestamp file indicating it had
already ran that day.  Subsequent attempts to run would simply be
skipped, since apt.systemd.daily checks the timestamp file, and aborts
if the stamp file indicates it has already ran that day.

The update failing because the script was run before network interfaces
were up, after a suspend/wake, was apparently a known issue, as a
partial work around was already included in apt-daily.service.

apt-daily.service includes the line:
ExecStartPre=-/usr/lib/apt/apt-helper wait-online

But, 'apt-helper wait-online' does nothing when using ifupdown.  You can
down the interface entirely, and 'apt-helper wait-online' will still,
immediately, return with a zero exit code, the apt.systemd.daily script
will attempt, unsuccessfully, to do the apt update, but the script will
set the flag, as if it was successful, so no further attempts will run
that same day.

Per this Ubuntu bug report where it appears 'apt-helper wait-online' was
introduced, there was no attempt to make it work for anything but
systemd networkd and network manager.

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1699850

Per that bug report, 'apt-helper wait-online' waits up to 60 minutes for
the network to become available.  The following hacky line, added to
apt-daily.service, accomplishes the same, and works regardless of how
network interfaces are managed.

# try once per minute for 60 minutes to see if network ready, if yes,
continue
ExecStartPre=/bin/bash -c "for ((i=1; i<60; i++)); do nc -z -w 1
deb.debian.org 80 2>/dev/null  && break || sleep 60; done"

This work around gets apt periodic's update to work again, so
unattended-upgrades works again; unattended-upgrades has picked up
updates each of the 3 days since putting this hacky work-around into
place; while failing to work each of the 40 days prior to putting the
work-around hack into place.

It seems there are multiple bugs:

1. apt periodic update should not touch the "stamp" file if there was an
error that prevented it from performing the update.

2. 'apt-helper wait-online' should work for all supported network
management tools, including ifupdown.

3. if systemd had sane behavior, 'apt-helper wait-online', and my
similar hack, would not be necessary, in the first place.

#1120743#10
Date:
2025-11-15 19:22:15 UTC
From:
To:
This is wrong, we also support connman.

This is not an acceptable way of waiting for online connectiveness.

It works correctly, touching the stamp file for the run, but not the
succesful stamp.

If ifupdown provides active network monitoring and a wait-online command
it can be added, otherwise you're on your own.

At some point when we have an apt daemon it may monitor netlink events
and then try to check for updates when it may be online, but it's a bit
off still.

Unfortunately the network-online.target is not dynamic, yes.

#1120743#15
Date:
2025-11-15 19:34:49 UTC
From:
To:
Sorry I did not add detail; I'm in a terrible stressed out place from
the boredom of a week off work.

So we can't go randomly phone home for privacy reasons; the
real solution is to check if a source is online and then update it
which is I think where we might just go once the daemon exists.