Dear Maintainer,
* What led up to the situation?
After upgrading from buster to bullseye I determined that unattended-upgrade is
working but no update is found. But I know that there must be some updates
available. After some research I found out, that the package-cache
(var/cache/apt/srcpkgcache.bin) is not updated. After doing "apt update"
manually unattended-upgrade is installing the available updates.
The cause of this issue is, that the systemd-timer "apt-daily.timer" is not
executed, because my computer is often powered off during the time the "apt-
daily.timer" should be executed. In the configuration of "apt-daily.timer" the
option "Persistent=true" is set, so the timer should be executed immediately
after the next boot if the execution was missed. This is never happening and so
unattended-upgrades won't find any updates to install.
* What exactly did you do (or not do) that was effective (or
ineffective)?
After running "systemctl edit apt-daily.timer" and changing the Option
"RandomizedDelaySec=12h" to "RandomizedDelaySec=0h" the timer works as
expected. If the execution time was missed the timer is executed immediately
after next boot.
Output of "systemctl list-timers":
NEXT LEFT LAST PASSED
UNIT ACTIVATES
Wed 2021-10-27 14:11:36 CEST 12min left n/a n/a
systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2021-10-27 14:51:15 CEST 52min left Tue 2021-10-26 21:10:27 CEST 16h ago
apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2021-10-27 22:29:19 CEST 8h left Sun 2021-10-24 19:34:38 CEST 2 days
ago apt-daily.timer apt-daily.service
Thu 2021-10-28 00:00:00 CEST 10h left Wed 2021-10-27 13:56:41 CEST 2min 34s
ago exim4-base.timer exim4-base.service
Thu 2021-10-28 00:00:00 CEST 10h left Wed 2021-10-27 13:56:41 CEST 2min 34s
ago logrotate.timer logrotate.service
Thu 2021-10-28 00:00:00 CEST 10h left Wed 2021-10-27 13:56:41 CEST 2min 34s
ago man-db.timer man-db.service
Thu 2021-10-28 00:00:00 CEST 10h left Wed 2021-10-27 13:56:41 CEST 2min 34s
ago mlocate.timer mlocate.service
Sun 2021-10-31 03:10:09 CET 3 days left Sun 2021-10-24 11:02:10 CEST 3 days
ago e2scrub_all.timer e2scrub_all.service
n/a n/a Wed 2021-10-27 13:57:55 CEST 1min 20s
ago anacron.timer anacron.service
9 timers listed.
Pass --all to see loaded but inactive timers, too.
Content of "/etc/systemd/system/timers.target.wants/apt-daily.timer":
[Unit]
Description=Daily apt download activities
[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
[Install]
WantedBy=timers.target
Hello, unfortunately, I can not confirm the issue (thus marking the bug report accordingly). Persistent timers are correctly "remembered" when shutdown/rebooted. Have you seen that apt-daily-upgrade.service has ConditionACPower=true Maybe your system is not updated because it ran on battery when the timer fired? Can you send me the "journalctl -u apt-daily-upgrade.service" log after the dist upgrade until you locally modified the timer. Regards, Michael
Hello Michael, my system isn't running on battery because it's a Desktop-PC and not a laptop. I've observed this behaviour now on 3 of my computers (2 Desktop-PCs and 1 laptop). Attached you can find the output of your "journalctl", but on this system I haven't already changed the timer. If you need the output from a system which has already a modified timer please contact me. Regards Hendrik
I've attached the output of "journalctl -u apt-daily.service" and as you can see, it has not been running as often as "apt-daily-upgrade.service"
- Is this issue only reproducible with apt-daily.timer and apt-daily-upgrade.timer works correctly? - Does this issue happen reproducibly? Say "systemctl status apt-daily.timer" shows that the timer is about to elapse in 5 hours. If you shut down the system before that and start it after that, apt-daily.service is not run with 100% certainty?
Can you mark in the log, when you upgraded your system and at what times you shutdown/started the system and where exactly you expected the timer to fire Looking at the log, I currenly only see that it is executed rather regularly on a daily basis, i.e. the log looks rather normal.
I think so, but I will observe it for the next days. Yes, that's correct. I think it was at 15. August, one day after bullseye was released. That's different, but it's easy to see when you compare both logs. As you can see the last execution of "apt-daily" was "Okt 24 19:34:45". In the log of "apt-daily-upgrade" you see that the system was running at "Okt 25 16:08:30" and "Okt 26 21:10:30" and "Okt 27 15:45:00". So with the options of "apt-daily.timer" (OnCalendar=*-*-* 6,18:00 RandomizedDelaySec=12h) it should have been executed since then because there were missed timers. For normal the computer is not running for only 1 minute so that the timespan is too short for execution (apt-daily-upgrade had also enough time for execution).
Hello once again So, I reread the documentation: file which indicates, when the corresponding service was last activated. This is true in my case. Important is this sentence: "Such triggering is nonetheless subject to the delay imposed by RandomizedDelaySec=" Keep in mind that RandomizedDelaySec= is not stored on disk, only the timestamp of the last execution. So, let's assume the service was last triggered 2 days ago and you shut down the system yesterday. You boot the system this morning at 9:00, so the timer has elapsed. But a RandomizedDelaySec= is applied, which is computed dynamically. Let's say the delay is 7 hours. If you shut down the system before that, your service won't be executed. The next day when you start your system again, systemd will notice again, that the timer has elapsed and again will apply a RandomizedDelaySec=12h, in this case maybe 9 hours. If you only ever use your system for a very short period of time, this would mean that you never actually see the service being executed. A RandomizedDelaySec=12h is rather large, if you run your system for only 6 hours a day, you have a chance of hitting the timer by 50% I'd say.
This would be consistent with your observation that by reducing RandomizedDelaySec= to 0h you are reliably triggering the timer. So, what you see is actually consistent with the documented behaviour I'd argue. Nonetheless, if I understand you correctly there is a difference of behaviour between buster and bullseye if I understand you correctly. I went through the changelog/NEWS file of systemd. This seems to be related: v247: and from the man page
I'm putting Julian (the maintainer of apt and therefor apt-daily.timer) into the loop here. Julian, please have a look at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=997943#42 and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=997943#47 What were your expectations regarding the timer and the currently documented behavior? Afaics, Persistent=true is functional, but the behaviour of RandomizedDelaySec= changed slighly which might affect timers which chose a long delay.
Regarding the title of this bug report: Afaics the Persistent= attribute is not ignored but the interpretation of RandomizedDelaySec= has changed
Possibly related https://github.com/systemd/systemd/pull/11608
The commit message explains it rather well: https://github.com/systemd/systemd/pull/11608/commits/a87c1d3a979f8c2641471bed93577558a1027a24
I believe that on the one hand this is a significant improvement for us, as the old situation meant that all your containers would run unattended-upgrades at boot which was a terrible experience. Or was it at resume where I always had problems with it? But users not getting their systems updated once a day is a major concern as well. Unfortunately the answer is hard: You might only resume your laptop for 5 minutes a day, so even a RandomizedBootDelaySec option would not help if we'd set it to 30 min (I assume resume acts the same way as boot? I have not used either in a long time with timers). Need to think more, but certainly not running directly at boot is something we actively want.
Hello Michael and Julian, that explains why the behaviour of "RandomizedDelaySec" together with "Persistent" is different between buster and bullseye. If that's the intended behaviour, my systems seems to work like they should and it's no bug. So for me I will change the delay from 12h to maybe 5min. I think that will fit better to my use case. Thanks a lot for your research and explanation of the timers. Regards Hendrik
It's definitely a change of behaviour I wasn't aware of myself. While it fixes the thundering herd issue during boot, it is definitely problematic that for systems that aren't up for a longer period of time, a timer with a large RandomizedDelaySec= will basically never fire. I do think that we have a similar thundering herd issue with timers that elapse while the system is suspended and ttbomk, systemd will *not* reapply a RandomizedDelaySec= on resume. This appears to be inconsistent imho. Somehow I think systemd could be more clever here and scale down RandomizedDelaySec= on boot depending on how far in the past the timestamp of the stampfile is. Say the last activation of the service was 12 hours ago, then scale down RandomizedDelaySec=12h by dividing it by 12, so it would become RandomizedDelaySec=1h. The further in the past, the shorter the delay would become and the more likely that the timer fires eventually. And for consistencies sake, a (scaled) RandomizedDelaySec=1 should also be reapplied on resume. Regards, Michael
codesearch.debian.net shows the following timers with a RandomizedDelaySec of 12h, which will likely run into the same problem. fwupd_1.5.7-5/data/motd/fwupd-refresh.timer [Timer] OnCalendar=*-*-* 6,18:00 RandomizedDelaySec=12h Persistent=true apt_2.3.11/debian/apt-daily.timer [Timer] OnCalendar=*-*-* 6,18:00 RandomizedDelaySec=12h Persistent=true plocate_1.1.12-1/plocate-updatedb.timer [Timer] OnCalendar=daily RandomizedDelaySec=12h AccuracySec=20min Persistent=true python-certbot_1.18.0-1/debian/certbot.timer [Timer] OnCalendar=*-*-* 00,12:00:00 RandomizedDelaySec=43200 Persistent=true josm-installer_0.0.1+svn18171/josm-installer.timer [Timer] OnCalendar=*-*-* 00,12:00:00 RandomizedDelaySec=12h Persistent=true
That's right, if systemd would be capable of this behaviour, it would be very helpfull to not end up in a never updated system for example (like it might happen to my systems).
Hello, I first came across this problem on a computer I maintain for an elderly relative who uses it only occasionally, typically two or three times a week for about an hour each time. As that includes online banking, the system was configured to use the "unattended-upgrades" package to keep up with security updates. After the upgrade from Debian 10 to Debian 11 (switch from anacron to systemd as the entity triggering automatic upgrades), I noticed that while unattended-upgrades was still being executed, it didn't find any packages to upgrade any more although such packages had been released. At the time I fixed that by brute force with a daily anacron job for "apt-get update". Recently I observed similar behaviour on my own PC (three days without installing security updates although such packages existed already for at least two days). I therefore looked closer into this behaviour and traced it to the problem described in this report, i.e., the "RandomizedDelaySec=12h" line in the default configuration for apt-daily.timer (which is part of the "apt" package and not of the "systemd" package as implied by the current assignation of this report). In the scenario as described above (typical run time 1 hour), this leads to the system having only a chance of 1/12 (8 %) per boot session to update its package lists, irrespective of how often it boots. (Note that in contrast apt-daily-upgrade.timer is configured with "RandomizedDelaySec=60m".) Systems running continuously should obviously have no problems with the default configuration, while for the use case above I would consider this behaviour totally unacceptable for security reasons. In my opinion the "apt" package should either provide a default configuration which achieves the (presently misleading) "daily" claim implied by the name of the units or at least prominently document the limitations of the configuration so users will know when they have to take action. For those who prefer to do the latter by restoring the old behaviour I append a shell script I'm presently using with anacron. Because apt-compat refuses to act if systemd is running at all, irrespective of the status of the apt timers, we can't use it unmodified in its present form; I suggest that it should be changed to behave similar to my script which would make the latter obsolete. The initial comments in the script describe how it is intended to be used. Regards, Martin Lottermoser
Hello, I encountered the same issue here. Since bookworm update, apt upgrade happens only once in a few days. I can confirm that this happens because apt-daily timer executes with random delay up to 12 hours, which on my home computer, turned on only for few hours a day, often does not happen while the system is up, even multiple days. The result is as reported, that the apt-deily-upgrade (which uses work of apt-daily) often happen with delay of days. Creating override unit using "systemctl edit apt-daily.timer" and defining "RandomizedDelaySec=1h" does fix the behaviour.