#976101 fwupdtool keeps / fs busy on shutdown/reboot

Package:
fwupd
Source:
fwupd
Description:
Firmware update daemon
Submitter:
Josh
Date:
2020-12-25 15:54:05 UTC
Severity:
normal
#976101#5
Date:
2020-11-29 18:23:57 UTC
From:
To:
Dear Maintainer,

On two recent clean installations of Buster, I am noticing that choosing to
install package updates through gnome-software, and therefore the systemd
offline update system, results in the root filesystem not being unmounted
properly once packages are installed and therefore gives an fsck error on
reboot:

Failed to remount '/' read-only; Device or resource busy

Followed by journal recovery on reboot.

I haven't noticed any serious problems subsequently but it seems like asking
for trouble for the filesystem to not be cleanly unmounted right after packages
are updated. I have seen this on two machines, neither using 3rd party repos
etc.

Thanks for your time.
#976101#10
Date:
2020-11-29 18:45:00 UTC
From:
To:
Am 29.11.20 um 19:23 schrieb Josh:

Can you please provide a (debug) journal log from such an offline update.

Michael
#976101#17
Date:
2020-12-01 17:02:35 UTC
From:
To:
Sorry which log do you mean by debug journal log? Apologies for my
ignorance.

(Incidentally this problem has occured on two out of three offline
updates in recent days)
#976101#22
Date:
2020-12-01 17:42:14 UTC
From:
To:
Am 01.12.20 um 18:02 schrieb Josh Jones:

Enable persistent journal (see
/usr/share/doc/systemd/README.Debian.gz)

When booting into the offline based upgrade, add
"systemd.log_level=debug" to the kernel command line.

After booting into your regular system again after the upgrade, dump the
journal from the previous boot (as root)

journalctl -alb-1

(b-1 means, boot - 1, i.e. previous boot)

Michael
#976101#27
Date:
2020-12-01 22:46:47 UTC
From:
To:
Am 01.12.20 um 18:02 schrieb Josh Jones:

So, I tried this in a test VM.
The offline update went just fine (see attached journal.txt), so I can't
reproduce the issue locally and we will need at the very least at
journal log.

Michael
#976101#34
Date:
2020-12-01 22:54:44 UTC
From:
To:
No worries - I will send you the debug journal at the next package update - or if I have time tomorrow I will try to install outdated versions of the packages last updated to force it to happen. Whatever is happening is pretty consistent for me by the looks of it.

I have just enabled persistent logging.

Josh
#976101#39
Date:
2020-12-06 11:45:02 UTC
From:
To:
Please see attached debug log from last offline update. It failed to
unmount cleanly again. Apologies it took a few days to provide this.

Josh
#976101#44
Date:
2020-12-06 15:08:00 UTC
From:
To:
Am 06.12.2020 um 12:45 schrieb Josh Jones:

I don't see a failed unmount attempts in this log unfortunately. Or am I
missing something?
Which file system(s) in particular did not unmount cleanly?

You might also have a look at
https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/README.Debian

"Debugging boot/shutdown problems"
to gather logs after journald has been shut down.



Michael
#976101#49
Date:
2020-12-06 16:04:41 UTC
From:
To:
It is the root filesystem which is failing to be remounted read-only -
this is a simple setup with only one filesystem for the whole system.

It seems to be happening right at the very very end of the shutdown
sequence - I have attached a picture of the error; quality is poor
because I had to capture it from video footage. Is there any way to
even have anything saved to disk so late in the sequence?

Thanks for your help and attention with this; I'm willing to try
whatever you suggest to help work out what's going on - if that is even
possible..!


Josh
#976101#54
Date:
2020-12-06 18:28:00 UTC
From:
To:
Am 06.12.20 um 17:04 schrieb Josh Jones:

With the shutdown hook described in README.Debian and
https://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1
(install the hook as /lib/systemd/system-shutdown/debug.sh without the
/usr prefix) you can gather logs right until the end
#976101#59
Date:
2020-12-06 20:36:34 UTC
From:
To:
Am Sonntag, den 06.12.2020, 19:28 +0100 schrieb Michael Biebl:


Copy the attached script to /lib/systemd/system-shutdown/, make it
executable with chmod +x, then add
systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M
to the kernel command line and after the reboot, attach the file
/shutdown-log.txt to this bug report.

Michael
#976101#64
Date:
2020-12-08 19:39:16 UTC
From:
To:
Amazing, that has taught me something very useful!

shutdown-log.txt attached and error is visible!

Josh
#976101#69
Date:
2020-12-08 20:05:34 UTC
From:
To:
Am 08.12.20 um 20:39 schrieb Josh Jones:

Thanks.
So from a quick glance it seems fwupd keeps your / busy and prevents it
from being remounted ro.


https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953679 looks related

If you don't have a system/BIOS vendor supported by fwupd, you might
consider purging the package to verify that the problem is solved.



Michael
#976101#74
Date:
2020-12-08 22:25:56 UTC
From:
To:
Am 08.12.20 um 21:05 schrieb Michael Biebl:

Given the existing information, I'm going to reassign this to fwupd


Regards,
Michael
#976101#87
Date:
2020-12-25 15:48:40 UTC
From:
To:
I don't know enough to be sure of this, but I think perhaps this bug
should be reassigned against plymouthd rather than fwupd; I've attached
the output of "lsof /" both with and without plymouth, plymouthd and
libplymouth installed. You can see that plymouth has files open read-
write right at the end of the shutdown process.

When plymouth is installed, this bug occurs. When not installed it does
not - though admittedly different packages were updated in the working
example, after some time and various package updates later I have still
not seen this bug after removing plymouth.


Josh