Dear Maintainer,
* What led up to the situation?
(previous bug at #788193).
Automatic run of e2scrub reported:
So sorry, the automatic e2scrub of /tmp on mahogany.vultrusercontent.com failed.
A log of what happened follows:
× e2scrub@-tmp.service - Online ext4 Metadata Check for /tmp
Loaded: loaded (/lib/systemd/system/e2scrub@.service; static)
Active: failed (Result: exit-code) since Sun 2023-12-03 03:11:03
CET; 137ms ago
Docs: man:e2scrub(8)
Process: 25375 ExecStart=/sbin/e2scrub -t /tmp (code=exited,
status=1/FAILURE)
Main PID: 25375 (code=exited, status=1/FAILURE)
CPU: 227ms
Dec 03 03:11:00 mahogany systemd[1]: Starting e2scrub@-tmp.service -
Online ext4 Metadata Check for /tmp...
Dec 03 03:11:01 mahogany e2scrub@-tmp[25387]: Logical volume
"tmp.e2scrub" created.
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: e2fsck 1.47.0 (5-Feb-2023)
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: Pass 1: Checking inodes,
blocks, and sizes
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: Pass 2: Checking
directory structure
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: Pass 3: Checking
directory connectivity
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: /lost+found not found.
Create? yes
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: Pass 4: Checking reference counts
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: Pass 5: Checking group
summary information
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: /dev/system/tmp.e2scrub:
***** FILE SYSTEM WAS MODIFIED *****
Dec 03 03:11:01 mahogany e2scrub@-tmp[25415]: /dev/system/tmp.e2scrub:
21/124928 files (0.0% non-contiguous), 17515/499712 blocks
Dec 03 03:11:01 mahogany e2scrub@-tmp[25375]: /tmp: Scrub FAILED due
to corruption! Unmount and run e2fsck -y.
Dec 03 03:11:01 mahogany e2scrub@-tmp[25420]: tune2fs 1.47.0 (5-Feb-2023)
Dec 03 03:11:01 mahogany e2scrub@-tmp[25420]: Setting filesystem error
flag to force fsck.
Dec 03 03:11:01 mahogany e2scrub@-tmp[25422]: Logical volume
"tmp.e2scrub" successfully removed.
Dec 03 03:11:03 mahogany systemd[1]: e2scrub@-tmp.service: Main
process exited, code=exited, status=1/FAILURE
Dec 03 03:11:03 mahogany systemd[1]: e2scrub@-tmp.service: Failed with
result 'exit-code'.
Dec 03 03:11:03 mahogany systemd[1]: Failed to start
e2scrub@-tmp.service - Online ext4 Metadata Check for /tmp.
Dec 03 03:11:03 mahogany systemd[1]: e2scrub@-tmp.service: Triggering
OnFailure= dependencies.
* What exactly did you do (or not do) that was effective (or
ineffective)?
Ineffective: put /tmp on a separate filesystem, effective; add lost+found
to /usr/lib/tmpfiles.d;
cat /usr/lib/tmpfiles.d/tmp-lost-and-found.conf
x /tmp/lost+found
I can't find an exeption in the source nor in /usr/lib/tmpfiles.d/*
root@mahogany:/tmp/src/systemd-252.17/src/tmpfiles# ed tmpfiles.c
158825
/.journal
".journal",
-3,+3n
764 S_ISREG(sx.stx_mode) &&
765 sx.stx_uid == 0 &&
766 STR_IN_SET(de->d_name,
767 ".journal",
768 "aquota.user",
769 "aquota.group")) {
770 log_debug("Skipping \"%s\".", sub_path);
Maybe I'm missing something but if not can it be added to either
/usr/lib/tmpfiles.d or as a patch to tmpfiles.c?
I can confirm adding it to tmpfiles.d solves the problem for me.
Happy holidays, kind regards,
Axel
Control: tags -1 moreinfo mahogany.vultrusercontent.com failed. You have a separate, non-tmpfs filesystem on /tmp? How did you end up with such a setup?
Hi Luca, I always configure separate /var, /tmp and /home filesystems. In this case the system is also low on ram so tmpfs is not really an option (I think?). Is using tmpfs the preferred way to handle this on debian? Or maybe ext4 as a filesystem is not preferred in this case? Kind regards, Axel
I think the default is simply a directory in the rootfs in Debian - the more common setup is to have a tmpfs on it. But the main point is, it's fine if you do a custom local setup with the appropriate local configuration, but then you also need to add the appropriate config for tmpfiles.d. You can either mask or replace tmp.conf, simply add your own file as /etc/tmpfiles.d/tmp.conf and it will have priority, and do what you need for your custom local setup.
Oh, ok. I admit I have a traditional unix background where it's common practice to have these a separate real partitions instead. I'll just keep the exception I already made then. I have some users on the system which sometimes store larger things in there. Thanks, Kind regards, Axel
On Mon, 4 Dec 2023 20:39:21 +0100 Axel Scheepers <axel.scheepers76@gmail.com> wrote: /tmp very prominently (see attached screenshot). Whether this is still a good idea nowadays is debatable. As Luca said, if we offer /tmp as a separate partition, it should probably be tmpfs now and not an (ext4/xfs/...) partition. For reference, I also include the legacy cleanup routine for SysV: https://salsa.debian.org/debian/sysvinit/-/blob/master/debian/src/initscripts/lib/init/bootclean.sh#L119-128 So if there should be an exclusion for lost+found (for ext4), there should probably also be exclusion for quota related files. Whether those exclusions should be shipped directly by systemd or created by d-i (as suggested by Luca on IRC), is something I don't have a strong opinion about. The downside of letting d-i create such a tmpfiles snippet is that we wouldn't cover existing systems. As for d-i itself, my preferred solution for this would be to change it to uses tmpfs for /tmp. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=245465 (that's an old bug report) Michael
The code to exclude lost+found appears to be still there: https://github.com/systemd/systemd/blob/main/src/tmpfiles/tmpfiles.c#L720 Can you create a lost+found directory and then run SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles --prefix /tmp ?
Am 04.12.23 um 23:09 schrieb Michael Biebl: I haven't really debugged this fully, but from a cursory glance it appears that systemd-tmpfiles-setup.service is responsible for nuking /tmp/lost+found systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev doesn't preserve /tmp/lost+found which looks like a regression.
Yes you are right, I overlooked the existing exception, it is indeed probably the setup which causes it. I had rebooted the box previously and tested only the clearing service. Kind regards, Axel Op di 5 dec. 2023 00:38 schreef Michael Biebl <biebl@debian.org>: