#887828 needrestart: Possible false positive on postifx and wazuh-agent running in LXC container #887828
- Package:
- needrestart
- Source:
- needrestart
- Submitter:
- Chris
- Date:
- 2021-07-30 09:51:02 UTC
- Severity:
- normal
- Tags:
Dear Maintainer, having Postfix and the wazuh-agent package from [1] on a current Debian Stretch 9.3 running within an LXC container shows the following services as required for a restart even if the services, the container or the host was freshly restarted: postfix@-.service wazuh-agent.service Running needrestart with the -v parameter shows this output: [main] eval /etc/needrestart/needrestart.conf [main] needrestart v2.11 [main] running in root mode [Core] Using UI 'NeedRestart::UI::stdio'... [main] detected systemd [main] #372 uses non-existing /var/ossec/bin/ossec-agentd [main] #372 is not a child [main] #1047 uses non-existing /usr/lib/postfix/sbin/pickup [main] #1047 is a child of #438 [main] #372 exe => /var/ossec/bin/ossec-agentd [main] #372 is wazuh-agent.service [main] #438 exe => /usr/lib/postfix/sbin/master [main] #438 is postfix@-.service [Kernel] Linux: kernel release 4.13.13-5-pve, kernel version #1 SMP PVE 4.13.13-36 (Mon, 15 Jan 2018 12:36:49 +0100) [Kernel/Linux] Did not find any linux images. Failed to retrieve available kernel versions. Restarting services... Services to be restarted: Restart «postfix@-.service»? [Ynas?] n Restart «wazuh-agent.service»? [Ynas?] n Services being skipped: systemctl restart postfix@-.service systemctl restart wazuh-agent.service No containers need to be restarted. No user sessions are running outdated binaries. The two mentioned binaries which doesn't exist according to needrestart output are there and accessible: ls -la /var/ossec/bin/ossec-agentd
Hi, similar happens with a few other services as well running on the same setup: Dovecot ------- [main] eval /etc/needrestart/needrestart.conf [main] needrestart v2.11 [main] running in root mode [Core] Using UI 'NeedRestart::UI::stdio'... [main] detected systemd [main] #348 uses non-existing /usr/lib/dovecot/anvil [main] #348 is a child of #321 [Core] #473 is a NeedRestart::Interp::Perl [Perl] #473: could not get a source file, skipping [Core] #475 is a NeedRestart::Interp::Perl [Perl] #475: could not get a source file, skipping [Core] #552 is a NeedRestart::Interp::Perl [Perl] #552: could not get a source file, skipping [Core] #4506 is a NeedRestart::Interp::Perl [Perl] #4506: could not get a source file, skipping [main] #12544 uses non-existing /usr/lib/dovecot/imap-login [main] #12544 is a child of #321 [main] #12545 uses non-existing /usr/lib/dovecot/imap-login [main] #12545 is a child of #321 [main] #12550 uses non-existing /usr/lib/dovecot/imap-login [main] #12550 is a child of #321 [main] #12552 uses non-existing /usr/lib/dovecot/imap-login [main] #12552 is a child of #321 [main] #12554 uses non-existing /usr/lib/dovecot/imap-login [main] #12554 is a child of #321 [main] #12559 uses non-existing /usr/lib/dovecot/imap-login [main] #12559 is a child of #321 [main] #12561 uses non-existing /usr/lib/dovecot/imap-login [main] #12561 is a child of #321 [main] #12566 uses non-existing /usr/lib/dovecot/imap-login [main] #12566 is a child of #321 [main] #12593 uses non-existing /usr/lib/dovecot/imap-login [main] #12593 is a child of #321 [main] #12974 uses non-existing /usr/lib/dovecot/imap-login [main] #12974 is a child of #321 [main] #12975 uses non-existing /usr/lib/dovecot/imap-login [main] #12975 is a child of #321 [main] #13320 uses non-existing /usr/lib/dovecot/imap-login [main] #13320 is a child of #321 [main] #321 exe => /usr/sbin/dovecot [main] #321 is dovecot.service [Kernel] Linux: kernel release 4.13.13-5-pve, kernel version #1 SMP PVE 4.13.13-36 (Mon, 15 Jan 2018 12:36:49 +0100) [Kernel/Linux] Did not find any linux images. Failed to retrieve available kernel versions. Restarting services... Services to be restarted: Restart «dovecot.service»? [Ynas?] n Services being skipped: systemctl restart dovecot.service No containers need to be restarted. No user sessions are running outdated binaries. loolforkit ---------- from https://www.collaboraoffice.com/code/#getting_set_up ls -la /usr/bin/loolforkit -rwxr-xr-x 1 root root 951784 Dez 18 20:20 /usr/bin/loolforkit Output from needrestart -v [main] eval /etc/needrestart/needrestart.conf [main] needrestart v2.11 [main] running in root mode [Core] Using UI 'NeedRestart::UI::stdio'... [main] detected systemd [main] #519 uses non-existing /usr/bin/loolforkit [main] #519 is a child of #517 [main] #517 exe => /usr/bin/loolforkit [main] #517 is loolwsd.service [Kernel] Linux: kernel release 4.13.13-5-pve, kernel version #1 SMP PVE 4.13.13-36 (Mon, 15 Jan 2018 12:36:49 +0100) Failed to load NeedRestart::Kernel::kFreeBSD: [Kernel/kFreeBSD] Not running on GNU/kFreeBSD! [Kernel/Linux] Did not find any linux images. Failed to retrieve available kernel versions. Restarting services... Services to be restarted: Restart «loolwsd.service»? [Ynas?] n Services being skipped: systemctl restart loolwsd.service No containers need to be restarted. No user sessions are running outdated binaries.
tags 887828 upstream thanks Hi, you are running needrestart outside of the container? For some reason needrestart seems to not detect that the services are belonging to a LXC container. And the postfix service name looks suspicious. Could you please provide the content of /proc/$PID/cgroup of the postfix's master service? Thanks, Thomas Chris <Fisch.666@gmx.de> writes:
tags 887828 upstream thanks Hi, you are running needrestart outside of the container? For some reason needrestart seems to not detect that the services are belonging to a LXC container. And the postfix service name looks suspicious. Could you please provide the content of /proc/$PID/cgroup of the postfix's master service? Thanks, Thomas Chris <Fisch.666@gmx.de> writes:
Hi, thanks for your reply. The posted outputs are from running needrestart within the container. Sorry that i havn't clarified this from the beginning of my post. within the container: [main] #338 uses non-existing /var/ossec/bin/ossec-agentd [main] #338 is not a child [main] #25460 uses non-existing /usr/lib/postfix/sbin/pickup [main] #25460 is a child of #430 [main] #338 exe => /var/ossec/bin/ossec-agentd [main] #338 is wazuh-agent.service [main] #430 exe => /usr/lib/postfix/sbin/master [main] #430 is postfix@-.service cat /proc/338/cgroup ------------- 12:cpuset:/ 11:hugetlb:/ 10:perf_event:/ 9:blkio:/ 8:net_cls,net_prio:/ 7:memory:/ 6:rdma:/ 5:cpu,cpuacct:/ 4:freezer:/ 3:pids:/system.slice/wazuh-agent.service 2:devices:/system.slice/wazuh-agent.service 1:name=systemd:/system.slice/wazuh-agent.service cat /proc/25460/cgroup ---------------------- 12:cpuset:/ 11:hugetlb:/ 10:perf_event:/ 9:blkio:/ 8:net_cls,net_prio:/ 7:memory:/ 6:rdma:/ 5:cpu,cpuacct:/ 4:freezer:/ 3:pids:/system.slice/system-postfix.slice/postfix@-.service 2:devices:/system.slice/system-postfix.slice 1:name=systemd:/system.slice/system-postfix.slice/postfix@-.service cat /proc/430/cgroup -------------------- 12:cpuset:/ 11:hugetlb:/ 10:perf_event:/ 9:blkio:/ 8:net_cls,net_prio:/ 7:memory:/ 6:rdma:/ 5:cpu,cpuacct:/ 4:freezer:/ 3:pids:/system.slice/system-postfix.slice/postfix@-.service 2:devices:/system.slice/system-postfix.slice 1:name=systemd:/system.slice/system-postfix.slice/postfix@-.service As you have mentioned cgroups i'm also getting the following output from the postfix services within the containers: Jan 28 15:51:51 example systemd[1]: postfix.service: Failed to reset devices.list: Operation not permitted Jan 28 15:51:51 example systemd[1]: postfix.service: Failed to set invocation ID on control group /system.slice/postfix.service, ignoring: Operation not permitted Not sure if this is related here.
Hi, Chris <fisch.666@gmx.de> writes: this is by design of the wazuh-agent and might trigger a false positive in needrestart - putting binaries into /var looks something special. Is your postfix chrooted? Could you please post: stat /usr/lib/postfix/sbin/pickup stat /proc/25460/root/usr/lib/postfix/sbin/pickup Regards, Thomas
Hi, and thanks again for your reply. Ah, thats a good and valid point. I think if this is no best practice of wazuh-agent then i can live with that and handle this special case in my local needrestart config. Yes, it seems most processes of postfix are chrooted by default in Debian Stretch (plain install of Postfix via apt-get install postfix): /usr/share/postfix/master.cf.dist used/installed by /var/lib/dpkg/info/postfix/postfix.postinst is e.g. chrooting the mentioned process: pickup unix n - y 60 1 pickup File: /usr/lib/postfix/sbin/pickup Size: 14408 Blocks: 32 IO Block: 4096 regular file Device: 715h/1813d Inode: 142070 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-02-08 01:06:13.281395346 +0000 Modify: 2017-09-27 04:56:28.000000000 +0000 Change: 2018-01-26 14:10:42.474783916 +0000 Birth: - [main] #4262 uses non-existing /usr/lib/postfix/sbin/pickup [main] #4262 is a child of #478 stat: cannot stat '/proc/4262/root/usr/lib/postfix/sbin/pickup': No such file or directory and it seems the pickup is at: File: /proc/478/root/usr/lib/postfix/sbin/pickup Size: 14408 Blocks: 32 IO Block: 4096 regular file Device: 715h/1813d Inode: 142070 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-02-08 01:06:13.281395346 +0000 Modify: 2017-09-27 04:56:28.000000000 +0000 Change: 2018-01-26 14:10:42.474783916 +0000 Birth: - I've also had a look at the previously mentioned dovecot which seems to be chrooted as well: "Login processes (imap-login, pop3-login) are chrooted by default into an empty non-writable directory." -> https://wiki.dovecot.org/Chrooting and indeed the same happening here: [main] #24776 uses non-existing /usr/lib/dovecot/imap-login [main] #24776 is a child of #13446 File: /usr/lib/dovecot/imap-login Size: 31336 Blocks: 64 IO Block: 4096 regular file Device: 70ah/1802d Inode: 920400 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-02-08 13:49:54.190058675 +0100 Modify: 2017-06-30 21:01:28.000000000 +0200 Change: 2017-08-22 14:24:29.284898620 +0200 Birth: - stat: cannot stat '/proc/24776/root/usr/lib/dovecot/imap-login': No such file or directory File: /proc/13446/root/usr/lib/dovecot/imap-login Size: 31336 Blocks: 64 IO Block: 4096 regular file Device: 70ah/1802d Inode: 920400 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-02-08 13:49:54.190058675 +0100 Modify: 2017-06-30 21:01:28.000000000 +0200 Change: 2017-08-22 14:24:29.284898620 +0200 Birth: - Thanks
Hi, Chris <fisch.666@gmx.de> writes: I did install a vanilla Debian Stretch VM, setup an LXC container inside (using Stretch again) and installed postfix inside the container. Running needrestart inside of the container does *not* detect any false positives on postfix. So it seems that your setup has something special... On which linux distri and kernel are you running your LXC container? From the kernel string it seems to be proxmox, isn't it? I assume there is something special in /proc/$PID/maps or the /proc/$PID/map_files/$MADDR links are missing which are used by needrestart. As fallback needrestart looks for /proc/$PID/root/$FILENAME which fails for chrooted processes. Regards, Thomas Needrestart looks for any executable mapped files
Hey, Indeed this seems to be related and even the reason for the difference. The installation is a vanilla Debian Stretch with the Proxmox VE 5.1 from their repository "on top" running on a 4.13.13-5-pve Kernel. Luckily i had one single container running in "privileged" mode [1] (the others are "unprivileged" containers) and it seems this makes the difference: On all "unprivileged" container needrestart shows this behavior where the "privileged" doesn't falsely detect postfix and wazuh-agent as to be restarted. Any suggestion how to proceed with this? Closing this issue as it seems not directly related to Debian but to LXC / Proxmox? [1] https://pve.proxmox.com/wiki/Linux_Container#pct_settings Thanks,
Hi,
It appears i have some similar issue inside a systemd-nspawn container
running buster on a host running buster as well. No proxmox/lxc
involved.
# strace -e lstat,stat -s1024 stat /proc/6314/map_files/556995da1000-556995dc5000
lstat("/proc/6314/map_files/556995da1000-556995dc5000", {st_mode=S_IFLNK|0400, st_size=64, ...}) = 0
File: /proc/6314/map_files/556995da1000-556995dc5000 -> /usr/lib/dovecot/imap
Size: 64 Blocks: 0 IO Block: 1024 symbolic link
Device: 32h/50d Inode: 3693525 Links: 1
Access: (0400/lr--------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-07-30 08:25:26.120210107 +0200
Modify: 2021-07-30 08:24:50.492904140 +0200
Change: 2021-07-30 08:24:50.492904140 +0200
Birth: -
+++ exited with 0 +++
but:
# strace -e stat -s1024 needrestart -v 2>&1|grep -F 556995da1000-556995dc5000
stat("/proc/6314/map_files/556995da1000-556995dc5000", 0x563265c23478) = -1 EPERM (Operation not permitted)
So the stat executable is using lstat on the symlink. The naive quick
fix attempt
# diff -u {/usr/sbin,.}/needrestart
--- /usr/sbin/needrestart 2019-05-24 11:56:40.000000000 +0200
+++ ./needrestart 2021-07-30 10:26:18.302473557 +0200
@@ -507,7 +507,7 @@
$maddr =~ s/^0+([^-])/$1/;
$maddr =~ s/-0+(.)/-$1/;
my @paths = ("/proc/$pid/map_files/$maddr", "/proc/$pid/root/$path");
- my ($testp) = grep { -e $_; } @paths;
+ my ($testp) = grep { warn $_; -e $_ || -l $_; } @paths;
unless($testp) {
unless($path =~ m@^(/var)?/tmp/@ ||
$path =~ m@^(/var)?/run/@ ||
didn't work because the elements of @testp are stated again at line
524 (and failing there). I'd like to see this fixed one way or another
but don't have time to deal with it right now. What do you think?