#986748 syslog-ng: stucks on writev(), mostly around nightly rotate and cause _everything_ to stuck until killed

Package:
syslog-ng
Source:
syslog-ng
Submitter:
Peter Gervai
Date:
2021-04-11 10:57:03 UTC
Severity:
important
Tags:
#986748#5
Date:
2021-04-11 10:21:35 UTC
From:
To:
I sincerely apologise since this is not a good bugreport, so if you really
feel like completely clueless you can close it, but the problem will
probably presist.

The environments use SysV init (none of the systems are systemd-infected).
(It is possible that it have happened on a systemd-infected server, too,
but I am not sure now as it was many weeks before and back then I was not
aware about the cause of the problem.)

What happens:

* usually around 03:26 (cron.daily / logrotate) syslog-ng gets stuck on a
writev() call, which is supposed to write the actual incoming log line into
a physical file on the system. strace shows that syslog-ng gets signals
(like HUP or TERM), handles them and ignores them and go back to writev().

* during this time the syslog service blocks (after a while).

* this in turn blocks everything using syslog service, and I mean everything.
Notable mentions are dhcp server, vpn server, various cron services. (Blocking
cron may also mean system load goes sky high due to many cron spawns running at once.)

* the disk is not full.

* HUP, TERM and normal signals get ignored. ILL or KILL kills the daemon and the
system starts working again as syslog-ng gets respawned.

Log rotation is done using 'invoke-rc.d syslog-ng reload' which uses
`start-stop-daemon` to send SIGNAL1 (SIGHUP) then uses
`syslog-ng-ctl stats` to see when the daemon reports back, which never happens:
the call gets stuck as well.
[I would guess SIGHUP handler runs amok.]

This seems to happen in 3.19.1-5 and 3.28.1-2 as well, but only on the fraction
of the servers, but there it is recurring around weekly. Nothing relevant is in
dmesg or elsewhere. :-(