#990756 rsyslog: Please send user errors only to user.log and rotate that file based on size.

Package:
rsyslog
Source:
rsyslog
Description:
reliable system and kernel logging daemon
Submitter:
Date:
2021-07-06 10:57:03 UTC
Severity:
wishlist
#990756#5
Date:
2021-07-06 09:51:01 UTC
From:
To:
Dear Maintainer,

My Debian box's hard disk filled up due to a single buggy user
application spewing messages at syslog. While I blame the application,
Debian's rsyslog should be more robust by default.

This is something I've seen happening more and more frequently.
A typical response people get when googling the problem is
"sudo rm /var/log/* and then set that up to run daily as a cronjob."
That's terrible advice and makes Debian seem shoddy.

Ideally, I'd like to see some sort of rate-limiter added to rsyslog.

In the meantime, here is a simple three step fix which would improve
stability for many people using Debian.


1. /etc/rsyslog.conf: add user.none to /var/log/syslog, debug, and messages.
   That way, the spewage is limited to a single file, user.log.

2. /etc/logrotate.d/rsyslog: add "maxsize" so that log files that are
   greater than a certain size are rotated even if their time
   criteria are not due. I suggest "maxsize 1G" is reasonable for
   most people and can be adjusted for those who want more.

3a. Move /etc/cron.daily/logrotate to /etc/cron.hourly/ so that file
    sizes are checked more often. (This also fixes the "bug" where an
    hourly entry in logrotate.conf only gets rotated daily.)

3b. /usr/lib/systemd/system/logrotate.timer: Likewise for systemd.

    [Timer]					[Timer]
    OnCalendar=daily		-->		OnCalendar=hourly
    AccuracySec=1h				AccuracySec=1m
    Persistent=true				Persistent=true


Steps 1 and 2 ensure that the disk will not fill up and that important
system messages won't be rotated away too quickly based on file size.

Step 3 is necessary because the time in which a log file can fill up
/var is no longer measured in days. Currently, I have a single
chromium process sending over thirty thousand messages per second and
I doubt that's even close to the maximum possible.

Thank you.


*** Reporter, please consider answering these questions, where appropriate ***

   * What led up to the situation?
   * What exactly did you do (or not do) that was effective (or
     ineffective)?
   * What was the outcome of this action?
   * What outcome did you expect instead?

*** End of the template - remove these template lines ***

#990756#10
Date:
2021-07-06 10:08:13 UTC
From:
To:
Am 06.07.21 um 11:51 schrieb Ben Wong:
https://www.rsyslog.com/how-to-use-rate-limiting-in-rsyslog/

The problem is: a couple of GB of log data is something that a busy log
server can generate in a couple of minutes which perfectly valid.

On a laptop, a couple of GB of log data is probably unusual.

The problem is to pick a default which suits everyone, which is guess is
not possible.

I'm not really sure we can win this game.
Next time, it's not a user application generating a lot of data but some
system service.

That said, there is certainly value in cleaning up the default Debian
rsyslog default config, see e.g.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=508376
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=580552
to avoid duplication in log messages.

Depends. I'd argue that the systemd journal nowadays is better suited
for end user systems anyway and syslog/rsyslog more targetted at the
enterprise area. So I don't think it makes sense to optimize for the former.

I don't think, changing the cron job to hourly is really going to help
if you have an application running amok. 1 hour is more then sufficient
to fill up your disk.


Fwiw, my plan is for bookworm to have rsyslog removed from a default
installation and rely on journald only (which does size-based log
truncation), as I think journald is a better fit for end-user systems.

rsyslog would then only be installed by those who explicitly need it
(those users will most likely will tweak its settings anyway to suit
their needs).

Regards,
Michael

#990756#17
Date:
2021-07-06 10:15:45 UTC
From:
To:
Am 06.07.21 um 11:51 schrieb Ben Wong:

This is basically a duplicate of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954069

Keep in mind that /etc/cron.daily/logrotate is not under the control of
the rsyslog package.

I'm inclined to merge your bug report accordingly.

#990756#22
Date:
2021-07-06 10:52:17 UTC
From:
To:
Am 06.07.21 um 12:08 schrieb Michael Biebl:

https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#syssock-ratelimit-interval

Apparently this feature caused problems, so it was turned off again.