#1000518 logcheck: separate filtering for apt term.log and or unattended-upgrades-dpkg.log etc?

#1000518#5
Date:
2021-11-24 13:37:36 UTC
From:
To:
Currently logcheck focuses on /var/log/syslog and /var/log/auth.log but
it would be nice to have separate filtering for other types of logs
that normally don't get merged into syslog or the journal.

One of those types of logs is apt upgrade logs. When apt itself is
invoked, it sends terminal output (including \r but not colours) to the
apt term.log file. If unattended-upgrades is being run then the same
output and also output of the apt hooks goes to the additional log file
unattended-upgrades-dpkg.log (which also contains \r but not colours).
The unattended-upgrades code may also send a mail with that log output
but without any \r or colors.

The unattended-upgrades wrapper around apt print various uninteresting
messages. The cron-apt/aptitude-robot alternatives probably also do the
same. apt itself prints a lot of messages that aren't interesting.
The apt hooks shipped by various packages print various uninteresting
messages. The maintainer scripts shipped by various packages print
various uninteresting messages.

I'm currently using the attached hacky script with lots of regexes to
implement apt upgrade log filtering. It seems to me that a better way
to do this would be to integrate separate apt filtering into logcheck
and then integrate into each package (including apt) logcheck ignore
configs containing the regexes that represent uninteresting messages.

There could be an apt hook to use logcheck to filter the apt term.log
to an apt term-interesting.log and an unattended-upgrades hook to
filter unattended-upgrades-dpkg.log to a corresponding file and an
option/hook to filter unattended-upgrades mails. The same could be done
for cron-apt/aptitude-robot.

#1000518#10
Date:
2021-11-30 22:05:08 UTC
From:
To:
Hi Paul

If my memory is correct. I should know since I wrote the cron-apt software
but it was quite some time ago. ... Checking the code now to refresh my
memory...

Emails are sent by default only when apt fails.

All output is logged to the cron-apt log file.

Logs are sent to syslog on "upgrade". By default upgrade is not enabled.

I'm not sure this helps in any way, but I thought I could at least respond
with what I know.

Cheers

// Ola

#1000518#15
Date:
2021-11-30 22:05:08 UTC
From:
To:
Hi Paul

If my memory is correct. I should know since I wrote the cron-apt software
but it was quite some time ago. ... Checking the code now to refresh my
memory...

Emails are sent by default only when apt fails.

All output is logged to the cron-apt log file.

Logs are sent to syslog on "upgrade". By default upgrade is not enabled.

I'm not sure this helps in any way, but I thought I could at least respond
with what I know.

Cheers

// Ola

#1000518#20
Date:
2021-12-01 00:54:14 UTC
From:
To:
Thanks for that info.

If this feature gets implemented in logcheck, there could be cron-apt
options to apply it to either syslog or mails for each of the types of
output cron-apt sends. An option to send mail/syslog for success too
would be good for users of this logcheck feature and possibly also
users who do processing of the results either in a mail client plugin
or in a syslog database and search system like Elasticsearch or Loki.

#1000518#25
Date:
2021-12-01 08:54:26 UTC
From:
To:
Hi Paul

The current possibilities in cron-apt are:

SYSLOGON="upgrade"

  When to log the cron-apt results to syslog.
  Value:
    error   (syslog on error runs)
    upgrade (when packages is upgraded)
    changes (syslog when change in output from an action)
    output  (syslog when output is generated)
    always  (always syslog)
    never   (never syslog)
            (else never syslog)


MAILON="error"

  When to send email about the cron-apt results.
  Value:
    error   (send mail on error runs)
    upgrade (when packages are upgraded)
    changes (mail when change in output from an action)
    output  (send mail when output is generated)
    always  (always send mail)
    never   (never send mail)
            (else never send mail)

I guess "output" is what you are after.


// Ola

#1000518#30
Date:
2021-12-01 15:41:08 UTC
From:
To:
Hi Paul,

Paul Wise wrote:

aptitude-robot does this in
https://github.com/elmar/aptitude-robot/blob/master/reporting-helpers
which is sourced by the actual hooks in
https://github.com/elmar/aptitude-robot/blob/master/mail-log-on-error
and https://github.com/elmar/aptitude-robot/blob/master/xymon-report

But it does this on its own log file which captures the whole terminal
output of dpkg/apt/aptitude.

		Regards, Axel

#1000518#35
Date:
2021-12-02 00:03:52 UTC
From:
To:
Interesting. This detects instances of a few "warning" and "error"
types of strings. This is different to the approach I was going for
where logcheck would hide "uninteresting" strings and highlight
everything else. Perhaps logcheck could add these different levels of
interest in messages too, so you could say only show me errors or show
me warnings and errors or show me everything "interesting".

#1000518#40
Date:
2023-05-15 05:42:27 UTC
From:
To:
Here is an updated version of the script.
#1000518#45
Date:
2023-05-15 11:10:08 UTC
From:
To:
I'm sorry, but i read the bug and i am not entirely sure i understood what
the request for logcheck is! but if you want to use logcheck to scan apt
logs this is already possible.

You just need to add the name of the log to a file in the logfiles.d
directory. Your script using sed could be then be replaced by a list of
regexps and they would be used by logcheck in its usual run, and matches
reported in the usual way. (I do this locally on other log files and it
works. One limitation is that logcheck cannot report which line came from
which log and i wouldnt see that changing)

If you want to run logcheck totally separately from the periodic check,
with different config, so that the reported lines are not mixed up with the
normal log entries (eg in an apt hook), this is also possible - you will
need to invoke logcheck as the logcheck user and use the relevent command
line options. (the autopkgtests do this).

.... But if this is really about having
apt/unattended-uogrades report errors better it would be better to do that
change directly in those tools i would suggest

I would also echo the recommendion for using cron-apt which can already
send an email (or write a log entry for logcheck to report) if anything
went wrong, (or always - apart from cron-apt not yet having a systemd timer
it is quite easy to configure and works very well). it seems better to use
that existing functionality than to insert logcheck. but perhaps i didnt
understand the request.

#1000518#50
Date:
2023-05-18 03:45:21 UTC
From:
To:
Thanks for the info and thoughts.

The idea would do something like your second suggestion; run logcheck
on apt logs separately, but within Debian instead of just on my system.
Perhaps we could also distribute the ignore regexes across packages
like logcheck itself does. Since not everyone is going to want this,
probably this would be in a separate apt logcheck package.

Since apt term.log collects all apt output, but doesn't collect output
from apt hooks (which is often useful), unattended-upgrades-dpkg.log
collects only automated apt output, but *does* collect output from apt
hooks, this idea is going to require some changes to one or either of
those packages so that users don't get duplicate entries.

The apt/u-u tools already report errors fine, this isn't about that,
but about reporting anomalous messages that aren't apt errors, like
notices or warnings from maintainer scripts or apt hooks etc.

The cron-apt error check is likewise not useful for this idea.

PS: do you know if logcheck supports printing nearby ignored lines for
context for each non-ignored log message that was printed? That would
be essential for this apt log filtering because the non-ignored
messages are usually produced by package maintainer scripts and the
ignored nearby lines contain the name of the relevant packages.

#1000518#55
Date:
2023-05-18 11:36:51 UTC
From:
To:

(i wonder how well this distributed model works  -  i dont think most
packages have understood the violations layer, and rules do not seem to be
kept updated (also true of logcheck-database). it also means
logcheck-database is less useful for other consumers, and creates cruft for
people that do not use logcheck or who want to write all the rules
themselves)

it doesnt out of the box - yet: but adding support for replacing grep wih
faster tools like rg or ag is already on my list for bookworm and the
implementation-in-progress allows other options such as -C etc to be passed
through.

 I assume grep does something sensible when outputting context with -f  but
there would be some subtleties with how it interacts with the violations
and violations-ignore layers and sorting/collating of rules.... simplifying
that mess is also on the list.