Hi!

I have logcheck running and running, for very long time now.
This is mostly because I have very big logfiles currently
due bug in one of deamons. It take so long that
it take few hours to process (and logcheck is started every 2 hours,
no enough to process this). I often need to killall -9 -u logcheck,
as it starts to be eating lots of CPU and making my laptop really hot. :/

So, I looked why actually it take so long for logcheck
and ways of improving.

One of things I immediately see, was that in top I see things
like this:


.... egrep --text -v -f /tmp/logcheck.SVwK21/ignore/dovecot
/tmp/logcheck.SVwK21/checked


In this particular moment it runs (already for 10 minutes) negative matching
using grep -E, with preprocessed dovecot pattern files
(probably removing empty lines and lines beging with # - comments - I guess).
On a file /tmp/logcheck.SVwK21/checked, and this files have about 310k lines
and 61MB.

Immediately I asked myself, why logcheck doesn't actually create one pattern
file
from all files in ignore.d, and pass it to egrep ? (Similar for positive
matches).

So, finding no excuse I filling this report.

Merging all pattern files (both for positive and negative match passes), will
greatly
improve performance by doing only single pass over log file. It will also
in principle allow egrep to optimize regular expressions better (because
there is lots of commonality beetwen patterns), make it even
slightly faster overally.

Please implement it, it is very simple change. Do not have particular
benchmark numbers, but if you want I can perform some.

Thanks.

I think the current design of logcheck with all the reportlevels and
violation/cracking layers makes this approach hard to do correctly.

But if we simplify the layout of rules it might be worth considering.

(another idea is to track how many times each file matches so more
frequently-matching files can run first)