Yesterday, while running logcheck against all my syslogs for the week, I
started bemoaning how long the whole thing was taking (over 9 minutes
for 4 megs).  I wondered if maybe one bad regex was stalling the whole
thing, but the debug output showed that all rulefiles were taking up
time proportional to their size.  (Besides, egrep being based on a DFA,
it doesn't care much about how a regex is written.)

Out of curiosity, and realizing that an egrep regex should, AFAIK, work
just the same in Perl, I whipped up a one-liner to test out one
rulefile.  egrep took over 20 seconds to match ignore.d.server/spamd
against my logs; perl took less than 2 to produce the same results.

So, I wrote up the attached script as a quick hack to try out perl as a
substitute for egrep.  This brought the run time down to less than a
minute and a half.  As Mr. Brian Norris said: "I'm convinced".  :)


Now, I'm not advocating immediate action, as such a switch should
certainly not be taken lightly, especially given the security role of
logcheck.  Nevertheless, I think it's something worth mulling over,
given the speed difference.  What do you think?

Bug 445215 is about  egrep being a ton slower than doing the same thing in PERL.

Bug 450649 is about egrep spending 50% of its time clearing memory
using a for() loop rather than memset().

There's obviously some kind of relationship between these issues :-).

  Regards //Johan

Even with the egrep memory clearing issue fixed (it was long time
ago), grep still uses up ungodly amounts of CPU time while logcheck is
running.  If the above perl hack is the way to go, please implement
it!

Or maybe logcheck is becoming big enough that it shouldn't be in
shellscript any more?

  //Johan

also sprach Johan Walles <johan.walles@gmail.com> [2009.12.21.1522 +0700]:

It should have never been a shell script. If you are interested in
a rewrite, have a look at http://wiki.logcheck.org/logfilter and
don't hesitate to speak to me for input, should you want it.

Not sure there is much of relevance, but It may still be worth
investigating whether logcheck should use silver-searcher (ag) instead
of grep