Dear Maintainer,
since bookworm rsyslog defaults to timestamps in short-iso-precise format,
while logcheck rules (and journald) still default to the old rule format,
and while the default logcheck rules in the package are easily switched,
this poses problems for larger installations with local logcheck rules
used on systems running different suites.
So I'd recommend to add some advice to logcheck/debian/NEWS based on this
conversation on #debian-devel just now:
<h01ger> mbiebl: given #475303 as context, what's your advice on resolving this: rsyslog now uses new time format, journald uses the old format and logcheck rules are in the old format. does journald use the old format because this is bookworm upgraded and not fresh install? should i simply remove/ignore the timestamps in my logcheck rules? should we add some hint to the release notes?
- zwiebelbot- | (#debian-devel) Debian#475303: Enable support for high precision timestamps - https://bugs.debian.org/475303
<mbiebl> | h01ger: I wasn't aware that logcheck checks the journal until 2 weeks ago someone asked about it
<mbiebl> https://github.com/systemd/systemd/issues/26639 was the result of this discussion
<h01ger> yeah, its a new feature (and sensible! i want it!)
<h01ger> mbiebl: issues/26639 seems sensible too. will/shall that land in bookworm?
<mbiebl> atm, it doesn't look like
<h01ger> dropping timestamps from all logcheck rules could migate this and is an easy way to run mixed suite setups
<h01ger> though it makes me wonder why i kept those for the last 10 or so years, if they now suddenly are not needed ;)
<h01ger> breaking habbits..
<mbiebl> maybe you could make the existing parsing/regexps work with both formats
<mbiebl> 2023-03-16T12:45:45.159206+01:00
<mbiebl> vs
<mbiebl> 2023-03-16T12:50:13.503482+0100
<mbiebl> you'd basically just need an optional ':' in the timezone information
<mbiebl> that is rsyslog and journalctl --output=short-iso-precise
<h01ger> doesnt help with systems not yet running bookworm.
<h01ger> (and those are not all running bullseye either, but older releases too)
<mbiebl> I thought this was about fixing it in bookworm
<h01ger> well, its also about using logcheck for all 'my' systems. i (co-)maintain several setups using logcheck...
<h01ger> and i'm sure i'm not the only one who'll encounter this
<h01ger> since when do both rsyslog and journalctl support --output=short-iso-precise ?
<h01ger> #475303 is from 2008, so i assume changing rsyslog format for old systems could work
<mbiebl> rsyslog uses rfc 3339 by default since bookworm (has supported for 10+years), journald supports short-iso-precise since I can reemember
<h01ger> cool, so i'll switch to short-iso-precise everywhere at once
<mbiebl> systemd, just checked: since v234
<h01ger> i guess this could be a NEWS entry for logcheck
<mbiebl> | h01ger: so you'd miss o-o-stable (v232)
<h01ger> mbiebl: can i put this conversation in a wishlist bug against logcheck, asking to document this in NEWS?
<mbiebl> It was my impression that logcheck changed the regexps which match the timestamps in a way that both matched the old and new format?
<mbiebl> sure, feel free to quote this wherever you like
<h01ger> mbiebl: everyone using logcheck has local rules which need to be changed
<h01ger> hmpf, one setup has 13 machines still running stretch...
<h01ger> mbiebl: & thanks! ("feel free..")
<mbiebl> we do provide backports fwiw
<mbiebl> not sure if that is option in your case
<h01ger> it is, thanks!
<h01ger> systemd | 241-5~bpo9+1 | stretch-backports
<h01ger> cool cool. happy this is conceptually solved now ;) i've migrated very few systems to bookworm yet and have been noticing those 1-2 new daily mails since 2 weeks or so, knowing this will need solving eventually...
<h01ger> mbiebl: thank you very much for this conversation!
<mbiebl> | h01ger: thanks for the bug report, but something got mixed up there: rsyslog uses rfc3339, not short-iso-precise <mbiebl> short-iso-precise is an output format of journalctl that is similar to rfc3339 but differs in the presentation of the timezone information <h01ger> *g*, thanks <h01ger> mbiebl: this is also why you said this: <h01ger> [12:51] < mbiebl> maybe you could make the existing parsing/regexps work with both formats <h01ger> (above) <mbiebl> yes
<mbiebl> a properly placed [:]? might be all that's need
aaaaaaaaaaah, thanks! I only checked /usr/share/doc/logcheck/NEWS.Debian.gz but not /usr/share/doc/logcheck-database/NEWS.Debian.gz no, because it also effects logcheck users not using logcheck-database at all. ;) that's great, maybe that's the best way forward indeed. so maybe reassign this bug to src:release-notes? ;) I very much appreciate your efforts bringing logcheck back in shape! *g* I trust your judgement. Thanks for maintaining logcheck.
Is it not the first entry, from version 1.4.0 from dec 2022 in /usr/share/doc/logcheck-database/README.logcheck-database.gz ?? at least on my system it is there. i think my version is non-standard (systemd unit is coming for trixie) While I can sort of see an argument for putting this in logcheck's news instead (or as well) that doesnt seem correct to me...logcheck-database is what provides the rules for normal users - it is recommended by logcheck. I would assume people not using it know what they are doing. If you really want to catch all users shouldnt it be in rsyslog's NEWS.Debian ? What do you think the best way forward is? (I do intend to write something for debian's release notes about the rsyslog change, if no-one else does.) The wider issue is that logcheck has not been a package that works out of the box without significant configuration and has had minimal attention for several debian releases. we are trying to change that, but please give us some time while we understand the gap - i think debian is slightly fortunate to be releasing bookworm with a logcheck package that works at all I suspect most of the rules in debian are so old they never match anything, and there are definitely many updates needed. but i dont think anyone has the desire to do so before bookworm. i personally dont think it is worth even contemplating that work until we have revised the way rules are selected and the format they use.
it's not. just checked the NEWS from 1.4.2 and it only explains that systemd's journal is now also checked. no word about different time formats. this is also about giving advice what to do with *local* logcheck rules. we need some solution/workaround/configuration/advice for bookworm, else people will just not use logcheck, if it creates too much noise. i'm basically wondering why to have timestamp regexes in logcheck rules at all. *not* having them has two benefits, I guess: a.) (very) slightly faster, b.) easier to maintain/read by humans. what benefits *does* having them? cool. so this sounds like easy advice to give for updating local rules! :) and that's what I'm asking for to be done in debian/NEWS.
I dont understand - logcheck rules cater for both formats since 1.4.1 iirc and this is already explained in NEWS.Debian. (and i thought that included instructions for updating local rules in that) can you clarify what the request for logcheck is here? Did you maybe not upgade logcheck-database to latest version? and while the default logcheck rules in the package are easily switched, the longer term solution is perhaps macros in rules, which may happen in trixie. then rules can start ^@TIMESTAMP @HOSTNAME:.....$ (or whatever syntax is chosen) and you could set TIMESTAMP to whatever you liked.... it's actually not a new feature, was possible in at least bulleye, just enabled it by default recently given the downgrade of rsyslog <h01ger> mbiebl: issues/26639 seems sensible too. will/shall that land in not sure the package should drop the prefixes, <h01ger> though it makes me wonder why i kept those for the last 10 or so yes!
I dont understand - logcheck rules cater for both formats since 1.4.1 iirc and this is already explained in NEWS.Debian. (and i thought that included instructions for updating local rules in that) can you clarify what the request for logcheck is here? Did you maybe not upgade logcheck-database to latest version? and while the default logcheck rules in the package are easily switched, the longer term solution is perhaps macros in rules, which may happen in trixie. then rules can start ^@TIMESTAMP @HOSTNAME:.....$ (or whatever syntax is chosen) and you could set TIMESTAMP to whatever you liked.... it's actually not a new feature, was possible in at least bulleye, just enabled it by default recently given the downgrade of rsyslog <h01ger> mbiebl: issues/26639 seems sensible too. will/shall that land in not sure the package should drop the prefixes, <h01ger> though it makes me wonder why i kept those for the last 10 or so yes!
now that I read it and followed the advice and the very nice sed example there, I can they that it worked flawlessly and was very easy to do. Thank you for that NEWS entry! this question is still open... though maybe cloning the bug is even better, I'd really appreciated a small pointer to logcheck-database's NEWS file in the NEWS for logcheck...
I have submitted something against release-notes so that is in hand. rsyslog has #1031827 which seems to at least have had a response in 2023 I dont mind adding an entry for logcheck's NEWS as well as/instead of logcheck-database's NEWS, @Mathias Gibbens what do you think? The one drawback i see is that 99.9% of people will upgrade both logcheck and logcheck-database together so will get 2 emails from apt-listchanges if we put it in both..... So we should delete it from logcheck-database's NEWS I think? - logcheck does require the same layout of rules even if you dont use logcheck-database so i think this makes sense. I hope this would not crash apt-listchanges fir unstable users if the NEWS file shrinks/disappears due to whatever culls.old entries...?
On Sat, 18 Mar 2023 18:55:25 +0000 Richard Lewis <richard.lewis.debian@googlemail.com> wrote: Think we may as well close this bug, unless anyone objects. bookworm release-notes cover the issue. Next big change im planning to document in logcheck's NEWS.Debian to catch all users - watch this space!