Dear Maintainer,

rsyslogd currently produces two different timestamp formats at the start of a
log line with the default (now also Debian default) rfc3339 format.

Local log lines include the sub-seconds part like:
2024-02-16T22:05:52.315463+01:00 tux [...]

while remote logs (in that case from virtual machines on the same host) do not
include the sub-seconds part:
2024-02-16T22:06:02+01:00 tux1 [...]

Logcheck currently deals only with the first format. This results in no
logcheck pattern matching for remote host log entries.

Fortunately logcheck also still supports the 'traditional' format which I've
reverted to.

I would expect rsyslog to only use a single format, but failing that I think
that logcheck should not drop support for the old 'traditional' timestamp
format until the issue in rsyslogd is resolved.

Logcheck *may* want to support both rfc3339 formats (the sub-seconds part *is*
optional in the RFC).

I forgot to mention:
There is an upstream (rsyslog) bug-report at
https://github.com/rsyslog/rsyslog/issues/5332

And a debian bug report for rsyslog at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064385

Thanks
Ralf

Upstream has decided that it is not a bug and that both timestamp
formats are valid RFC 3339 (I've checked, the grammar explicitly defines
the sub-seconds part of the timestamp as optional). See link above.
They also think, logcheck should cope with both formats.

So I guess that logcheck should be prepared to receive both kinds of
timestamps, the 32-byte version and the 25-byte version (without the
subseconds timestamp).

In the downstream bug-report of rsylogd (which I suppose will be closed
soon) I've mentioned how to configure remote clients to send a timestamp
*with* sub-seconds part.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064385

But there may be other clients out there (which may not be rsyslogd)
which still send the traditional format and thus will be logged without
a subseconds part.

So logcheck should be prepared to receive both formats.

Thanks
Ralf

what is the default, and does logcheck cope with that? there's a limit to
how much to suport out of the box - especially as rsyslog is no longer the
default.

if you configure a logger to produce a certain format it's not unreasonable
to also have to edit logcheck rules accordingly

But a longer-term solution is perhaps to allow easier customisation of
rules via "macros"/variables --- a proof-of-concept for this is in
progress, but not.yet ready for testing

The current default of Debians rsyslog (after a long time where it was
the 'traditional' format) it is now RFC 3339 timestamps. This comes in
two variants, with or without the sub-seconds part. Logcheck only
supports the variant *with* sub-seconds.

By default, logcheck supports the 'traditional' format and the 32-byte
header, the pattern in most logcheck rules is

^(\w{3} [ :[:digit:]]{11}|[0-9T:.+-]{32})

The first alternative matched by this is something like

Feb 18 00:01:36

while the second is

2024-02-16T20:59:34.218904+01:00

The short form also produced by rsyslog is

2024-02-16T22:06:02+01:00

The third (short) form with no sub-seconds part is currently not matched
by logcheck.

You might want to simply set the match pattern to

^(\w{3} [ :[:digit:]]{11}|[0-9T:.+-]{25,32})

Although rsyslog would probably never produce it, RFC 3339 allows the
sub-seconds part to be short (min 1 digit). There is no maximum in RFC
3339 but RFC 5424 prohibits more than 6 digits:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.2.3.1
For RFC 3339 see p.7 section 5.6 in
https://www.rfc-editor.org/rfc/rfc3339#section-5.6

So it makes sense to match a range of lengths.

I'm talking about the new Debian rsyslog package's default.
And, yes, but that would mean to edit logcheck rules for each installed
package? And the new default of the rsyslog package is the two variants
of RFC 3339. Unfortunately the default for remote logging does *not*
transmit the sub-seconds part. So you end up with two timestamp formats
in the same logfile. Which is fine according to the syslog standard in
RFC 5424.

Nice!

Thanks
Ralf

Hi

On Thu, 22 Feb 2024 19:01:05 +0000 Richard Lewis <richard.lewis.debian@googlemail.com> wrote:
default. That said, I would still consider rsyslog the default syslog
daemon in Debian. Packages that depend on system-log-daemon typically do
this via a "Depends: rsyslog | system-log-daemon"

Making the individual fields more flexible/customizable sounds like a
good idea.

Just wanted to add, that syslog-ng also supports RFC3339 timestamps [1].
So any improvements in that regard will also benefit other sysloggers.

As for this specific issue, Ralf has found a way to make ensure that
remote syslog messages also carry a subseconds timestamp by explicitly
specifying the format when forwarding the messages [2]

Michael

[1]
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/ts-format
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064385#29

thanks - agree logcheck should cope with a default rsyslog output. ... i
just dont know what that default output is: does the below mean the
subseconds are now always present?

or: what regexp should logcheck use as prefix?



As for this specific issue, Ralf has found a way to make ensure that

Am 27.02.24 um 09:50 schrieb Richard Lewis:

For locally generated messages, the time stamp includes subsecond precision.
If rsyslog is setup to receive remote messages, it depends on the sender.

According to RFC 3339 everything from no subseconds to 6 subsecond
digits can be present (which is seven chars including the decimal dot).
So far (in the wild) I've seen only the two variants with no subseconds
or with 6 subsecond digits. But anything between these is possible.

I've already suggested to modify the regex for the timestamp part from

^(\w{3} [ :0-9]{11}|[0-9T:.+-]{32})

to

^(\w{3} [ :0-9]{11}|[0-9T:.+-]{25,32})

That would match the two extremes (with/without sub-seconds) but would
also cope with anything in between (which is legal according to the
RFCs). The first part would continue to match the "traditional" syslog
format. This is also in wide use today.

Thanks
Ralf