Dear Maintainer,
When using the package I see many messages like:
/usr/lib/python3/dist-packages/robot_detection.py:635: SyntaxWarning: invalid escape sequence '\/'
  'ng\/1\.', # put at end to avoid false positive
/usr/lib/python3/dist-packages/robot_detection.py:636: SyntaxWarning: invalid escape sequence '\/'
  'ng\/2\.', # put at end to avoid false positive
/usr/lib/python3/dist-packages/robot_detection.py:641: SyntaxWarning: invalid escape sequence '\/'
  'java\/[0-9]'   # put at end to avoid false positive
/usr/lib/python3/dist-packages/robot_detection.py:652: SyntaxWarning: invalid escape sequence '\s'
  'bot[\s_+:,\.\;\/\\\-]',
/usr/lib/python3/dist-packages/robot_detection.py:653: SyntaxWarning: invalid escape sequence '\s'
  '[\s_+:,\.\;\/\\\-]bot',

The fix seems to be to double most of the backslashes in robot-detection.py

I'd suggest removing the backslashes instead.

Doubling the backslashes is for when you want literal backlash
characters in the text. Here the backslash are intended to
escape characters like hyphens:

robot_useragents = [
        ...
        'googlebot',
        'google\-sitemaps',
        'gullive',
        ...
        ]

/usr/lib/python3/dist-packages/robot_detection.py:13: SyntaxWarning: invalid escape sequence '\-'
  'google\-sitemaps',

This escaping isn't required though and Python >= 3.12 (Trixie has 3.13)
will show a `SyntaxWarning` since `\-` isn't a proper escape sequence.

Unfortunately the upstream project appears to be archived.

Ben Sturmfels <ben@sturm.com.au> writes:

Oops, my mistake. These strings are used as regular expressions, so removing the
backslashes would cause the patterns not to match properly.

I'd suggest instead marking all these strings as "raw" strings, eg:

robot_useragents = [
        ...
        r'googlebot',
        r'google\-sitemaps',
        r'gullive',
        ...
        ]

This is equivalent the same as doubling the backslashes, just a little more
readable.

Hello!

just FYI the issue also applies upstream, so I was intending to report
it there and tag this bug report as upstream.

However, the upstream repository on github is currently archived and new
issues cannot be reported there anymore. It has been archived on June
7th 2024. I have no idea if a fork of this repository is suggested for
continuation or not.

So this means that we should apply the change suggested by Ben as a
local patch.

I can confirm that changing all strings in the list of patterns into raw
strings makes the warnings go away and the matching is still working.
e.g. after changing the code:

robot-detection$ python3 -c 'from robot_detection import is_robot;
print(is_robot("blah"))'
False
robot-detection$ python3 -c 'from robot_detection import is_robot;
print(is_robot("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko;
compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"))'
True

Cheers!

Hello,

this module is used by `mailman3-web`. It is called from a cron-job
every minute and the warnings trigger cron to send an email, as the
output of the cron job is no longer empty.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1082541

As Ben Sturmfels has already noticed, the module contains a massive list
of strings, which are regular expressions. To fix this:
- replace \- and \/ with - respective / as both are no special regular
  expression characters.
- change any remaing string containing \. to be a raw string, e.g.
  r'…\.…'.
- there is one \', which is to NOT terminate the Python string.
- there are two \s, which represent the class of white-space characters.
- there are two \\, which represent a backslash itself.


If's unfortunate, that upstream is archived but understandable, as the
list of robots is not static and ever changing. Even its upstream source
for the DB seems dead: http://www.robotstxt.org/db.html

PS: The implementation is also very inefficient as it uses
	any(re.match(…) for re in …).
Generally it's much more efficient to combine all regular expressions
with | and compile them into a single regular expression. Even better is
to use a Trie as Python builds a very inefficient regular expression
automaton when given a very large set of alternatives.
Rust has https://docs.rs/regex/latest/regex/struct.RegexSet.html for
this.

Philipp