#1080025 fetchmail: Return-Path hdrs w/out domain causes fetchmail to generate an RFC non-compliant envelope

Package:
fetchmail
Source:
fetchmail
Description:
SSL enabled POP3, APOP, IMAP mail gatherer/forwarder
Submitter:
Manny
Date:
2025-01-22 17:27:01 UTC
Severity:
normal
Tags:
#1080025#5
Date:
2024-08-29 18:17:29 UTC
From:
To:
The failure scenario played out like this:

① sent a message that was larger than the receiving server accepts
② the email provider’s SMTP server rightfully generated a bounce message and delivered it to the inbox of the sender. But the Return-Path: lacked a domain and was therefore RFC non-complaint. Specifically, the header was→ “Return-Path: <MAILER-DAEMON>”
③ fetchmail fetched the bounce message then used the Return-Path header to attempt to construct a valid envelope-from address in the local SMTP session. It sent this→ “MAIL FROM:<MAILER-DAEMON@127.0.0.1> BODY=8BITMIME SIZE=4623”. That is also RFC non-compliant by today’s standards. IP addresses must be bracketed in order to be valid. (I think in the 1990s a naked IP address was valid)
④ Postfix locally generates another bounce message containing this:

  In:  MAIL FROM:<MAILER-DAEMON@127.0.0.1> BODY=8BITMIME SIZE=4623
  Out: 501 5.1.7 Bad sender address syntax

The first defect to manifest is the email provider generating a
malformed Return-Path (obviously not a fetchmail bug). The second bug
is fetchmail botching the “MAIL FROM” header in attempt to salvage the
bad Return-Path.

It’s nice that fetchmail tries to be forgiving and work with bad
input. But if it’s going to take that route then it needs to generate
valid output. A low-effort fix is to simply add brackets around the IP
address.

A better fix is likely to extract a domain name from the “From:”
header and use that instead of the localhost IP address. But note that
it may be non-trivial considering that the “From:” field could legally
contain mutliple addresses. Or the “From:” field could also be invalid
in the same way, in which case finding a sensible domain might become
more of a crap shoot. In my particular scenario, my email provider had
a trivial and proper “From:” field as: MAILER-DAEMON@mx.server.tld.

Alternative /possible/ fix:

Perhaps outsourcing this to formail is sensible?  I sometimes have to
process email locally by feeding messages with missing headers into
procmail. E.g.

  $ formail < msg_with_bad_or_missing_headers > proper_msg.mbox

I just did a couple tests with formail. The first msg was the same msg
that fetchmail botched. When that same msg is fed to formail, formail
generates a “From ” header at the top of the msg. Note the space after
the From. That is apparently another way to embed the
envelope-from. Formail ignored the malformed Return-Path header and
used the From: header to generate the “From ” header. In another test,
the Return-Path header on the input was made proper by adding an
“@domain.tld”. In that case formail favored the Return-Path header
over the From: header.

Fetchmail could either duplicate that logic, or perhaps better
outsource the job to formail to make use of code reuse benefits.

#1080025#10
Date:
2025-01-21 21:35:47 UTC
From:
To:
Am 29.08.24 um 20:17 schrieb Manny:
I take it your mail provider uses Postfix and you claim it messes up?

The postconf(5) manual page (command: man 5 postconf) states:

So that's upstream business, they have the means to make that valid --
and not fetchmail's business to fix.

Are you using multidrop? =>
<https://www.fetchmail.info/fetchmail-FAQ.html#G3>
Agreed, but 6.4 is frozen upstream, so changes would be in 6.5 and
available for backporting through the Debian package maintainer.

Not going to happen. We shall not confuse envelope and header addresses
only to delay the problem. And if the envelope sender isn't the null
sender (which you reported as MAILER-DAEMON), why would we know it's a
bounce and we want special handling?

The general requirement is that whatever upstream accepts, must be
accepted by the server that fetchmail forwards to. The assumption is
that the latter is under your control, so you can relax policies for the
localhost.

Also not going to happen. I am not going to touch that pile of bitrot
(not even with a barge pole) that had bogus concepts 30 years ago and
hasn't been maintained for more than a quarter century and source all
the procmail bugs in. That includes resorting to empiric second-guessing
behavior of antediluvian software such as procmail/formail.

#1080025#15
Date:
2025-01-21 21:46:03 UTC
From:
To:
Reading source code, I need to understand the code path to generate such
headers, and proper verbose traces (link to instructions in previous
message).

Reason is that fetchmail has had, since fetchmail-5.9.13 (Sat Jun 22
17:53:00 EDT 2002), recognition for dotted-quads and would enclose those
in square brackets. And I don't see obvious code that would both fail to
add those square brackets and at the same time add options. Fetchmail's
bounces are created without options such as BODY=8BITMIME or SIZE=1234,
these only happen for forwarded stuff which brackets addresses.

So please provide fetchmail -vv logs.

#1080025#20
Date:
2025-01-22 17:25:38 UTC
From:
To:
I never said what server software my ESP¹ uses. My mention of postfix
referred to my own local host. I run postfix locally but it is not
WAN-facing (thus all inbound mail is by fetchmail grabbing from pop3
servers).

¹ESP: email service provider

But incidentally, you are correct. My ESP in fact also uses
postfix. And they use the default setting, as you suggest:

Indeed. It’s a bit surprising postfix would default to an RFC
non-compliant value. And I suppose the take-away there is that my ESP
is not doing something unusual, which means this scenario is more
common than ideal.

$ env LC_ALL=C fetchmail -V | grep -i drop

^ this indicates single-drop mode is used for all my configs.

To reproduce this, I had to attempt to send an oversized message to
the same recipient as that of august last year. Luckily the same
server limits are in place and nothing changed (which means my ESP has
not fixed the problem on their end that I reported [annoying] -- but
that actually benefits this reproduction to get more detailed logs).

===8<----------------------------------------
fetchmail: Trying to connect to 127.0.0.1/13552...connected.
fetchmail: POP3< +OK [XCLIENT] howdy, ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< AUTH-RESP-CODE
fetchmail: POP3< STLS
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN LOGIN
fetchmail: POP3< .
fetchmail: 127.0.0.1: WARNING: server offered STLS, but sslproto '' given.
fetchmail: POP3> USER manny
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 1 4567
fetchmail: 1 message for manny at mannysESP-onion (4567 octets).
fetchmail: POP3> LIST 1
fetchmail: POP3< +OK 1 4567
fetchmail: POP3> RETR 1
fetchmail: POP3< +OK 4567 octets
fetchmail: reading message manny@127.0.0.1:1 of 1 (4567 octets)About to rewrite Return-Path: <MAILER-DAEMON>...
...rewritten version is Return-Path: <MAILER-DAEMON@127.0.0.1>.
fetchmail: About to rewrite From: MAILER-DAEMON@mx0.mannysESP.net (Mail Delivery System)...
...rewritten version is From: MAILER-DAEMON@mx0.mannysESP.net (Mail Delivery System).
fetchmail: About to rewrite To: myalias@mannysESP.net...
...rewritten version is To: myalias@mannysESP.net.
fetchmail: Trying to connect to ::1/25...connected.
fetchmail: SMTP< 220 gudrun ESMTP Postfix (Debian/GNU)
fetchmail: SMTP> EHLO mannys.host
fetchmail: SMTP< 250-gudrun
fetchmail: SMTP< 250-PIPELINING
fetchmail: SMTP< 250-SIZE 40960000
fetchmail: SMTP< 250-VRFY
fetchmail: SMTP< 250-ETRN
fetchmail: SMTP< 250-ENHANCEDSTATUSCODES
fetchmail: SMTP< 250-8BITMIME
fetchmail: SMTP< 250-DSN
fetchmail: SMTP< 250-SMTPUTF8
fetchmail: SMTP< 250 CHUNKING
fetchmail: forwarding to localhost
fetchmail: SMTP> MAIL FROM:<MAILER-DAEMON@127.0.0.1> BODY=8BITMIME SIZE=4567
fetchmail: SMTP< 501 5.1.7 Bad sender address syntax
fetchmail: SMTP error: 501 5.1.7 Bad sender address syntax
fetchmail: SMTP> RSET
fetchmail: SMTP< 250 2.0.0 Ok
fetchmail:  not flushed
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 2.0.0 Bye
fetchmail: 6.4.37 querying mannysESP-onion (protocol POP3) at Wed 22 Jan 2025 02:22:22 PM CET: poll completed
fetchmail: New UID list from mannysESP-onion:
 <empty>
fetchmail: not swapping UID lists, no UIDs seen this query
===8<----------------------------------------

I should also elaborate on this line to clear up confusion:

  reading message manny@127.0.0.1:1 of 1 (4567 octets)About to rewrite Return-Path: <MAILER-DAEMON>...

That looks as if the ESP is my localhost. But it’s not. There is a
fetchmail limitation (unless I’m wrong) that fetchmail cannot handle
onion hosts. So I have this config stanza:

===8<----------------------------------------
skip mannysESP-onion via 127.0.0.1
        protocol       pop3
        port           13552
        username       "manny"
        sslproto       ''
        fetchall
===8<----------------------------------------

So running fetchmail is complicated. I run fetchmail this way:

===8<----------------------------------------
$ socat TCP4-LISTEN:13552,reuseaddr,fork SOCKS4A:127.0.0.1:ungodlyverylongassonionversion3hostnamehere.onion:110,socksport=9050 &
$ local pid=$!
$ fetchmail -d0 mannysESP-onion
$ kill "$pid"
===8<----------------------------------------

Fetchmail thinks it’s fetching from a local server, but
127.0.0.1:13552 is actually the end of a tunnel over Tor to an onion
host.  Hmm.. I just realized that may be why fetchmail decides to
append @127.0.0.1 to MAILER-DAEMON.