hi,
I was puzzled by some RE failure on a constant part of msgs sent by a script.
Eg. the recipe has the following RE
* ^Subject: .+ word1 word2
of course I'm sure sent msgs always have subject ending with 'word1 word2' -
that's wired into the script.
Turned out that RE fails whenever the '.+' part is long enough that mailer/
MTA folds the header, so msg is received like
!Subject: blah blah blah word1
! word2
but procmail fails to properly unfold the header, the recipe's RE would see
"Subject: blah blah blah word1 word2"
^
note the extra LWSP. According to rfc822,
"... Unfolding is accomplished by regarding CRLF immediately
followed by a LWSP-char as equivalent to the LWSP-char."
which of course makes sense, otherwise no RE with [ \t] would be reliable -
ie 'word1 word2\tword3' should be written instead as
'word1[ \t]+word2[ \t]+word3'.
thanks
--
paolo
Hello.
I've received the following report from the Debian bug system.
The submitter does not say it explicitly, but I guess it's about
behaviour of "formail -c". Manpage says:
-c Concatenate continued fields in the header. [...]
but it's not what it really does, nor it is what RFC 2822 calls
"unfolding". Quote:
2.2.3. Long Header Fields
[...]
Unfolding is accomplished by simply removing any CRLF
that is immediately followed by WSP.
however this is what formail -c seems to do, from src/fields.c in the
source code:
void concatenate(fldp)struct field*const fldp;
{ register char*p;register size_t l; /* concatenate a continued field */
l=fldp->Tot_len;
if(!eqFrom_(p=fldp->fld_text)) /* don't concatenate From_ lines */
while(l--)
if(*p++=='\n'&&l) /* by substituting all newlines except the last */
p[-1]=' ';
}
Converting a newline into a white space does not really concatenate
fields, as it adds an extra space.
Package: procmail
Version: 3.22-16
Severity: normal
hi,
I was puzzled by some RE failure on a constant part of msgs sent by a script.
Eg. the recipe has the following RE
* ^Subject: .+ word1 word2
of course I'm sure sent msgs always have subject ending with 'word1 word2' -
that's wired into the script.
Turned out that RE fails whenever the '.+' part is long enough that mailer/
MTA folds the header, so msg is received like
!Subject: blah blah blah word1
! word2
but procmail fails to properly unfold the header, the recipe's RE would see
"Subject: blah blah blah word1 word2"
^
note the extra LWSP. According to rfc822,
"... Unfolding is accomplished by regarding CRLF immediately
followed by a LWSP-char as equivalent to the LWSP-char."
which of course makes sense, otherwise no RE with [ \t] would be reliable -
ie 'word1 word2\tword3' should be written instead as
'word1[ \t]+word2[ \t]+word3'.
thanks
--
paolo
hm, it looks explict to me: I'm talking of procmail. procmail is supposed to do like that as well. right, both procmail and formail do the wrong thing here - thanks for pointing it out, I had a note around for same bugrep for formail as well. yep, different wording, but same as rfc822 ... right - and breaks the RE in the recipe: thanks -- paolo
...
well, seems it's a known bug: man procmail:
...
The embedded newlines in a continued header should be skipped when
matching instead of being treated as a single space as they are now.
...
so upstream is aware of it; I guess s/\n/ / was just easier than RFC ;)
code shared between both formail and procmail):
#avoid duplicated messages
# (if testing, don't do duplicate test)
:0 Whc: msgid.lock
* $ ${TESTMAIL+!}
| formail -D 16384 .msgid.cache
When I get a message via two different systems, which break a long message
id differently:
Message-ID:
<F6B9883A73AC594592041E20C59A484601C3B5EBBAC6@BOM-VMBX-HO.bom.gov.au>
Message-ID:
<F6B9883A73AC594592041E20C59A484601C3B5EBBAC6@BOM-VMBX-HO.bom.gov.au>
I end up with these two entries in formail, and they aren't deduplicated:
<F6B9883A73AC594592041E20C59A484601C3B5EBBAC6@BOM-VMBX-HO.bom.gov.au>
<F6B9883A73AC594592041E20C59A484601C3B5EBBAC6@BOM-VMBX-HO.bom.gov.au>
Maintainer:
Could you kindly forward this bug upstream? I know procmail is ... very
mature (this bug was first mentioned on the mailing list in 2000, applying
to a 1994 version of the code):
http://www.mhonarc.org/archive/html/procmail/2000-05/msg00185.html
Considering the wording in the RFCs, I have to admit that it is not according to spec, and since folding apparently happens more frequently these days, the difference gets noted. The code is not shared between procmail and formail, the methodology *is*, however (for consistency's sake). The reason why procmail did it the way it does, is because it was better for performance (i.e. the mail didn't need to be moved in memory). If I recall correctly, procmail simply replaces the offending newlines with spaces (temporarily) while running the regexp engine on the whole buffer, then puts back the newlines again afterward. Fixing this would involve copying and moving the mail in memory, since the regexp engine needs a contiguous mail for matching. Then again, this problem is in the header only, so maybe we could restrict the moving back and forth to that region. Consider it fixed, formail will receive a similar update to keep the implementations in sync.
Considering the wording in the RFCs, I have to admit that it is not according to spec, and since folding apparently happens more frequently these days, the difference gets noted. The code is not shared between procmail and formail, the methodology *is*, however (for consistency's sake). The reason why procmail did it the way it does, is because it was better for performance (i.e. the mail didn't need to be moved in memory). If I recall correctly, procmail simply replaces the offending newlines with spaces (temporarily) while running the regexp engine on the whole buffer, then puts back the newlines again afterward. Fixing this would involve copying and moving the mail in memory, since the regexp engine needs a contiguous mail for matching. Then again, this problem is in the header only, so maybe we could restrict the moving back and forth to that region. Consider it fixed, formail will receive a similar update to keep the implementations in sync.