#879204 MIME::Words::encode_mimewords: splits in the middle of a multibyte character, invalid MIME

Package:
libmime-tools-perl
Source:
libmime-tools-perl
Submitter:
Thorsten Glaser
Date:
2017-10-20 20:45:07 UTC
Severity:
normal
#879204#5
Date:
2017-10-20 12:48:51 UTC
From:
To:
tglase@helpdesk:~$ perl -MEncode -MMIME::Words -e 'print MIME::Words::encode_mimewords(Encode::encode("UTF-8", "Re: Bildungsurlaub für CCC-Fahrt? [THD#1424195]"), Charset => "UTF-8", Field => "Subject") . "\n";'
Re:=?UTF-8?Q?=20Bildungsurlaub=20f=C3?= =?UTF-8?Q?=83=C2=BCr=20CCC?=-Fahrt?=?UTF-8?Q?=20?=[THD#1424195]

Expected output is something along the lines of:

tglase@tglase:~ $ php
<?php
mb_internal_encoding('UTF-8');
echo mb_encode_mimeheader('Subject: Re: Bildungsurlaub für CCC-Fahrt? [THD#1424195]', 'UTF-8', 'Q', "\015\012") . "\n";
Subject: Re: Bildungsurlaub =?UTF-8?Q?f=C3=BCr=20CCC-Fahrt=3F=20=5BTHD=23?=
 =?UTF-8?Q?=31=34=32=34=31=39=35=5D?=
#879204#10
Date:
2017-10-20 13:20:28 UTC
From:
To:
% perl -MEncode -MMIME::Words -e 'print MIME::Words::encode_mimewords(Encode::encode("UTF-8", "Re: Bildungsurlaub für CCC-Fahrt? [THD#1424195]"), Charset => "UTF-8", Field => "Subject") . "\n";'
Re: Bildungsurlaub =?UTF-8?Q?f=C3=83=C2=BCr=20?=CCC-Fahrt? [THD#1424195]


Looking at the upstream Changes:

5.504     2013-01-30  Dianne Skoll <dfs@roaringpenguin.com>
[..]
        * Fix long-standing bug in encode_mimewords that can
          break multibyte-encodings (eg, utf-8)
https://rt.cpan.org/Public/Bug/Display.html?id=5462


Cheers,
gregor
#879204#15
Date:
2017-10-20 20:12:58 UTC
From:
To:
gregor herrmann dixit:

Indeed, see #879205, which is why I reported this as two
separate issues, considering wheezy is still supported-ish.

I just noticed this in an eMail I got from OTRS earlier
today and thought to forward it *somewhere* where people
in the know can care about it.

bye,
//mirabilos
#879204#20
Date:
2017-10-20 20:37:35 UTC
From:
To:
Hi,

This is not a bug in MIME::tools.

The OP misunderstands how Perl works.  He typed UTF-8 source code in and
is double encoding it.  Here's a test program:

#===================================================================
use MIME::Words;
use Encode;

my $sample = "Re: Bildungsurlaub für CCC-Fahrt? [THD#1424195]";

my $utf8 = Encode::encode('UTF-8', $sample);
my $out = MIME::Words::encode_mimewords($utf8, Charset => 'UTF-8');

print "Out: $out\n";
#===================================================================

If I run:

   perl test-utf8.pl

Output is:

   Out: Re: Bildungsurlaub =?UTF-8?Q?f=C3=83=C2=BCr=20?=CCC-Fahrt? [THD#1424195]

But that's because the word "für" is *already* UTF-8.  If I tell Perl
to convert UTF-8 in the source code to native Perl Unicode, the result
is very different:

  perl -Mutf8 test-utf8.pl

Output is:

  Out: Re: Bildungsurlaub =?UTF-8?Q?f=C3=BCr=20?=CCC-Fahrt? [THD#1424195]

The OP should read "perldoc utf8" and should also not use UTF-8 directly
as Perl source code; use \x{FC} rather than ü, etc.

Regards,

Dianne.