#661736 FormatText: UTF-8 character gets killed

Package:
libhtml-format-perl
Source:
libhtml-format-perl
Submitter:
Jörg Sommer
Date:
2012-12-06 02:30:03 UTC
Severity:
normal
#661736#5
Date:
2012-02-29 13:20:32 UTC
From:
To:
Hi,

% perl -MHTML::FormatText -e 'print HTML::FormatText->new->format_string("à")' |hd
00000000  c3 20 0a                                          |. .|
00000003

The character à gets replaced by \xC3\x20. I don't know why this happens,
but the string should remain unchanged.

Bye, Jörg.
#661736#10
Date:
2012-12-06 02:26:04 UTC
From:
To:
This issue is this line (line 135):

  $text =~ tr/\xA0\xAD/ /d;

This works great if your data is in a Unicode string.  It also works
great if your data is a byte string using Latin-1.  It works very poorly
if your UTF-8 data is in a byte string.  In the example given in the
original bug report, -Mutf8 was not used, so the data is treated as a
series of (two) Latin-1 characters.

vauxhall ok % perl -MHTML::FormatText -Mutf8 -C6 -E 'print HTML::FormatText->new->format_string("à")' |hd
00000000  c3 a0 0a                                          |...|
00000003
vauxhall ok % perl -MHTML::FormatText -Mutf8 -E 'print HTML::FormatText->new->format_string("à")' |hd
00000000  e0 0a                                             |..|
00000002

I suspect the correct fix for this bug is documentation.