Hi myspell-hu maintainer!

it seems that there is non-u8 data in /usr/share/hunspell/hu_HU_u8.aff
-- it causes an error message to be emitted in pg_updatedicts when it is
converted to the postgres form (see transcript below).

It appears to be due (at least) to ISO-8859-1 characters in the
comments in an otherwise-UTF-8 file:

0 dkg@stylus:~/tmp$ grep ^SET </usr/share/hunspell/hu_HU_u8.aff
SET UTF-8
0 dkg@stylus:~/tmp$ hd </usr/share/hunspell/hu_HU_u8.aff | head -n6
00000000  23 20 54 68 69 73 20 64  69 63 74 69 6f 6e 61 72  |# This dictionar|
00000010  79 20 69 73 20 62 61 73  65 64 20 6f 6e 20 74 68  |y is based on th|
00000020  65 20 48 75 6e 67 61 72  69 61 6e 20 77 6f 72 64  |e Hungarian word|
00000030  6c 69 73 74 20 61 6e 64  20 61 66 66 69 78 65 73  |list and affixes|
00000040  20 63 72 65 61 74 65 64  20 20 20 20 20 20 20 0a  | created       .|
00000050  23 20 62 79 20 4c e1 73  7a 6c f3 20 4e e9 6d 65  |# by L.szl. N.me|
0 dkg@stylus:~/tmp$

note the use of 0xe1 for á, 0xf3 for ó and 0xe9 for é, which is
ISO-8859-1.

this affects pg_updatedicts, because that proram calls iconv on the
file like this:

  system 'iconv', '-f', $enc, '-t', 'UTF-8', '-o', "$cachedir/$locale.affix", $aff

where $enc comes from:

sub get_encoding {
    open F, $_[0] or die "cannot open $_[0]: $!";
    while (<F>) {
        if (/^SET ([\w-]+)\s*$/) { return $1; }
    }
    return undef;
}


There are many non-UTF-8 characters in hu_HU_u8.aff as well, though.
a simple transformation of all the comments to UTF-8 doesn't appear to
be enough to make the pg_updatedicts invocation complete without a
noisy warning.

If you think the file is legitimate as it stands (i confess i don't
really understand the affix file format), please reassign this bug to
postgresql-common (which owns pg_updatedicts) so that
postgresql-common can do better file handling.

Arguably, this is also a bug in postgresql-common because it is not
verifying that the iconv transformation worked correctly at all, but
just pushes ahead.

Regards,

tags upstream 689997
thanks

Hi Daniel,
Thank you for the report, it will be fixed in next upstream release.

Cheers,
Balint

reassign 689997 postgresql-common
tags 689997 =
affects 689997 - postgresql-common
thanks

Hi,

Further discussion with upstream clarified the situation.
The comments will be fixed to be UTF-8 encoded, but the file will still
contain 8 bit characters which will make iconv fail.

The files are valid affix files, processing them using iconv is not
expected to result success and a valid affix file.

Reassigning to postresql-common.

Cheers,
Balint

tag 689997 wontfix
thanks

Balint Reczey [2014-01-24 23:19 +0100]:

Thanks for your followup!

I'm afraid I'm between a rock and a hard place then. PostgreSQL's
tsearch requires the affix files to be UTF-8 encoded, hence my attempt
to convert them using iconv. It doesn't break the package install, you
just get that error message and Hungarian stemming is then just not
available.

I committed a test case to postgresql-common that non-ASCII stemming
works to ensure that it at least is the right approach for other
my/hunspell dictionaries.

Tagging "wontfix" as I don't really know what to change on my end the.

Thanks!

Martin

clone 689997 -1
reassign -1 myspell-hu
tags -1 - wontfix
severity -1 wishlist
retitle -1 Affix files shouldn't use 8bit chars for rules
block 689997 by -1
thanks

Re: Martin Pitt 2014-01-29 <20140129145548.GA3409@piware.de>

IMHO the real problem is in the weird usage of 8bit characters in
upstream's .affix file. Instead of using

... /<FF>...
SFX <FF> ...

they should just pick some ascii char in place of <FF>.

Christoph

Not only myspell-hu errors out here:


	Building PostgreSQL dictionaries from installed myspell/hunspell
packages...
	ERROR: no ecoding defined in /usr/share/hunspell/ar.aff, ignoring
	  bg_bg
...
	  hu_hu
	iconv: ungültige Eingabe-Sequenz an der Stelle 131
	ERROR: Conversion of /usr/share/hunspell/hu_HU.aff failed
	  id_id
	  is_is
	  it_it
	ERROR: no ecoding defined in /usr/share/hunspell/kk_KZ.aff, ignoring
	  kmr_latn