#976746 lintian: false positive for national-encoding

Package:
lintian
Source:
lintian
Submitter:
Phil Morrell
Date:
2020-12-07 20:27:10 UTC
Severity:
normal
#976746#5
Date:
2020-12-07 16:57:36 UTC
From:
To:
Hi, please find attached a sample file that appears to be valid UTF-8
from all the checks I can think of (file, iconv, isutf8) but perl's
$item->is_valid_utf8 appears to disagree.

    40de920fc1da70f25bfe7ae2bf66f332  CorsixTH/Lua/languages/traditional_chinese.lua
#976746#10
Date:
2020-12-07 19:12:40 UTC
From:
To:
Hi Phil,

With the kind help of Grinnz on #perl-help, we determined that the
document you sent causes the following error:

    "Can't interchange noncharacter code point U+FFFF in position 46198"

According to the documentation for Unicode::UTF-8 [1], the use of that
codepoint is not allowed in open interchange of Unicode text data

Kind regards,
Felix Lechner

[1] https://metacpan.org/pod/Unicode::UTF8#Can't-interchange-noncharacter-code-point-U+%X-in-position-%u
#976746#15
Date:
2020-12-07 19:12:40 UTC
From:
To:
Hi Phil,

With the kind help of Grinnz on #perl-help, we determined that the
document you sent causes the following error:

    "Can't interchange noncharacter code point U+FFFF in position 46198"

According to the documentation for Unicode::UTF-8 [1], the use of that
codepoint is not allowed in open interchange of Unicode text data

Kind regards,
Felix Lechner

[1] https://metacpan.org/pod/Unicode::UTF8#Can't-interchange-noncharacter-code-point-U+%X-in-position-%u
#976746#20
Date:
2020-12-07 20:24:27 UTC
From:
To:
Hi Felix, thank you for the diagnosis, hopefully that will help fix it.

https://github.com/CorsixTH/CorsixTH/issues/1760

Interestingly, it seems that is no longer true as of Unicode 7 - it is
valid in UTF-8 encoded files, but has no "standard, interchangeable
meaning". I will leave it up to the maintainers to interpret that in the
context of lintian, probably still a false positive, but the point of
avoiding national encodings is to provide standardised meaning of text.

https://www.unicode.org/versions/corrigendum9.html