When using -k on a file which contains a single UTF-8 character, preconv
misdetects the text as some other encoding, even though the locale in
use is UTF-8.  Since UTF-8 is nearly universally used for text files on
Unix, this leads to bizarre behaviour and misencodings.

For example, given the first file below, groff prints a warning and then
proceeds to insert an incorrect character.  However, when a second UTF-8
character is included, the file works.

My recommendation here is that when detecting character sets, if the
data is valid UTF-8, then UTF-8 be used as the encoding.  The uchardet
detection of "MAC-CENTRALEUROPE" may be acceptable for some web pages,
where encoding can be specified explicitly at the HTTP level, but it is
not a prudent choice for documents on Debian (which has never supported
this as a valid system encoding) in 2022.  I very much doubt this would
be a prudent encoding on macOS in 2022, either, which, as I understand
it, has used UTF-8 exclusively since 10.0, released over two decades
ago.

Command line:

  LC_ALL=fr_CA.UTF-8 groff -Tps -dpaper=com10l -P-pcom10 -P-l -k envelope.me >envelope.ps

broken
----
.nf
.po 0.5c
.sp 0.5c
.ft P
Toronto City Hall
100 Queen Street W
Toronto ON M5H 2N2
Canada
.sp 2c
.in 8.5c
New York City Hall
1 City Hall
New York NY 10007-1298
États-Unis
----

working
----
.nf
.po 0.5c
.sp 0.5c
.ft P
Hôtel de Ville de Toronto
100 Rue Queen O
Toronto ON M5H 2N2
Canada
.sp 2c
.in 8.5c
New York City Hall
1 City Hall
New York NY 10007-1298
États-Unis
----

Hi Brian,

I can't reproduce the behavior you describe with groff 1.22.4-5 (and
while it's worth trying that older version, I don't think Colin has
changed anything since then that would break this).

I don't get any such warning, nor incorrect output.

$ preconv -d ./EXPERIMENTS/carlson1.groff
default encoding: 'UTF-8'
file './EXPERIMENTS/carlson1.groff':
  no encoding tag
  len: 173
  uchardet read: 173 bytes
  charset: UTF-8
  encoding used: 'UTF-8'
.lf 1 ./EXPERIMENTS/carlson1.groff
.nf
.po 0.5c
.sp 0.5c
.ft P
Toronto City Hall
100 Queen Street W
Toronto ON M5H 2N2
Canada
.sp 2c
.in 8.5c
New York City Hall
1 City Hall
New York NY 10007-1298
\[u00C9]tats-Unis

Please follow up with the diagnostic message(s) you're getting, and/or
the output of "preconv -d" on your broken input file (which I called
"carlson1.groff" above).

Regards,
Branden

I don't happen to have that version available in APT, but I do have
1.22.4-6.  Downgrading groff to that version doesn't appear to make a
difference.

I expect the relevant difference here is libuchardet0.  With 0.0.6-3, I
get correct output, but I get incorrect output with 0.0.7-1.

Sure.  Here's the output of running groff:

----
$ LC_ALL=fr_CA.UTF-8 groff -Tps -dpaper=com10l -P-pcom10 -P-l -k envelope.me >envelope.ps
troff: envelope.me:14: warning: can't find special character 'u0043_030C'
----

And here's the output of running preconv with libuchardet0 0.0.7-1 (sid,
broken):

----
$ preconv -d envelope.me
default encoding: 'UTF-8'
file 'envelope.me':
  no encoding tag
  len: 173
  uchardet read: 173 bytes
  charset: MAC-CENTRALEUROPE
  encoding used: 'MAC-CENTRALEUROPE'
.lf 1 envelope.me
.nf
.po 0.5c
.sp 0.5c
.ft P
Toronto City Hall
100 Queen Street W
Toronto ON M5H 2N2
Canada
.sp 2c
.in 8.5c
New York City Hall
1 City Hall
New York NY 10007-1298
\[u221A]\[u010C]tats-Unis
----

And the output with libuchardet0 0.0.6-3 (buster, working):

----
$ preconv -d envelope.me
default encoding: 'UTF-8'
file 'envelope.me':
  no encoding tag
  len: 173
  uchardet read: 173 bytes
  charset: UTF-8
  encoding used: 'UTF-8'
.lf 1 envelope.me
.nf
.po 0.5c
.sp 0.5c
.ft P
Toronto City Hall
100 Queen Street W
Toronto ON M5H 2N2
Canada
.sp 2c
.in 8.5c
New York City Hall
1 City Hall
New York NY 10007-1298
\[u00C9]tats-Unis
----
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

reassign 1009249 libuchardet0
found 1009249 0.0.7-1
thanks

Hi Brian,

It's hard for me to imagine how this isn't a libuchardet bug.

I can suggest some workarounds.  When calling groff, supply the "-K
utf8" option.  Alternatively, add a groff comment to the source document
that will skew libuchardet's heuristics toward detection of UTF-8 over
"MAC-CENTRALEUROPE".  You noted that a single additional UTF-8-encoded
character (that is, one requiring a multi-byte sequence) is enough to
achieve this.

Reassigning.

Regards,
Branden