#1032173 identity recoding is too identical

Package:
recode
Source:
recode
Description:
Character set conversion utility
Submitter:
Zefram
Date:
2023-03-02 23:33:03 UTC
Severity:
normal
#1032173#5
Date:
2023-03-01 01:30:17 UTC
From:
To:
recode(1) usually checks that its input conforms to the specified input
encoding, and signals an error if it doesn't:

$ echo $'L\xe9on' | recode utf8..utf7
Lrecode: Invalid input in step `UTF-8..UNICODE-1-1-UTF-7'

But if the output encoding happens to be the same as the input encoding
then this checking doesn't happen, and invalid output can be produced:

$ echo $'L\xe9on' | recode utf8..utf8 | od -tc
0000000   L 351   o   n  \n
0000005

The invocation with both encodings the same superficially looks like
it's requesting an identity transformation, and it would correctly have
the behaviour of an identity transformation on input that were correctly
encoded.  Because of the input checking that recode(1) usually provides,
it seems like this kind of invocation would be useful, as something
that copies its input while checking the encoding.  Apparently it's
being optimised incorrectly, to a pure identity transformation without
the checking.
#1032173#10
Date:
2023-03-02 21:54:12 UTC
From:
To:
I have done some work on it, but I don't yet have something I can release.

Here's the existing upstream issue:
https://github.com/rrthomas/recode/issues/37
#1032173#15
Date:
2023-03-02 23:09:27 UTC
From:
To:
Reuben Thomas wrote:
some progress.

Quick thought about a way this could be tackled: internally you could
explicitly represent the input checking step distinct from a "mere copy"
operation, you interpret "UTF-8..UTF-8" as a checking step, and then
some checking steps can be optimised out of the operation sequence.
Checking that the input conforms to particular charset immediately after
conversion to that same charset can be optimised out, checking conformance
to any 8-bit single-byte charset is null and can be optimised out, and
there are some cases where checks for different charsets are equivalent.

Further refinement of the above: in some cases there might be value
in splitting a conversion step into a checking step followed by a
non-checking conversion.  The value here is that that checking step
might then be able to be optimised out depending on the prior step of
the pipeline.  At a later stage of optimisation, maybe the checking
step and non-checking conversion recombine into an ordinary checking
conversion of the kind you already have.