#348909 recode_perform_task() returns wrong error code

Package:
librecode0
Source:
recode
Description:
Shared library on which recode is based
Submitter:
Bas Zoetekouw
Date:
2021-09-22 04:49:00 UTC
Severity:
normal
Tags:
#348909#5
Date:
2006-01-19 20:28:26 UTC
From:
To:
According to the info page, recode_perform_task() should return the
error code RECODE_UNTRANSLATABLE in task->error_so_far if the input
contains characters that cannot be represented in the output charset.

However, it returns RECODE_INVALID_INPUT when trying to translate
certain chars from utf8 to latin1, even if the input is valid utf8.

Below's an example C program that show the bug.  It tries to translate
the string "á ç  α ζ" from utf8 into latin1.  The á and ç work fine,
but it chokes on the alpha (as it should, because latin1 doesn't
contain an alpha).  However, the error code it returns is 4
(==RECODE_INVALID_INPUT) instead of 3 (==RECODE_UNTRANSLATABLE).

This bug obviously makes it impossible to distinguish between invalid
inputs (which, in a user application, should throw an error) or
characters that simple cannot be represented in the desired charset
(which could be replaced by a ? for example).

#include <stdio.h>
#include <stdbool.h>
#include <recodext.h>
#include <string.h>

int
main ()
{
    /* utf8 test string: 2 chars ('a, ,c) representable in latin1,
     * followed by 2 chars (alpha, zeta) that cannot be represented
     * in latin1 */
    char greek_utf_str[] = "\303\241 \303\247  \316\261 \316\266";
    char buf[100] = "";

    RECODE_OUTER outer = recode_new_outer (false);
    RECODE_REQUEST request = recode_new_request (outer);
    RECODE_TASK task;
    bool success;

    recode_scan_request (request, "utf-8..latin1");

    task = recode_new_task (request);
    task->input.buffer = &(greek_utf_str[0]);
    task->input.cursor = task->input.buffer;
    task->input.limit  = task->input.buffer + sizeof(greek_utf_str);
    task->output.buffer = &(buf[0]);
    task->output.cursor = task->output.buffer;
    task->output.limit = task->output.buffer + sizeof(buf);

    success = recode_perform_task (task);

    printf("task completed with error %i\n", task->error_so_far);
    printf("output buffer: ");
    while (task->output.buffer < task->output.cursor) {
        printf("%02X ", (unsigned char) *(task->output.buffer++));
    }
    printf("\n");

    return 0;
}
#348909#8
Date:
2006-01-19 21:17:08 UTC
From:
To:
Hello François.

I just received this from the Debian bug system.

What about the new recode release you were preparing? It would be wonderful
if you could take a look at the list bugs reported to Debian:

http://bugs.debian.org/recode

[ Note: The only bug not in forwarded state is #313925, you can ignore it,
  as I expect it to be fixed by the Free Translation Team ].

Package: librecode0
Version: 3.6-12
Severity: normal

According to the info page, recode_perform_task() should return the
error code RECODE_UNTRANSLATABLE in task->error_so_far if the input
contains characters that cannot be represented in the output charset.

However, it returns RECODE_INVALID_INPUT when trying to translate
certain chars from utf8 to latin1, even if the input is valid utf8.

Below's an example C program that show the bug.  It tries to translate
the string "á ç  ÿÿ ÿÿ" from utf8 into latin1.  The á and ç work fine,
but it chokes on the alpha (as it should, because latin1 doesn't
contain an alpha).  However, the error code it returns is 4
(==RECODE_INVALID_INPUT) instead of 3 (==RECODE_UNTRANSLATABLE).

This bug obviously makes it impossible to distinguish between invalid
inputs (which, in a user application, should throw an error) or
characters that simple cannot be represented in the desired charset
(which could be replaced by a ? for example).

#include <stdio.h>
#include <stdbool.h>
#include <recodext.h>
#include <string.h>

int
main ()
{
    /* utf8 test string: 2 chars ('a, ,c) representable in latin1,
     * followed by 2 chars (alpha, zeta) that cannot be represented
     * in latin1 */
    char greek_utf_str[] = "\303\241 \303\247  \316\261 \316\266";
    char buf[100] = "";

    RECODE_OUTER outer = recode_new_outer (false);
    RECODE_REQUEST request = recode_new_request (outer);
    RECODE_TASK task;
    bool success;

    recode_scan_request (request, "utf-8..latin1");

    task = recode_new_task (request);
    task->input.buffer = &(greek_utf_str[0]);
    task->input.cursor = task->input.buffer;
    task->input.limit  = task->input.buffer + sizeof(greek_utf_str);
    task->output.buffer = &(buf[0]);
    task->output.cursor = task->output.buffer;
    task->output.limit = task->output.buffer + sizeof(buf);

    success = recode_perform_task (task);

    printf("task completed with error %i\n", task->error_so_far);
    printf("output buffer: ");
    while (task->output.buffer < task->output.cursor) {
        printf("%02X ", (unsigned char) *(task->output.buffer++));
    }
    printf("\n");

    return 0;
}
#348909#13
Date:
2006-01-19 21:37:48 UTC
From:
To:
Hi François, Santiago!

Hopefully you won't mind if I add an other comment here.

Note that with the program below, after the error, the
task->input.cursor is still pointing to the begining of
task->input.buffer, even though part of the string has already bene
translated.

AFAICS, this makes it impossible to tell which character exactly caused
the error, and which part of the string has already been translated.
Also, the info page says "Its value will internally advance as the
recoding goes, until it reaches the value of `input.limit'.", which
doesn't seem in line with this behaviour.

Hopefully, you'll be able to fix this (or point out to me what I'm doing
wrong).

Thanks!

Bas.

You wrote:
#348909#14
Date:
2006-01-22 18:27:59 UTC
From:
To:
[Santiago Vila]

Hi, Santiago.  Let me wish you a Nice Year!

Yes, I received it too (you invited me to subscribe to this system).

Still in the works (I regularly work on Recode every week, for a few
hours, yet not many).  Hard to predict when things will be ready.  A few
months, most likely.  I'll keep you informed.
#348909#17
Date:
2014-05-24 13:41:29 UTC
From:
To:
Sehr geehrte/r Arbeitsuchender,

folgendes Arbeitsangebot ist für alle geeignet, da diese Arbeit ohne besondere Anforderungen auch von zu Hause zu bewerkstelligen ist. Der Arbeitnehmer hat keine Ausgaben und muss keine besonderen Kenntnisse mitbringen. Die benötigte technische Ausrüstung wird von uns frei zur Verfügung gestellt. Zu Ihrem Arbeitsinhalt gehört die Betreuung von Onlinebibliotheken, die Koordinierung, das Erstellen von Buchwerken, das Erstellen von Ebooks und das Erstellen von Mediatheken. Wir bieten eine attraktive Bezahlung in Höhe von 17€ pro Stunde.

Unser Betrieb verfügt über internationale Niederlassungen auf der ganzen Welt und wir arbeiten im Bereich der Mediendigitalisierung. Im Moment suchen wir nach neuen Mitarbeitern.

Worauf wir bei unseren Mitarbeiter Wert legen wäre: Sie verfügen über ein Paar Stunden Zeit am Tag, sie verfügen über ein Paar Stunden Zeit am Tag, Sie arbeiten gern von zu Hause, Grundkenntnisse von MS-Office sind von Vorteil, Sie besitzen Flexibilität und Sie besitzen eine teamorientierte Arbeitsweise.

Haben wir Ihr Interesse geweckt? Dann freuen wir uns über Ihre Bewerbung! Senden Sie Ihre vollständigen Unterlagen an: BayCastlete@gardener.com
Wir freuen uns auf Ihre Bewerbung.

Hochachtungsvoll

Gabriel Katz Gbr
#348909#18
Date:
2015-02-09 05:08:37 UTC
From:
To:
Sehr geehrter Bewerber,

unser Kunde ist ein erfolgreiches, europaweit agierendes Unternehmen und sucht zur Zeit aktive Angestellte zur Verstärkung des Teams im europäischen Raum.

Die Leistungen werden Eu weit gebraucht, und Sie haben die Möglichkeit unabhängig von Ihrem Wohnort tätig zu werden. Wir bieten Stellen für jedermann. Die Beschäftigung kann sowohl von Rentnern, Hausfrauen als auch nebenberuflich aufgenommen werden.
Verpackungsaushilfe, Sekretärservice, Bewerter, und vieles mehr wird derzeit angeboten.

Es werden Ihnen vorhandene Aufträge und die dazugehörige Bezahlung angeboten und Sie treffen die Auswahl. Jede Tätigkeit wird verschieden bezahlt, im Schnitt bekommen Sie bei 3-5 Stunden am Tag 1500 bis 1900 Euro Brutto monatlich.
Sie haben keinerlei Ausgaben und können sofort bei uns starten.

Kennziffer MN-416-1352 Es sind 8 freie Stellen zu besetzen.

Die benötigte technische Ausrüstung wird von uns frei zur Verfügung gestellt. Die Position kann gerne von Rentnern, Hausfrauen und auch nebenberuflich besetzt werden.
Ihre Voraussetzungen wären verantwortungsvolle, selbstständige Arbeitsweise, Ehrlichkeit, Zielstrebigkeit, Flexibilität, Freundlichkeit
Möchten Sie sich dieser interessanten und herausfordernden neuen Aufgabe stellen? Dann freuen wir uns auf Ihre aussagekräftigen und vollständigen Bewerbungsunterlagen bevorzugt an: quuchenaexy@webname.com Sie erhalten weitere Einzelheiten zugeschickt.


Ihre persönlichen Unterlagen behandeln wir vertraulich.

Hochachtungsvoll

Hartmann SA
Rue de la Cossonnerie 74
Brest 5359
#348909#23
Date:
2018-01-17 22:49:19 UTC
From:
To:
I looked into this bug. The problem starts with the fact that iconv returns
EILSEQ (invalid input) when in fact the input is simply untranslatable.

It is possible to diagnose this situation by running another conversion with
the output encoding the same as the input (so that it will always succeed on
valid input) at the same point.

Unfortunately, I can’t work out what to do next: in particular, there’s no C
API I can use in general to skip the valid but untranslatable character. (At
least, none I can see; ideas welcome!)

For now, I’ve implemented a partial workaround: untranslatable text is
detected, and one byte is skipped. The typical result is that invalid input
is diagnosed on the next step, resulting in the same problem as at present.

There are two possible workarounds:

1. Set abort_level to RECODE_UNTRANSLATABLE.
2. Don’t use iconv.

I have documented these.
#348909#28
Date:
2021-05-18 06:23:34 UTC
From:
To:
Greeting to you once again



I am writing for the second time as I did not receive any reply from you to
my previous message.



Kindly reply so we can discuss the subject as soon as possible.



Sincerely,



Arthur Clerk
#348909#29
Date:
2021-05-18 06:23:34 UTC
From:
To:
Greeting to you once again



I am writing for the second time as I did not receive any reply from you to
my previous message.



Kindly reply so we can discuss the subject as soon as possible.



Sincerely,



Arthur Clerk
#348909#34
Date:
2021-09-22 04:24:44 UTC
From:
To:
Hello,

Good morning,

We have gone through your samples from a partner and Here is our  Order
List. Please do bear in mind that we are very much in  need of this
order, quote your competitive prices.

Kindly send the Order confirmation.

Your early reply will be much appreciated.

Best Regards,

Maryanah Erwin.

PT FINDORA INTERNUSA

Jln Pahlawan 66 Kec. Arjawinangun

45162 CIREBON West-Java INDONESIA

tel : +62 231 357334

fax: +62 231 357260

email: marketing@findora.com
#348909#35
Date:
2021-09-22 04:24:44 UTC
From:
To:
Hello,

Good morning,

We have gone through your samples from a partner and Here is our  Order
List. Please do bear in mind that we are very much in  need of this
order, quote your competitive prices.

Kindly send the Order confirmation.

Your early reply will be much appreciated.

Best Regards,

Maryanah Erwin.

PT FINDORA INTERNUSA

Jln Pahlawan 66 Kec. Arjawinangun

45162 CIREBON West-Java INDONESIA

tel : +62 231 357334

fax: +62 231 357260

email: marketing@findora.com