On Sun, 25 May 2014 02:06:38 +0200 Stefan Skoglund wrote:

[...]

Hello Stefan, thanks for your bug report.

I see that you test LANG=sv_SE.utf-8 with lowercase "utf", which
reminds me of bug #725496 [1]. That other bug should however be fixed
in current ruby-gettext, so it's probably not the same issue you are
experiencing...

[1] https://bugs.debian.org/725496

I will however try to reproduce the bug soon and then I'll get back to
you... Please stay tuned!  ;-)

[...]

I am under the impression that the issue in how Ruby handles locales,
hence sniffing the network traffic won't probably be of much use... Or
maybe not? I don't know...

The direct trigger of the crash is a peculiar character in a bug report
for sound-juicer (bug #717391.)

Take a look in the report from Ben Finney (message 1.)

It could be a lack of input sanitizing in bugs.debian.org ?

Still, this together with localization shouldn't cause a crash in
ruby's localization module (if the weakness is there.)

The older report (725496) regarding this had another trigger:

for some reason the reporter's desktop environment saved a value of
"de_DE.utf8". He destroyed the setting in
"var/lib/AccountsServices/users" and did a reselect of his environment
(dpkg-reconfigure -plow locales) and re-login with the proper LANG
selected.

That cured the symptom in that case.

For me this crashes:
---
[stefan@LOKELDARN@compaq:~]$ echo $LANG
en_GB.UTF-8
[stefan@LOKELDARN@compaq:~]$ LC_MESSAGES=sv_SE.UTF-8 apt-listbugs list
sound-juicer
Hämtar felrapporter... Klar
Tolkar hittad/tillrättad information... Klar
grave bugs of sound-juicer (→ ) <Resolved in some Version>
/usr/share/apt-listbugs/apt-listbugs/logic.rb:611:in `block (4 levels)
in display_bugs': incompatible character encodings: ASCII-8BIT and UTF-8
(Encoding::CompatibilityError)
        from /usr/share/apt-listbugs/debian/bug.rb:66:in `block in
each_by_category'
        from /usr/share/apt-listbugs/debian/bug.rb:65:in `each'
        from /usr/share/apt-listbugs/debian/bug.rb:65:in
`each_by_category'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:594:in `block
(3 levels) in display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:590:in `each'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:590:in `block
(2 levels) in display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:589:in `each'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:589:in `block
in display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:588:in `each'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:588:in
`display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:375:in `view'
        from /usr/sbin/apt-listbugs:513:in `<main>'
---

But this works:
---
[stefan@LOKELDARN@compaq:~]$ LANG=sv_SE.UTF-8 LC_MESSAGES=en_GB.UTF-8
apt-listbugs list sound-juicer
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
grave bugs of sound-juicer (→ ) <Resolved in some Version>
 #717391 - sound-juicer: Refuses to start: “The plugin necessary for
file access was not found” (Fixed: 3.8.0-2 sound-juicer/3.11.90-1)
Summary:
 sound-juicer(1 bug)
---

Ruby isn't sensitive to 'utf-8' or 'UTF-8' but 'utf8' or 'UTF8' was once
sensitive.

This crashes:
---
[stefan@LOKELDARN@compaq:~]$ LANG=sv_SE.UTF-8 LC_MESSAGES=de_DE.UTF-8
apt-listbugs list sound-juicer
Laden der Fehlerberichte ... Erledigt
»Found/Fixed«-Informationen werden ausgewertet ... Erledigt
grave Fehler von sound-juicer (→ ) <Resolved in some Version>
/usr/share/apt-listbugs/apt-listbugs/logic.rb:611:in `block (4 levels)
in display_bugs': incompatible character encodings: ASCII-8BIT and UTF-8
(Encoding::CompatibilityError)
        from /usr/share/apt-listbugs/debian/bug.rb:66:in `block in
each_by_category'
        from /usr/share/apt-listbugs/debian/bug.rb:65:in `each'
        from /usr/share/apt-listbugs/debian/bug.rb:65:in
`each_by_category'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:594:in `block
(3 levels) in display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:590:in `each'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:590:in `block
(2 levels) in display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:589:in `each'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:589:in `block
in display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:588:in `each'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:588:in
`display_bugs'
        from /usr/share/apt-listbugs/apt-listbugs/logic.rb:375:in `view'
        from /usr/sbin/apt-listbugs:513:in `<main>'
[stefan@LOKELDARN@compaq:~]$
---

Control: reassign -1 ruby-locale 2.1.0-2
Control: affects -1 + apt-listbugs

Yes, I reproduced the issue with the following command:

  $ LC_ALL=sv_SE.UTF-8 apt-listbugs list sound-juicer

I performed a number of tests and it seems to me that the problem lies
in the subject line of bug #717391 which includes non-ASCII UTF-8
quotes “ and ” .

For some reason that I cannot understand, Ruby at some point seems to
consider this string as ASCII-8BIT and cannot concatenate it with a
UTF-8 string. Hence, when using a localization where the translation of
"Fixed" includes non-ASCII UTF-8 characters (one such localization is
the Swedish "Tillrättade"), an Encoding::CompatibilityError is
generated...

As I said, I cannot understand why Ruby is convinced that the subject
line of bug #717391 is encoded in ASCII-8BIT.
By using the -d option of apt-listbugs, I took a look at the SOAP wire
dump and it seems to me that the Debian BTS SOAP interface sends this
subject line as

<subject xsi:type="xsd:base64Binary">c291bmQtanVpY2VyOiBSZWZ1c2VzIHRvIHN0YXJ0OiDigJxUaGUgcGx1Z2luIG5lY2Vzc2FyeSBmb3IgZmlsZSBhY2Nlc3Mgd2FzIG5vdCBmb3VuZOKAnQ==</subject>

but:

  $ echo c291bmQtanVpY2VyOiBSZWZ1c2VzIHRvIHN0YXJ0OiDigJxUaGUgcGx1Z2luIG5lY2Vzc2FyeSBmb3IgZmlsZSBhY2Nlc3Mgd2FzIG5vdCBmb3VuZOKAnQ== | base64 -d
  sound-juicer: Refuses to start: “The plugin necessary for file access was not found”

and:

  $ echo c291bmQtanVpY2VyOiBSZWZ1c2VzIHRvIHN0YXJ0OiDigJxUaGUgcGx1Z2luIG5lY2Vzc2FyeSBmb3IgZmlsZSBhY2Nlc3Mgd2FzIG5vdCBmb3VuZOKAnQ== | base64 -d | file -
  /dev/stdin: UTF-8 Unicode text, with no line terminators

Hence, I seem to understand that this subject line is sent as a
base64-encoded UTF-8 string. I failed to figure out where Ruby begins
to consider it as ASCII-8BIT.

I am reassigning your bug report to package ruby-locale, since I am
more and more convinced that the problem lies in how Ruby handles
string encodings.
I hope the maintainers of ruby-locale may investigate and fix
the issue.
[...]

Definitely, lowercase "utf" or uppercase "UTF" does not seem to make
any difference for the issue you are experiencing.

Thanks for taking the time to check this as well!


Now let's hope that the issue may be fixed soon.
Bye.

On Mon, 26 May 2014 00:07:40 +0200 Francesco Poli wrote:

[...]
[...]
responsibility of the SOAP library to return UTF-8 strings with the
encoding correctly set, so that Ruby knows how to handle them.

I am reassigning the bug report to package ruby-soap4r, in the hope
that the actual issue may be fixed there.


Please note that, if I force apt-listbugs to interpret the subject line
of bugs as UTF-8 encoded strings (by modifying a single line of code
in  /usr/share/apt-listbugs/debian/bug.rb , see the attached diff),
the issue with
  $ LC_ALL=sv_SE.UTF-8 apt-listbugs list sound-juicer
seems to vanish.
However, I am afraid that this would *not* be a general solution: what
if, for some other bug subject line, the BTS SOAP interface returns a
string which is not UTF-8 encoded? I think that funny characters would
be displayed by apt-listbugs, if it took every subject line as always
encoded in UTF-8... Or am I wrong?
This is the reasoning that made me think the issue is actually in
ruby-soap4r. I hope it may be fixed there in a proper way.


Bye.

Hi,

seems like I was just bitten by this when installing apt-cacher-ng:

# aptitude -u
Performing actions...
Laden der Fehlerberichte … Erledigt
»Found/Fixed«-Informationen werden ausgewertet … Erledigt
serious Fehler von apt-cacher-ng (→ 3.7.4-1+b2) <In einigen Versionen gelöst>
/usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:828:in `+': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:828:in `block (4 levels) in display_bugs'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:69:in `block in each_by_category'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in `each'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in `each_by_category'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:807:in `block (3 levels) in display_bugs'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:803:in `each'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:803:in `block (2 levels) in display_bugs'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:802:in `each'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:802:in `block in display_bugs'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:801:in `each'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:801:in `display_bugs'
        from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:451:in `view'
        from /usr/bin/apt-listbugs:626:in `<main>'
E: Unterprozess /usr/bin/apt-listbugs apt hat Fehlercode zurückgegeben (1)
E: Failure running script /usr/bin/apt-listbugs apt
Press Return to continue, 'q' followed by Return to quit.
q
root@fenchel:~# LANG=C.UTF-8 aptitude -u
Performing actions...
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
serious bugs of apt-cacher-ng (→ 3.7.4-1+b2) <Resolved in some Version>
 b1 - #980923 - acngtools eats all the CPU and doesn’t finish daily cron with merged pdiffs (Fixed: apt-cacher-ng/3.6.2-1)
   Merged with: 977611
Summary:
 apt-cacher-ng(1 bug)
Are you sure you want to install/upgrade the above packages? [Y/n/?/...]

Same here on apt upgrade, it stops on wine:



Analyse des informations Trouvé/Corrigé… Fait
bogues de gravité grave sur python3-jedi (0.18.2-1 → 0.19.1+ds-1) <En
attente de traitement>
  b1 - #1063491 - python3-jedi: unvendoring python3-typeshed breaks
other packages
bogues de gravité serious sur adwaita-icon-theme (45.0-2 → 46~beta-1)
<En attente de traitement>
  b2 - #1063640 - adwaita-icon-theme: cursor theme not found
bogues de gravité serious sur fwupd (1.9.11-1 → 1.9.13-1) <En attente de
traitement>
  b3 - #1061731 - fwupd: Failed to load daemon: failed to load engine:
Failed to load config: Key file does not have group “redfish”
bogues de gravité serious sur wine (8.0.2~repack-2 → 9.0~repack-2)
<Résolus dans une version donnée>
/usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:842:in `+': incompatible
character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:842:in
`block (4 levels) in display_bugs'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:69:in
`block in each_by_category'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in
`each'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in
`each_by_category'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:821:in
`block (3 levels) in display_bugs'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:817:in `each'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:817:in
`block (2 levels) in display_bugs'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:816:in `each'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:816:in
`block in display_bugs'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:815:in `each'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:815:in
`display_bugs'
         from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:465:in `view'
         from /usr/bin/apt-listbugs:632:in `<main>'
E: Le sous-processus /usr/bin/apt-listbugs apt a renvoyé un code
d'erreur (1)
E: Failure running script /usr/bin/apt-listbugs apt





Passing LANG=C seems to work around it:

Parsing Found/Fixed information... Done
grave bugs of python3-jedi (0.18.2-1 -> 0.19.1+ds-1) <Outstanding>
  b1 - #1063491 - python3-jedi: unvendoring python3-typeshed breaks
other packages
serious bugs of adwaita-icon-theme (45.0-2 -> 46~beta-1) <Outstanding>
  b2 - #1063640 - adwaita-icon-theme: cursor theme not found
serious bugs of fwupd (1.9.11-1 -> 1.9.13-1) <Outstanding>
  b3 - #1061731 - fwupd: Failed to load daemon: failed to load engine:
Failed to load config: Key file does not have group “redfish”
serious bugs of wine (8.0.2~repack-2 -> 9.0~repack-2) <Resolved in some
Version>
  b4 - #1061655 - wine: FTBFS with error: unknown conversion type
character ‘I’ in format [-Werror=format=] (Fixed: wine/9.0~repack-3)
Summary:
  adwaita-icon-theme(1 bug), fwupd(1 bug), wine(1 bug), python3-jedi(1 bug)




Could it be the UTF-8 single quotes around the I that triggers it?
00000000  e2 80 98 49 e2 80 99 0a                           |...I....|

Another case, #1122404 (```
dynare: FTBFS: Error: ‘b’ at (1) has the CONTIGUOUS attribute but is not an array pointer or an assumed-shape or assumed-rank array
```)

Note that there are, as my Konsole said when I pasted this from the
browser, control symbols there, they are visible in my vim in the line
above right now but may not be visible for you: ```Error: â<80><98>bâ<80><99>```.

`apt-listbugs list gcc-15` fails with "incompatible character encodings:
ASCII-8BIT and UTF-8" but `LANG=C apt-listbugs list gcc-15` exits after
printing the first â:

```
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
serious bugs of gcc-15 (-> ) <Forwarded>
  b1 - #1119850 - netpbm-free: FTBFS on arm* (testsuite error) with gcc-15
serious bugs of gcc-15 (-> ) <Resolved in some Version>
  b2 - #1122404 - dynare: FTBFS: Error: â%
```

(the final % is from zsh, showing that the newline wasn't printed before
exiting)