- Package:
- ruby-debian
- Source:
- ruby-debian
- Description:
- ruby interface for dpkg
- Submitter:
- Yavor Doganov
- Date:
- 2014-12-19 14:27:11 UTC
- Severity:
- normal
$ dhelp Starting browser (using local filesystem) ... w3m: Can't load /usr/share/doc/HTML/index.html. /usr/bin/sensible-browser: Failed to open /usr/share/doc/HTML/index.html: Отказано свързване $ ls /usr/share/doc/HTML/ README $ sudo dpkg-reconfigure dhelp Building HTML tree...ArgumentError: invalid byte sequence in UTF-8 (/usr/lib/ruby/vendor_ruby/debian.rb:914:in `block in initialize' /usr/lib/ruby/vendor_ruby/debian.rb:727:in `block in parse' /usr/lib/ruby/vendor_ruby/debian.rb:726:in `each' /usr/lib/ruby/vendor_ruby/debian.rb:726:in `parse' /usr/lib/ruby/vendor_ruby/debian.rb:720:in `parseArchiveFile' /usr/lib/ruby/vendor_ruby/debian.rb:913:in `initialize' /usr/lib/ruby/vendor_ruby/debian.rb:46:in `new' /usr/lib/ruby/vendor_ruby/debian.rb:46:in `status' /usr/lib/ruby/vendor_ruby/dhelp/exporter/cgimap.rb:58:in `package_status' /usr/lib/ruby/vendor_ruby/dhelp/exporter/cgimap.rb:71:in `installed?' /usr/lib/ruby/vendor_ruby/dhelp/exporter/cgimap.rb:87:in `info2www_link' (erb):30:in `make_page' /usr/lib/ruby/2.1.0/erb.rb:850:in `eval' /usr/lib/ruby/2.1.0/erb.rb:850:in `result' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:45:in `make_page' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:100:in `export_section_page' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:64:in `block in export' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:62:in `each_pair' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:62:in `export' /usr/sbin/dhelp_parse:153:in `rebuild_html_index' /usr/sbin/dhelp_parse:210:in `main' /usr/sbin/dhelp_parse:216:in `<main>') done. Reindexing documentation in the background
Hi, Yavor Doganov wrote (05 Nov 2014 08:02:47 GMT) : I cannot reproduce that in a sid GNOME VM, so I guess this is caused by some documentation file you have installed, but I haven't. Could you please try to get dhelp to tell you what file it's parsing when it fails? IMO dhelp should *not* crash on invalid UTF-8, but just skip the affected file, display a warning, and move on with its life. Your input could be very useful to create a test case that helps fixing the actual bug. Cheers, -- intrigeri
to do that. I tried to move away all UTF-8 files from /var/lib/doc-base/documents and it fails again. Then I moved all files but "dc" and it fails, too. FWIW, if I run dhelp_parse with "LC_ALL=C" the error message is ArgumentError: invalid byte sequence in US-ASCII (/usr/lib/ruby/vendor_ruby/debian.rb:914:in `block in initialize' (The backtrace is the same). Could it be that debian.rb from ruby-debian is unable to parse the Packages file(s)? The log file in /var/lib/dhelp/tmp is empty, as well as /var/lib/dhelp/documents.index. I also tried with different doc-base files, the result is always the same. I'd be glad if I can assist in reproducing the issue, but I I'd need some directions.
I don't know if it helps, but I got a similar error from the weekly cron task. LANG=C sudo /etc/cron.weekly/dhelp ArgumentError: invalid byte sequence in US-ASCII (/usr/lib/ruby/vendor_ruby/dhelp.rb:185:in `===' /usr/lib/ruby/vendor_ruby/dhelp.rb:185:in `block in initialize' /usr/lib/ruby/vendor_ruby/dhelp.rb:183:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:183:in `initialize' /usr/lib/ruby/vendor_ruby/dhelp.rb:309:in `new' /usr/lib/ruby/vendor_ruby/dhelp.rb:309:in `block (2 levels) in each' /usr/lib/ruby/vendor_ruby/dhelp.rb:306:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:306:in `block in each' /usr/lib/ruby/vendor_ruby/dhelp.rb:305:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:305:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:456:in `_register_docs' /usr/lib/ruby/vendor_ruby/dhelp.rb:387:in `rebuild' /usr/sbin/dhelp_parse:204:in `main' /usr/sbin/dhelp_parse:216:in `<main>') It seems to work fine with my default locale: LANG=es_CO.utf8 sudo /etc/cron.weekly/dhelp I added a "puts @path" in /usr/lib/ruby/vendor_ruby/dhelp.rb:184 and it shows this: ... /var/lib/doc-base/documents/bogofilter-bogotune-faq /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/developers-reference ArgumentError: invalid byte sequence in US-ASCII (/usr/lib/ruby/vendor_ruby/dhelp.rb:186:in `===' /usr/lib/ruby/vendor_ruby/dhelp.rb:186:in `block in initialize' /usr/lib/ruby/vendor_ruby/dhelp.rb:183:in `each' ... The problem rises when it parses the fourth line in developers-reference: Author: Adam Di Carlo, Josip Rodin, Raphaël Hertzog, et al It doesn't complain when I use fr_FR.UTF-8 ! (which is not consisent :P) Cheers, Santiago
I don't know if it helps, but I got a similar error from the weekly cron task. LANG=C sudo /etc/cron.weekly/dhelp ArgumentError: invalid byte sequence in US-ASCII (/usr/lib/ruby/vendor_ruby/dhelp.rb:185:in `===' /usr/lib/ruby/vendor_ruby/dhelp.rb:185:in `block in initialize' /usr/lib/ruby/vendor_ruby/dhelp.rb:183:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:183:in `initialize' /usr/lib/ruby/vendor_ruby/dhelp.rb:309:in `new' /usr/lib/ruby/vendor_ruby/dhelp.rb:309:in `block (2 levels) in each' /usr/lib/ruby/vendor_ruby/dhelp.rb:306:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:306:in `block in each' /usr/lib/ruby/vendor_ruby/dhelp.rb:305:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:305:in `each' /usr/lib/ruby/vendor_ruby/dhelp.rb:456:in `_register_docs' /usr/lib/ruby/vendor_ruby/dhelp.rb:387:in `rebuild' /usr/sbin/dhelp_parse:204:in `main' /usr/sbin/dhelp_parse:216:in `<main>') It seems to work fine with my default locale: LANG=es_CO.utf8 sudo /etc/cron.weekly/dhelp I added a "puts @path" in /usr/lib/ruby/vendor_ruby/dhelp.rb:184 and it shows this: ... /var/lib/doc-base/documents/bogofilter-bogotune-faq /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/developers-reference ArgumentError: invalid byte sequence in US-ASCII (/usr/lib/ruby/vendor_ruby/dhelp.rb:186:in `===' /usr/lib/ruby/vendor_ruby/dhelp.rb:186:in `block in initialize' /usr/lib/ruby/vendor_ruby/dhelp.rb:183:in `each' ... The problem rises when it parses the fourth line in developers-reference: Author: Adam Di Carlo, Josip Rodin, Raphaël Hertzog, et al It doesn't complain when I use fr_FR.UTF-8 ! (which is not consisent :P) Cheers, Santiago
Attached is a diff with a change to dhelp_parse.rb which sets Encoding.default_external explicitly, so that even if LANG=C, it uses UTF-8 instead of US-ASCII as the default for opening files. By my (limited) understanding of Encoding.default_external, this should have the same effect on opening files as replacing LANG=C with LANG=xx_XX.UTF-8 would. On my machine, without the patch, I see the same errors with LANG=C as the others here. With the patch, I do not. Hope to help, - Dan Getz
On Sat, 06 Dec 2014 01:33:58 -0100, Daniel Getz wrote: I can reproduce the problem with LC_ALL=C LANG=C /etc/cron.weekly/dhelp Works for me as well. Since I don't speak any ruby I'm a bit hesitant to upload; maybe some ruby speaker knowing Encoding.default_external can confirm that's the correct way forwards? (And: Are we sure all doc-base files are us-ascii or utf-8 encoded? At least on my machine they are, so maybe that's a non-concern.) Cheers, gregor
UTF-8 should be the right format for doc-base files, according to https://lintian.debian.org/tags/doc-base-file-uses-obsolete-national-encoding.html I also don't know ruby, but from my research setting Encoding.default_external is considered the "wrong" thing to do, the "right" way being to pass "-E UTF-8" as an option to ruby via the command line, or the environment variable RUBYOPT. I had to explicitly silence a warning because of this. See http://docs.ruby-lang.org/en/2.1.0/Encoding.html#method-c-default_external-3D However, neither of those "right" ways to set the encoding work well with using a ruby file directly as a script. (Is ruby not intended to be used in scripts?!) In the ruby docs, it says the problem is if code gets run before the change to the encoding. That's avoidable, and I believe I avoided it in my patch by placing the encoding change before any require imports. An alternative is to explicitly set the encoding to UTF-8 each time a file is opened. If someone feels that's a better way, I'm willing to do that and create a new patch. But like I said, I don't know ruby, so I can't guarantee correctness beyond trying it and seeing that it works. - Dan
Dear maintainer, I've prepared an NMU for dhelp (versioned as 0.6.21+nmu6) and uploaded it to DELAYED/5. Please feel free to tell me if I should delay it longer. Regards.
We believe that the bug you reported is fixed in the latest version of
dhelp, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 768127@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
gregor herrmann <gregoa@debian.org> (supplier of updated dhelp package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Fri, 12 Dec 2014 22:02:20 +0100
Source: dhelp
Binary: dhelp
Architecture: source all
Version: 0.6.21+nmu6
Distribution: unstable
Urgency: medium
Maintainer: Georgios M. Zarkadas <gz@member.fsf.org>
Changed-By: gregor herrmann <gregoa@debian.org>
Description:
dhelp - online help system
Closes: 768127
Changes:
dhelp (0.6.21+nmu6) unstable; urgency=medium
.
* Non-maintainer upload.
* Fix "Fails to build the index when invalid UTF-8 is met":
apply patch from Daniel Getz:
+ Load files as UTF-8, regardless of $LANG
(Closes: #768127)
Checksums-Sha1:
62974affb2cd887685e7bd4abda6500281c795be 1658 dhelp_0.6.21+nmu6.dsc
943b14a40d857967a17d27583100ddab09a5f1bb 70260 dhelp_0.6.21+nmu6.tar.xz
919edb21558f6e9c0a0fe6f35f68413d397c2954 65560 dhelp_0.6.21+nmu6_all.deb
Checksums-Sha256:
1999e5ff20e60a1b38488296b9f87e42e76541c0c742e63b61f1ff01de35f96b 1658 dhelp_0.6.21+nmu6.dsc
e4d5f36ca3ecdd06321d0d71d2ba382da7203392c733b2367a533de7d674c2e6 70260 dhelp_0.6.21+nmu6.tar.xz
c2c40c6bd91fdb077a316ed904507ea3fdcbd6c79329c9445868aac0319ba28e 65560 dhelp_0.6.21+nmu6_all.deb
Files:
8ebd587b894ffd4a6d3e59490b909662 1658 doc optional dhelp_0.6.21+nmu6.dsc
c4a33e5a5354e35135cc949f8ef7a17a 70260 doc optional dhelp_0.6.21+nmu6.tar.xz
22bfda252a507033db09babcd21e5441 65560 doc optional dhelp_0.6.21+nmu6_all.deb
iQJ8BAEBCgBmBQJUi1gKXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXREMUUxMzE2RTkzQTc2MEE4MTA0RDg1RkFC
QjNBNjgwMTg2NDlBQTA2AAoJELs6aAGGSaoGgy4P/i9yJ6FsBX4z6qsCP5A7tEef
nbcFhKiAhjvCsfiz73sJbFdMdRUCJvOr6BFLZwU6X02UCh5l31EbV2bYWvILzSYD
W1wcjcbxLF0g+WBUaZyPuk2ZcbRrMa4r/4Sw79yebF0cUjN6wDns3XoMpYu0bX3T
11B/8PGVxgaceuPzPmD8o225MaDaX+DZT5ocu9ESwlm9BZhiA9g0nJPCm9kdyb8e
nD0I9AJ3LqFS/IiVtAqu9D2suPenFSgttUeIeDsz+LYZVgRnGrgquNi/52MA4jW/
6cxP7QIyg/tFn9z124zHJcJowDfa07/NwlO3H0NxnPwHMJA8roVa2oZp7GDC0xKU
nVchE+GBXSH4MpnLIcq03zy1phi8JOKBsbY/Rn4RO5vqz8mMfa5ECARPonsXOVt0
Rs3o2FCkrRcxi/oz2HwNOwUPSNR0dKhZwNVhjusl9AmL7L2cmRTsOgVjCi6PmrYk
W81/oXF5p+jr3lO7fDU7jcfwXeyFdjtOOqPI2JRupxl77nM6iqKHOAy0I2DPhR66
CzXnf8rMxQ09TOOYb8EN7FL4Erv3c4Iivf5bQR7sgM73z2A2R6qaBEJgWeGn5+CG
XjZ8pHpS1NAHBm5gEyiGbUhnWgeoVfHyd4EsEIYEIICC960aff90B8F5ibu3dmNO
4t6zafPglPlFBczATefh
=Z8MG
-----END PGP SIGNATURE-----
reopen 768127 notfixed 768127 0.6.21+nmu6 thanks Thanks for your work, but unfortunately I experience exactly the same problem with the new version. $ isutf8 /var/lib/doc-base/documents/* $ echo $? 0
Ouch. I'm sorry to hear this. Could you please provide a bit more information? I guess it would be helpful if you could try to - add a "puts @path" in /usr/lib/ruby/vendor_ruby/dhelp.rb:184 (see Santiago's message #24); - copy the output of `locale' and `/etc/cron.weekly/dhelp'; - maybe try `LC_ALL=x LANG=x /etc/cron.weekly/dhelp' for different versions of x. (Hm, maybe we should set LANG/LC_ALL in the cron script. But before that I'd like to see where it fails for Yavor.) Thanks in advance, gregor
gregor herrmann wrote: LANG=bg_BG.UTF-8 LANGUAGE=bg:en_GB LC_CTYPE="bg_BG.UTF-8" LC_NUMERIC="bg_BG.UTF-8" LC_TIME="bg_BG.UTF-8" LC_COLLATE="bg_BG.UTF-8" LC_MONETARY="bg_BG.UTF-8" LC_MESSAGES="bg_BG.UTF-8" LC_PAPER="bg_BG.UTF-8" LC_NAME="bg_BG.UTF-8" LC_ADDRESS="bg_BG.UTF-8" LC_TELEPHONE="bg_BG.UTF-8" LC_MEASUREMENT="bg_BG.UTF-8" LC_IDENTIFICATION="bg_BG.UTF-8" LC_ALL= [ Sorry for the long output. ] /var/lib/doc-base/documents/dc /var/lib/doc-base/documents/abi-compliance-checker /var/lib/doc-base/documents/aptitude-doc-en /var/lib/doc-base/documents/autoconf /var/lib/doc-base/documents/automake-1.14 /var/lib/doc-base/documents/bash /var/lib/doc-base/documents/bashref /var/lib/doc-base/documents/bc /var/lib/doc-base/documents/bzip2 /var/lib/doc-base/documents/bzr /var/lib/doc-base/documents/bzr-builddeb /var/lib/doc-base/documents/comerr-manual /var/lib/doc-base/documents/copyright-format-1.0 /var/lib/doc-base/documents/cpp-4.9 /var/lib/doc-base/documents/cppinternals-4.9 /var/lib/doc-base/documents/cvs-doc /var/lib/doc-base/documents/cvs-doc-client /var/lib/doc-base/documents/cvs-doc-faq /var/lib/doc-base/documents/cvs-doc-intro /var/lib/doc-base/documents/cvs-doc-paper /var/lib/doc-base/documents/cvs-doc-rcsfiles /var/lib/doc-base/documents/dbuskit-api-docs /var/lib/doc-base/documents/dbuskit-manual /var/lib/doc-base/documents/debconf-spec /var/lib/doc-base/documents/debian-constitution-text /var/lib/doc-base/documents/debian-faq /var/lib/doc-base/documents/debian-mailing-lists /var/lib/doc-base/documents/debian-manifesto /var/lib/doc-base/documents/debian-menu-policy /var/lib/doc-base/documents/debian-perl-policy /var/lib/doc-base/documents/debian-policy /var/lib/doc-base/documents/debian-reporting-bugs /var/lib/doc-base/documents/debian-social-contract /var/lib/doc-base/documents/debian-tex-policy /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/doc-base /var/lib/doc-base/documents/docbook-xsl-doc /var/lib/doc-base/documents/everyday-git /var/lib/doc-base/documents/exim4-filter-txt /var/lib/doc-base/documents/exim4-readme-debian /var/lib/doc-base/documents/exim4-spec-txt /var/lib/doc-base/documents/expat /var/lib/doc-base/documents/feynmf-manual /var/lib/doc-base/documents/fhs /var/lib/doc-base/documents/findutils /var/lib/doc-base/documents/fontconfig-devel /var/lib/doc-base/documents/fontconfig-user /var/lib/doc-base/documents/gawk-doc-gawk /var/lib/doc-base/documents/gawk-doc-gawkinet /var/lib/doc-base/documents/gcc-4.9 /var/lib/doc-base/documents/gccint-4.9 /var/lib/doc-base/documents/gccintro /var/lib/doc-base/documents/gdl2api /var/lib/doc-base/documents/gdl2intro /var/lib/doc-base/documents/git-api /var/lib/doc-base/documents/git-bisect-lk2009 /var/lib/doc-base/documents/git-buildpackage /var/lib/doc-base/documents/git-howtos /var/lib/doc-base/documents/git-index-format /var/lib/doc-base/documents/git-pack-format /var/lib/doc-base/documents/git-protocol /var/lib/doc-base/documents/git-reference-manual /var/lib/doc-base/documents/git-shallow-clone-design /var/lib/doc-base/documents/git-tools /var/lib/doc-base/documents/git-trivial-merge-rules /var/lib/doc-base/documents/git-user-manual /var/lib/doc-base/documents/glibc-manual /var/lib/doc-base/documents/gnu-coding-standards /var/lib/doc-base/documents/gnu-maintainers-information /var/lib/doc-base/documents/gnustep-base-additions /var/lib/doc-base/documents/gnustep-base-programming-manual /var/lib/doc-base/documents/gnustep-base-reference /var/lib/doc-base/documents/gnustep-base-tools /var/lib/doc-base/documents/gnustep-coding-standards /var/lib/doc-base/documents/gnustep-gui-additions /var/lib/doc-base/documents/gnustep-gui-programming-manual /var/lib/doc-base/documents/gnustep-gui-reference /var/lib/doc-base/documents/gnustep-make-manual /var/lib/doc-base/documents/gnustep-netclasses-docs /var/lib/doc-base/documents/gnustep-performance /var/lib/doc-base/documents/gnustep-sqlclient /var/lib/doc-base/documents/gorm.app /var/lib/doc-base/documents/initramfs-maintainer /var/lib/doc-base/documents/install-docs-man /var/lib/doc-base/documents/jade /var/lib/doc-base/documents/kbd-font-formats /var/lib/doc-base/documents/libao /var/lib/doc-base/documents/libexif-api /var/lib/doc-base/documents/libffi /var/lib/doc-base/documents/libfreetype6-dev /var/lib/doc-base/documents/libidn11 /var/lib/doc-base/documents/libio-stringy-perl /var/lib/doc-base/documents/libpng12 /var/lib/doc-base/documents/libsdl1.2-dev /var/lib/doc-base/documents/libsndfile /var/lib/doc-base/documents/libtasn1 /var/lib/doc-base/documents/libvorbis /var/lib/doc-base/documents/libxml-parser-perl /var/lib/doc-base/documents/libxslt1-dev /var/lib/doc-base/documents/lintian /var/lib/doc-base/documents/lynx-cur /var/lib/doc-base/documents/make-doc /var/lib/doc-base/documents/man-db /var/lib/doc-base/documents/mc-faq /var/lib/doc-base/documents/menu /var/lib/doc-base/documents/mutt /var/lib/doc-base/documents/nano /var/lib/doc-base/documents/nano-faq /var/lib/doc-base/documents/nettle-dev /var/lib/doc-base/documents/oolite-advice-for-new-commander /var/lib/doc-base/documents/oolite-readme /var/lib/doc-base/documents/oolite-reference-sheet /var/lib/doc-base/documents/openjade /var/lib/doc-base/documents/pbuilder /var/lib/doc-base/documents/po-debconf /var/lib/doc-base/documents/prerex /var/lib/doc-base/documents/preview-latex-style /var/lib/doc-base/documents/pstoedit-man /var/lib/doc-base/documents/python-policy /var/lib/doc-base/documents/quilt /var/lib/doc-base/documents/renaissance-manual /var/lib/doc-base/documents/renaissance-tutorial /var/lib/doc-base/documents/rsskit-api-docs /var/lib/doc-base/documents/ruby-bdb /var/lib/doc-base/documents/sane-backends /var/lib/doc-base/documents/shared-mime-info /var/lib/doc-base/documents/sp /var/lib/doc-base/documents/steptalk-docs /var/lib/doc-base/documents/svn-buildpackage-howto /var/lib/doc-base/documents/tex-on-debian /var/lib/doc-base/documents/the-racy-git-problem /var/lib/doc-base/documents/time /var/lib/doc-base/documents/tla-doc /var/lib/doc-base/documents/urlview /var/lib/doc-base/documents/users-and-groups /var/lib/doc-base/documents/valgrind /var/lib/doc-base/documents/whohas-intro /var/lib/doc-base/documents/xinetd-faq /var/lib/doc-base/documents/xterm-ctlseqs /var/lib/doc-base/documents/xterm-faq /var/lib/doc-base/documents/dc /var/lib/doc-base/documents/abi-compliance-checker /var/lib/doc-base/documents/aptitude-doc-en /var/lib/doc-base/documents/autoconf /var/lib/doc-base/documents/automake-1.14 /var/lib/doc-base/documents/bash /var/lib/doc-base/documents/bashref /var/lib/doc-base/documents/bc /var/lib/doc-base/documents/bzip2 /var/lib/doc-base/documents/bzr /var/lib/doc-base/documents/bzr-builddeb /var/lib/doc-base/documents/comerr-manual /var/lib/doc-base/documents/copyright-format-1.0 /var/lib/doc-base/documents/cpp-4.9 /var/lib/doc-base/documents/cppinternals-4.9 /var/lib/doc-base/documents/cvs-doc /var/lib/doc-base/documents/cvs-doc-client /var/lib/doc-base/documents/cvs-doc-faq /var/lib/doc-base/documents/cvs-doc-intro /var/lib/doc-base/documents/cvs-doc-paper /var/lib/doc-base/documents/cvs-doc-rcsfiles /var/lib/doc-base/documents/dbuskit-api-docs /var/lib/doc-base/documents/dbuskit-manual /var/lib/doc-base/documents/debconf-spec /var/lib/doc-base/documents/debian-constitution-text /var/lib/doc-base/documents/debian-faq /var/lib/doc-base/documents/debian-mailing-lists /var/lib/doc-base/documents/debian-manifesto /var/lib/doc-base/documents/debian-menu-policy /var/lib/doc-base/documents/debian-perl-policy /var/lib/doc-base/documents/debian-policy /var/lib/doc-base/documents/debian-reporting-bugs /var/lib/doc-base/documents/debian-social-contract /var/lib/doc-base/documents/debian-tex-policy /var/lib/doc-base/documents/developers-reference /var/lib/doc-base/documents/doc-base /var/lib/doc-base/documents/docbook-xsl-doc /var/lib/doc-base/documents/everyday-git /var/lib/doc-base/documents/exim4-filter-txt /var/lib/doc-base/documents/exim4-readme-debian /var/lib/doc-base/documents/exim4-spec-txt /var/lib/doc-base/documents/expat /var/lib/doc-base/documents/feynmf-manual /var/lib/doc-base/documents/fhs /var/lib/doc-base/documents/findutils /var/lib/doc-base/documents/fontconfig-devel /var/lib/doc-base/documents/fontconfig-user /var/lib/doc-base/documents/gawk-doc-gawk /var/lib/doc-base/documents/gawk-doc-gawkinet /var/lib/doc-base/documents/gcc-4.9 /var/lib/doc-base/documents/gccint-4.9 /var/lib/doc-base/documents/gccintro /var/lib/doc-base/documents/gdl2api /var/lib/doc-base/documents/gdl2intro /var/lib/doc-base/documents/git-api /var/lib/doc-base/documents/git-bisect-lk2009 /var/lib/doc-base/documents/git-buildpackage /var/lib/doc-base/documents/git-howtos /var/lib/doc-base/documents/git-index-format /var/lib/doc-base/documents/git-pack-format /var/lib/doc-base/documents/git-protocol /var/lib/doc-base/documents/git-reference-manual /var/lib/doc-base/documents/git-shallow-clone-design /var/lib/doc-base/documents/git-tools /var/lib/doc-base/documents/git-trivial-merge-rules /var/lib/doc-base/documents/git-user-manual /var/lib/doc-base/documents/glibc-manual /var/lib/doc-base/documents/gnu-coding-standards /var/lib/doc-base/documents/gnu-maintainers-information /var/lib/doc-base/documents/gnustep-base-additions /var/lib/doc-base/documents/gnustep-base-programming-manual /var/lib/doc-base/documents/gnustep-base-reference /var/lib/doc-base/documents/gnustep-base-tools /var/lib/doc-base/documents/gnustep-coding-standards /var/lib/doc-base/documents/gnustep-gui-additions /var/lib/doc-base/documents/gnustep-gui-programming-manual /var/lib/doc-base/documents/gnustep-gui-reference /var/lib/doc-base/documents/gnustep-make-manual /var/lib/doc-base/documents/gnustep-netclasses-docs /var/lib/doc-base/documents/gnustep-performance /var/lib/doc-base/documents/gnustep-sqlclient /var/lib/doc-base/documents/gorm.app /var/lib/doc-base/documents/initramfs-maintainer /var/lib/doc-base/documents/install-docs-man /var/lib/doc-base/documents/jade /var/lib/doc-base/documents/kbd-font-formats /var/lib/doc-base/documents/libao /var/lib/doc-base/documents/libexif-api /var/lib/doc-base/documents/libffi /var/lib/doc-base/documents/libfreetype6-dev /var/lib/doc-base/documents/libidn11 /var/lib/doc-base/documents/libio-stringy-perl /var/lib/doc-base/documents/libpng12 /var/lib/doc-base/documents/libsdl1.2-dev /var/lib/doc-base/documents/libsndfile /var/lib/doc-base/documents/libtasn1 /var/lib/doc-base/documents/libvorbis /var/lib/doc-base/documents/libxml-parser-perl /var/lib/doc-base/documents/libxslt1-dev /var/lib/doc-base/documents/lintian /var/lib/doc-base/documents/lynx-cur /var/lib/doc-base/documents/make-doc /var/lib/doc-base/documents/man-db /var/lib/doc-base/documents/mc-faq /var/lib/doc-base/documents/menu /var/lib/doc-base/documents/mutt /var/lib/doc-base/documents/nano /var/lib/doc-base/documents/nano-faq /var/lib/doc-base/documents/nettle-dev /var/lib/doc-base/documents/oolite-advice-for-new-commander /var/lib/doc-base/documents/oolite-readme /var/lib/doc-base/documents/oolite-reference-sheet /var/lib/doc-base/documents/openjade /var/lib/doc-base/documents/pbuilder /var/lib/doc-base/documents/po-debconf /var/lib/doc-base/documents/prerex /var/lib/doc-base/documents/preview-latex-style /var/lib/doc-base/documents/pstoedit-man /var/lib/doc-base/documents/python-policy /var/lib/doc-base/documents/quilt /var/lib/doc-base/documents/renaissance-manual /var/lib/doc-base/documents/renaissance-tutorial /var/lib/doc-base/documents/rsskit-api-docs /var/lib/doc-base/documents/ruby-bdb /var/lib/doc-base/documents/sane-backends /var/lib/doc-base/documents/shared-mime-info /var/lib/doc-base/documents/sp /var/lib/doc-base/documents/steptalk-docs /var/lib/doc-base/documents/svn-buildpackage-howto /var/lib/doc-base/documents/tex-on-debian /var/lib/doc-base/documents/the-racy-git-problem /var/lib/doc-base/documents/time /var/lib/doc-base/documents/tla-doc /var/lib/doc-base/documents/urlview /var/lib/doc-base/documents/users-and-groups /var/lib/doc-base/documents/valgrind /var/lib/doc-base/documents/whohas-intro /var/lib/doc-base/documents/xinetd-faq /var/lib/doc-base/documents/xterm-ctlseqs /var/lib/doc-base/documents/xterm-faq ArgumentError: invalid byte sequence in UTF-8 (/usr/lib/ruby/vendor_ruby/debian.rb:914:in `block in initialize' /usr/lib/ruby/vendor_ruby/debian.rb:727:in `block in parse' /usr/lib/ruby/vendor_ruby/debian.rb:726:in `each' /usr/lib/ruby/vendor_ruby/debian.rb:726:in `parse' /usr/lib/ruby/vendor_ruby/debian.rb:720:in `parseArchiveFile' /usr/lib/ruby/vendor_ruby/debian.rb:913:in `initialize' /usr/lib/ruby/vendor_ruby/debian.rb:46:in `new' /usr/lib/ruby/vendor_ruby/debian.rb:46:in `status' /usr/lib/ruby/vendor_ruby/dhelp/exporter/cgimap.rb:58:in `package_status' /usr/lib/ruby/vendor_ruby/dhelp/exporter/cgimap.rb:71:in `installed?' /usr/lib/ruby/vendor_ruby/dhelp/exporter/cgimap.rb:87:in `info2www_link' (erb):30:in `make_page' /usr/lib/ruby/2.1.0/erb.rb:850:in `eval' /usr/lib/ruby/2.1.0/erb.rb:850:in `result' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:45:in `make_page' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:100:in `export_section_page' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:64:in `block in export' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:62:in `each_pair' /usr/lib/ruby/vendor_ruby/dhelp/exporter/html.rb:62:in `export' /usr/sbin/dhelp_parse:158:in `rebuild_html_index' /usr/sbin/dhelp_parse:215:in `main' /usr/sbin/dhelp_parse:221:in `<main>') There is no difference.
Thanks! Ok. No worries, that was expected :) [..] /var/lib/doc-base/documents/xterm-faq can't be interpreted as UTF-8. Which is interesing for two reasons: - First, I have the file installed and don't have any problems with dhelp (although not with a bulgarian locale but UTF-8 should be UTF-8?!) - Second, at least on my system, it's plain ASCII: % file -i /var/lib/doc-base/documents/xterm-faq /var/lib/doc-base/documents/xterm-faq: text/plain; charset=us-ascii And the file itself looks completely innocent: #v+ Document: xterm-faq Section: Terminal Emulators Title: XTerm Frequently Asked Questions (FAQ) Author: Thomas Dickey Abstract: This document provides answers to frequently asked questions about the XTerm terminal emulator as it ships with the X.Org distribution of the X Window System. Format: HTML Index: /usr/share/doc/xterm/xterm.faq.html Files: /usr/share/doc/xterm/xterm.faq.html Format: text Files: /usr/share/doc/xterm/xterm.faq.gz #v- This is getting slightly mysterious. Could you please try `file -i /var/lib/doc-base/documents/xterm-faq' as well and open the file in a pager/editor to see if it's the same on your machine? If I'm understanding this correctly, the "origin" is /usr/share/doc-base/xterm-faq which has, according to /var/lib/dpkg/info/xterm.md5sums, a md5sum of 4f81e4dd965c918abc250beeb54131fb. Confirmed locally: % md5sum /usr/share/doc-base/xterm-faq 4f81e4dd965c918abc250beeb54131fb /usr/share/doc-base/xterm-faq Maybe you could check this out as well to rule out a corrupted file? Ok, thanks again! Cheers, gregor
gregor herrmann wrote: I don't think this has anything to do with xterm-faq, it's just that this is the last file in alphabetical order. If I move the file away I get the same failure -- it chokes on xterm-ctlseqs then. Yes, it's the same: $ file -i /var/lib/doc-base/documents/xterm-faq /var/lib/doc-base/documents/xterm-faq: text/plain; charset=us-ascii $ md5sum /usr/share/doc-base/xterm-faq 4f81e4dd965c918abc250beeb54131fb /usr/share/doc-base/xterm-faq Too bad I don't know Ruby and I'm completely clueless. It seems that there are two different problems -- Santiago's failure that he posted on the bug log is at dhelp.rb:185 while mine is at debian.rb:914 (which is why I suggested it might be a ruby-debian issue). If you and Daniel have reproduced and fixed Santiago's bug it is not surprising that the NMU does not address the bug I am observing. (At least that is how I explain the mystery for the time being, with my limited knowledge.)
Yes, that's my understanding, too. Can you run with the attached patch to debian.rb, and see if it will show which entry of which file triggers the error?
D'oh. Then my guesses were really off track. Same here. Right, sorry for missing this. Let's hope that Daniel's debugging idea helps ... Cheers, gregor
Daniel Getz wrote: Error parsing file /var/lib/dpkg/available Contents of info: Package: ayuda Priority: extra Section: misc Installed-Size: 204 Maintainer: Javier Vi�uales Guti�rrez <vigu@matrio.com> Architecture: all Version: 0.1-4 Suggests: manpages-es, doc-linux-es, doc-debian-es Filename: pool/main/a/ayuda/ayuda_0.1-4_all.deb Size: 31710 MD5sum: 79c8ded94cce4b054ad883cb139500d7 Description: Help for spanish-speakers This package contains a help program called 'ayuda' useful for users that speak spanish, and are new to the world of Debian GNU/Linux. . The help provided covers many topics from administration to daily use. (Ruby backtrace follows.) The maintainer name is not valid UTF-8: $ isutf8 /var/lib/dpkg/available /var/lib/dpkg/available: line 17427, char 1, byte offset 22: invalid UTF-8 code
Wow, that's an interesting finding. There is no package ayuda in Debian (anymore; it was removed in 2005, according to https://packages.qa.debian.org/a/ayuda.html -- which also shows the encoding problems :)) Ok, so what are we doing now? While I would like dhelp to handle this situation a bit more gracefully, I suggest to downgrade the severity of the bug since it shouldn't affect anyone running packages contained in recent and upcoming Debian releases, and keeping it out of jessie for these cornercases seems a bit to strong for me. (Of course a fix would be best :)) Cheers, gregor
In terms of fixing dhelp, we could in theory catch the UTF-8 error, log a warning somehow, and continue on to the next package description. One missing package from the documentation index isn't the end of the world (and in my experience, dhelp doesn't index all my HTML documentation anyway.) However, the code in question is in ruby-debian, which is a separate library used by other packages. Might not be correct behavior for other users of the library?
Control: clone -1 -2 Control: reassign -2 ruby-debian Control: affects -2 dhelp Control: close -1 768127 0.6.21+nmu6 Right, cloning+reassigning to ruby-debian might make sense. Let's do this :) (And close the original bug since it does fix another problem.) Probably the severity in the cloned bug in ruby-debian shoule be lowered; the problem might not be present if a non-UTF-9 file is not opened as UTF-8 ... Cheers, gregor
gregor herrmann wrote: Thanks. I used "dpkg --update-avail" to get rid of that ancient package and now dhelp works as expected. (Thought dpkg was doing this automatically these days...) You're probably right; it's up to the ruby-debian maintainers. Thanks to both of you for your efforts.
Control: severity 773485 normal yes
Great! Already done by Antonio in the meantime, thanks. Thanks for your patience :) Cheers, gregor