After noticing that cindex silently skipped some files in my codebase, I
found that this happened to files that contained non-ASCII characters
that were encoded as Latin-1.

Here's a quick reproducer:

$ pwd
/home/bengen/tmp/cs-test
$ ls -l
total 12
-rw-r--r--. 1 bengen bengen  8 Aug 16 09:33 ascii.txt
-rw-r--r--. 1 bengen bengen  8 Aug 16 09:33 latin1.txt
-rw-r--r--. 1 bengen bengen 11 Aug 16 09:33 utf-8.txt
$ rm ~/.csearchindex
$ hd ascii.txt
00000000  61 6f 75 20 66 6f 6f 0a                           |aou foo.|
00000008
$ hd latin1.txt
00000000  e4 f6 fc 20 66 6f 6f 0a                           |... foo.|
00000008
$ hd utf-8.txt
00000000  c3 a4 c3 b6 c3 bc 20 66  6f 6f 0a                 |...... foo.|
0000000b
$ cindex `pwd`
2013/08/16 09:37:33 index /home/bengen/tmp/cs-test
2013/08/16 09:37:33 flush index
2013/08/16 09:37:33 merge 0 files + mem
2013/08/16 09:37:33 19 data bytes, 371 index bytes
2013/08/16 09:37:33 done
$ csearch foo
/home/bengen/tmp/cs-test/ascii.txt:aou foo
/home/bengen/tmp/cs-test/utf-8.txt:äöü foo


There's a check for invalid UTF-8 sequences in the indexWriter.Add()
function to weed out binary data. If a file contains an invalid UTF-8
sequence, it is skipped entirely.

This behavior is surprising. It only makes sense in a controlled
environment where you can guarantee that everything you want to index is
valid UTF-8 (or even ASCII). This was certainly not a valid assumption
in the case where I wanted to index a rather large internal collection
of source code that contained the occasional German message or comment.

The first patch simply removes this check. It has worked well for me for
several months (before the codesearch package appeared in Debian). I
haven't really checked index sizes, but I can't imagine this patch doing
much harm because there are other checks that are there to keep the
index from being filled up with arbitrary trigrams.

Please consider also adding the second patch to enable logging of files
that are not added to the index for whatever reason.

Cheers,
-Hilko

Hi Hilko,

Hilko Bengen <bengen@debian.org> writes:
The issue I have with that is that it will break the assumption that
everything which is in the index is findable. Given that codesearch is
used with UTF-8 search terms, you would not be able to find “Grüße” when
searching for it.
I agree that the second patch makes more sense, even though I am not
entirely sure whether a separate flag would be more appropriate
(nitpick).

Can you please send the second patch upstream?

Hi Michael,

Right. To be honest, it never occured to me to search for non-ASCII
content in source code. :-)

Codesearch has been very useful in answering questions such as "Are we
sure that none of our code uses function X any more ... can we just get
rid of X in the next release?". So to me, having an index that contains
every source text file been more important than worrying about not being
able to find Latin1-encoded strings.

BTW, I just tried passing 'äöü' as a Latin1-encoded string (bytes e4 f6
fc) to csearch. This led to regexp/syntax failing with an "invalid
UTF-8" error, so this does not work, even if the character encoding of
the search term matches that of the index.

A "proper" solution would probably involve guessing the character set of
a text file and convert it if necessary before indexing. Meh.

How are you dealing with this in codesearch.debian.net?

Will do.

Cheers,
-Hilko

Hi Hilko,

Hilko Bengen <bengen@debian.org> writes:
Yep, that is what I suspected. Only UTF-8 is supported.
I just assume everything is UTF-8. If it is not, and actually contains
non-ASCII characters, it needs to be converted to UTF-8. I mean,
common. This is 2013. Just convert it the files already! :-)

Hi,

your bug showed up as Bug of the Day[1] and I took the freedom to
migrate the packaging from Alioth collab-maint to debian/ on Salsa.
I also upgraded to the latest upstream version.  Since the bug log
suggested to forward your patch to upstream I would like you to
check the just uploaded version 1.2 whether the problem is actually
solved.  If so, please close this bug.  In case it is not fixed
yet I would be happy if you could open an upstream issue and tag
this bug uptream as well as providing Forwarded info here.

Thanks a lot for your cooperation
    Andreas.


[1] https://salsa.debian.org/tille/tiny_qa_tools/-/wikis/Tiny-QA-tasks

* Andreas Tille:

The problem has not been solved – and there are multiple issues listed
in https://github.com/google/codesearch/issues that look similar enough
that there's no point in adding a new issue:

- https://github.com/google/codesearch/issues/26
- https://github.com/google/codesearch/issues/37
- https://github.com/google/codesearch/issues/80

Cheers,
-Hilko

Hi Hilko,

Am Sun, Aug 18, 2024 at 11:56:01PM +0200 schrieb Hilko Bengen:

Thanks for confirming
    Andreas.