Dear Maintainer,

Jigdo version is now 0.8.0.1 for bullseye testing and 0.7.3.5 for buster
stable.

We can use jigdo for downloading ISOs from the servers and it works fine for
both versions.

The problem with version 0.8.0.1 is when we want to make ISOs from a previous
(bigger) image.

For a precise example we have a big BD ISO image :
debian-10.7.0-amd64-BD-1.iso, mounted on /media/user/ (or /mnt/cdrom/).
From that big one, we want to make up the 5 first DVD ISOs
debian-10.7.0-amd64-DVD-1.iso to debian-10.7.0-amd64-DVD-5.iso.

With Jigdo 0.7.3.5 everything works perfectly fine. Jigdo reads the big mounted
BD-1 image. It writes a "jigdo-file-cache.db" cache info data file at the first
reading of the BD-1 image, so that it does not need to re-read BD-1 next times.
So the making up of DVD-2 to DVD-5 is very quick.
Of course Jigdo finds everything it needs in BD-1 to make DVD-1 to DVD-5,
without having to connect to the servers (Internet). And that is the point
(what we want), to be able to do that, in case we don't have access to
Internet.

But with Jigdo 0.8.0.1 :
- First jigdo does not even seem to re-use the jigdo-file-cache.db (by the way
twice bigger with 0.8.0.1 than with 0.7.3.5) it has created the first time. So
Jigdo has to read and analyse BD-1 all over again every time. And it is very,
but veeeery long, as it seems the checking method (for every *.deb file) is
different with Jigdo 0.8.0.1.
- And second, jigdo fails to recognize some existing files in BD-1. For
instance, DVD-1 and 2 are fine but for DVD-3 Jigdo fails to fetch
/pool/contrib/b/b43-fwcutter/b43-fwcutter_019-4+deb10u1_amd64.deb in BD-1 and
has to fetch it on the servers. For DVD-4 Jigdo fails to fetch
/pool/contrib/a/alsa-tools/alsa-firmware-loaders_1.1.7-1_amd64.deb. I did not
try DVD-5 (so irritatingly slow!) but from a previous try that I made two
months ago, I think I remember there were more than one *.deb file missing.

So these little bugs are very saddening because Jigdo is such a beautiful and
useful tool. This little guy can really do a great job and I wish it would be
on DVD-1, regardless of actual usage statistics.

Cordially,
Pascal.

Dear Maintainer,

I have been worrying about those bugs and sensing the trouble would probably be
with "jigdo-file-cache.db", I made new tests beginning with the last of the
five first DVDs.

The version used for jigdo-file is always 0.8.0-1 with debian bullseye.

So I mounted "debian-10.7.0-amd64-BD-1.iso" under "/media/user/" and then I
typed :

$ jigdo-lite debian-10.7.0-amd64-DVD-5.jigdo

I chose "Files to scan : /media/user/" when jigdo asked me. The scanning for
the whole BD takes 35mn on my computer. This slowness is not surprising as I
understood that with jigdo 0.8.0-1 checksums for each *.deb file were made with
SHA256. With jigdo 0.7.3-5 it was with MD5, much quicker.

The surprising thing is that all worked fine. Jigdo created "jigdo-file-
cache.db" for the first time and it could find in
"debian-10.7.0-amd64-BD-1.iso" everything it needed to make up
"debian-10.7.0-amd64-DVD-5.iso".

Following my intuition I removed "jigdo-file-cache.db", or rather displaced it
into another directory. Then I remade a try with
"debian-10.7.0-amd64-DVD-4.jigdo" and then "debian-10.7.0-amd64-DVD-3.jigdo",
each time removing the preceding "jigdo-file-cache.db" before lauching jigdo-
lite.

And again it worked fine. Jigdo could find all it needed from
"debian-10.7.0-amd64-BD-1.iso" to make up DVD-4 and DVD-3 ISO images.

So I would say that the trouble seems to come from the overwriting of "jigdo-
file-cache.db". Comparing each "jigdo-file-cache.db" from the preceding one
with the "cmp -l" command line, I remarked that they were very different from
one another. The one for DVD-3 is even a little bigger :

DVD 5 : Found 1814 files  "jigdo-file-cache.db" : 16 568 320 bytes
DVD 4 : Found 2759 files  "jigdo-file-cache.db" : 16 568 320 bytes
DVD 3 : Found 2480 files  "jigdo-file-cache.db" : 16 576 512 bytes

After overwriting "jigdo-file-cache.db" jigdo seems to be induced into errors.
As if something went wrong in the process of overwriting versus writing for the
first time. Since this cache file is obviously not re-used correctly, jigdo
would work well if "jigdo-file-cache.db" was automatically erased at each jigdo
launching.
But "jigdo-file-cache.db" should be re-used in all cases (and particularly in
our case) to save time. Or else jigdo has to re-read the BD-1 image (or any
other ISO) all over again, and again.

With Jigdo 0.7.3-5, I have a backup of "jigdo-file-cache.db" (only to be used
with that version of jigdo because of different *.deb checksums). With that
cache file, Jigdo 0.7.3-5 does not even check the BD-1 ISO content. It produces
DVD-1 to 5 ISOs directly, and at the end it says the DVD-1 to 5 images (MD5
check) are correct.

Cordially,
Pascal.

Hello,

I just wanted to report that I have noticed the same problem.

I usually mount all old DVD images in one parent directory. On update
jigdo just scanned this whole directory for the first new DVD and
created a jigdo-file-cache.db file. For all other following new DVDs
jigdo-file-cache.db was used and no additional scan of the directory was
done.

After upgrading to bullseye (and using jigdo-file 0.8.0-1) I see the
same behaviour as reported in this bug. A complete rescan of the whole
directory is done for EVERY DVD and jigdo-file-cache.db seems to be
ignored. This is very time consuming compared to the old behaviour.

I just downgraded to the buster version 0.7.3-5+deb10u1 and the
behaviour is like before and there is only one scan for the first new
DVD and the created jigdo-file-cache.db is then used for the following
new ones.

(Testing was done with the 11.2.0 jigdo images)

So it seems a bug was introduced which corrupts (?) jigdo-file-cache.db
for directories with a (very) large number of files.

If needed I can do some debugging with my setup.

Best regards,

Mark

I've also observed this problem, that Jigdo ignores the cache and re-
scans all the files every time.  I did some debugging and tracked it
down to a one-line mistake that's easy to fix:

diff --git a/src/scan.cc b/src/scan.cc
index 9ce598e..b031680 100644
--- a/src/scan.cc
+++ b/src/scan.cc
@@ -109,7 +109,7 @@ size_t FilePart::unserializeCacheEntry(const Ubyte* data, size_t dataSize,
   Paranoid(serialSizeOf(md5Sum) == 16);
   Paranoid(serialSizeOf(sha256Sum) == 32);
   // All blocks of file present?
-  if (blocks == MD5sums.size() + SHA256sums.size()) {
+  if (blocks == MD5sums.size() && blocks == SHA256sums.size()) {
     setFlag(MD_VALID);
     data = unserialize(md5Sum, data);
     data = unserialize(sha256Sum, data);

A cache entry contains hashes of individual 1k blocks of the file, and
this code is checking that the entry contains the expected number of
them.  The number of blocks is simply the file's size divided by the
block size (1k), rounded up, and the cache entry should contain that
many MD5 block hashes, followed by the same number of SHA256 block
hashes.  The "blocks" variable is the length of *each* of the two block
lists in the entry (since they're always equal-length), not their sum.
So, the bug is that the deserialization code thought the cached hash
data was invalid, because it expected the wrong number of blocks.

Using this patch, I upgraded my set of Debian 11.3 DVD images to 11.4,
and it behaved as expected: scanned all the input files (the union of
all the 11.3 DVD contents) in the first run, to produce the first
output file, and then re-used the cached hashes in later runs to
produce the other files.

(The rest of the function's code is a little confusing, btw: if the
number of blocks doesn't match, it clears the MD_VALID flag and ignores
the whole-file hashes, but then proceeds to deserialize the block
hashes anyway, in a way that'll go off the end of the MD5sums and
SHA256sums vectors if they're not long enough.  But there's a debug
assertion earlier that MD5sums and SHA256sums are sized correctly based
on the file size, and a cache entry *should* always have the correct
number of block hashes based on the file size, so I don't think it's a
problem in practice.  Brittle code, though.)

I can confirm that the problem is still present in the latest versions of jigdo-file.

The patch from Mike fixed the issue for me. I was able to build my own debian package with it.

Any chance we can get the patch added to the Debian package?

Please let me know if I should add any more information or contribute in some way. :-)

I can confirm that the patch works (applied on jigdo 0.8.2).
I'm surprised that it still hasn't made it into the jigdo git
repository?!?!?

On Sun, 10 Jul 2022 21:53:34 -0400 Mike Paul <mike@wyzardry.net> wrote:
 >
 > diff --git a/src/scan.cc b/src/scan.cc
 > index 9ce598e..b031680 100644
 > --- a/src/scan.cc
 > +++ b/src/scan.cc
 > @@ -109,7 +109,7 @@ size_t FilePart::unserializeCacheEntry(const
Ubyte* data, size_t dataSize,
 > Paranoid(serialSizeOf(md5Sum) == 16);
 > Paranoid(serialSizeOf(sha256Sum) == 32);
 > // All blocks of file present?
 > - if (blocks == MD5sums.size() + SHA256sums.size()) {
 > + if (blocks == MD5sums.size() && blocks == SHA256sums.size()) {
 > setFlag(MD_VALID);
 > data = unserialize(md5Sum, data);
 > data = unserialize(sha256Sum, data);