#837681 diffoscope: 'ERROR guestfs can't be launched' when trying to compare to qubes ISOs

Package:
diffoscope
Source:
diffoscope
Submitter:
Holger Levsen
Date:
2018-12-15 15:05:20 UTC
Severity:
normal
#837681#5
Date:
2016-09-13 13:35:58 UTC
From:
To:
Hi,

so I build an Qubes ISO, twice and ran diffoscope against it:

holger@profitbricks-build3-amd64:~$ sudo schroot --directory /tmp -c source:jenkins-reproducible-unstable-diffoscope diffoscope -- --html /tmp/q.html --text /tmp/q.txt /tmp/q1 /tmp/q2
1473773034.873399    ERROR guestfs can't be launched
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/fsimage.py", line 43, in open_archive
    self.g.launch()
  File "/usr/lib/python3/dist-packages/guestfs.py", line 5398, in launch
    r = libguestfsmod.launch (self._o)
RuntimeError: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
1473773034.929533    ERROR If memory is too tight for 512 MiB, try running with LIBGUESTFS_MEMSIZE=256 or lower.
1473773035.369462    ERROR guestfs can't be launched
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/fsimage.py", line 43, in open_archive
    self.g.launch()
  File "/usr/lib/python3/dist-packages/guestfs.py", line 5398, in launch
    r = libguestfsmod.launch (self._o)
RuntimeError: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
1473773035.373510    ERROR If memory is too tight for 512 MiB, try running with LIBGUESTFS_MEMSIZE=256 or lower.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 246, in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 217, in run_diffoscope
    parsed_args.path1, parsed_args.path2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 124, in compare_root_paths
    return compare_directories(path1, path2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 105, in compare_directories
    return FilesystemDirectory(path1).compare(FilesystemDirectory(path2))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 158, in compare
    my_file, other_file, source=name)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 144, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 213, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 183, in _compare_using_details
    details.extend(filter(None, self.as_container.compare(other.as_container)))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 147, in compare_commented_files
    difference = compare_files(file1, file2, source=source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 144, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 213, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 183, in _compare_using_details
    details.extend(filter(None, self.as_container.compare(other.as_container)))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 147, in compare_commented_files
    difference = compare_files(file1, file2, source=source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 144, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 213, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 183, in _compare_using_details
    details.extend(filter(None, self.as_container.compare(other.as_container)))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 147, in compare_commented_files
    difference = compare_files(file1, file2, source=source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 133, in compare_files
    if file1.has_same_content_as(file2):
  File "/usr/lib/python3/dist-packages/diffoscope/__init__.py", line 64, in tool_check
    return original_function(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 195, in has_same_content_as
    my_size = os.path.getsize(self.path)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils.py", line 246, in path
    self._path = self.container.extract(self._name, self._temp_dir.name)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/fsimage.py", line 68, in extract
    self.g.tar_out("/", dest_path)
  File "/usr/lib/python3/dist-packages/guestfs.py", line 9307, in tar_out
    r = libguestfsmod.tar_out (self._o, directory, tarfile, compress, numericowner, excludes, xattrs, selinux, acls)
RuntimeError: tar_out: call launch before using this function
(in guestfish, don't forget to use the 'run' command)
Exception ignored in: <bound method Archive.__del__ of <diffoscope.comparators.fsimage.FsImageContainer object at 0x7f2109003908>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils.py", line 279, in __del__
    self.close_archive()
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/fsimage.py", line 56, in close_archive
    self.g.umount_all()
  File "/usr/lib/python3/dist-packages/guestfs.py", line 9543, in umount_all
    r = libguestfsmod.umount_all (self._o)
RuntimeError: umount_all: call launch before using this function
(in guestfish, don't forget to use the 'run' command)
Exception ignored in: <bound method Archive.__del__ of <diffoscope.comparators.fsimage.FsImageContainer object at 0x7f2108ffb3c8>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils.py", line 279, in __del__
    self.close_archive()
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/fsimage.py", line 56, in close_archive
    self.g.umount_all()
  File "/usr/lib/python3/dist-packages/guestfs.py", line 9543, in umount_all
    r = libguestfsmod.umount_all (self._o)
RuntimeError: umount_all: call launch before using this function
(in guestfish, don't forget to use the 'run' command)
holger@profitbricks-build3-amd64:~$ ls /tmp/q1 /tmp/q2
/tmp/q1:
Qubes-DVD-x86_64-20160913.iso

/tmp/q2:
Qubes-DVD-x86_64-20160913.iso


Am I missing some dependencies or what does that mean and what should I do now?
#837681#10
Date:
2016-09-13 14:08:00 UTC
From:
To:
Since guestfs works by running a modified kernel in an VM to parse the
file system, I think it fails to start the VM (nested virt disabled,
OOM, ...).

So I think you should first try if guestfs works at all (without
diffoscope) and/or enable the debug loggin like suggested in the error
message.
#837681#13
Date:
2016-09-13 16:22:29 UTC
From:
To:
I wanted to quickly try myself, but you already deleted the files.
Could you please save them somewhere (I suppose they are too big to be
attached to the bug?)

guestfs is not called directly, but through
ISTR the consensuos was in diffoscope printing an
error but continuing nonetheless (proposing an e.g. binary diff).

And reading the code suggest that it should do (it's catching
RuntimeError and returning back safely), but it is doing that only in
open_archive, not in e.g. close_archive which seems to be where the
actual crash is:

If you want to compare it right now you might want to uninstall
python3-guestfs so it falls back to binary comparison, but that's
probably not that much helpful..
#837681#18
Date:
2016-09-13 17:41:15 UTC
From:
To:
they are in /home/holger/q(1|2) on pb3, please help yourself! :-)
[...]

thanks but no thanks.

:)
#837681#23
Date:
2016-09-13 17:43:23 UTC
From:
To:
a.) I should definitly retry with debug logging…
b.) guestfs works on jenkins.d.n which is the same hardware (kvm…) and
the same software (jessie)
#837681#28
Date:
2016-09-15 09:26:38 UTC
From:
To:
just now I enabled debugging like this:

the result is attached.
#837681#33
Date:
2016-09-15 09:33:10 UTC
From:
To:
I've now also tried with LIBGUESTFS_MEMSIZE=2048 just in case it needs
more memory. the result is again attached :)
#837681#38
Date:
2016-09-15 14:02:15 UTC
From:
To:
Hi,

these qubes ISO image files are (temporarily) now also available for download
at https://jenkins.debian.net/userContent/qubes/
#837681#43
Date:
2016-09-15 14:58:00 UTC
From:
To:
Holger Levsen:

It fails to find a kernel for the VM image it creates on the fly:

diffoscope-qubes-debug:

On what kind of VM is this run, and what kernels are installed?
#837681#48
Date:
2016-09-15 16:28:38 UTC
From:
To:
ah, wow.

3.16.0-4-amd64 is the kernel (so jessie standard), and the host this is
running on is a profitbricks VM, so running on kvm.
#837681#53
Date:
2016-09-15 16:38:00 UTC
From:
To:
Holger Levsen:

I assume you have this kernel only installed on the VM not the chroot.
So try installing linux-image-amd64 _inside_ the chroot.
#837681#58
Date:
2016-09-15 17:36:50 UTC
From:
To:
the problem is, that this is a sid chroot, so linux-image would have
version 4.7.2, while the host has 3.16…
#837681#63
Date:
2016-09-15 17:40:00 UTC
From:
To:
Holger Levsen:

I don't think this mismatch is a problem.
#837681#68
Date:
2017-07-27 11:09:42 UTC
From:
To:
The attached patch should help the comparison to continue instead of crashing.

A few points are still open/unclear for me:
* When FsImageContainer.open_archive returns None, the rest of the code still thinks there are files to extract
  and calls get_member_names? Is this OK? Should i behave more like if an exception was raised?
* Is it OK that the extracting error shows up in the HTML output?
* Should we create a test for missing guestfs and guestfs launch failed?
* Should we tell the user about the possibility that unreadable kernels under /boot can cause this problem?
  And should this information be included in the HTML output or only in the log outupt
* Is it possible to have a mechanism like @tool_required or raise RequiredToolNotFound for missing python modules like "guestfs"
#837681#73
Date:
2018-12-15 15:02:50 UTC
From:
To:
forwarded 837681 https://salsa.debian.org/reproducible-builds/diffoscope/issues/3
thanks

I've forwarded this upstream here:

https://salsa.debian.org/reproducible-builds/diffoscope/issues/3


Regards,