#1088235 autopkgtest: autopkgtest-virt-qemu sometimes hangs when installing build dependencies

#1088235#5
Date:
2024-11-25 13:30:45 UTC
From:
To:
Dear Maintainer,

I'm observing that the podman package keeps failing with 'tmpfail' result:

https://ci.debian.net/packages/p/podman/testing/amd64/

Note that the package is configured to use qemu instead of debci.  This
is holding up migration to testing.

I've been able to reproduce the symptom locally. Using pystack, I was
able to determine where the process hangs:

(pystack) siretart@x1:/tmp $ pystack remote 32478
Traceback for thread 32481 (runcmd) [] (most recent call last):
    (Python) File "/usr/lib/python3.12/threading.py", line 1032, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/usr/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
        self.run()
    (Python) File "/usr/lib/python3.12/threading.py", line 1012, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/tmp/autopkgtest-qemu.ai0h585s/runcmd", line 37, in shovel
        time.sleep(0.01)

Traceback for thread 32480 (runcmd) [] (most recent call last):
    (Python) File "/usr/lib/python3.12/threading.py", line 1032, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/usr/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
        self.run()
    (Python) File "/usr/lib/python3.12/threading.py", line 1012, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/tmp/autopkgtest-qemu.ai0h585s/runcmd", line 37, in shovel
        time.sleep(0.01)

Traceback for thread 32478 (runcmd) [] (most recent call last):
    (Python) File "/tmp/autopkgtest-qemu.ai0h585s/runcmd", line 79, in <module>
        time.sleep(0.2)



Apparently, the runcmd script is looking for the 'exit' file to be
created.

This led me to looking at the job directory:

siretart@x1:/tmp/autopkgtest-qemu.tvv_adh3 $ find . -ls
     9699      0 drwxr-xr-x   3 siretart siretart      180 Nov 25 07:23 .
     9700      0 drwxrwxrwt   3 siretart siretart      120 Nov 25 07:24 ./shared
     9707      0 -rw-r--r--   1 siretart siretart        0 Nov 25 07:23 ./shared/done_shared
     9715      4 -rw-r--r--   1 siretart siretart        5 Nov 25 07:23 ./shared/normal_user
     9848      0 drwxr-xr-x   2 siretart siretart      120 Nov 25 07:24 ./shared/job.kdzgmbyv
     9849    288 -rw-rw-r--   1 siretart siretart   293351 Nov 25 07:24 ./shared/job.kdzgmbyv/stdout
     9850    440 -rw-rw-r--   1 siretart siretart   450097 Nov 25 07:24 ./shared/job.kdzgmbyv/stderr
     9851      0 -rwxrwxr-x   1 siretart siretart        0 Nov 25 07:23 ./shared/job.kdzgmbyv/stdin
     9852      4 -rw-rw-r--   1 siretart siretart        1 Nov 25 07:23 ./shared/job.kdzgmbyv/stdin_eof
     9855      4 -rw-r--r--   1 siretart siretart        2 Nov 25 07:24 ./shared/exit.tmp
     9702      0 srwxrwxr-x   1 siretart siretart        0 Nov 25 07:22 ./monitor
     9703      0 srwxrwxr-x   1 siretart siretart        0 Nov 25 07:22 ./hvc0
     9704      0 srwxrwxr-x   1 siretart siretart        0 Nov 25 07:22 ./hvc1
     9705      0 srwxrwxr-x   1 siretart siretart        0 Nov 25 07:22 ./ttyS0
     9706      0 srwxrwxr-x   1 siretart siretart        0 Nov 25 07:22 ./ttyS1
     9708      4 -rwxr-xr-x   1 siretart siretart     3252 Nov 25 07:23 ./runcmd


So as far as I can tell, the command terminated successfully and left
the 'exit.tmp' with the content of '0'.

Furthermore, moving the file `shared/exit.tmp` -> `./shared/job.kdzgmbyv/exit` allowed
autopkgtest to proceed with the tests and pass them successfully.


I am still wondering why this appears to happen only in some
conditions. Is it the amount of output caused by the large number of
build-depends? -- not sure.

#1088235#12
Date:
2024-11-26 11:23:01 UTC
From:
To:
Curiously, disabling the autopkgtest `builtin-tests` (which strictly
speaking doesn't even require qemu), makes the package pass in debci:

https://salsa.debian.org/go-team/packages/podman/-/commit/0d29ddc882b313e4a397826a9cbea7fa593f1f8c

What's so special about this particular test?

#1088235#17
Date:
2024-12-01 01:03:34 UTC
From:
To:
I've looked more into the issue, and found that reducing the number of
packages avoids this issue.
I believe it is possible that the number of dependency is hitting some
limit, possibly with how
long a command-line in /bin/sh can be? -- not sure.

For podman, I was able to identify some redundant build-dependencies
which seems to allow the
tests to pass again.