Heya,

This is sort of similar to #779559. Let me just explain the problem
briefly-

In Ubuntu we dispatch tests to fresh cloud instances. Each instance has
a ~3 minute boot and teardown overhead, which is quite significant when
there are many triggered tests - for example glibc uploads trigger
thousands of tests.

Full machine virtualisation is not actually required for the majority of
tests. autopkgtests has an lxd (or schroot, etc) backend that I would
like to use as they have a far smaller per-test overhead. The only
problem is that some tests have the 'isolation-machine' restriction
which means that you do require an entire machine - lxd won't do. I
don't currently have any way when dispatching the tests to know if this
is the case, short of downloading the source package and looking in
debian/tests/control. Exposing the restrictions in the index will let me
send tests to the right autopkgtest backend straight off.

There's a patch attached. It works for me, but review appreciated. I
didn't know if 'deps_iterate' was right to use for Restrictions - it's
dependency like in format but obviously not actually a dependency field.

Cheers,

s/Recommends/Restrictions/, obviously.

Hi!

Bug #847926 in package dpkg reported by you has been fixed in
the dpkg/dpkg.git Git repository. You can see the changelog below, and
you can check the diff of the fix at:

https://anonscm.debian.org/cgit/dpkg/dpkg.git/diff/?id=9899bdc
    dpkg-source: Generate Testsuite-Restrictions fields from test restrictions

    This information is currently only available in the Restrictions field in
    the debian/tests/control file.

    When dispatching tests, it might be inconvenient to have to download and
    unpack the source package beforehand. Let's expose this via the .dsc in
    a similar way we do for the Testsuite-Triggers field.

    Closes: #847926
    Based-on-patch-by: Iain Lane <laney@debian.org>

diff --git a/debian/changelog b/debian/changelog
index 6e83226..6bc2eaa 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -12,6 +12,9 @@ dpkg (1.18.19) UNRELEASED; urgency=medium
   * Refactor update-alternatives pathname existence check into a new function.
   * Avoid useless repeated lstat()s in update-alternatives.
   * Only check for debian/tests/control file once in dpkg-source.
+  * Generate Testsuite-Restrictions fields from the test restrictions in
+    dpkg-source into .dsc files. Closes: #847926
+    Based on a patch by Iain Lane <laney@debian.org>.
   * Portability:
     - On GNU/Hurd try to use the new process executable name attribute from
       libps, to properly match on start-stop-daemon --exec.

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 847926@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Fri, 27 Jan 2017 05:43:36 +0100
Source: dpkg
Binary: dpkg libdpkg-dev dpkg-dev libdpkg-perl dselect
Architecture: source
Version: 1.18.19
Distribution: unstable
Urgency: medium
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Description:
 dpkg       - Debian package management system
 dpkg-dev   - Debian package development tools
 dselect    - Debian package management front-end
 libdpkg-dev - Debian package management static library
 libdpkg-perl - Dpkg perl modules
Closes: 831524 843925 846164 847926 848705 849081 849913 851441 851889 851891
Changes:
 dpkg (1.18.19) unstable; urgency=medium
 .
   [ Guillem Jover ]
   * Stop emitting Built-For-Profiles from dpkg-gencontrol. The information
     is already provided in .buildinfo files, and including it in the binary
     packages makes them unreproducible even when the profile used would not
     alter its contents. Closes: #831524
   * Do not allow empty epochs and revisions in versions. When there's at
     least one colon or one dash, we should expect epoch and revision numbers.
   * Always set SOURCE_DATE_EPOCH in dpkg-buildpackage and dpkg-source. Use
     the current date if the changelog does not have one. Closes: #849081
   * Refactor update-alternatives pathname existence check into a new function.
   * Avoid useless repeated lstat()s in update-alternatives.
   * Only check for debian/tests/control file once in dpkg-source.
   * Generate Testsuite-Restrictions fields from the test restrictions in
     dpkg-source into .dsc files. Closes: #847926
     Based on a patch by Iain Lane <laney@debian.org>.
   * Improve the ELF ABI mismatch detector in dpkg-shlibdeps, by parsing the
     ELF header ourselves. While still not perfect (things like linux-i386 and
     hurd-i386 will still match), it will filter lots of previously matching
     objects that should have been ignored, and will work even when objdump
     does not know about the specific object details. Closes: #849913
   * Add initial support for DEB_BUILD_OPTIONS to dpkg-genbuildinfo. This will
     make it possible to enable or disable specific features that should be
     recorded in the .buildinfo file. For now only “all” and “path” are
     supported. Closes: #848705
   * Add again the architecture from the filename to .changes files for any
     artifact with one. This reverts the change introduced in dpkg 1.18.11.
   * Fold the filtering and checksumming of files to distribute in a .changes
     file in dpkg-genchanges into the initial loop. This way we do not include
     architectures for artifacts we are not going to distribute, and do not
     unnecessarily recompute the checksums for artifacts like the sources.
   * Do not compute the architecture list twice in dpkg-genchanges.
   * Include .buildinfo files also for source-only uploads in dpkg-genchanges.
     Closes: #846164
   * Fix check for expected number of binary artifacts in dpkg-genchanges, to
     only take into account the artifacts that we are distributing.
   * Fix parsing of Pre-Depends and Depends in dpkg-genbuildinfo, so that
     the code parses both and not just the first to appear in the stanza.
     Based on a patch by Johannes Schauer <josch@debian.org>.
   * Add support for signed .buildinfo files to dpkg-buildpackage. Add new
     -ui and --unsigned-buildinfo options. Closes: #843925
   * Portability:
     - On GNU/Hurd try to use the new process executable name attribute from
       libps, to properly match on start-stop-daemon --exec.
   * Perl modules:
     - Fix Debian architecture wildcard parsing so that matching four-tuple
       matchings work. Missed in dpkg 1.18.11.
       Reported by Julian Andres Klode <jak@debian.org>.
     - Add new import tags for Dpkg::Arch.
     - Abort on EOF in patch name prompt in Dpkg::Source::Package::V2,
       instead of getting into an infinite loop. Closes: #851441
     - Call anonymous subs via -> operator instead of casting with &, and fix
       bogus POD documentation to match the code.
     - Add new Auto-Built-Package field to Dpkg::Control::Fields.
     - Add a new debug() reporting function, and switch code to use it.
     - Add new Dpkg::BuildOption parse_features() method refactored from
       Dpkg::Vendor::Debian.
   * Documentation:
     - Cleanup software requirements in README.
     - Move control member file references from dpkg(1) to deb(5).
     - Fix typos in docs and code comments.
     - Document Auto-Built-Package field in deb-control(5).
   * Build system:
     - Disable disk pre-allocation by default, but let the builder re-enable
       it via a new configure option. This has been causing major performance
       issues on "modern" filesystems.
   * Packaging:
     - Add debsig-verify to dpkg Suggests. The code optionally supports this
       specific signed .deb verification program.
       Prompted by Stuart Prescott <stuart@debian.org>.
   * Test suite:
     - Generate and check all currently possible architecture wildcards.
     - Correctly iterate over all default and passed .dsc template substvars.
 .
   [ Updated programs translations ]
   * Dutch (Frans Spiesschaert). Closes: #851889
   * German (Sven Joachim).
 .
   [ Updated scripts translations ]
   * German (Helge Kreutzmann).
 .
   [ Updated man pages translations ]
   * Dutch (Frans Spiesschaer). Closes: #851891
   * German (Helge Kreutzmann).
Checksums-Sha1:
 b095dc40f8f1a76a1f0cafe3a4c33b9527cead67 2032 dpkg_1.18.19.dsc
 f8ec626d3503e0c8e6dfff5d11c95104811db9db 4516116 dpkg_1.18.19.tar.xz
 a24f616884b03619e07017518053202651875d5a 7301 dpkg_1.18.19_amd64.buildinfo
Checksums-Sha256:
 8b46dcac0a09b0c9ca9a462c1b23b2ece9ec5d5c5d9a4a1aa91406d83de7be78 2032 dpkg_1.18.19.dsc
 67c8b4d580497991892ecd6745267ed4be9f65d2cc842b75b758f999c6ee7bbb 4516116 dpkg_1.18.19.tar.xz
 683b0c34af65ea0ac7ded8e63395d937dd9494a97b7c317640def47a7d30c1e4 7301 dpkg_1.18.19_amd64.buildinfo
Files:
 b41ba9c5d6a34aba330ffec62a2f0cae 2032 admin required dpkg_1.18.19.dsc
 231a66f09747e1b77b236ff48cd71a9e 4516116 admin required dpkg_1.18.19.tar.xz
 d0fd205f0f98b27401700522514e1e37 7301 admin required dpkg_1.18.19_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEETz509DYFDBD1aWV0uXK/PqSuV6MFAliK348ACgkQuXK/PqSu
V6M6Dg//R5i0tqcZStNnQTKmwCfK10A49fS87qTHRydeuqpEZGC2jt8b41LEmmqm
p4dNInnpH3LJPpJ1+SWFWP91vX6AwrRbW1a+5KcSYrsfJQqPzi2TF77uofcPs5Dd
Uf9C1TQCU6TjV9Pb0JA4D1e3QLjIFX2QFyBVQbzqWb1XsLnyvXZ2l2Etj8f2EO97
OXVEDdTNy1s9biY//PYW2d+e+nX7IG+eNXwI5qfy7PXC05bvhUPcxEWA2LSMM+WB
0zjT0QkUFG+Nxf6Fm6FT62N4ms67xbCi4nvYg2PKMbXAGDmhwJYm+JAioFhQFmUv
0S1Y6X1v2RDYuhI/HJfTvJ3SreClvtgGKQR4H231axuv0+ZXNcXe17DXI0z5qe2l
vaUrmafPHcBPxtkl9pNzBJPYUvmev8jKRp2XOqxlwnDWoFkeupcMxjItyK3bTyfN
4alLG4H66BmHGVxyNSTOoSsqoz1Ew+HlhUX0TkacURqigKICLEThgwzVdmqbIm1d
BdI8UVB5trXAmG8nyEoMKkaGkh6nI5a5TKnKsl7iiXJdg6nhaqU8J/MXLeorNe8a
bzWFsp4SbfZuiQuG2OTXBevjmwGaLTa9VXllFtuf1CeiiVxn706lupN/eTieRtZH
VMec0ITJ8U6Hx2mxqmASORuSUfyTJ+AFrxThfkm7Cxc+/AsKmho=
=sLUv
-----END PGP SIGNATURE-----

From the patch from Iain Lane:

 +This field declares the comma-separated union of all test restrictions
 +(\fBRestrictions\fP fields in \fIdebian/tests/control\fP file).

This is quite awkward to use correctly, and is a hazard to correct
implementation.  For example, dgit's debian/tests/control contains
several tests marked with Dgit-specific Restrictions with
x-dgit-... names.  The result is that with 1.18.10ubuntu1, we would
see this in dgit's .dsc:

 Testsuite-Restrictions: x-dgit-intree-only x-dgit-git-only x-dgit-schroot-build

If not interpreted very carefully, this would give a test suite runner
the erroneous impression that none of the tests can be run.

Also, Iain's stated use case in #847926 does not require anything new
in the .dsc and hence Sources.gz.  All that is required to get the
full information (debian/tests/control) is to extract the source
package.  That extraction of the source package has to be done anyway
no matter how the tests will be run.

If we do need more information in Sources.gz then maybe the set of
combinations of restrictions ought to be listed.  But then the
features might be useful too.

I'm sorry that I'm reporting this bug now, due to happening to see the
message on debian-dpkg.  But really I think:

 * Changes to the .dsc format and hence to the Sources format should
   be discussed more widely than a bug on debian-dpkg.

 * Changes to the handling of autopkgtest (DEP-8) metadata should be
   discussed on autopkgtest-devel (or other DEP-8 related places).

Thanks,
Ian.

Hi!

Right, I see what you mean.

OTOH, the same could be said for several of the fields in the .dsc, such
as Build-Depends, but we still list them because it's more convenient to
not have to extract the source beforehand.

TBH, I hesitated a bit before adding this, because this percolates
into many other places. But considered that, while the information
could be retrieved in other ways, it made life easier for test
runners. And we can always remove it if it ends up being
unnecessary/undesirable.

But I certainly agree that this should have probably been discussed
more widely, which is something I overlooked, sorry about that. And
agree completely with your two points above. So I'll be adding an
entry to the FAQ detailing the process to add new information to
the .dsc file (and probably to the .changes and other interchange
formats).

Thanks,
Guillem

I've now updated the FAQ to include processes for field inclusions to
.dsc and .changes files:

  <https://wiki.debian.org/Teams/Dpkg/FAQ?action=diff&rev2=42&rev1=41>

Thanks,
Guillem

Guillem Jover writes ("Re: Bug#852820: Testsuite-Restrictions field is hard to use"):

Can you please add something to the documentation, sayinng that this
field MUST NOT be used to determine whether a package has any tests
that can be run ?  (Because it cannot be correctly so used.)

Build-Depends is not a (perverse) atttempt at a summary of some other
pieces of metadata.

You have permanently assigned the name `Testsuite-Restrictions' for
this unnatural meaning, and you and Iain intend for Iain's software to
start relying on it (so withdrawing it would be troublesomw).
Changing published metadata formats is not so easy, I'm afraid.

Thanks,
Ian.

I don't, or not really - see below. I plan on using this field
externally to choose between a couple of available backends to dispatch
to when constructing autopkgtest invocations - ssh (nova) or something
less heavyweight like lxd. In this case unknown or uninteresting values
will simply be ignored. I suppose you're worried about somebody
intersecting the field with the supported restrictions of particular
backends or something like that. That's not what I plan to do, but the
README says:

So it would seem to me (unless I'm missing something in there like an
exception for x-prefixed items that I can't see ATM) that this would be
legal and as a result I was not really expecting arbitrary values to be
present. Again, I'm not going to be doing that, so it's not a problem
now and this is apparently not currently implemented in autopkgtest
itself either, since dgit's tests are being run.

Exactly that. dpkg is constructing the information from the source
package, so it's always possible for somebody else to do the same work.
It's just much more convenient for it to be in Sources already.

It's definitely going to be necessary and desirable for me.

Yes, I concede that I should have posted on the list. I discussed with
Martin first, but that is not enough. Sorry about that.

Cheers,

Iain Lane writes ("Re: Bug#852820: Testsuite-Restrictions field is hard to use"):

Yes, I understand that.  But I think this field is too tailored to
your specific use case and too ripe for misuse.
those restrictions are not run by Debian's CI.  They can be run in
ad-hoc situations of various kinds.

Declaring test restrictions, using a private bit of the namespace, in
this way, seems like an appropriate way to achieve that effect.

For the avoidance of doubt: I'm not saying that your program would
mishandle correct packages.  (Is it happy with dgit's test suite,
despite the presence of some un-runnable tests?)

But as I say this field seems to invite people to use it for deciding
whether the package is testable, but it is not useable for that
purpose.

To put it another way: why not publish the intersection of the
restrictions ?  Why not also publish the union or intersection of the
features ?  As I wrote:

ISTM that your real problem is that you don't have an efficient way to
access debian/tests/control.  Perhaps the right answer is to provide
an additional side channel for this information, rather than burdening
the Sources file.  Many many, non-testing-related, programs need to
read Sources.  I think the test metadata is rather too rich to express
compactly.

Thanks.

Ian.

Hi!

This got reverted, let's reopen the bug so that we can discuss how
to go forward.

Thanks,
Guillem

I found this bug because we have a similar design issue in Debusine
(buried in a messy discussion about a tangentially-related problem on
our end -
https://salsa.debian.org/freexian-team/debusine/-/merge_requests/1863#note_613244)
and it occurred to me that Testsuite-Restrictions would have been
helpful for this if it existed.

In our case, we need to know which restrictions exist in order that we
can dispatch the autopkgtest task to a worker that has the appropriate
backends available.  But we don't want to unpack the source package in a
trusted (non-worker) context, because unpacking source packages does not
have a particularly great security history, as well as potentially being
expensive.

I agree that we need an efficient way to access either
debian/tests/control itself, or some better-summarized subset of it
(perhaps a deb822-style mapping from test names to restrictions).  I'm
not sure exactly what that representation should be.  While I take your
point about burdening the Sources file, this is the sort of information
we need to be able to get at when we don't have much else available.

Thanks,