Dear Maintainer,

Depending on the builder being used it is not possible to reach the
internet, making, e.g. opportunistic, tests to fail. This patch makes
opportunistic and cavp tests skippable and makes it exit 77 in case of
failure, as specified by:
https://salsa.debian.org/ci-team/autopkgtest/blob/master/doc/README.package-tests.rst

In Ubuntu, the attached patch was applied to achieve the following:

  * d/tests/control: make opportunistic and cavp skippable if builder
    doesn't have access to internet.
  * d/tests/opportunistic: if ping or wget fails then exit 77 to skip
    the test.
  * d/tests/cavp: if wget fails then exit 77 to skip the test.


Thanks for considering the patch.

Hi Eduardo--

Both of these tests are already marked as "needs-internet" -- shouldn't
that be sufficient for autopkgtest runners to know to skip them if
regular internet access is not available?

I'm not sure why we should use the skippable feature as well.  Can you
help me understand why we need both?  Given that the only reason to need
to skip is because the test needs the internet, it seems redundant (and
possibly too lax) to have both methods available to bypass the test.

(i might be misunderstanding something about how autopkgtest works --
please help me understand better if that's the case, i'm not opposed to
adjusting the tests in general!)

Hi Daniel,

It seems like this issue appeared after Launchpad builders started to
recognize the needs-internet restriction, and it seems that
needs-internet is not enough. Here is what the test was returning
before:
opportunistic        SKIP unknown restriction needs-internet
And after, when I was applying some fixes:
opportunistic        FAIL non-zero exit status 1

That's why we added the skippable and exit 77.

Not sure if that's helpful enough, but if there's any better option to
it we are certainly open to discuss it.

ok, but what does "recognize the needs-internet restriction" mean?  the
definition of "needs-internet" [0] is:

    The test needs unrestricted internet access, e.g. to download test
    data that's not shipped as a package, or to test a protocol
    implementation against a test server. Please also see the note about
    Network access later in this document.

[0] https://salsa.debian.org/ci-team/autopkgtest/blob/master/doc/README.package-tests.rst


The "Network access" section also says:

    […] In Ubuntu's infrastructure access to sites other than
    *.ubuntu.com and *.launchpad.net happens via a proxy (limited to DNS
    and http/https).

The patches you've proposed, if i'm understanding them correctly, cause
the test to be skipped because the test really does use unrestricted
Internet -- not only DNS and HTTP/HTTPS.

This makes me think that either Ubuntu's infrastructure shouldn't
consider itself as offering "unrestricted internet access" (meaning, it
should automatically skip any test with a "needs-internet" restriction),
or it should run any test that has a "needs-internet" restriction
outside of the proxied filter.

If there's a need for a different constraint (like
"needs-only-dns-and-http") to match what ubuntu's infrastructure offers,
that might be a separate question.  But either the documentation of what
"needs-internet" means needs to change, or Ubuntu's infrastructure
should not claim to support it, if i'm reading the docs correctly.

thanks for raising this issue!  I'm happy to talk about it more if
you've got a different interpretation.

Regards,