Package: lintian
Version: 2.1.6
Severity: wishlist

Hi everyone,

Thinking about how to improve the QA aspect of Debian I though it would be
nice if lintian encouraged the use of test suites if they exist. This should
hopefully help in guaranteeing the functionality of the packages during their
existence in Debian. Of course the benefits of running a test suite are only
helpful when the package is built in different archs (assuming the uploader
*does* test the package in their box, which means one less arch to test it
under) or when a package rebuild occurs (either archive wide, or a binNMU).

I'm not yet sure how to implent it, basically I've reduced it to two options:
a) Only warn if a well known kind of test suite exists (e.g. perl's
test_harness, *units, etc).
b) Do as a) plus a more FP-prone check like dh_auto_test's.

And as for knowing whether the test suite is run:
* dh, dh build, or dh_auto_test are run.
* any occurrence of m/MAKE.{1,60}(test|check)/i  in debian/rules.
* any other test suite-specific way to run the tests.

Comments, suggestions, volunteers (to for example help knowing about test
suite systems, as I do plan to work on this check)?

Cheers,

Actually, I'm not sure if checks/rules is the best place for it, as it needs
information that would make it part of checks/cruft, but doesn't completely
fit in there either. Maybe some sort of checks/testsuite? I don't feel
totally comfortable with that option either as I can't think about many other
checks that could be applied to the package's test suite.

What do you think?

Cheers,

Raphael Geissert <atomo64@gmail.com> writes:

Maybe we should introduce checks/build for all checks that try to analyze
the upstream build system?  A fix for #472219 would fit in there as well,
and if we wanted to, we could move the config.guess/config.sub/libtool
stuff to that check as well.  It only sort of belongs in cruft; it's
different than what the rest of that check does.

Hello,

I think that generally this is a good idea. However, such general
lintian check might be difficult to implement, but at least packages
built using maven-debian-helper could be checked.

I propose the following algorithm:

1. Check that package build depends on maven-debian-helper;

2. Check that there are tests (*.java files under src/test/java/ or
<module>/src/test/java/);

3. Check that debian/maven.properties contains uncommented line
'maven.test.skip=true'.

If all three checks are positive, upstream test suite exists, and is not
run during the build.

What do you think?

Andrius

Hi Felix,

I noticed you changed the title of the bug report from "lintian: Warn if
a test suite exists but is not run" to "lintian: Warn about unused
autopkgtests", and I doubt this reflects the original intent of this bug
report.

I read the initial Raphael's message as asking to detect test suites not
run at the build time (as per the subject of the message). Not run
autopkgtests are not less important, but this is a separate topic. I
suggest splitting this bug report into two:

* lintian: Warn about unused autopkgtests
* lintian: Warn about unused build time tests

What do you think?

Best wishes,
Andrius

Hi Andrius,

I agree with you, and picked yet another title. Please let me know
what you think.

Upon reflection and refactoring the code [1] I do not think it can be
implemented. With Test-Directory set to '.' all files in the source
package are candidates, and would produce hints for not being used.

I am not sure how to implement the original request. Your suggestions
are welcome.

Thank you for figuring out the purpose of this report!

Kind regards
Felix Lechner

[1] https://salsa.debian.org/lintian/lintian/-/commit/7428997ded64b57ccf9a7c5874c683b7b8fd188e#02c42ce39affeca72ca4ed5d708e75a7bc945171_203_207

Hi Felix,

I still think the original intent was to encourage the maintainers to
run upstream tests during build. Thus I would suggest renaming the bug to:

    lintian: flag packages not running upstream build time tests

Sorry, I was not clear enough here. My suggestion here was to flag
packages not supplying autopkgtests although they could do so. For
example, Perl packages having tests under t/ directory, but somehow not
having autopkgtest at all.
override_dh_auto_test followed by an empty line. This means upstream
tests would be run by debhelper, but for some reason they are turned off.

And there are buildsystem-specific rules. For example, for Maven
packages, a line

    maven.test.skip=true

in debian/maven.properties means that upstream tests are ignored during
build.

Thanks for prompt response!

Best,
Andrius