Hi, Lintian has a lot of tests which is great for coverage, but maybe some of them could be skipped in autopkgtest runs. Currently autopkgtest takes ~1 wall clock hours on Debian's amd64 CI [1], but Ubuntu runs autopkgtest on all architectures and arm64 runs for example take more than 3 hours [2]. Setting the tests to run in parallel helps with the wall clock time, but still this may not be the best use of CI CPU resources or if all tests are needed, maybe lintian itself could be sped up a bit. Cheers, Balint [1] https://ci.debian.net/packages/l/lintian/unstable/amd64/ [2] http://autopkgtest.ubuntu.com/packages/l/lintian/disco/arm64
tags 926409 + moreinfo thanks Hi Balint, Interesting. I guess I would have three follow-up questions here: * On what criterion or criteria could we include or exclude tests from the autopkgtest runs? Whilst we could skip the unit tests (as these are "just" Perl that is unlikely to vary) the most interesting ones to run in terms of detecting regressions in an real-world environment (the entire point of autopkgtests from my point of view) would be the tests of the checks themselves and these likely constitute the vast majority of the total time. * I'm not sure *how* we can speed up the tests. I mean, they all essentially involve building Debian packages with all the usual debhelper calls, etc. Speeding *this* up is somewhat out-of-scope of this Lintian wishlist issue, alas. * Why not simply increase Ubuntu's timeout? I would concede this is not the best use of CI resources, but the trade-off with "human" time would appear to be worth it here. However, perhaps Felix has some input here as he has been doing a lot of work on the test suite recently? Best wishes,
Hi Chris, One criterion that came to my mind is filtering by severity, including errors for sure, but not pedantic ones. The full suite can run during the build thus we don't loose a lot of coverage. I hoped to rely on Lintian maintainer's judgement about what to omit. A profiling round with perf would point out a few things IMO, but as a start I did a timestamped run here to find slowest tests: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-rbalint-scratch/disco/arm64/l/lintian/20190404_002129_96f0c@/log.gz If there many package builds in the tests just changing the faster compression helps, a lot with little effort (until zstd becomes the default ;-)). I agree, and we may increase the timeout, but the running the tests seems to take longer in general than seems reasonable. Cheers, Balint
About 95% of the time is spent building packages, even though they almost never change. The tests would run much faster if we shipped pre-built packages. One way to accomplish that would be to package the tests separately.
Balint Reczey wrote: Mmm, unfortunately I think we would still want to know if, for example, the runtime toolchain changes such that a pedantic tag changes behaviour. :( Felix Lechner wrote: Another way to accomplish this could be that we could cache/store them across autopkgtest runs? IIRC (at least) Gitlab supports caching stuff like this. Best wishes,
The pedantic setting may become the default for tests, but very little time is spent running Lintian. Things may speed up a little if we run only the checks being tested (using the '-C' option), but that won't make much of a difference in the overall run time of the tests, which is primarily spent building packages.
Upon reflection, each test should be packaged separately. That way I no longer have to worry about using chroot to build tests with potentially conflicting build dependencies.
Yep, that'd be the way to go IMO. You aren't trying to test dpkg-buildpackage or parts of the package-building toolchain - you're trying to test Lintian, which operates on the results of that. Shunting this part to a one-time operation would be eminently sensible. We do similar in some pkg-gnome packages, for example glib2.0 ships a -tests package that contains "installed tests" which are compiled as part of the package build and then executed during the autopkgtests. Cheers,
Yep, that'd be the way to go IMO. You aren't trying to test dpkg-buildpackage or parts of the package-building toolchain - you're trying to test Lintian, which operates on the results of that. Shunting this part to a one-time operation would be eminently sensible. We do similar in some pkg-gnome packages, for example glib2.0 ships a -tests package that contains "installed tests" which are compiled as part of the package build and then executed during the autopkgtests. Cheers,
Hi Chris, Should we ship all built test packages as part of our releases? I can't think of a better way to close this bug. Kind regards Felix Lechner
On Wed, 3 Jun 2020, 1:09 am Felix Lechner, <felix.lechner@lease-up.com> wrote: Now lintian autopkgtests take approximately 1 hour everywhere I checked. Honestly, I believe 1 hour to be acceptable.
It is broadly acceptable, but if you can reduce the time by assembling artifacts in advance, I think that it is still worth doing to save time and not repeat computation that doesn't need to be repeated. As a bonus you're then not also testing the package build toolchain with each Lintian CI run - you are (mostly) only testing Lintian itself. Cheers,
Felix Lechner wrote: I'd be -1 on that for difficult-to-elucidate reasons and there are likely other approaches to explore outside of this bug. I think we have sped up things enough to close this particular issue, at least for now. Regards,
Felix asked me to report the performance data for the lintian testsuite I'm getting on my new desktop computer. Let me first say I'm a very casual contributor to Lintian and I'm certainly not in a position to judge if the testsuite takes too much time to run or is too extensive by default. Take this as additional data to make informed decisions :) On my previous desktop computer (AMD FX-8530, 8GB DDR3 RAM, SATA SSD), running the testsuite used to take up to 15mins. Now with the new PC (AMD Ryzen 5900x, 64GB DDR4 RAM, M.2 NVME SSD), it takes less than 3 minutes. This is the result of running `private/runtests` from a newly created git clone of the current master branch: The test suite ran for 2 minutes and 59 seconds. ----------------------------------------------------------------------- Then again, that's a brand new machine running top of the line consumer-grade hardware, with a beefy cooler :) Cheers,
I have been trying to reach you, please get back to me.
ACHTUNG!! Herzlichen Glückwunsch zum Gewinn der diesjährigen COCA COLA FOUNDATION-Förderaktion in Höhe von 2.000.000,00 US-Dollar. Für weitere Informationen und zur Abholung senden Sie bitte eine E-Mail mit Ihrem Namen, Alter und Ihrer Telefonnummer an unser Treuhandbüro: caso2900122@foundation32.org<mailto:caso2900122@foundation32.org>
Ich habe versucht, Sie zu erreichen. Bitte kontaktieren Sie mich über meine neue E-Mail-Adresse: : caso2900122@foundation32.org<mailto:caso2900122@foundation32.org>
<mailto:sbakhshi817@gmail.com> 1 <mailto:Sandeep.Bakhsh333@hotmail.com> Jhalakathi Branch , Mobile IP Phone Ext Email Web Office Head Office : : : : jhalakathi@unionbank.com.bd : www.unionbank.com.bd : : Bahela Tower, 72, Gulshan Avenue Gulshan-1, Dhaka-1212, Bangladesh [logo] [https://www.unionbank.com.bd/img/union-bank-plc.jpeg] EMAIL DISCLAIMER: The contents and any attachment of this email are confidential and legally privileged. It is intended solely for recipient or recipients. Any use and/or expose of any part of the content of this e-mail and/or attachment by not intended recipient is fully unauthorized and unlawful. If you have received this e-mail by mistake please notify the sender immediately. Also note that any view or opinion presented in this e-mail is solely responsibility of the author and do not necessarily represent the Union Bank Limited. We strongly suggest that you should do the needful against virus while checking e-mail and/or attachments, Union Bank Limited bears no liability for any harm caused by any malicious code or virus transmitted by this e-mail.
I have a job for you, for more info contact me via: sandeep.bakhshi@egodough.com<mailto:sandeep.bakhshi@egodough.com>