I tried running lintian on a full amd64 build of linux.  lintian
allocated about 23 GiB of VM and then was OOM-killed before generating
any output.

I notice that <https://lintian.debian.org/sources/linux> has no
information, probably because of this.

This is a regression in lintian, but I don't know how long ago it
happened.

Ben.

Hi,

Thanks for reporting this!

That is one of several mysterious results of what happened when I
simplified the code. It's possibly due to excessive regex usage,
although Perl usually catches that. Another possibility is an
accidental recursion in Lintian's copy of your package's file index.
Either way, I will shortly add helpful profiling information that you
will be able to examine on our website.

On the positive side, I broke Lintian's 23 original checks into 353
pieces (and counting). It makes issues much easier to monitor and
isolate.

Kind regards
Felix Lechner

I'm not aware of regex processing ever requiring lots of memory.

I don't really want to examine it, I want you or some other lintian
contributor to fix it.

OK, but it's no longer working for me or probably anyone working on a
larger package.

I just tested with some older releases.  The performance got worse
between buster and bullseye, but the bullseye version is still usable
for me (~ 1.3 GiB VM used, 17 min wallclock time).  So the regression
is since bullseye.

Ben.

Hi,

So far, the most convenient way to reproduce this bug is the following
command. It exercises only a single, very simple check. [1]

    bin/lintian -C linda --debug --no-override --no-tag-display-limit
../picolibc-arm-none-eabi_1.7.4-1_all.deb

Kind regards,
Felix Lechner

[1] https://salsa.debian.org/lintian/lintian/-/blob/master/lib/Lintian/Check/Linda.pm#L32-39

Hello,

Bug #1003456 in lintian reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/lintian/lintian/-/commit/192e15ae2eda7535622d44d2f3f382783b110ee5
Works but is slow. Perhaps not quite so much of the readelf information needs
to be retained.

The impact on average library packages has yet to be evaluated.

It may only is be issue for some installables. The one from the bug, for
example (picolibc-arm-none-eabi_1.7.4-1_all.deb) seemed to generate several
million symbols whose purpose was not clear to me. That impression is dated
because I looked into the reason for the high memory usage in January.
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1003456

We believe that the bug you reported is fixed in the latest version of
lintian, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1003456@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Axel Beckert <abe@debian.org> (supplier of updated lintian package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Mon, 20 Jun 2022 13:23:02 +0200
Source: lintian
Architecture: source
Version: 2.115.0
Distribution: unstable
Urgency: medium
Maintainer: Debian Lintian Maintainers <lintian-maint@debian.org>
Changed-By: Axel Beckert <abe@debian.org>
Closes: 657390 932634 941656 963099 989381 995286 996740 999768 999810 1000234 1000977 1001655 1002828 1003131 1003272 1003353 1003456 1003668 1003817 1003913 1003941 1004231 1004239 1004240 1004660 1005046 1005184 1005762 1006390 1006859 1007140 1007257 1012090
Changes:
 lintian (2.115.0) unstable; urgency=medium
 .
   The Lintian Resurrection Release.
 .
   * Summary of tag changes:
     + Added:
       - alien-tag
       - chown-with-dot
       - conflicting-test-fields
       - declare-python-versions-for-test
       - drop-python-version-declaration
       - invalid-override-restriction
       - missing-prerequisite-for-pyproject-backend
       - old-devhelp-standard
       - stray-devhelp-documentation
       - test-leaves-python-version-untested
       - uses-poetry-cli
     + Removed:
       - crossing-screens
       - debhelper-compatibility-level-not-a-number
       - debian-tests-control-and-control-autodep8
       - exclusive-runtime-tests-field
       - package-contains-devhelp-file-without-symlink
 .
   [ Axel Beckert ]
   * Adopting Lintian. (Changes #1012289 from ITA to pure RFH.)
     + Remove Chris Lamb from Uploaders (see #1012289) and re-add myself.
   * Workarounds until
https://github.com/Perl-Critic/Perl-Critic/issues/925 is fixed:
     + Replace all occurrences of "Copyright ©" with "Copyright (C)" again.
     + Remove unnecessary usage of UTF-8 from bin/lintian.
     + Replace UTF-8 characters in mostly Copyright comments.
     + Replace UTF-8 characters in code with \N{…}.
   * Remove literal unicode character U+0334 COMBINING TILDE OVERLAY which
     likely had been added accidentally. (Triggered by the symptoms of
https://github.com/Perl-Critic/Perl-Critic/issues/925, but permanent.)
   * Update copyright years in debian/copyright.
   * Run perltidy over lib, bin/lintian, private/refresh-perl-provides,
     private/runtests and several files in t/scripts/.
   * data/…/perl-provides updated by running "debian/rules
     refresh-perl-provides".
   * Add Felix Lechner to debian/copyright based on copyright statements
     elsewhere. Thanks for all your contributions!
   * Update t/recipes/README: "debian/rules runtests" → "private/runtests"
   * Follow module renaming: Perl::Critic::Freenode → Perl::…::Community.
   * t/s…/h…/tag-coverage.t: Replace "$ENV{'LINTIAN_BASE'}" with
     "$ENV{'LINTIAN_BASE'} // '.'" to be able to run it with "prove -l".
   * init.d-general check: Avoid relying on line numbers in #DEBHELPER#
     replacement code.
   * very-long-line-length-in-source-file: Ignore files listed in new data
     file binary-file-extensions. (Closes: #1005046)
   * Fix false positives for adopted-extended-field with X- prefixed
     fields. (Closes: #999768)
     + Empty hints files seem to require a Test-Against field in desc.
   * Update own source lintian-overrides for "pointed hints".
     + Make them work with old and new lintian versions by using wildcards.
   * Rename README.developers to have a proper file suffix (.pod).
   * Switch syntax marker of README.developers.pod from "perl" to "pod".
   * Documentation update: Replace directory "frontend/" with "bin/".
   * Fix a bunch of "Use of uninitialized value $_ in concatenation"
     warnings when running tests with "prove -l" directly.
   * README.developers.pod: Explain the difference between check and test.
   * lintian(1): Drop mentioning of never existing --no-overrides option.
   * Replace unfitting Text::Glob with more flexible Regexp::Wildcards
     (Closes: #1003353)
     + Add unit test for Lintian::Util::match_glob. The current testsuite
       does not seem to be able to cover such a case.
   * Declare compliance with Debian Policy 4.6.1. (No changes needed.)
   * Refresh data using private/refresh-data. Skip unreleased policy though
     for now.
   * Fix "Use of uninitialized value $step in concatenation" in
     Lintian::Version which showed up as unrecognized tag (!) when running
     the test suite on the git repo already tagged for a release.
   * debian/gbp.conf: Declare so far used tag format so that gbp uses it.
   * Add lintian override for very-long-line-length-in-source-file in
     Lintian::Check::Cruft as well as test-leaves-python-version-untested.
   * Use versioned Breaks instead of Conflicts against lzd, see #1001655.
     Thanks Lintian for reporting ;-) and Paul Gevers for the sanity check!
 .
   [ Felix Lechner ]
   * Refresh manual references.
   * Use Text::Glob to match hint contexts with override patterns. Replaces
     a trusted homegrown routine. (Closes: #1003272)
   * Refresh list of available Debhelper commands.
   * Refresh list of installable fonts.
   * Generate section references for Lintian manual from repo; point to
     website.
   * Accept globbing patterns in profiles when enabling and disabling
     checks or tags.
   * Refresh data sources in parallel.
   * Add the New Maintainer's Guide to the list of quotable authorities.
   * Eliminate unpredictable output in the check siles/privacy-breach.
   * Honor the environment variable NO_COLOR as specified in
https://no-color.org/.
   * More attempts to eliminate unpredictable output in the check
     files/privacy-breach.
   * Drop the tag debian-tests-control-and-control-autodep8.
   * Set authority references apart from other data sources.
   * Provide rudimentary Emacs integration. (See: #968758)
   * Associate Emacs modules with the 'editors' archive section.
   * Recognize /usr/bin/raku as a known interpreter for scripts. (Closes:
     #1002828)
   * Do not depend on any particular Lzip implementation. (Closes:
     #1001655)
   * Exempt installables designated as documentation from warning about new
     Python2 packages. (Closes: #995286)
   * Update citations in two tags. (Closes: #1003131)
   * Drop version requirement from
     skip-systemd-native-flag-missing-pre-depends. (See: #1003271)
   * Import new CSS style sheet from the website.
   * Recognize dh-sequence-sphinxdoc as a valid prerequisite for
     dh_sphinxdoc. (Closes: #999810)
   * Tolerate multiarch acceptors in prerequisites for Debhelper commands
     and addons. (Closes: #1000234)
   * Issue yet more pointed hints.
   * Recognize pybuild-plugin-pyproject as a valid prerequisite for the
     python3 Debhelper plugin. (Closes: #1003668)
   * Exempt bullseye backports from changelog-file-missing-explicit-entry.
     (Closes: #941656)
   * Mask long source lines in autotools-generated files. (Closes: #996740)
   * Turn embedded-library into a classification tag. (Closes: #932634)
   * Require the targets build-arch and build-indep in debian/rules.
     (Closes: #657390)
   * Do not insist on a particular name for unversioned links to a shared
     library. (Closes: #963099)
   * Exempt the names of Debian folks associated with a package from
     spelling checks. (Closes: #989381)
   * Require py3version invocation consistent with presence of
     X-Python3-Version in d/control. (See: #1001677)
   * Exempt CGI scripts from executable-in-usr-lib. (Closes: #1003941)
   * CGI scripts can be ELF executables. (See: #1003941)
   * Exempt Python's .dist-info and .egg-info folders everywhere from
     documentation-outside-usr-share. (Closes: #1003913)
   * Flag an outdated Debian copyright just once; use the most recent
     year. (Closes: #1003817)
   * Implement '--no-show-overrides'; honor it for overrides and masks
     alike. (See: #1004240)
   * Allow the command-line option '--no-info' to reverse 'info=yes' in the
     configuration file. (Closes: #1004240)
   * Elide manual references to ancient Lintian versions; use modern
     examples. (Closes: #1004231)
   * Deprecate --no-tag-display-limit for '--tag-display-limit 0'; update
     documentation. (Closes: #1004239)
   * Also provide a default output width for
     lintian-annotate-hints. (Closes: #1004660)
   * Mask examples in tests from
     package-does-not-install-examples. (Closes: #1005184)
   * Recognize Java 18 in unstable, and Java 19 as otherwise
     available. (Closes: #1005762)
   * Leave default Java bytecode version at 56. (See: #1005762)
   * Adjust documentation reference to manual page for dh_make. (Closes:
     #1006390)
   * Warn about devhelp index files that use version 1. (Closes: #1006859)
   * Store ELF information from readelf in an MLDBM database. (Closes:
     #1003456)
   * Issue pedantic hint for dot in 'chown user.group' instead of a
     colon. (Closes: #1007140)
   * Upgrade missing-systemd-timer-for-cron-script to warning; no longer
     experimental. (Closes: #1007257)
 .
   [ Ryan Finnie ]
   * Provide a constant citation for
     systemd-service-file-uses-nobody-or-nogroup. (Closes: !385)
 .
   [ Louis-Philippe Véronneau ]
   * Check that tests pulling in all Python versions also query which ones
     are available. (Closes: !361)
   * Add new Python tags for pyproject.toml build backends according to
     PEP-517. (Closes: !384)
   * Rename 'python3-flit' to 'flit', as there is no 'python3-flit'
     package. (Closes: !386)
 .
   [ Daniel Kahn Gillmor ]
   * Correct lintian-annotate-hints manpage.
 .
   [ Simon McVittie ]
   * Silence a very widespread false positive for detached debug symbols.
     (Closes: #1000977, !387)
 .
   [ Simon Quigley ]
   * Add "kinetic" as a known Ubuntu distribution. (Closes: !392)
 .
   [ xiao sheng wen(肖盛文) ]
   * Add riscv64 support (Closes: #1012090, !394)
 .
   [ Damyan Ivanov ]
   * Update releases.json data for Debian policy releases (4.6.1 added;
     closes: !393)
 .
   [ Paul Wise ]
   * Add more obsolete domains for former source code hosting services.
Checksums-Sha1:
 863a51cffc5b8359ef133d6e6f150d9cac148eb3 2503 lintian_2.115.0.dsc
 e20194c63f481a7361969fab6ea7739f3f6e03d5 2170172 lintian_2.115.0.tar.xz
 3453b48f59dd2f58e2e78523e9018c58a42faebe 7274 lintian_2.115.0_source.buildinfo
Checksums-Sha256:
 431a025b52e185cac6cbf2fedca8b6c60e2f7dfa55d89d86fcd543a34e8232b9 2503 lintian_2.115.0.dsc
 f353d372d036daa6ad0341b418728cfe73c11688708b6c3a33d50acb445d2b53 2170172 lintian_2.115.0.tar.xz
 e1c1bd41b7f5a8014231f9db422c1711e97cf50146eab83a1d113370735f02d9 7274 lintian_2.115.0_source.buildinfo
Files:
 4484e31abe39a15def23b119f92ef3ab 2503 devel optional lintian_2.115.0.dsc
 9d7922b69d30a63693825120e049ea00 2170172 devel optional lintian_2.115.0.tar.xz
 12f2494d410df4c2bf03f82075ce5f60 7274 devel optional lintian_2.115.0_source.buildinfo
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEERoyJeTtCmBnp12Ema+Zjx1o1yXUFAmKwZiIACgkQa+Zjx1o1
yXUIng/5ARDjo8W4NggeM79xaeIgqtRbCao5XuBX6VyAE6hupaK8vHH/konnGmwg
jfRvPQK6YCXfqHggIKNQnToJgKUxdI+U3e10PYrbc5qt2UWFMiRfEHGqG3cfu+uA
JhqldKXfRkvmY4+9WPChWZ4C8qjdPZNcQwwdwm1Rg6gKMreYDGNIWJ94Hh7BkNsC
lNowSbDW6uE0XLBxVVeHhQGrn0Aki8fkx82W+Hod2A26sCXv/Gx7OSb8KVERa3+7
D05KDZd+ZGXBIdk/zRn/AeCxwcLXXMCS7qNDsfbJ8oUeYyy7y7QrQbGeFrBdOTyk
JbqqG0rTGbvfg3UcviOm/rvkr5QHdA5OMLUKRoljApMJEzbASGzYqNiUSmZNWH8x
v9EZ/qTHrcPFL/bIC9POvjD3FhSs3EkgQxXCuVPWZRruurgEme6DmnxHEj9TAe6L
GcUr+6DDHMw+j9SlO04VDDpUFVO2ftPfMcSckNCNM3eMxfig4XjPmyay3m6BjgaF
Rq6ckTUCUt5hsm7T48I0PAjXMRcAbemfI3PmVOUgwIf3BvzcdvJew0h90ULOY94N
cCWdsZ5SwC1hoz6EsBVmD16XxxYvZM7/wcf40PJkJ+YwtJu2Wm3dAv5WPK8UTrYr
qK1s62DfByho45eV9rXKeLnjvRN37rQYOWl7WwYy6YfpzQYDFWU=
=3dK3
-----END PGP SIGNATURE-----

Control: reopen -1
Control: found -1 2.115.0
[...]

This didn't fix the issue.  It is still reproducible by building linux
for amd64 and then running lintian over the resulting .changes file.

Running lintian over a single linux-image-<version>-amd64-dbg binary
package also shows the issue, but requires "only" about 12 GB of VM and
does complete on my development system.

Ben.

Hi Ben,

Ben Hutchings wrote:

Thanks for these details and for reopening!

Will also have a look at it.

Given that it is a debug package, I'll see if I can either

* find the cause by debugging the code,
* find the cause by bisecting, or
* reduce the amount of checks run against it as many
  make no sense for debug packages.

		Regards, Axel

When you look at this again please consider replacing the BerkeleyDB
storage backend with a different database so that we can get rid of
BerkeleyDB. See #1119186.