dpkg-buildpackage has become incredible slow. This seems to have started around end of 2024. When starting any build with "dpkg-buildpackage -B -nc", the problem gets visible in the process tree: 13317 13313 /bin/sh -c LC_ALL=C make -Rrnpsf debian/rules debhelper-fail-me 13320 13317 /usr/bin/make -Rrnpsf debian/rules debhelper-fail-me 19106 13320 /bin/sh -c DEB_BUILD_OPTIONS="nocheck parallel=8" DEB_BUILD_MAINT_OPTIONS="hardening=+all,-fortify future=+lfs" dpkg-buildflags --get OBJCFLAGS_FOR_BUILD 19109 19106 /usr/bin/perl /usr/bin/dpkg-buildflags --get OBJCFLAGS_FOR_BUILD the "debhelper-fail-me" target (which seems to come automatically) calls over-and-over lots of "dpkg-buildflags --get <some_flag>" subprocesses. This happens very often, and on machines where starting processes is slow (e.g. when running qemu-user for a non-native platform) building packages now suddenly takes endless time. There must have been some change in dpkg or debhelper packages (or something else) which triggers this changed behavior, as it was not like that a few weeks back. Any idea would be great how to track this down... Thanks! Helge
Control: tags -1 moreinfo Hi Helge To my knowledge, nothing note worthy has happened in dpkg-buildpackage or debhelper to make performance worse. The 1.22.13 should in fact improve the situation *by default*, but your package might already have opted into the optimization, so you might not see that difference (though). Additionally, I cannot spot anything obvious from the changelog of dpkg/1.22.11 that would have made performance worse (dpkg/1.22.12+ are from 2025 and therefore not the source of regression based on your timeline) This is a very old and very known problem (I cannot find the bug for it at the moment). Generally, ensure your package does **not** call dpkg-buildflags manually if at all possible. The `dh` helper will export them for you (as of compat 10), so you can rely on them in any hook target. When not using `dh` and you need something from `dpkg-buildflags` include `/usr/share/dpkg/buildflags.mk` instead. Manually calling `dpkg-buildflags` is the root to all performance problems in the current design. Tools that manually call `dpkg-buildflags` are equally so. Please show us the package source. A pstree can also be helpful in case the `dpkg-buildflags` are called from a tool rather than `debian/rules`. This behavior in dpkg and debhelper is decades old. Something else changed and is now triggering it. Such as changing from classic debhelper to `dh`. Look for any calls to dpkg-buildflags and replace it as above is the easiest way of solving it. If you introduced a new third-party helper tool, then it might contain sub-optimal calls that need to be weeded out. If you provide a link to the source in question, I can have a look and see if I spot anything obvious. Best regards, Niels
This happens for me with the "neomutt" or "vim" source packages from sid. Helge
Helge Deller: The `neomutt` package has two calls to `dpkg-buildflags` that looks like they could definitely be optimized. Though, I cannot see those two being introduced recently. In fact, `git blame` dates those two lines as 6 years old. For `neomutt`, assuming those two are only needed for `dh_auto_configure` (which is likely but untested) we could move them to the override_dh_auto_configure. Something like override_dh_auto_configure: EXTRA_CFLAGS_FOR_BUILD=$$(dpkg-buildflags --get CFLAGS) \ EXTRA_LDFLAGS_FOR_BUILD=$$(dpkg-buildflags --get LDFLAGS) \ dh_auto_configure -- \ ... Alternatively, perhaps including the dpkg makfile and use the relevant variables from it with lazy evaluation `EXTRA_CFLAGS_FOR_BUILD=...` (instead of `EXTRA_CFLAGS_FOR_BUILD:=...`) might help. For `vim`, I cannot spot anything obvious with the `dpkg-buildflags` (it uses the include). But it does have several hook targets plus self recursion into `debian/rules`, which tends to amplify the problem considerably. I suspect that inlining the `$(MAKE) -f debian/rules ...` would help here. Perhaps a "DPKG_EXPORT_BUILDFLAGS=1" might work as a work around, since it should make recursion a bit less costly. Sadly, the "conventional" knowledge for makefiles does not apply `debian/rules` when it comes to optimization, which does not help the problem. Best regards, Niels
make-dfsg (4.4.1-1) is introducing the issue.
Reverting "make" back down to version 4.3-4.1 solves the issue
and lets "dpkg-buildpackage" run lightning fast again.
Looking at the make changelog, this seems related:
* New upstream version 4.4. Closes: #1029106.
- Exports variables to commands started by $(shell ...).
Helge
Hi Helge, That is indeed probably the reason; I’ve seen more significant speed degradation in gcc-mingw-w64 for example (fixed by avoiding recursive expansions). Could you qualify “incredible slow”? In my tests, vim fails its tests so I can’t measure the speed difference, and neomutt went from a 1 min build in testing (on my 10-year-old system) to 1:30, which is significant but not incredibly slow in my book. There is a noticeable pause whenever debian/rules is loaded, so Make is definitely doing too much work. I’ll try to come up with a minimal reproducer and forward the issue upstream. Regards, Stephen
Hi Stephen, You probably won't notice on x86 and other fast machines. Sometimes there the build takes 2 minutes instead of prior 1 minute. So, it's 100% slower, but not a real problem. That's different on parisc: Physical parisc machines need to flush caches when a processes is spawned. Cache flushes are generally slow and I think this is a problem on other architectures (e.g. sparc?) as well. It's even worse is on those buildds which are running linux-user. Everytime a process is spawned, linux-user needs to assembly the file again, and the build gets slowed down a lot. Example: https://buildd.debian.org/status/logs.php?pkg=neomutt&arch=hppa On the physical machine "parisc" build time went up from 20 minutes (2024-11-24) to 1:10 hours (2025-01-05). strace -f dpkg-buildpackage -B -nc 2>&1 | grep buildflags > some_file wc -l some_file Thanks! Helge
Hi! I just got approached by Michael Hudson-Doyle (CCed) where something similar was noticed in Ubuntu [U]. We pondered whether the dpkg-buildflags had gotten very slow or the amount of calls had increased, Michael tested calling «debian/rules clean» for libvmime 0.9.2-8.2ubuntu1, with make 4.3 and 4.4, and noticed the following numbers on Ubuntu: 4.3 calls dpkg-buildflags 22 times 4.4 calls dpkg-buildflags 1492 times For me the numbers on Debian were 23 and 138 respectively. After testing whether removing the DPKG_BUILDFLAGS_EXPORT_ENVVAR support might help (which it did not), I noticed that the buildflags.mk is using the «or» make function for its variable caching, which can be problematic when the variables expand to an empty string. Changing that to check whether the cached variable was already defined (even if empty), takes my numbers from 138 to 40, which is still not ideal but much better. I've queued that patch for the next dpkg upload (attached). Then I noticed that the dpkg_buildflags_setvar variable uses «=» instead of «?=» for its internal variable assignment. Changing that takes it down from 40 to 20, but the problem is that this change could be problematic, as ISTM, it changes semantics, where «=» will overwrite a previously assigned variable, so this could break packages. :/ And I don't think this is a safe change to make. I'm attaching the change anyway for people to take a peek, but I'm not currently planning on queueing it. Maybe after a mass rebuild and mass autopkgtest run or similar, if stuff does not break then that could be considered. Michael also noticed tons of messages from «make --debug=all», like: /usr/share/dpkg//buildflags.mk:71: not recursively expanding CFLAGS to export to shell function It's still not clear what is going on with make 4.4, and what has caused these regressions. The multiple braking behavior changes might explain it, but the reason does not seem obvious. The buildflags.mk change to check for the cached variable being defined, seems correct anyway, and does not regress with make 4.3, so that looks like a good improvement non the less, but w/o the ?= change we are not back to the old numbers. [U] https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/2093107 Thanks, Guillem
the most extreme I've seen is php8.4, dpkg-buildpackage -B -nc calling dpkg-build flags 256915 times.
with the first patch, that's down to 2420.