Hi,

Thanks for providing guess-concurrency, it's a clear improvement over
the number of open-coded equivalents in many debian/rules file.

That said, I have observed that MemAvailable is sometimes significantly
lower than expected on some build daemons. I have tracked that to the
use of tmpfs as the build directory (which by the way is the default in
sbuild): until the data is flushed to the swap, the corresponding memory
amount disappears from MemAvailable. This causes the number of jobs to
be reduced for no real reason (the memory is still be available for
build), especially for packages with big sources.

Could you please consider using MemTotal instead? I think it represents
better the amount of memory a build daemon is able to use.

Regards
Aurelien

Hi Aurelien,

Thank you!

I see where you are coming from. When designing it, I was considering
that system services (and in case of building on a desktop or laptop)
also user applications may consume RAM. Simply going from MemAvailable
to MemTotal would degrade that use case. An earlier version did use
MemTotal, so it is not clear cut to me just yet.

It seems like we do want to use some amount of swap, but swapping out
everything might be too much. Can we find a middle ground?

How about using

    MemAvailable + min(SwapFree, (MemTotal - MemAvailable) / 2)

as a better approximation? I recognize that this is not going all the
way that you want to go. However, it would still consider min(SwapFree,
MemTotal / 2) as a minimum even if MemAvailable would be fully consumed
with tmpfs. On systems without swap, this would result in the same
behavior as before the change.

Can you give me an idea of whether that would improve the buildd use
case? Would it be a sufficient improvement in your view?

Helmut

guess-concurrency.

IMHO the most important fix would be to make it clear in the
documentation that guess-concurrency should only be used when
***really*** necessary.

Sometimes guess-concurrency is used with an insanely high value
(#1149937), people guessing numbers is just harmful.

From a decade of experience keeping packages alive on 32-bit I can tell
that it is usually exactly one ld call or the one compilation of the
swig-generated Python bindings for a C++ library that is the memory hog.
Using guess-concurrency does not make much sense in this typical case.
Packages where the maximum memory usage scales with the number of cores
do exist, but they are rare.

On 32-bit architectures, requiring more memory per process than the
address space available for a process is obviously wrong.

In [1] the removal of guess-concurrency in my 9.6.2+ds-2+test1 package
uses 6 instead of 2 cores on the buildd, resulting in a 2.76x build time
difference. A 2.76x speedup for a 3x increase in the number of cores is
as close to perfect scaling as might happen practice.

In practice we have to pick one of the following:
1. treat build time increases due to guess-concurrency as RC bugs, or
2. drop riscv64 as release architecture

And I mean 2. seriously, [2] increases the build time of a package
currently involved in a transition from 4 to 6 days on the slower
buildds due to only using 2 instead of 4 cores. Even on the new faster
buildds[3] it would likely also push the build time above 2 days.

And riscv64 is just the tip of the iceberg:
We spent a decade getting parallel builds working in nearly all packages,
we don't want to lose a large part of that when people start modifying
packages on a large scale with super-high guessed limits to
guess-concurrency - requiring in the need for more buildds on multiple
architectures.

And you also want to be conservative regarding how to calculate the
available memory. Not using all cores on a 4-core riscv64 buildd with
16 GB RAM because your proposed formula would use only 3 cores when
an uploader used 4 GB as guessed value[4] - that's just bad.

Can we agree that guess-concurrency increasing build times is a bad
thing that should usually result in an RC bug, or do you insist on
killing riscv64?

cu
Adrian

[1] https://buildd.debian.org/status/logs.php?pkg=vtk9&arch=i386
[2] https://buildd.debian.org/status/logs.php?pkg=vtk9&arch=riscv64
[3] rv-manda-0{2,4}
[4] which seems to be the most common uploader guess

Hi Helmut,

While that's definitely an improvement, it's only a small improvement.
For a typical buildd with 16GB of RAM, if a package consists in 1GB of
source files, requires 4GB of build-dependencies installed in a 1GB
chroot, this reduces the available memory size as guessed by
guess-concurrency from 16GB to 13GB instead of 16GB to 10GB.

At the end it still increases the package build time, and results in
an inefficient use of our buildd hardware.

Regards
Aurelien

Hi again,

The memory used by tmpfs, but not yet swapped, appears in "Shmem".
Therefore how about using

  MemAvailable + min(SwapFree, Shmem) ?

Regards
Aurelien

Hi Helmut,

apologies if this might have sounded a bit aggressive, it was not at all
intended as a personal attack against you.

We are having a real problem on the buildds with build time regressions
caused by guess-concurrency, basing everything on worst-case guesses
with large margins everywhere might have been well-intended, but it is
not workable if this is being done with many packages.

And riscv64 does not have any margin for slower builds.

cu
Adrian

Hi Adrian,

guess-concurrency is not breaking anything. It merely collects a pattern
that was vendored into several source packages into a binary package for
easier reuse. The guesswork (that people seem to be getting wrong)
remains with them.

Expanding the documentation likely is a good thing. Unfortunately, we
first have to agree what guidance we want to give. What it means to be
really necessary is not that obvious. From my pov, it is necessary when
the build OOMs the machine. The tricky part is figuring out what the
lowest machine specs are that we want to support.

Generally though, I agree that using guess-concurrency is more of an
exception as most packages should work with the default concurrency.

This is sad indeed.

I know about this behavior and took care of making guess-concurrency
somewhat support this common case. If you have a single large (e.g. 3GB)
link job and expect your other compiler invocations to not typically
exceed 1GB, you could pass --require-mem 3G,1G.

That's your view. My view is that suing guess-concurrency makes more
sense than having to manually retry your build several times due to it
being OOM killed.

In my experience it is most packages that scale with the number of
cores. I think what you meant to imply here is "scales linearly". I
agree that a linear approximation often is a bad match, but an affine
approximation is accurate more often. Effectively, guess-concurrency
supports piecewise linear approximation - a generalization of affine
functions.

This is a different problem not addressed by guess-concurrency in any
way.

That was using --require-mem=4G. And then to me there is a difference
between it building slowly and it requiring human intervention to build
at all.

I don't follow how you reach this conclusion. While debhelper implements
concurrent building by default, doing so still is not mandatory. For
instance, sqlite3 still is annoyingly sequential and I haven't seen a
bug about this. Much less an rc one. Nor is sqlite3 being used as a
reason to demote riscv64.

What you probably see as a problem here is that pessimistic use of
guess-concurrency tends to clog the build queue. This has been a problem
before when guess-concurrency was open coded into debian/rules in
various ways. In my view, the fix is not to stop using it but to pass
more sensible memory guesses.

The alternative that you do not highlight here is failing the build
after two days to have some developer give it back hoping that it does
not fail again. Not only does this waste buildd time, but also developer
time.

The vast majority of packages should not be using guess-concurrency.
Unfortunately, the ones where it does make sense are the ones that
typically take several days so any misuse delays builds a lot. Figuring
out what limits are sensible is a task that arises independently of
whether you implement it using guess-concurrency or not.

That's your view. My view is that less builds fail by OOMing the buildd
and thus *save* a lot of time.

No, we actually disagree about this.

Can we agree that the way guess-concurrency is presently being used in a
few packages is less than ideal?

Note that the forky/sid manual page is expanded compared to trixie. It
now includes a ready example for the linker behavior. I guess that guess
that some users read the trixie one.

Helmut

Hi Aurelien and Adrian,

Thank you for this suggestion. I implemented this in
https://salsa.debian.org/helmutg/guess_concurrency/-/commit/12e7a4d1d72d31fa0c731912985c0069210f3974.
Would you confirm that it adequately solves your problem?

Adrian, I also expanded the documentation given your feedback. Could you
have a look at
https://salsa.debian.org/helmutg/guess_concurrency/-/commit/55a7a06d2653a459855c476c914c31d5886f8c06
and let me know whether it adequately addresses your legitimate
concerns?

Helmut

Hi Helmut,

is guess-concurrency supposed to only replace various existing open
coded implementations/limits, or is it intended that people add new
guess-concurrency usage just because it exists?

The former is an improvement, but the latter is what is actually
happening. And this is the problem.

Primarily we want to support our buildds.

There is a large overlap between the packages with huge build times,
and the packages where the build uses a lot of memory.
3x --require-mem 4G
1x --require-mem 5G
1x --require-mem 8G (#1149937)
1x --require-mem 16G

The only non-round number (5G) has a comment indicating that the
maintainer tried to determine actual memory usage.

A semantic problem with guess-concurrency is that it is not clear
whether it is supposed to estimate typical maximum usage, or whether
it is supposed to set a hard maximum.

The comment for the "--require-mem 5G" in nextpnr says:
  # Building the BBA files requires about 5GB in the worst case

On a riscv64 buildds with 4 cores and 16 GB RAM I would guess
that fastest might be to use all 4 cores when the worst case
for a process is 5 GB, since it would be unusual that all
processes reach the worst case maximum and reach it at
the same time.

nextpnr building on 32-bit architectures without any memory-saving
tricks also makes me wonder how much margin above any actually
observed number is in the "worst case" 5G value.

The "--require-mem 5G" in nextpnr was added for OOM on the alpha
qemu builders, this was not a problem on any release architecture.

If this is the final link then that is wrong,
and --require-mem 3G,0,,1G would be correct.

I'd call it a bug in guess_concurrency that it doesn't support the
zero value needed here.

I've seen the vast majority of FTBFS on our buildds in the past
10 years, and builds failing due to being OOM killed on a release
architecture happens basically never.

Using guess_concurrency only when an actual problem has been observed
would be good, but that's not what is happening.

The problem is not what guess-concurrency supports, the problem is how
it is being used.

The package with "--require-mem 16G" builds successfully on sh4, where a
process has 2 GB address space.

But building slowly is much worse than building faster, and that's what
actually happened here.

riscv64 is already considered a problem by the release team due to
buildd speed, there is no margin for even slower builds.

Nearly all large packages do, and many of them were already using
--parallel before compat 10 became available 10 years ago.

I did a lot of FTBFS reporting and fixing when packages switched to
compat 10, and I do remember around ~1% of all packages having breakage
due to that.

For tiny old software where it was not trivial to fix a handwritten
Makefile I was just adding --no-parallel to many packages.

Some odd package like sqlite3 taking 3 hours instead of 1 hour on
riscv64 is not a real problem.

Many packages in two large transitions (fmtlib, boost1.92) are currently
waiting for vtk9 to finish building before they can start building, and
adding guess-concurrency usage did increase the build time of vtk9 from
4 days to 6 days. If that alone is not already the final straw for the
release team to remove riscv64 from the forky release architectures,
then an even larger buildd backlog due to increased build times for
many packages where people add new guess-concurrency usage would do it.

No, the only reasonable fix is to stop guessing memory usage unless this
is needed to actually fix an observed problem.

I am not highlighting this because it does not happen.

The first developer who sees a FTBFS on our buildds and looks at why it
fails is very often me, and alpha is the only architecture where we
sometimes have this problem.

That's only true when there was an actual problem observed that needed
fixing.

My observation is that builds failing by OOMing the buildd is not a
problem we actually have on the buildds of release architectures.

In the cases where this ever happened the maintainer already added a
--max-parallel or -j years ago. Replacing a hardcoded or open coded 
value for that with guess-concurrency makes sense, but this is not
how people are using guess-concurrency so far.

The 6 packages that are currently using guess-concurrency are not a
promising start.

Among the first 6 packages that use guess-concurrency is one that added
"guess_concurrency --require-mem 16G" but builds on an architecture with
2 GB address space - when talking about how/when to use guess_concurrency
it is important to keep in mind that the technical skills of many DDs
are rather poor.

Humans guessing numbers and then hardcoding them in debian/rules where
they might stay unchanged for decades - that only makes sense when it
is fixing an actual problem today.

If replacing the hardcoded 1 in --max-parallel=1 in mlpack with
something like "guess_concurrency --require-mem 4G" would reduce
the build time on riscv64 from half a week to 1 day, then that
would be a usage where guess-concurrency would bring real benefits.

cu
Adrian

Hi Helmut,

thanks, that looks better.

cu
Adrian

Hi Helmut,

Yes, this solves the issue. It still return a bit less than the RAM
available, depending on what is running on the host on the moment
guess-concurrency is run (e.g. puppet run), but this is already a
significant improvement.

For the vtk9 case, that will increase the number of jobs on a host with
16GB RAM from 2 to 3. The next step is to work with the maintainer to
define the correct required memory size.

Regards
Aurelien

Hi Adrian and Helmut,

I think most of the confusion comes from here. I see two main use cases:

1) Avoid an OOM on a typical developer laptop with a lot of of cores,
but not that much memory, especially given there are many other
processes running. In that case we the number of jobs should really
determined from the maximum memory used by a single process during the
build.

2) Optimize the build time by limiting the use of swap for active
processes. In that case the amount of memory is just a typical maximum
usage. We accept that some gcc/g++ process will just use swap if it
still improves the overall build time. This is what we used on the
buildd so far, they have enough swap that an OOM is almost impossible
(at least for the official buildds).

It is not fully clear what use case guess-concurrency is designed for. I
would say for the scenario 2, given it doesn't take the swap into
account (except for backing tmpfs with the change in git) and it returns
1 instead of failing if requiring more memory than available.

I then looked at a practical case, the vtk9 package. It makes almost
11000 calls to gcc/g++ with an average of 215 MiB, and a median of
171 MiB.  Only 22 calls use more than 2GiB and only 3 calls use more
than 4 GiB, with a maximum of 6.1 GiB. In the scenario 1) we probably
want to compute the number of jobs using 6.1 GiB, while in the scenario
2) we probably want to compute the number of jobs using 2 GiB or less as
the gain of doing most of the build with more jobs it not lost by
*possibly* running a few big jobs in parallel.

Regards
Aurelien

We believe that the bug you reported is fixed in the latest version of
guess-concurrency, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1149922@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Helmut Grohne <helmut@subdivi.de> (supplier of updated guess-concurrency package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Mon, 12 Oct 2026 17:50:56 +0200
Source: guess-concurrency
Architecture: source
Version: 0+20260512
Distribution: unstable
Urgency: medium
Maintainer: Helmut Grohne <helmut@subdivi.de>
Changed-By: Helmut Grohne <helmut@subdivi.de>
Closes: 1149922
Changes:
 guess-concurrency (0+20260512) unstable; urgency=medium
 .
   * Clarify how guess_python can return None
   * Revamp --detect semantics
   * Add a --debug option
   * Change the estimation of available memory to consider tmpfs as
     swappable (Closes: #1149922)
   * Document downsides of using guess_concurrency in Debian
Checksums-Sha1:
 f657b49c937e60a145033cfb2686ba621dffad11 1763 guess-concurrency_0+20260512.dsc
 9b393c92b109144b70f1f5befe0bc6f456b65445 7028 guess-concurrency_0+20260512.tar.xz
Checksums-Sha256:
 fdc6ac221961d170d54fcd3770c4530675b2dad151d7b3676e62f0dd5c3bf84a 1763 guess-concurrency_0+20260512.dsc
 1d9dfa6544c7895403b322f857fab6002c287c4de44ca7a0b5de1918c9a0e58e 7028 guess-concurrency_0+20260512.tar.xz
Files:
 0947760dd4e38f2479fd2c88141df5de 1763 devel optional guess-concurrency_0+20260512.dsc
 88b7363f25bd1aea098351985fccdbb6 7028 devel optional guess-concurrency_0+20260512.tar.xz
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEETMLS2QqNFlTb+HOqLRqqzyREREIFAmrNAgMACgkQLRqqzyRE
REL0Eg//fJQt3q5ujkXW5SIlmWwuJLpQhuSAQ7icbRsztxXaMGnn8mdekTfkA+dv
AjxY/x37mhfS+M+VN0icz++Akyh2by9STv0DBvWJPngdFjneYL00B3mpoY++oTh9
WMvmvO70z9U+r5nGCeSjjaIDclx5IIx9gpNhMjcyHVJmzPeCGgP8h7FOFRElfLFJ
If+poR3wGnfdakE9Qa3Mh0xfZkbR0d3E/URUuceIGyT4C+n2/7y3vXFb7LjQ3BxX
itVnta9N8zDMnimCCoobXeSdiQckpTN0zsTX+zr4L/EJJoCdww1fsFXR1Yi0Bthq
aji3BXGN0O3OqU63K3aeJygAKxlGdhX/gnCIEo9h69KcqXimwZwLtyT2iBBV7xG0
N5Dh1G6AK1IUmUUMGBu7u//amYlgTjfU4d1SbtML4Zyezl6o0Wke3cJxI6mG0ZI2
WDY4tbE4kRq+DqFanxHQAkvhQzYsPcry29MOvCM2Ag3wB0uSNQN/vmVWfRgwEoRU
NTfj2BVO7VmmN/OlD+KL8HAzhDxAWxeHipJkHnlg9/ECQpRHyPoJm2RorkAknXrH
rPyVcmhCu8LmBHqFdEU+hVl07nlHXGCQ2c9nZYZE82Hxv+61KjNzbyo0SZBLqY/l
D3gjWoisWsyhhHz5iFuyJ8biWxFLn6bmG3XrKmtx6VAP5QFqeIM=
=ivCv
-----END PGP SIGNATURE-----

Hi Aurelien,

Thank you for spelling out the use cases.

Given that I build very many packages locally (and some OOM), this
definitely is a use case I had in mind.

Note that Santiago Vila has been initially using quite small VMs for his
archive rebuilds and encountered quite some failures that way. I'd like
us to not only consider Debian's buildds as relevant machines for
building but making builds just work locally such that users are enabled
to easily modify packages.

This is not a use case I had in mind, but it sounds sensible to me. Note
that when you get to a point where compilers are being swapped, you
typically already swapped out all of tmpfs. I guess my own machines
simple have insufficient swap. On a system with 32GB swap a fair number
of packages stopped building about three years ago. 64GB swap likely is
insufficient today. Does that match your numbers?

I fear it is more complex than this. We need to consider two variants of
the use cases here. One is when the build tree is backed by tmpfs and
the other is when it is not. Note that guess-concurrency does not
presently distinguish them. When using tmpfs, a lot of RAM can be
occupied with page cache. Several packages use more than 64GB of build
trees. This needs to fit into swap. So in the tmpfs case, the assumption
kinda is that swap is able to hold all of tmpfs usage and it is not
accounted otherwise. When not using tmpfs, swap could be accounted
indeed. Though as you point out in 2) doing so slows down the build at
some point.

Tricky. I don't think avoiding swapping at all cost is the way to go. A
pessimistic parameter would be --require-mem 6g,5g,,4g,2g. On a machine
with 16GB, this would likely produce a concurrency of 2. That quite
definitely is not an ideal outcome.

I have to admit that I'm not quite sure where to go from here. Quite
evidently, the model that I used is too simplistic and the way it is
being used in Debian causes harm to riscv64. I'm sorry about that.

Last but not least swap is not all the same. Some build machines swap to
rotating storage (maybe none of Debian's) while most swap to NVMe.

I shall upload the pending change soon to provide some relief, but I
doubt it to be sufficient. This bug will be marked as closed. Don't
hesitate to reopen or open a new one if you think we need to change
more. But we also need to change e.g. src:openbabel as --require-mem 16G
is not sensible.

Thanks for bearing with me

Helmut

Hi Helmut,

I understand this use case, as long as it doesn't mean having to
significantly increase the resources we need at the build daemon level.
Hosting build daemons is difficult those days. And buying additional
hardware even more.

It seems you are mixing swap and size of the tmpfs. In normal use case,
you need to have at minimum as much swap as the size of your tmpfs used
for building. Then you need extra swap for handling the cases were big
processes run concurrently, even if the goal is to limit that case as
much as possible.

The minimum disk size needed to build all packages (being ext4, tmpfs or
other fs), is around 150G on arm64 (for the linux package), and around
100GB on other architectures for linux, rustc and/or gcc-16. Therefore
the tmpfs is sized to 150GB on all architectures except s390x where it
is 100GB (as we currently do not have enough more diskk space).

Then we configured the swap as tmpfs size + 20GB, so 170GB on all
architectures, except s390x where it is 120GB.

We just use tmpfs because it's fast, doesn't need special file system
option or wrapper like eatmydata to optimize the speed, doesn't break
and always come clean after a reboot. In addition it allows a
significant part of the packages to be built without disk I/O, an for
other packages to have the I/O performed slowly in the background.
Finally even if it lacks some features, it's an easy way to standardize
the filesystem and ensure reproducibility.

But to me the question of swap vs memory vs number of jobs, has to be
decorrelated from tmpfs, the same issue exists with an ext4 build
directory. At the end you should just ensure that the swap covers at
least the size of the tmpfs. Seen otherwise, just subtract the size of
the tmpfs from the swap to see how much memory can be overcommitted for
the running processes.

You are trying to define the required memory from 0.03% of the GCC calls
(admittedly a bit more if we look at the CPU time). And I only looked at
all the calls, not the makefiles. Maybe there is no chance at all that
they run concurrently.

riscv64 is currently the most affected, and to be fair, we need to
upgrade the hardware asap, but it's not always easy. That said if more
packages starting using guess-concurrency in a bad way, other
architectures will get affected relatively quickly, as it tends to be
used in big packages.

We only have 2 build daemons with rotating storage. All the other are
backed by NVMe, except s390x which is backed by amazingly fast storage.

But at the end, even in the vtk9 case, the goal is to not get the
running process to get swapped. It's just there for the unlikely case
where the 3 bigs jobs end up running in parallel.

Thanks, that's already a significant improvement.

Regards
Aurelien

Santiago also tried hard to make FTBFS on single-core RC, claiming this
would be important for users doing QA work.

The result of the single-core discussion was that anyone rebuilding
packages has to be as close as possible to the specs of our buildds.

2-4 GB RAM / core might be a reasonable default setting for
--max-parallel for anyone aiming at touching random packages
on a machine with insufficient RAM.

More interesting would be the amount of RAM per core.

The i386 buildd where vtk9 has this near-linear scaling has 6 cores and
AFAIR 12 GB RAM.

2 GB RAM / core is in my experience a bit tight for building software,
especially when it's not a 32-bit architecture like i386 where the
address space cannot be more than 4 GB per process.

riscv64 is just the tip of the iceberg, because it currently has no
margin of error for build times.

Similar to your sqlite3 example, vtk9 taking 4:50 hours instead
of 1:45 hours on i386 alone is not a big issue.
A general pattern of doubling the build times of large packages would
be a problem for all release architectures.

If guess-concurrency was used in a way that doubled the build time of
chromium, then we'd immediately need more x86 buildds.

cu
Adrian