#1110404 Consider increasing builder worker parallelism

#1110404#5
Date:
2025-08-04 16:19:38 UTC
From:
To:
Currently we have only one worker.  We may want to increase this,
even though we have only one vcpu.

Each worker will do some combination of local "disk" IO, network
activitym and CPU use.  Effective use of these resources probably
means greater parallelism than the vcpu count.  (I think very little
of our processing code makes effective use of multicore pallelism.)

My experience is that maximum performance with a mixed workload on a
desktop-sized machine occurs at about 1.5x cpu "overcommitment".

This analysis depends stronly on the situation on the underlying
physical host, which presuably has more than one pcpu and is
presumably doing other work too.

Another argument in favour of greater parallelism is that it can
significantly reduce maximum latency especially for small packages
mixed in with a few big ones.

My guess is that we should:

 - Plan to change tag2upload-oracled to default the number of workers
   to `ceil(nproc * 1.5)`, for a minimum of 2.  (Not sure if
   1.5 should be configurable.)

 - Consider the RAM and disk space implications and talk to DSA about
   that.  (We'd almost certainly need more disk space both for the
   additional running containers, and for working space for multiple
   possibly-large source packages.)

We can this put off until we start to see nontrivial queueing.

Ian.

#1110404#10
Date:
2025-08-04 21:41:32 UTC
From:
To:
Hello,

Given that we have to ensure there is enough RAM and disk space before
doing any increases, I would have thought that leaving the default as 1
and just passing --workers=2 in the service file would be better than
calculating it based on nproc.

For example, if one day DSA were to increase the cores but not the other
resources (perhaps as a side effect of some other maintenance they were
doing) the service would then try to start more workers than the host
could really handle, and things would go wrong.