Hi,

more and more packages implement a technique called profile guided
optimization. The general idea is that it performs a build that is
instrumented for profiling first. It then runs a reasonable workload to
collect profiling data, which in turn is used to guide the optimizer of
a second build which is not thus instrumented. The idea is that this
second build probably is faster than a regular build.

Quite obviously this approach completely breaks cross building. It also
is unclear how it affects reproducible builds since such builds depend
on the performance characteristics of the system performing the build.
This makes it very obvious that the pgo technique has downsides that
warrant disabling it in some situations.

A number of packages have agreed on disabling such optimization when
DEB_BUILD_OPTIONS contains nopgo. I'm aware of the following packages:
 * binutils
 * cross-toolchain-base
 * gcc-VER
 * halide
 * pythonVER

I'll also be filing a patch for foot to support this option.

Is this sufficient coverage to document the option already? If not, this
bug report can serve as a central point for discussing it and its
adoption.

Proposed wording:

    This tag requests that any optimization performed during the build
    should not rely on performance characteristics captured during the
    build. Such optimization is usually called profile guided
    optimization.

The proposed tag intentionally is fairly narrow. It does not cover link
time optimization. It also does not cover the case where profiling
information is recorded ahead of upload and included in the source
package[1]. In both cases, neither cross building nor reproducibility is
impacted.

As for cross builds, it is not clear to me where we want to put the
responsibility to disable pgo. At the time of this writing, most
packages automatically disable pgo when performing a cross build. On the
flip side, we could have any cross builder set nopgo like they set
nocheck already. Doing so would allow performing a pgo-enabled cross
build to i386 on amd64 while still benefiting from the larger address
space for instance. I consider this aspect to be a separate matter
though.

Helmut

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2022-June/002638.html

Helmut Grohne <helmut@subdivi.de> writes:

Yes, I think that's plenty.  I would love to have Policy be a bit more
proactive about documenting things like this.

Could you specifically mention cross-building (and possibly reproducible
builds) as an example of why someone may want to set this option?  I think
having those sorts of explanations add a lot of value to Policy, since
they help explain to the reader how Debian is designed beyond just a
mechanical set of instructions.

If you have a chance, feel free to send a proposed diff to add this to the
DEB_BUILD_OPTIONS section.

It is worth noting that the performance (execution time) of a
build-system does not matter for profiling, so it is possible to achieve
reproducible builds with PGO enabled. It is just hard.

See https://build.opensuse.org/request/show/499887
linked in
https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/pgo

In that gzip patch we even had to hide the name of the temporary file
from gzip to not get variations from a 'tolower' call that would be
optimized for different amounts of upper/lower-case letters.

Parallel builds and variations in ordering are also problematic, because
some of the performance-counter-logic in gcc is not commutative, so
running A and B calls produces different results from first calling B
and then A.


With all that said, in openSUSE we also have a %do_profiling value to
disable PGO for gcc, python and some others, because these profiling
runs are too large to make deterministic.

Ciao
Bernhard M.

Hi!

Is it possible to save the profile data to reperform the second build later ?

Cheers,