Hi,

with the upcoming time_t & friends 64-bit transition, dpkg-buildflags
will be used to configure the ABI in use.  Thus all compiler
invocations *must* use the flags specified by dpkg-buildflags to avoid
ABI inconsistencies like this one:

  struct T { time_t a; time_t b; };

If this struct is accessed from both `libfoo1t64` (built respecting
dpkg-buildflags and thus time_t is 64-bit) and `bar` (built by a user
invoking gcc themselves), the result is probably not what one wants.

Thus `dpkg-buildflags` *must* be used by all packages *and* all users,
including users building their own software.  There is one exception
when libraries provide both 32-bit and 64-bit time_t ABIs like glibc
itself (but I doubt there are many of those).

Any compiler invocation missing these *should* be a serious bug.
(This should probably be mentioned in user documentation as well.)

Currently Debian Policy does not mention dpkg-buildflags at all.

Ansgar

This decision comes from the wrong premise that the use of dpkg-buildflags
is universal, which is not the case. Hence it needs to be reconsidered.
There is not magic that will make all packages use dpkg-buildflags consistently
in the timeframe of this migration.

No, it is required that packages use correctly the right compiler flags.
Since packages need to be reuploaded anyway this is not a real issue.
dpkg-buildflags is not required for that, and does not necessarily achieve it
either, since upstream build system might not honor the environment variables
CFLAGS  etc. consistently.

This a different issue than mandating dpkg-buildflags.
dpkg-buildflags leads to build flags which are significantly different from
upstream, that have not been tested by users building from source, and they
can change without notice. It is very reasonable not to trust them.
In general we discourage diverging from upstream for similar reasons.

I offered #724621 as an option which would let packages maintainers control
which buildflags are used while still providing the benefit of buildflags.

Cheers,

If packages do not use dpkg-buildflags, but use any ABI changed by the
transititon, they will just be broken and must already be fixed.

The interface for the compiler flags is dpkg-buildflags. See
the transition annoucnement.
I did not come up with this interface, but am not aware of an
alternative.

(C++ ABI changes were handled differently, but this transition did
choose a different way; I'm not aware of the details why.)

Packages must ensure that these flags are passed correctly to the
upstream build system. As mentioned above, any package not doing so is
already broken (or will be by the transition).

You can also establish another interface, but currently the only one
I'm aware of is dpkg-buildflags.

Ansgar

Ansgar <ansgar@debian.org> writes:

This seems correct to me. We can at least make it a should in Debian
Policy, and I can see the argument that it might be a must given the
issues with time_t on 32-bit architectures.

I'm not sure the best place to start with this given that we currently
don't talk about dpkg-buildflags at all, as you point out. Maybe it's as
simple as adding a 4.9.3 section along the following lines?

This is very rough wording; feedback welcome. In particular, I'm a bit
worried this is too strong when it comes to flags from dpkg-buildflags
that are fine to override, such as changing the optimization level.

``debian/rules`` and dpkg-buildflags
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Source packages that invoke a compiler that is supported by
:command:`dpkg-buildflags` should use that command to obtain the default
flags to pass to the compiler. See :manpage:`dpkg-buildflags(1)` for more
information. In many cases, this is handled automatically by the ``dh``
tool provided by the debhelper package.

Packages that require special handling of compiler flags may selectively
override the results of :command:`dpkg-buildflags` or avoid that command
entirely, but only if the package maintainer is prepared to track future
changes to the default compiler flags and update the package accordingly.
This is normally only appropriate for packages such as ``glibc`` that have
unique build considerations.

I am not in favor of requiring the use of dpkg-buildflags in its current form.
At least dpkg-buildflag should be versionned so that package maintainers have
the ability to validate any compiler flags change before it is applied to a
package.

Using different compiler flags than upstream is not very different from
patching the source code and should be treated with the same care.

The concept of building all packages with the same flags do not really make
sense anyway.  Packages serve different purpose.  dpkg-buildflag targets
'security flags' which are not relevant or suitable for HPC software
whose users require performance instead.

Cheers,
Bill.

Bill Allombert <ballombe@debian.org> writes:

While I do agree with you on everything here, when I went back to read
the proposed text from Russ, I think it works since it allows
maintainers to opt out when needed.  Did you intend your e-mail as
opposition to the proposal?  If so, could you explain why the opt out
isn't sufficient?

My perception is that opt-out is necessary and useful (for the reasons
you explain) for many more packages than glibc though.  I'm thinking of
crypto packages like libmceliece, lib25519, lib1305 and probably many
others where GCC/CLANG compiler flags are really security sensitive
decisions, and one removed or added flag may mean that a severe timing
side-channel is introduced.  I believe there is some protection in the
libraries I managed (careful checking of generated assembler output) but
I expect these to be excellent examples compared to many others which
just assume the compiler did what the upstream author intended to tell
it (which dpkg-buildflags can break).  I agree many other packages will
need it for performance reasons, or other matters rather than security.
Further, I think this set of packages will grow over the coming years.

Still, I suspect more use of dpkg-buildflags will be generally useful,
so I find the proposal good [unless I'm missing something].

/Simon

Simon Josefsson <simon@josefsson.org> writes:

Correct, the intention is not to say that everyone must do this, which is
why it's a should, not a must. My intention is to make it a should with an
exception: *if* your package is particularly sensitive to compiler flags
for some reason, *then* you can ignore dpkg-buildflags (or selectively
override it), but you are now responsible for keeping your compilation
flags up-to-date.

I think this is one of those cases where the goal is to get the 95% of the
packages that probably shouldn't have strong opinions about compiler flags
onto the standard machinery (or, to be more accurate, to document that we
have largely already done this and give the stragglers a bit of a push),
without ruling out the remaining 5% of exceptions that have good reasons
to control their flags. It's also secondarily to try to say that it is
worth the effort to plug dpkg-buildflags into the build system of packages
that aren't easily supported with dh.

It's possible that my proposed wording is too strong, but I'm hoping we
can build a consensus that something *like* this is the right idea and
then find the right wording for the exception. Being able to add new flags
via dpkg-buildflags is immensely powerful makes transitions much easier,
so I think it's a good idea to ask maintainers to not break this without a
good reason.

Also, hopefully some of these cases can be handled via dpkg-buildflags
feature areas rather than abandoning the tool entirely.

Hello,

Russ Allbery [07/Feb 10:31am -08] wrote:

LGTM, though as suggested by subsequent discussion, maybe you could
expand the last sentence's discussion to include reference to the other
sorts of packages that Bill and Simon raise -- especially the crypto
libs case.

Sean Whitton <spwhitton@spwhitton.name> writes:

Here this is turned into an actual patch.

I would like to add that the flags set by dpkg-buildflags are not defined by the
Debian policy.

Cheers,

Bill Allombert <ballombe@debian.org> writes:

I think that's true, if I understand what you mean by "defined," but I'm
not sure why it matters. Could you say more?

A lot of Debian packages contain libraries and headers files that can be used by
Debian users to compile software. These packages are not supposed to be restricted
to the purpose of building other Debian packages.

On the other hand Debian users are not expected to use the same flags as
dpkg-buildflags set, so the libaries provided by Debian should not require the
use of dpkg-buildflags.

In the original issue leading to this bug report, it was suggested to add a
flag to dpkg-buildflags that would change the ABI on some architecture.
Doing so would have required users to use the same flags when compiling software
using the libraries.

Ultimately, the flags were set directly at the compiler level which sidesteps this
issue as long as the user use a Debian-provided compiler.

In conclusion, any requirement to build Debian package with a certain compiler flag
must address the issue of user-build software and not rely on dpkg-buildflags.

Thus I think this sentence to be overstated:

'the package maintainer is prepared to track future changes to the default
compiler flags and update the package as necessary'

Cheers,

Bill Allombert <ballombe@debian.org> writes:

Oh, I see. Yes, this is a very good point. That means that the utility of
dpkg-buildflags is probably limited to cases where we want to set default
flags for Debian packages by (lowercase) policy, but not cases where we
want to change the ABI in ways that users of libraries need to know about.
In other words, cases where the flags are desirable but not mandatory (and
the original bug to which I'm responding was focused on mandatory flags).

I do think there are a lot of remaining cases that fall into the former
category, though. The one that comes up a lot is hardening flags, which we
have sometimes pushed into the compiler and sometimes managed with
dpkg-buildflags. Another use case is correctly handling reproducible
builds, which downstream users are not as concerned with.

Given that, I think this part still does apply?

For example, if we add new flags for reproducible builds, the packages
that don't use dpkg-buildflags really should pick those up, because
reproducible builds is a general distribution goal. "Should," not "must,"
but it feels to me like should is appropriate here.

I suppose if we're committed to always changing the compiler, then this
isn't needed and dpkg-buildflags is more of an "ought," but my impression
was that we preferred to use dpkg-buildflags where possible.

But then tracking Debian policy is sufficient (and mandatory already),
so tracking dpkg-buildflags is not needed.

To get back to my previous message, a change to dpkg-buildflags cannot
by itself change Debian policy, thus it is not necessary to track it to
follow policy.

Cheers,

Bill Allombert <ballombe@debian.org> writes:

Hm, am I missing something? I don't see any mention of, e.g.,
-ffile-prefix-map in Debian Policy. So far as I know, that is tracked
primarily in dpkg-buildflags.

Right, that's the current state that this change would change. That's the
point of this bug.

Reproducible build is required by policy, while -ffile-prefix-map is a mean to
that end, not a goal in itself.

We could update Debian policy to mandate the effect provided by
-ffile-prefix-map (reproducible filename in log) more precisely than just
mandating reproducible build (which is not well-specified in Debian policy).

Cheers,

Bill Allombert <ballombe@debian.org> writes:

Sure, I agree, we in theory could do that. In practice, I suspect we
won't, though, even if we intend to do so.

I see this as part of a long trend in Policy maintenance. Originally, in
part due to its origins in a world that had far fewer tools, Debian Policy
(by way of the Packaging Manual) was a fairly comprehensive guide to
everything that went into creating a Debian package from the ground up.
With only that one document, in theory you could work out how to create a
valid Debian package from first principles and common UNIX tools.

Over time, it has stopped being that, for a variety of reasons. One major
reason is that the tools have both become much better and more universal,
and the underlying tools have their own documentation. Spending Policy
maintenance time exhaustively documenting precisely what dpkg-buildpackage
does under the hood has therefore been less and less appealing because the
audience for such documentation is tiny and dpkg is already maintaining
largely parallel documentation. People therefore stopped working on it,
that documentation in Policy in some cases fell out of date, and it has
only been fitfully updated.

The same has been true for new requirements. If there is a tool that
already does what Policy wants to happen, we have increasingly been
documenting use of the tool rather than describing in detail precisely
what the tool does. This is a tradeoff, since it does make it harder to
write new tools, but I think it's a realistic tradeoff given that we are
not suffering from idle hands or an excess of resources.

In a world in which Policy is not keeping up with very long-standing
changes (triggers, multiarch) that we have not found the time to document
properly and which *are* packager-facing and cannot be easily addressed by
using a standard tool, and have also not been keeping up with the regular
requests for changes, I'm looking for more places where Policy can specify
the high-level desired state and delegate the details to a standard tool.
This lets us focus Policy maintenance on the places where we provide the
most benefit (telling packagers about things they need to pay direct
attention to) and not spending time documenting the internal behavior of
tools that by and large one can simply use.

That general desire does not imply that any specific example of a tool
will fall on one side or the other of a line. For example, I am still
reluctant to remove from Policy all the things that debhelper handles
automatically; I think it's healthy for Policy to be an (incomplete,
necessarily, due to lack of resources) specification that debhelper
implements. But the dpkg suite is one of the primary examples of a set of
tools to which we delegate a lot of details without exhaustively
documenting them, in large part because dpkg maintains its own exhaustive
documentation.

I'm not sure if I understand the primary source of your objection. If the
primary issue is that you would prefer Policy to be comprehensive and not
delegate the details to tools, well, I understand that and I can see a lot
of ways in which that would be ideal, but I don't think it's realistic. If
the objection is to having to sync with dpkg-buildflags specifically
because it's another place you have to look besides Policy, I'm not sure
that's really that bad? For the packages that can't just use
dpkg-buildflags directly for whatever reason, you can just occasionally
run dpkg-buildflags from the top level of your Debian packaging tree and
eyeball the relevant flags and see if there's anything new you don't
recognize. I would expect this to take at most a minute or two for
packages where the maintainer is already so deeply familiar with the
workings of the compiler that they're doing custom flag management.

I really do not want to have to push through a Policy change every time we
discover a new compiler flag that is helpful for, e.g., reproducible
builds. I would rather let other people handle that in a division of labor
and focus on places where Policy is adding unique value. Also, just on a
practical sense, I don't think we *will* handle such updates in a timely
fashion, just based on historic evidence, although of course I always
aspire to do better.

I completely agree that it would be a mistake for Policy to say that one
*must* use dpkg-buildflags or sync to its output, but I do think *should*
is the appropriate level and the appropriate practical compromise here.

This trend to be short-sighted, as this will lead to the ossification of
Debian. Debian Policy should stay true to its purpose.

All the packaging improvement that occured this last 20 years were possible
because Policy left a large latitude how packaging was done and allowed
improvements to happen.

One reason Policy is lacking is that the TC has sidestepped Debian policy in a
number of decisions without requesting that a detailed technical policy was
devised by the proposers (which then leads to ambiguities, disagreement and more
referall to the TC).

In this instance, Debian policy should not create an incorrect expectation that
dpkg-buildflags is used by all package or that it is suitable for all packages,
at least in its current unversioned form.  There has already be a MBF claiming
just that.

Cheers,

Russ Allbery [08/Feb  9:17am -08] wrote:

Seconded.

I don't think Bill's valid point about being careful with our flags such
that we don't cause problems for compilation done by users means we need
to change anything in your proposed wording.

Russ Allbery [10/Feb  1:10pm -08] wrote:

I agree with all this.

For the record:

It is not possible for a change to dpkg-buildflags to affect the status of the
package with respect of Debian policy, so tracking future change is simply not
needed, thus including this sentence is not in the spirit of Debian policy.

There is an easy change to dpkg-buildflags that would all ow all packages to use
it even if they cannot apply the default flags: allow
export DEB_BUILD_MAINT_OPTIONS=-all
to disable all built-in flags without overriding DEB_flag_SET/APPEND/etc.).

Cheers,