Hi,

Ben asked me to file a bug report with the linux package describing the
issues I see with linux-libc-dev being Arch:all.

A. Multi-Arch: foreign is a lie

linux-libc-dev declares being Multi-Arch: foreign, but it does not live
up to the involved promise and cannot. Earlier, a dependency on
linux-libc-dev:hurd-i386 would be considered unsatisfiable. With the
current setup, it is trivially satisfied even though that doesn't make
sense in any way. In a similar way, linux-libc-dev:sparc is now
considered satisfied. While that makes some sense, the actual package
does not ship any sparc files as 32bit sparc is dead. As such,
linux-libc-dev now makes dependencies look satisfied when they should
not be.

B. Inability to figure out what architectures linux-libc-dev supports

When linux-libc-dev was arch:any, architecture bootstrap would have to
cross build it. Now that it is arch:all, architecture bootstrap can use
the existing package. Sometimes. For release architectures and a few
more, it is good, but fore exotic and dead architectures (e.g. sparc),
one has to rebuild it with a modified configuration. In order to support
figuring out when a rebuild was required, Bastian added provides for
linux-libc-dev-$arch-cross due to my request. Neither of us saw how that
would break cross toolchains and these provides are now reverted as a
result. In any case, we're now back to the state where one cannot figure
out whether a rebuild is needed.

Ben asked me whether linux-libc-dev could simply support all relevant
architectures, but there often is a check&egg problem. The linux package
can only support architectures that dpkg knows about, but we want to
perform test bootstraps way before adding a triplet to dpkg and one of
the first things a test bootstrap does is building linux-libc-dev. So I
generally expect that linux-libc-dev can never provide support for all
architectures of interest and it shouldn't have to. Only once the
bootstrap is tested and the parameters are known do we want to file the
patches for a new architecture.

C. Inability to produce a linux-libc-dev-$arch-cross package

Now that linux-libc-dev-$arch-cross is no longer provided, dependencies
on it are no longer considered satisfied, but the toolchain packages
express dependencies on such packages. The architecture bootstrap
tooling does not use cross-toolchain-base and instead creates such
packages using dpkg-cross, but dpkg-cross cannot operate on arch:all
packages. Therefore, we can no longer turn linux-libc-dev:$arch into
linux-libc-dev-$arch-cross as is required by the cross toolchain.

I note that I do not have a good understanding why the change from
arch:any to arch:all was done. It does seem to reduce the cumulative
size consumed by binary .debs on mirrors and note that a similar effect
can be achieved by having a linux-libc-dev-common for the architecture
generic headers and a per-arch symlink farm in linux-libc-dev.

Ben, can you give some feedback on whether this report is useful for you
in moving things forward? I'm not sure how actionable this actually is
and welcome ideas on how architecture bootstrap should be adapted to
cope with the changed packages.

Helmut

Hi

What you describe here is a headers only library.  It is consensus
within Debian that those exist and are shipped as arch:all.  They don't
provide further architecture clues in the dependencies, but you need to
know or convey this information by failing to build.

Sparc is dead, so any Debian using Linux that does not support it any
longer will not provide the basic support to use it.  Sure, you could
hold on to an ancient linux-libc-dev, but this is then not longer
Debian.  Heck, we even consider running a mix of stable and unstable as
unsupported.  Why would we consider running a mix of
Debian-supporting-sparc and Debian-not-supporting-sparc a supportable
configuration?  Technically all of this can work, but still not
considered supported.

All the examples of yours will immediatelly fail to build anything with
it.  So what harm is done?

I have no idea what you are talking about.  I added the provides to
replace the existing cross packages with the functional identical
linux-libc-dev.  You where not involved in this at all.

I have to disagree.  With this setup, we can support architectures that
are neither known by dpkg nor dak.  This was not possible before and
required patching, aka making a derivative of the package.  You can
still patch and rebuild it how you want and inject that modified package
into your workflow.

But now you can have a port without hand patched linux that is built in
a non-standard way and relying on internal and unstable package API,
just be talking and getting it added to the package first.

This sounds like a bug in dpkg-cross, as it uses a not agreed on
interface to do it's work.

No, it can not.  We can not have arch:any -> arch:all dependencies
within the same source package this deep in the toolchain, especially as
those require = dependencies.

If we talk about size, please go to gcc, which ships unstripped
binaries parts of the time.  We don't need to talk about a highly
compressible 10 MB installed (up from 7 or so before) package.

Bastian

Hi Bastian,
...

This is practically untrue. Even with current unstable packages, the
sparc (32) bootstrap still succeeds. But then the use of sparc here was
an example rather than a concrete issue. linux-libc-dev claims to
support an architecture that it technically does not. That problem works
the same way for obsolete architectures (such as sparc) as it does for
new ones. The harm is being done. Quite simply, linux-libc-dev should
ensure that dependencies on it are not considered satisfied when the
architecture of the dependency is not practically supported.
...

Ok, you may see it that way (except for the "functional identical" part
that evidently doesn't hold). I don't think it matters much who caused
the mess we find ourselves in now. Let's skip the history bit.

Fair enough. Please add arc, cksy, mipsr6el, sh3, sparc, and
musl-linux-any. Thanks in advance.
...

I stand corrected. dpkg-cross actually can convert arch:all packages in
a reasonable way. Please consider item C a non-issue.

I read this as you saying that size was not a motivation to change
linux-libc-dev from Arch:any to Arch:all. But then, I do not understand
the benefits of the change and why it was done in the first place.
Obviously, we are still dealing with the fallout of an Arch:all
linux-libc-dev after more than half a year. This bug is evidence of that
dealing and demonstrates the significance of the downsides. Please
enlighten me about the benefits.

Helmut

[...]
[...]

Currently gencontrol.py runs dpkg-architecture to find the multiarch
triplet for each Debian architecture.  So it's not possible to add any
architecture that's unknown to dpkg in unstable.

Are you proposing to extend our config schema so we can define those
triplets within src:linux?

Do we really want to be on the critical path for the early stages of
defining a new architecture, including any renaming that might happen
before it's added to dpkg upstream?


Ben.

[...]
[...]

We would need to know the multiarch triplet for csky.

Ben.

Hi,

I skipped replying here earlier, because I felt like I would not add any
content. Stefano indicated that my impression was wrong. Hence replying
now.

In retrospect, the use of irony was not the best of moves here as it did
not add clarity to a discussion that was already heated. I'm sorry about
that.

Yes, exactly. Let me reiterate the reasons for why I prefer a different
solution here.

  * In general terms, the multiarch triplet is something that
    can change (and has changed) during the early phase of an
    architecture bootstrap.
  * When an architecture becomes available in toolchain packages, it is
    not clear whether it takes off. or1k very much looked like the next
    thing and now it is deleted. The story for tilegx is similar.
    Initially, riscv64 looked a lot like or1k. In not adding these to
    src:linux (and removing them later), we avoid a pile of unnecessary
    interaction. The excessive addition of architectures also interacts
    with #1081826.
  * It is very useful to locally add a new architecture and try a
    bootstrap to give feedback to upstreams as to what piece is missing.
    Not requiring src:linux to cooperate significantly speeds up this
    evaluation process.

I generally see src:linux as one of the later pieces relevant to
architecture cross bootstrap in the sense that while it is one of the
earliest packages needed for its headers, adding real support is often
delayed, because too many out-of-tree patches are required during the
initial bootstrap phase and thus everyone uses a self-built vendor
kernel there.

Automatically modifying src:linux is quite feasible. It has changed a
couple of times and now roughly looks like this:

    cat >debian/config.local/defines.toml >/dev/null <<EOF
    [[kernelarch]]
    name = '$kernel_arch'
      [[kernelarch.debianarch]]
      name = '$deb_arch'
    EOF
    ./debian/rules debian/rules.gen

This is quite mechanical and therein lies value. The difficulty now has
become that src:linux is the only package in the entire architecture
cross bootstrap that builds an architecture-dependent Arch:all package.
I have not yet found a good way to deal with this oxymoron.

Helmut

Hi Bastian (2024.10.19_09:35:21_-0400)

I think in most (all?) cases, they don't have architecture-specific
headers.  So that applying that consensus here may be not be that useful
to this case.

In general arch:all packages are preferable to arch:any, for the sake of
the size of the archive. But sometimes violating that is necessary.

It sounds like changing linux-libc-dev to arch:any could be helpful to
the cross-bootstrappers. That's something that happens outside the
archive, but is also part of the lifeblood of the project. Architectures
come and go, and we've been trying to make this process easier. It's one
of our strengths as a distribution.

I see a concern on a -common package: There are risks of arch:all and
arch:any packages building at different times on the buildds, and
possibly being published in separate mirror pushes. Leaving
build-essential uninstallable if the builds don't all succeed before the
arch-all package publishes.

If so, avoiding a -common package and having linux-libc-dev be pure
arch:any could be a reasonable solution? (At a cost of size)

Stefano

I don't understand what succeeds.  Because missing asm/* headers pretty
much makes it impossible to use anything from linux/*.  So the bootstrap
either have it's own headers or it will bail out if it tries to include
any kernel interface.

Can you please share the log for it?

Please make proper requests that are actionable.

- arc: in dpkg. bootstrap plan is where?
- cksy: not in dpkg, so not actionable, as triplet is not known.
  bootstrap plan is where?
- mipsr6el: which one do you mean?
  /usr/include/mipsisa64r6el-linux-gnuabi64?
- sh3: was never in debian. bootstrap plan?
- sparc: was removed years ago. bootstrap plan?
- musl-linux-any: how is that supposed to work?

Bastian

I did add that support first.
- Before the arch reaches anything related to Debian.  In this stage, it
  does not matter if the package it all or any, it can be patched.
- While it lives on -ports, but dak does not know about it.  In this
  stage, the package needs to be managed manually.
- While it lived on -ports and dak knows about.  In this stage, support
  can be added as arch:any.

Bastian

Hi Bastian,

This is the stage I am concerned about. You claim that it would not
matter whether the package is all or any, but my experience is
otherwise. With the exception of linux, the cross bootstrap tooling does
not build any Arch:all packages at all. Adding that support has not been
trivial and the way it is implemented still is violating internal
principles making its code fragile. In particular, for each combination
of package name and architecture, there would only be one binary package
- with the exception of linux where the would now be two (the unpatched
one from the archive and the patched one). I'd have to somehow remove
linux-libc-dev from an unstable amd64 mirror.

The all versus any also poses the problem that it becomes unclear
whether it has to be rebuilt. When it was :any, it always had to be
built, but given that its M-A:foreign is a lie and that we had to remove
the Provides, it is difficult to tell whether a rebuild is needed.

In now find it all surprising that we (at least Ben, Bastian and me)
agree on patching linux at this stage being the right approach. My
earlier impression was that Bastian preferred src:linux to participate
in this early stage of the bootstrap and I have expended mails to argue
otherwise, but that seems unnecessary now.

Instead, the question becomes how we can make that patching easy and the
disagreement becomes whether Arch:all makes that sufficiently easy. Even
though I cite building Arch:all as difficult in the bootstrap context,
the more pressing issue from my side is the inability to tell whether a
rebuild is needed by looking at package relations. Dealing with
unsatisfiable dependencies is something, I've developed ways to deal
with. Dealing with wrongly satisfied dependencies is a totally different
thing to me. As it stands, linux currently is the only package involved
in architecture cross bootstrap that produces false positives in
dependency satisfiability tests used for scheduling builds.

As an architecture enters -ports for the first time, it must have its
triplet included in dpkg. At that time, I think it is reasonable to ask
src:linux to include its headers. linux also is uploaded so frequently
that it will not be practically blocking any port progress. The step
that usually takes much time and convincing is getting an architecture
into dpkg. I do not see an Arch:all linux-libc-dev as being a hindrance
to these later stages.

Helmut

Hi Bastian,

I think much of what you are asking here is already answered in my reply
to Ben's followup.

You may find all the logs at
https://jenkins.debian.net/view/rebootstrap. Indeed, I missed the fact
that sparc is being patched in. And with patching it in, the complete
sparc bootstrap is still successful at this time. So yeah, sparc
immediately fails, but the main reason for that is linux and not some
other component.

I'm not gonna argue about sparc in particular. I'll just keep it alive
in rebootstrap as long as it is low effort. Once actual porting work
becomes necessary, I'll poke John Paul Adrian Glaubitz and if nothing
happens, discard sparc from my bootstrapping efforts.

So these architectures only served as examples here. mipsel and armel
look like they could be the next sparc.
actionable. The architectures referenced are work in progress to
different extents. You'll find more detail in my other mail.

Helmut

So this tooling does
- patch the source,
- _not_ change the version,
- build non-all?

This sounds broken, as it already relies on those packages producing
compatible all/non-all packages from two different sources.  Yes, in
practice this will work.

To make the double package problem easier, just change the version with
a sufficiently high epoch?

What kind of information can this dependency satisfiability test use?

Something like this?

| Provides:
|   linux-libc-dev-supports (= amd64-0),
|   linux-libc-dev-supports (= arm64-0),
|   linux-libc-dev-supports-multiarch (= aarch64-linux-gnu-0),
|   linux-libc-dev-supports-multiarch (= x86-64-linux-gnu-0),

Bastian

Hi Bastian,
into the version may technically work, but it feels really strange.
Earlier, I proposed encoding it into the provided package name like
linux-libc-dev-arm64-cross, but we all know how that went and I would
not have proposed it if I had seen how it broke other pieces. Still if
we were to just drop the "-cross" suffix and go for a very similar
version.

Provides: linux-libc-dev-amd64, linux-libc-dev-arm64, ...

In any of these cases, I can inject extra dependencies and have
dose-builddebcheck correctly determine whether the existing
linux-libc-dev satisfies what is needed.

As stated elsewhere, I still don't understand what we gained by
switching from Arch:any to Arch:all, but maybe I don't have to. A
solution that adds any of these Provides is what I'd call good enough in
practical terms for the purpose of bootstrapping.

Helmut

Hi Bastian:

We have two proposed provides schemes here, can we select one and add
it?

It would be helpful if this question could be answered. I think Matthias
was asking the same question in #1081826.

Hi

Something like simple providers is the easiest to do.

And I currently have no idea what to say, except to repeat myself.
Just a: this is wrong.  Not, what is broken by it, which makes it wrong.

Bastian

Hi Bastian (2025.02.04_14:05:07_+0000)
toolchain can determine which architectures are supported, and have a
target to Depend on.

The resolution of the issue brought to the tech-ctte requires figuring
out whether linux-libc-dev will take over providing these headers for
cross toolchain building, or whether cross-toolchain-base continues to
own the linux-libc-dev-${arch}-cross virtual package names (and provides
symlinks or duplicates of headers).

Thanks again for temporarily releasing the -cross virtual package names
while this is being discussed.

Before we can come to a conclusion on that, the elephant in the room
seems to be the question of the motivation for the migration to an
architecture all linux-libc-dev package. Let me try to distil this
question:

What is being gained by the transition to an architecture independent
package?

I see several pros and cons:

1. Advantage: Reduced overall disk usage on mirrors. 1x 2MiB
   package replaces per-architecture 2MiB packages. For context, the
   linux source produces ~1GiB of binaries per arch.
2. Advantage: This allows taking over linux-libc-dev-{arch}-cross, as
   virtual packages, reducing the need for cross-toolchain-base to
   duplicate linux source. And avoiding the need for cross-building to
   create this architecture: all binary packages in bootstrap.
3. Disadvantage: Users get ~40 extra files (totalling 100kiB) per
   architecture in Debian installed when they install linux-libc-dev.
   (bug #1081826)
4. Disadvantage: linux-libc-dev is installable on architectures that it
   doesn't actually support. No other package involved in
   cross-bootstrapping exhibits this. This only applies to architectures
   that aren't supported in the linux source package (helmut's A)
5. Disadvantage: cross-build tooling needs special heuristics to
   determine which architectures linux-libc-dev actually supports.
   (helmut's B) The Provides scheme above supports that.
6. Disadvantage: cross-build tooling doesn't currently support building
   architecture independent packages. And that's quite understandable,
   they should never need to be built unless we hit a situation like
   this.

Is there anything that we're missing here?

Item 6 leads to more complex questions like how do we handle versions of
such packages? That's the fundamental issue here and it's where it gets
gnarly. It begs the question of whether it reasonable to patch sources
and build a binary package without a bumped source version.

When I get to this point, my feeling is: Yes, cross-building is breaking
assumptions that the rest of the archive is making. But it's (until now)
been working fairly successfully while keeping the approach as simple as
possible. That sounds like on balance a good thing.

Should some of those assumptions be codified into policy, if we want
them? Probably.

The change of linux-libc-dev to architecture: all has the effect of
causing a lot of non-trivial work for people doing cross-bootstrapping.

The balance of advantages vs disadvantages does depend a bit on how
valuable you see item 2 being. I imagine Matthias doesn't see any value
there.

With the pros and cons that I've seen, the weight of balance seems to go
against the arch all package. This is why people keep asking what the
motivation for the change was. If you have other motivations, please
share them.

Obviously, undoing the change would be a lot of work for you and the
kernel team, and we wouldn't want to put that on you. I'm not pushing
you to reverse the decision.

It would help if everyone saw the impact that they were having on each
other, and was able to weigh the costs of each option. Once we have some
common ground, we can come to conclusions on the questions raised.

Thanks for taking the time to understand people's questions and answer
them.

Stefano

Hi
talking about.  It uses dpkg-cross, so is already using undefined
behavior of linux-libc-dev to do it's work.

My current version of it does this now:
| Provides: linux-libc-dev+support-debian+alpha (= ${binary:Version})

Yes.  It only supports architectures that are defined as Debian.

You mean "bootstrap tooling".  And yes, that part I missed in my initial
assessment,
arch:all linux-libc-dev.  But it might have failed with any change of
linux-libc-dev, because it relies of undefined behaviour.  So not a
disadvantage, but inherent instability of the dpkg-cross approach.

Or actually an advantage, because we can remove this instability and
reduce the complexity.

- It can provide headers for multiple architectures at the same time.
- It could even handle debian architectures mapping to multiple
  multiarch identifier and kernel arches (like hppa did kind of).
- debian-ports can import it without going through the manually handled
  "unreleased" part of their archive, even before ftp-master knows about
  an architecture.
- Reduces the part of the package API that is used in an undocumented
  way via dpkg-cross.

I don't know.  Right now the whole cross-building stuff violates the
policy, how would we clean this up?

I'm not even sure about arch:all.  We currently have no problems with
such packages only working on some architectures, if by dependency or
actually failing to work.

The impact was rather small.  Only two other parts where actually
affected, and none of them was our endusers.

The bootstrap stuff was affected, which I missed at first.  But is all
contained within two scripts that control everything.  And the rest of
that script is patching other packages quite havily already.

The other is cross-toolchain-base.  But here the impact is still
completelly unknown or at least does not show up within Debian.

Bastian

Hi,

In addition to all listed points, it also temporarily breaks
creating chroots for archs when the src:linux build for a given arch
and for arch:all end up in different dinstall windows. In this time
linux-libc-dev will not be visible to debootstrap. This behaviour
was observed f.e. on 2025-02-03.

Chris

Hi Bastian (2025.02.04_13:25:08_-0400)

I've had a few IRC conversations with both Bastian and Matthias about
what comes next. No conclusions yet, but the ground we've covered is:

Matthias is asserting that the -cross package namespace and
/usr/<triplet> sysroot directories belong to cross-toolchain-base. He'd
like to keep control of those packages.

Bastian is concerned that if both linux-libc-dev and
linux-libc-dev-*-cross provide linux headers in different paths they
should conflict with each other to avoid compilers accidentally sourcing
the wrong headers. linux-libc-dev installs in the usual multiarch
include directory (/usr/include/$ARCH/linux/). linux-libc-dev-*-cross
installs in the sysroot directory (/usr/$ARCH/include/linux/)

I would imagine that any conflicts like that would make
linux-libc-dev-*-cross practically uninstallable in any situation where
you'd need them (libc6-dev depends on linux-libc-dev).

So, we're back to trying to unpick this knot. The Techncial Committee is
not supposed to do design work, but I can see some possible ways out of
this.

I could see a scenario where linux-libc-dev-*-cross are tiny wrapper
packages, depending on linux-libc-dev and including symlinks from one
include directory to the other. I assume that wouldn't need the
Conflicts?

Or if the Conflicts were more version-targetted and allow linux-libc-dev
and linux-libc-dev-*-cross to co-install if they were built from the
same linux source version, would that be acceptable?

Do either of you have any thoughts on how we should figure this out?

Stefano

Yes.  An unversioned conflict would make the whole cross stuff
effectively uninstallable.

This is what I implemented as well, a symlink farm.  But those symlinks
needs to be belong to the same source package, as the set of required
symlinks change over time.  Or we are back at: requires conflicts.

I only consider symlinks to files acceptable.  With symlinks to
directories we are back in aliased directories land, with all the same
problems as merged-/usr.

Only if the *-cross maintainer is fine with updating it within a day,
every week or so for unstable, but even for stable or security.  So, not
possible, given that the existing conflict for the same reason against
glibc was only solved by NMU after a week or so.

Bastian

This conflict is currently escalating with the glibc side, see #1100544.
The glibc maintainers decided to not longer take up with it after the
release of trixie.

Bastian