Hi!

This is prompted by #1095862 (RC-bug on LLVM/rustc for
unsoundness/miscompilations on current Debian i386, because LLVM and
thus rustc don't support Debian's i386 baseline).

The issue was discussed back in November on debian-release/-devel,
including a roll call for opinions on raising i386's baseline by elbrus
[0] (with 6 pro voices, mine included, and no objections), and an
analysis of affected hardware[1].

IMHO the most sensible change to the baseline would be to move from:

- i686 since Debian 12 'bookworm'. There's no MMX nor SSE. [2]

to

- i686 with MMX and SSE/SSE2 ("Pentium 4") since Debian 13 'trixie'

which is what I want to suggest by filing this issue. It seems to be
sometimes refered to as i786, but that doesn't seem to be the universal
nomenclature - but i686 is already meaning different things as well.

I recently picked up the discussion on IRC (05.02.25):

12:35:13 <smcv> amacater, f_g: I think rust having a realistic/working
baseline on i386 would be better than dropping rusty things from i386,
because sooner or later I expect something like Mesa will pick up a Rust
dependency, and that's necessary for the legacy-binaries/Wine/Steam
use-case

20:03:03 <elbrus> f_g: if the baseline could be raised across the
toolchain packages in sync (I mean, if they agree what to raise it too),
than I (as a Release Team member) are fine with the raise; much more
appreciated than removal of rustc on i386

I am not worried about the rebuild side for rustc, as we (well, mostly
you ;)) rebuild the Rust world roughly every 6 weeks anyway with each
new upstream release, and we still plan on uploading 1.85 for Trixie
before the freeze starts. For rustc and LLVM an updated baseline would
actually align us closer with upstream rather than deviating from it, so
I expect the majority of fallout to be dropping patches that worked
around this fact.

Yesterday I reached out to the LLVM and GCC maintainers. Sylvestre (for
the LLVM side) would be fine with raising its target to match a bumped
baseline like described above, doko (for the GCC side) thinks it is too
late in the release cycle to do that because of the number of rebuilds
and potential performance implications it entails.

@Sylvestre, @Doko : please expand the above if you feel like something
is missing!

A mixed approach (raising the baseline to allow LLVM and rustc to bump
their targets, but keeping GCC's default target and thus the majority of
packages at the current/old baseline and skip the rebuilds there) might
be possible, but also has potential perfomance or bugginess implications
because of switching back and forth between x87 and SSE floating
operations.

So it seems we are at an impasse now with the toolchain freeze rapidly
approaching - and might have to remove i386 support from rustc (and in
theory, LLVM as well, but I will of course defer to the respective
team's opinion on that) for Trixie after all. Any further input that
might help get this unstuck highly appreciated.

I am sorry that this came up this late in the release cycle, and that I
didn't push faster between November and now to get this resolved.

Thanks for the consideration,
Fabian

0: https://lists.debian.org/debian-devel/2024/11/msg00389.html
1: https://lists.debian.org/debian-release/2024/11/msg00459.html
2: https://wiki.debian.org/ArchitectureSpecificsMemo#i386-1

Hello,

Fabian Grünbichler, le jeu. 13 févr. 2025 08:11:14 +0100, a ecrit:

Which kind of bug? AIUI raising the baseline does not change the
function call ABI, and within a function we use only one compiler?

Samuel

I am not aware of any (and my quick check yesterday indicates that
mixing like that shouldn't be an issue) - but I also haven't done any
in-depth research that would make me confident in saying that I'd rule
them out ;) the potential performance issue was raised by doko on IRC.

Regards,
Fabian

Fabian Grünbichler, le jeu. 13 févr. 2025 10:39:00 +0100, a ecrit:

When switching between functions, the floating-point operations are
normally already over, so I don't think we would suffer very much? (if
we were mixing within a function that would pose problem, sure).

Also, I don't think we would very often switch between floating-point
code compiled by llvm and gcc?

Samuel

yes.

can't speak for the archive as a whole, but in the Rust world this is
quite common:

- C library compiled with gcc (usually)
- Rust code compiled with rustc (/LLVM) linked with such a library

but like you (and others) pointed out - this shouldn't normally be an
issue, as the interface and calling conventions are well-defined (else
none of that would work anyway ;))

I don't think such changes should be made a few weeks before a freeze,
or if that change should be made at all. This is a discussion that
should be made during the freeze and then implemented at the start of
the next release cycle.

The current baseline is i686-linux-gnu, without access to any SSE
instructions.  Apparently other toolchains are actively raising these
baselines, or are unaware about what is used as a baseline upstream (as
seen in here for LLVM and rustc).

Mixing fp87 and SSE instructions leads to a performance penalty (I don't
have references to that, so people might want to investigate).

fp87 instructions also show some excess precision, so test cases and
test suites of packages might be affected by this change as well.

When preparing for such a change (maybe during the freeze when the
archive is not active), please do at least a comparative test rebuild
with these changes, also checking that affected toolchains adhere to the
new baseline.  A test rebuild still does not cover the use of the
rebuilt binaries as dependencies.

Is this really worth doing that change, alternatives could also include
to retire the i386 port, or to keep the status quo, and document that
packages built by toolchains with unknown baselines might not run on
some hardware.

Matthias

Matthias Klose, le jeu. 13 févr. 2025 12:35:51 +0100, a ecrit:

When they are mixed within a function, sure.

But between functions (and actually it's even about between libraries
here, so even less often), I don't see why.

So again, I don't see why bumping in rust (which will really fix things
there) should imply bumping in the rest of the toolchain. Raising the
*allowed* i386 baseline does not suddenly force the whole toolchain to
actively try more to use more than it currently does.

The references I can find only talk about within a function.

Indeed. In the case of the rust toolchain, it's actually in the good
direction: upstream tests do assume not using fp87.

To avoid the fp87 unsoundness in rust that does produce bugs, yes.

Samuel

Obviously, if this issue had been known (on our/Debian end) earlier then
end of Nov '24 it would have been raised earlier. Alas, none of us have
a time machine..

That LLVM and rustc upstream have a different baseline has been known
for basically ever. Until the referenced discussions in November, at
least I was under the impression that the only downside of this
difference is the different FP behaviour (and resulting patches, mostly
to test suites, assuming the upstream default semantics/precision).

Neither rustc nor LLVM packages in Debian violate the current i386
baseline. They are just (quite subtly!) broken under it, and the only
fix according to both upstream projects is to raise the baseline.

Note that rustc at least will forbid what we are currently doing soon
for exactly this reason (which we can of course patch out again, but at
that point I'd honestly rather drop i386 altogether).

I am fully prepared to handle that on the Rust side, and can help with
the LLVM side as well - this will mostly be dropping Debian-specific
patches and verifying the results, as upstreams projects using rustc or
LLVM will assume the usptream baseline, not the (from their PoV,
strange) Debian one.

The number of Rust cdylibs in the archive is tiny, everything else is
covered by a rebuild.

The issue is unfortunately not that they might not run on some hardware
(that would be the case if we were to selectively violate the baseline
by bumping it in rustc/LLVM while still pretending Debian's i386
baseline doesn't require SSE2/MMX, which nobody is suggesting!), but
that they are miscompiled and then segfault (depending on execution
path, so hard to catch until people run into it) both on hardware that
supports and hardware that doesn't support SSE2.

that would mean our amd64 users lose the ability to run 32-bit Wine, as
well as legacy 32-bit Linux binaries (most commonly game-related). In
my view an i386 port that is not optimally fast, or that is not
feature-complete, would be preferable to no i386 port at all.

I notice in particular that in Ubuntu, where the i386 port has been
cut down to the minimum of packages required for these legacy-binary
use-cases, significant effort has gone into cutting down packages'
dependencies so that they can still be built on i386, suggesting that
Ubuntu also values the existence of backward compatibility with legacy
binaries.

Similarly, our colleagues in distros that don't have the advantage
of multiarch have generally found that they need to keep at least a
subset of libraries available via multilib: for example the Fedora/Red
Hat ecosystem still has i686 RPMs, Arch has the lib32- family of packages,
and even intentionally unusual distros like NixOS have their equivalent.
I personally think an i386 port that is a multiarch foreign architecture,
either a nearly-full one like in Debian or a small subset like in Ubuntu,
would be a much less painful way to achieve this than expanding the
coverage of multilib.

I don't really want to have to remove Rust code from i386 because
that would itself be a rather large transition. For example, librsvg
alone has around 250 direct rdeps, which would need either removal or
sourceful changes; one of those is that if we want to keep 32-bit libwine
(which I believe we do), then we cannot remove libavcodec from i386, and
instead we would have to make sourceful changes to disable its optional
librsvg support.

But, if it comes down to "remove all Rust code from i386, or remove i386"
then I think removing Rust code from i386 would be a painfully large
amount of work, but still the less damaging option.

If raising the baseline in general is ruled out, then I actually think
selectively violating the baseline in rustc/LLVM would be a less damaging
option than removing all Rust code from i386.

The options as I see them, *including* the options that I would personally
prefer to rule out, are:

- Status quo: don't change anything. As Fabian says, Rust code on i386
  will sometimes be miscompiled and might crash.

- Raise baseline to i686+SSE2+MMX and make gcc require/assume this

- Raise "official" baseline to i686+SSE2+MMX, leave gcc producing
  code that would have worked with the previous baseline by default,
  but rustc/LLVM may require/assume i686+SSE2+MMX

- Officially keep current baseline, intentionally violate the baseline in
  rustc (and maybe LLVM?) so that rustc produces working code, and
  have the release team announce that the resulting baseline violations
  are not to be considered RC bugs

- Do a transition to remove all Rust code from i386

- Remove i386 completely, re-introduce the equivalent of ia32-libs so we
  can still provide 32-bit Wine and Mesa
  (I'm mentioning this for completeness but I suspect the release team
  would veto this, because nobody liked ia32-libs)

- Remove i386 completely, no more 32-bit Wine, no more ability to
  install Steam, etc.

Does anyone see others?

    smcv

Simon McVittie, le jeu. 13 févr. 2025 16:32:30 +0000, a ecrit:

Are rustc and LLVM necessarily coupled? AIUI the rustc baseline is
configured in rustc itself, in
./compiler/rustc_target/src/spec/targets/i686_unknown_linux_gnu.rs :

    base.cpu = "pentiumpro".into();

vs

    base.cpu = "pentium4".into();

So we could raise the baseline only in rustc.

Samuel

rustc uses LLVM as its backend compiler, and LLVM on its own is affected
by the same issue as LLVM internally assumes Pentium 4 semantics for
i686 targets when it does its optimization passes[0]. LLVM in Debian
carries a similar patch downgrading the definition of its i686
targets[1]:

-    // Fallback to p4.
-    return "pentium4";
+    // Fallback to i686.
+    return "i686";

it just downgrades further (to the pre-bookworm baseline, and not the
bookworm and later one).

0: https://github.com/llvm/llvm-project/issues/89885
1: https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/blob/19/debian/patches/clang-baseline-fix-i386.patch

dear release team,

I would appreciate some indication/answer which of the options below we will
target for Trixie. I did a rebuild of all immediate rdeps of rustc with the
baseline raised there (well, with the two patches lowering it dropped ;)).

out of the 3222 packages rebuilt, there were 82 build failures (either FTFBS in
sid, or my rebuild env not having enough disk space). 3 of packages actually
failed to build with the bumped baseline, but did build successfully in a
recheck build with the stock rustc:

rust-tint_1.0.1-2 (fails cause of test patched to workaround x87 floating
behaviour, likely fixed by dropping that patch)
rust-kuznyechik_0.8.2-1 (actually broken upstream, but looks fixable)
fish_4.0.0-3 (a second build with the bumped rustc worked, so this seems like
a flaky test breaking the build sometimes)

my rebuild did not include testing the same change for LLVM itself, nor running
autopkgtest which my guess is where most fallout would be if we were to do this
for real. I still think it gives a rough idea of what to expect if we
selectively bump the baseline.

thanks smvc for the great summary of available options!

I can live with this, provided we raise the baseline for forky (or drop i386 at
least for Rust). it should probably be mentioned in the i386 release notes if we
go down this route.

given doko's objection above, this is likely not an actual option.

I am willing to work through fallout during the freeze if we choose this option,
and would prefer it to the ignore-for-trixie variant.

same

this is probably more work and churn then the two partial-baseline-raising
variants above, and probably makes more people unhappy

I have no opinion on either of these as a rustc maintainer, but guess these
would make even more people unhappy.

thanks for your time reading this (and work on getting the Trixie ball
rolling!)

Fabian

I'm the one who added the warnings in rustc (that will eventually become hard
errors) that show up when disabling SSE2 on our primary x86-32 targets. Happy to
answer any questions you may have about that!

As a Rust compiler developer and Debian user, I'd love to see Debian no longer
ship a patched rustc that uses a configuration which we cannot support with
reasonable effort. :) So having the baseline raised here would be amazing -- if
I had known that this is even an option, I'd have tried to push for it sooner.
:D  That would remove a significant portability hazard from the Rust ecosystem.
It's one thing to have people opt-in to a tier 3 target and then get strange
behavior; it's another thing entirely for people to "apt install rustc" and get
strange behavior.

The lack of support in Rust, as already mentioned, is due to lack of support in
LLVM, which I would say is ultimately due to general lack of interest in making
a platform with such quirky behavior work well in today's world where basically
everything implements IEEE FP semantics (at the expected precision levels).
Someone would have to put in that work, but the community of people keeping
non-SSE2 x86-32 alive do not seem to have the capacity for that, unfortunately.

If you need to keep the patch for Trixie, and if you want to ship Rust 1.86 in
Trixie, the patch will need updating due to
<https://github.com/rust-lang/rust/pull/137037>. I left some comments
specifically for distributions in the code to hopefully help with that; again
let me know if you have any questions.

Kind regards,
Ralf

Hi,

Sorry for the long wait, somehow I thought we replied.


I think we're effectively going with this option, given the upload in
[1] that we are aware of and didn't object to.

Paul

[1]
https://tracker.debian.org/news/1627384/accepted-llvm-toolchain-18-11818-17-source-into-unstable/