gtk4 had a recent test failure regression on s390x and other big-endian
architectures like ppc64 (#1057782). I sent this upstream to
https://gitlab.gnome.org/GNOME/gtk/-/issues/6260 and proposed a patch in
https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/6653, but upstream is
reluctant to apply the patch because they think it is the wrong solution:

librsvg also has long-standing unsolved endianness-related issues, most
likely in one of its dependencies (#1038447, which has affected bookworm
since September 2022).

The GNOME team does not have big-endian hardware where we can run manual
tests, so we do not know how much of an impact this has on practical
usability of GTK and librsvg on big-endian architectures: it's entirely
possible that they have always been misrendered or broken on big-endian,
but the bug was never reported because there were no users, and we
are only noticing this now as a result of wider test coverage being
introduced.

If porters are interested in having GTK and librsvg continue to be
available on big-endian, please work with upstream to get them to a point
where endianness-specific bugs can be taken seriously in the upstream
projects. I do not consider doing this downstream-only to be a solution.

If endianness-specific issues become a blocker for the Debian release
process at some point in the future, then it is likely that I will have
to start the process of doing architecture-specific removals for these
packages and their reverse dependencies. For s390x this is likely to
have little user-visible effect, because I find it unlikely that there
are genuinely users running GUI applications on IBM mainframes, but for
-ports architectures this will probably be a larger regression.

Thanks,
    smcv

Hello Simon!

Is there someone working on that?

The fact that librsvg isn't fixing these issues is surprising to me.

Back when the library was converted to Rust, I was raising concerns
that this will reduce portability which was dismissed by the main
author as unjustified.

There are both QEMU-based solutions available as well as big-endian machines
in the GCC Compiler Farm [1]. There is also the OpenPOWER Openstack platform
which has both little- and big-endian targets available [2]. So, I don't think
the argument there is no way to test on big-endian targets is justified.

Looking at both Fedora [3] and openSUSE [4], tests aren't executed on s390x
for these distributions either.

Well, removing librsvg will certainly disable quite large number of packages
on s390x. Maybe someone should CC an engineer from IBM on the bug report
and ask for help to fix the testsuite issues.

Adrian

Hi Simon,

I am working As a Debian maintainer on s390x in IBM. Sorry I missed out this bug because I am engaged with the other issues like librsvg related.
In librsvg also we have seen some test failures and regressions. when we debugged, we came to know that pixman code changes are triggering the regression in librsvg. In the current issue of gtk4 When I tried to reproduce the issue on s390x there are 1507 tests that are failing. Can you please suggest that version of dependency packages being used/issue reproduce steps/logs.

Log:
Ok:38
Expected Fail:0
Fail:1507
Unexpected Pass:  0
Skipped: 1
Timeout: 0


Thanks,
Gayathri.

Gentle Reminder...!

Thanks,
Gayathri.

That's good to know. I see in
https://gitlab.freedesktop.org/pixman/pixman/-/issues/78 that there is
some confusion over which layer of the stack is the right one to be doing
this byteswapping: the original change was made to fix a bug seen in
gnome-characters, but then that change caused the librsvg bug.

In my experience the only way to solve endianness issues without causing
regressions is to have clear documentation of what endianness each layer
of the stack is aiming to use, so that you can check the behaviour of
each function against that documentation/specification and say "this is
correct" or "this is wrong" with confidence. The majority of computers this
decade are little-endian, so it's mainly the people who are interested in
supporting big-endian computers who need to get this right.

For instance, if a function is returning 32-bit RGBA data, you need
to be able to know whether the correct format is "red first" or "alpha
first" or "red in the least significant byte of the first 32-bit word"
or "alpha in the least significant byte". Otherwise, you can't know
whether a change is actually correct, or whether it's a workaround for
the layer above or below also being wrong.

I think this is what Simon Ser means in their comments on
https://gitlab.freedesktop.org/pixman/pixman/-/merge_requests/92.
In your commit message you said "This will write out the pixels as BGRA
on big endian systems but obviously that's wrong", but that isn't obvious
(at least not to me, and probably not immediately obvious to the other
Simon either).

In https://gitlab.freedesktop.org/pixman/pixman/-/merge_requests/92
you mentioned that before reverting that change, pixman was failing its
test suite on big-endian, and reverting the change makes its test suite
pass. That's probably relatively good evidence that the revert is the
correct thing to do, but that means it deserves to be mentioned in the
commit message.

This SDL change might be a good example of making sure that an intended
endianness is documented clearly:
https://github.com/libsdl-org/SDL/pull/8318/files

and this one might be a good example of justifying why a change is the
correct change:
https://github.com/libsdl-org/SDL/pull/8819

    smcv

I'm sorry, I don't understand the question. You say you can reproduce 1507
test failures: if that's the case, why do you need steps-to-reproduce?
It seems like you can already reproduce a problem that needs solving?

    smcv

Hi @Simon McVittie<mailto:smcv@debian.org> / @John Paul Adrian Glaubitz<mailto:glaubitz@physik.fu-berlin.de>



We understood that it’s a regression bug and the gtk package was working fine to the previous releases of “4.12.4” version... We also understand that only one test failure i.e. “gtk:gdk / memorytexture” noticed by you in the later releases.



We are having a Debian unstable environment(sid) on s390x.. We think, it's better to reproduce the issue with git cloned repo instead of using “sbuild” method, so that we can get the commit history and see the code changes.



Can you please provide the exact steps to reproduce the bug?.. What commands did you use when you encountered this bug?.



We cloned the gtk repo and built it with the following meson commands.. But we see that almost all of the tests are failing.

So, we are suspecting that we might be missing something here.. There could be some dependency packages that we are missing?...  Kindly clarify more on this.

  *
meson setup _build
  *   meson compile -C_build
  *
meson test -C_build

Thanks,
Gayathri.

Hi!

The problem seems to be that XDG_RUNTIME_DIR is not set:

	Gdk-DEBUG: error: XDG_RUNTIME_DIR is invalid or not set in the environment.

Looking at the debian/rules of the gtk4 package, there is a dedicated
script in the package that the Debian package uses to run the tests:

It's reference from here:

Or check what upstream states about running the tests.

Adrian

I see. In that case, you will still need to install the build-dependencies
of GTK 4 (or more generally, whatever package you're testing), and you will
need to either follow whatever steps Debian uses to run the tests, or make
your own build steps and workarounds that have similar behaviour.

If the bug you are referring to is #1057782, I didn't reproduce it myself
at first: our official autobuilders reproduced it, which they did by
building the package using sbuild.

When I reproduced it on the s390x porterbox 'zelenka', I followed steps
similar to these, which are the same for all Debian packages:

- create a new s390x unstable chroot
- install gtk4's build dependencies into it
- unpack gtk4 source and cd into it
- dpkg-buildpackage -us -uc -rfakeroot -B

If you have chosen to build from upstream source, then you're welcome to do
so, but the way that Debian packages are always built is to use
dpkg-buildpackage, and depending on the package, that might need to do
Debian-specific things as a result of the package's upstream design choices.
Those Debian-specific things should always be a good starting point for how
you can do a build from upstream source.

The debhelper tools log what they do, so if you look at the output from a
Debian build, you will see the exact commands that were used. For example,
in https://buildd.debian.org/status/fetch.php?pkg=gtk4&arch=s390x&ver=4.12.4%2Bds-1&stamp=1701103591&raw=0
you will find this command:

	cd debian/build/deb && DEB_PYTHON_INSTALL_LAYOUT=deb LC_ALL=C.UTF-8 meson setup ../../.. --wrap-mode=nodownload --buildtype=plain --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=lib/s390x-linux-gnu -Dpython.bytecompile=-1 -Dauto_features=enabled -Dbroadway-backend=true -Dx11-backend=true -Dcolord=enabled -Dman-pages=true -Dsysprof=enabled -Dwayland-backend=true -Ddocumentation=false -Dbuild-examples=true -Ddemos=true -Dinstall-tests=true -Dmedia-ffmpeg=disabled -Dcloudproviders=enabled

... which codifies exactly how Debian did the "meson setup" step for
that particular version of GTK. (In this case we're building in
debian/build/deb instead of _build, and we're setting a lot of options.)

Later in the log, you'll see the commands that were run instead of
"meson compile" and "meson test".

Yes, this.

This is partly because upstream expects that the test suite will be
invoked when you are already in a working X11 or Wayland environment,
but autobuilders and similar non-interactive or text-based systems
generally do not have that, so you have to run the tests under a mock
X11 environment (with xvfb-run) or under a mock Wayland environment
(we use weston --backend=headless-backend.so for this). In the Debian
packaging, this is encapsulated in debian/tests/run-with-display.

This is also partly because GTK can be either an X11 client or a Wayland
client. In Debian, we want to know that both of those work, so we run
the whole test suite twice: once under X11, and once under Wayland.

The debian/run-tests.sh script also skips some tests or adjusts the
thresholds used to to work around various issues:

- Some tests have slightly different rendering on i386 and other
  architectures with non-IEEE floating point. Upstream doesn't consider
  this to be a problem because they don't actively support non-x86_64
  architectures. There is a Debian-specific patch in the packaging that
  adds a mechanism by which we can tell the test suite to accept small
  rendering differences; I tried to upstream this, but upstream refused
  to apply it.

- Some tests have slightly different rendering with the versions of Pango,
  fonts and other dependencies in Debian. Upstream only really fully
  supports whatever exact versions they happen to have installed on their
  CI and development systems at the time (mostly Fedora).
  Again, there is a Debian-specific patch that adds a mechanism by which
  we can tell the test suite to accept small rendering differences.

- Sometimes tests fail because of architecture-specific issues (often bugs
  in Mesa or another dependency, rather than GTK), and are skipped on
  the affected architectures if we can establish that the impact on users
  of those architectures is likely to be non-release-critical.

That isn't actually a fatal error, only a debug message, despite having
the word "error" in it: XDG_RUNTIME_DIR is required for the Wayland
backend, but is not required for the X11 backend. But, if you are going
to run the tests with the Wayland backend forced (as we do in the Debian
packaging), then you will need working Wayland.

    smcv