Dear Maintainer,

The rocfft callback tests were passing with rocfft 5.5.1 on hip 5.2.3,
but began failing when hip was updated to 5.7.1. These failures are
specific to gfx900 and gfx1030 to gfx1036. The failures are not observed on
gfx803, gfx906, gfx1010, gfx1100 or gfx1101. This problem remains
unchanged by the update of rocfft to 5.7.1.

This is a sample of the failing test output [1]:

161s [ RUN      ] rocfft_UnitTest.default_load_callback_complex_single
161s clients/tests/default_callbacks_test.cpp:280: Failure
161s Expected equality of these values:
161s   rocfft_execute(plan, &in_ptr, &out_ptr, info)
161s     Which is: 1
161s   rocfft_status_success
161s     Which is: 0
161s
161s clients/tests/default_callbacks_test.cpp:310: Failure
161s Expected: (diff.l_inf) < (type_epsilon<Tbound>()), actual: 32.230823516845703 vs 3.75e-05
161s
161s [  FAILED  ] rocfft_UnitTest.default_load_callback_complex_single (349 ms)

It would be good to capture output with AMD_LOG_LEVEL=4 set in the
environment to view more information about the calls that are being made
to the HIP runtime.

Sincerely,
Cory Bloor

[1] https://ci.rocm.debian.net/data/autopkgtest/unstable/amd64+gfx1030/r/rocfft/9454/log.gz

I tried to reproduce the rocfft callback bug with a W6800 (gfx1030). I
used a Debian Unstable docker container on an Ubuntu Noble host, but the
tests all passed. This made me realize that the test failure pattern on
the CI is that all the qemu-based workers are failing and all the
podman-based workers are passing.

This issue seems to be somehow related to the qemu+rocm autopkgtest
environment.

Sincerely,
Cory Bloor

Hey Cory,

thank you for the analysis. I'll try to reproduce and lock this down on
my end, too.

Best,
Christian

The issue is already visible with AMD_LOG_LEVEL=1, it's the lack of PCIe
atomics:
additional causes, too.)

In an older ROCm ticket, a workaround to enable PCIe atomics in the
guest was discussed [1], but I never got this to work. The relevant bit
is not set after invoking setpci.

I don't know how to best address this. A workaround would be to skip
these tests if the host is a guest VM, but that would reduce coverage.
However, switching everything to podman would reduce coverage even more
if we only use the latest kernel.

Best,
Christian

PS: Full AMD_LOG_LEVEL=4 attached, for reference.

[1] https://github.com/ROCm/ROCK-Kernel-Driver/issues/26#issuecomment-313857180

In a more recent issue [2], a lack of PCIe atomics was also discovered
on physical hardware (it can depend on the CPU and/or the PCIe slot).

In that issue, it was stated that updating to ROCm 6.0 (and PyTorch)
resolved the issue.

I just rebuilt rocfft to 6.0.2 but the issue is still present. But that
was naive, there are other < 6.0 components in the stack that could
affect this.

[2] https://github.com/ROCm/ROCm/issues/2429

Ah. That makes sense. Thanks, Christian!

The problem appeared in rocfft 5.5.1 when rocm-hipamd was updated from
5.2.3 to 5.7.1. It was working fine even when every package in the stack
below it aside from rocm-compilersupport and rocm-hipamd were on 5.7.1.

I suspect that rocm-hipamd alone determines whether rocfft requires PCIe
atomics.