Dear Maintainer,

my backport of llama.cpp/ggml for trixie starts to fail when upgrading
libraries from rocr-runtime from 6.1 to 6.4.

llama.cpp and ggml were recently removed from testing for dependency
reasons, but since this is nevertheless easily reproducible, I need to
suspect that rocr-runtime is broken, therefore filing with severity:
serious to prevent migration to testing.


Steps to reproduce on a vanilla trixie system with an AMD GPU:

// Prerequisites
# cat /etc/apt/sources.list
deb     http://192.168.0.21:9999/debian trixie main contrib non-free-firmware
deb-src http://192.168.0.21:9999/debian trixie main contrib non-free-firmware

deb     http://192.168.0.21:9999/debian unstable main contrib non-free-firmware
deb-src http://192.168.0.21:9999/debian unstable main contrib non-free-firmware

# cat /etc/apt/preferences.d/unstable.pref
Package: *
Pin: release a=unstable
Pin-Priority: 50


// We need some build dependencies from unstable, primarily for #1116585
# apt-get install -y pkg-rocm-tools/unstable libamdhip64-dev/unstable libhiprtc-builtins5/unstable

// Set to whatever test arch you are using, to speed things up
# export ROCM_TARGET_ARCH_FIXED=gfx1030

// Build the package for trixie
# apt-get build-dep -y ggml
# apt-get source ggml && cd ggml-*
# dpkg-buildpackage

// Install ggml with the right backends
# dpkg -i ../libggml0_*.deb ../libggml0-backend-{cpu,hip}_*.deb

// Install llama.cpp from unstable
# apt-get install -y llama.cpp-tools/unstable

// Download a model
# wget https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf

// Known good case produces benchmark statistics
# llama-bench -m gemma-3-1b-it-Q4_K_M.gguf

// Upgrade rocr-runtime
# apt-get install -y libhsa-runtime-dev/unstable libhsa-runtime64-1/unstable libhsakmt1/unstable

// llama-bench now errors out
# llama-bench -m gemma-3-1b-it-Q4_K_M.gguf

KFD does not support xnack mode query.
ROCr must assume xnack is disabled.
free(): invalid pointer
Aborted


Best,
Christian

Thank you for the report.

I was testing with packages built against newer versions of the HIP
Runtime, so I didn't catch this regression. I'd only tested the basics
with rocminfo, which worked fine for this case because it doesn't use
the broken function.

The problem seems to be that upstream changed the ABI of
hsa_agent_iterate_isas by changing the types for the callback function:

Old Callback Data Definition:

     std::pair<hsa_isa_t, bool> comparison_data

New Callback Data Definition:

     struct callBackData {
         std::pair<hsa_isa_t, bool> comparison_data;
         const unsigned int codeGenericV;
     };

Sincerely,
Cory Bloor

Control: reassign -1 libamdhip64-5 5.7.1-6

I was mistaken. I tried to fix the ABI breakage only to find nothing
there. AFAICT, the expected data layout for the callback function is
defined by the caller, not by libhsa-runtime64-1. I created a minimal
reproducer to compare the traced output when linked libamdhip64-5 and
libamdhip64-6.

$ cat main.hip
#include <hip/hip_runtime_api.h>

int main() {
   float *dArr;
   hipMalloc(&dArr, 1);
   return 0;
}

$ clang++-21 main.hip -o hip5
$ apt install -t experimental libamdhip64-dev
$ clang++-21 main.hip -o hip6
$ ltrace -C -e "hsa*" -e "hip*" ./hip5  2> ltrace5.txt
$ ltrace -C -e "hsa*" -e "hip*" ./hip6 2> ltrace5.txt

The ltrace5.txt and ltrace6.txt logs are attached. The output indicates
the crash is in libamdhip64-5.

Enabling logging indicates that this is the fault of libamdhip64-5, as
it incorrectly assumes there cannot be more than one supported ISA for a
given device.

$ AMD_LOG_LEVEL=5 ./hip5
:3:rocdevice.cpp            :442 : 102386455878 us: [pid:127063
tid:0x7fbecb179ac0] Initializing HSA stack.
:1:rocdevice.cpp            :590 : 102386476567 us: [pid:127063
tid:0x7fbecb179ac0] HSA device gfx906 (PCI ID 66af) has 2 ISAs but can
only support a single ISA
:1:rocdevice.cpp            :506 : 102386476578 us: [pid:127063
tid:0x7fbecb179ac0] Error creating new instance of Device.
free(): invalid pointer
Aborted                    AMD_LOG_LEVEL=5 ./hip5

This bad assumption was fixed in libamdhip64-6 6.4.3-1~exp2.

Sincerely,
Cory Bloor

Hi Cory,

Regression tests like these should be testable in our CI, by testing the
reverse dependencies from testing while pinning the new package from
unstable. This is what the official debci does to prevent migration.

For reverse build dependencies, bin:ratt helps.
because there is nothing yet stopping rocr-runtime from being rebuilt
from source with the "wrong" libamdhip64-dev.

I believe either rocr-runtime's B-D of libamdhip64-dev needs to be
bumped to 6.4.3-1~exp2, or libamdhip64-5 needs to be fixed in stable.

Best,
Christian

Hi Christian,

You have the dependency order backwards. The libamdhip64-5 package
depends on librocr-runtime64-1.

I suspect we can fix this in libamdhip64-5, but I'll have to take a
look. I'd be backporting some code from rocdevice.cpp to avoid "Error
creating new instance of Device".

Sincerely,
Cory Bloor

Oh, got it. Thanks for clarifying.

In that case, I think a

  Breaks: libamdhip65-5 (<= $somever)

would be warranted for libhsa-runtime64-1, to prevent the upgrade of the
latter on systems where this would trigger this case.

Where $somever would be the most recent version broken, so 5.7.1-7
unless I'm mistaken.
That might be even better. Obviously my own primary interest is for
ggml, but if this is a general issue, I suspect others could run into it
as well.

If a fix comes out, the Breaks could be adjusted for it.

Best,
Christian

On Thu, 23 Oct 2025 20:15:12 +0200 Christian Kastner <ckk@debian.org> wrote:
 > If a fix comes out, the Breaks could be adjusted for it.

The fix is quite simple [1]. I made the obvious fix in
d/p/0042-support-generic-isas.patch and then checked what upstream did.
Their fix was identical to mine, so I used their patch. I've proposed an
update to testing, although it doesn't sound like it will be accepted
[2]. I suppose I'll propose an update to stable later.

Sincerely,
Cory Bloor

[1]:
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=2;bug=1118212;filename=rocm-hipamd_5.7.1-7%2Bdeb14u1.debdiff;msg=85
[2]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1118212#85

Hi,

A bunch of things happened in the rocm stack last months. Do you still
think this bug still correctly block rocm-hiparmd from migrating to
testing? If yes, is there a plan to resolve this in the near future?
This package is trying to migrate to testing for 69 days now.

Paul

Ping.

Paul

Hello Paul,

We are working on migrating ROCm stack in Debian to >= 7.2.3, leveraging the work has been done in Ubuntu.

We started working on the migration already and packages are already being updated in salsa it is just matter of time
for the packages below to land in Debian Experimental first and then propagate.

```
| Package        | Salsa debian/experimental | Debian Experimental | Debian Unstable |
| -------------- | ------------------------- | ------------------- | --------------- |
| amdsmi         | 7.2.3-1~exp1              | 7.2.3-1~exp1        | 7.2.0-3         |
| hipify         | 7.2.4+dfsg-1~exp1         | —                   | 7.0.2+dfsg-1    |
| rocm-cmake     | 7.2.4-1~exp1              | —                   | 6.4.3-1         |
| rocm-core      | 7.2.4-1~exp1              | 6.4.4-1~exp6        | —               |
| rocm-smi-lib   | 7.2.4-1~exp2 UNREL        | 7.2.4-1~exp1        | 6.4.1-1.1       |
| rocm-llvm      | 7.2.4+dfsg-1~exp1         | 7.2.4+dfsg-1~exp1   | 7.0.2+dfsg-3    |
| rocr-runtime   | 7.2.4+dfsg-1~exp1         | 7.2.3+dfsg-1~exp1   | 6.4.3+dfsg-5    |
```

These are only the packages in dependency group 1 but sharing to report some progress on the work we are taking
under.

Best regards,
Talha

Hi Talha,


Thanks for the update. While not an immediate solution, it's good to see
that the problem is on your radar and you have a plan.

Paul

Hi,


2.5 months have passed, how is it progressing?

Paul

Hi,
Nearly 2 months further. While no freeze has been agreed or announced
yet, the past releases started freezing in the first months of odd
years. There should really be action to get this issue resolved well
before then.

Paul

Hi Paul,

I should probably have said earlier that I don't think Talha's
discussion of the update to a newer ROCm is relevant. This bug is
ultimately that src:rocm-hipamd < 6.4 contains a pointless assertion
that fails with src:rocr-runtime >= 6.4. It's already fixed for anything
depending on the binaries from src:rocm-hipamd >= 6.4. The only problem
is in packages left behind by the libamdhip64-5 -> libamdhip64-6
transition. Moving packages from ROCm 6 to ROCm 7 won't change anything.

So, this is not really a problem that is resolved by updating
rocr-runtime. The only thing we should do is remove that silly assertion
from src:rocm-hipamd on Trixie. We don't have to... but it would be
nice. I've been meaning to do that for ages. I'll make that my task for
today.

Sincerely,
Cory Bloor

Hi,

So are you saying that the issue doesn't affect unstable? Or are you
saying that without a fix in trixie this will be an issue during
upgrading to forky? This issue is blocking migration for months now.

Paul

Hi Paul,

It affects three rdepends on unstable (none of which are on testing).
The libamdhip64-6 transition was pretty messy (mea culpa). apt-rdepends
reports libamdhip64-5 has these reverse dependencies on unstable:

libamdhip64-5
   Reverse Depends: librccl1 (>= 5.4.3-3+b1)
   Reverse Depends: librccl1-tests (>= 5.4.3-3+b1)
   Reverse Depends: libstdgpu-hip0d (>= 1.3.0+git20220507.32e0517-6+b2)

I need to address those. They're still depending on libamdhip64-5
because they FTBFS on unstable. The rccl problem would legitimately be
resolved by updating it to 7.2 from experimental (or any other version
that successfully builds). So, I suppose the ROCm 7.2 update would
resolve that part of the problem. I'm not sure about stdgpu, as it
seemed to have more problems than just ROCm.

With regards to the Trixie -> Forky upgrade, libhsa-runtime64-1 already
has a Breaks: libamdhip64-5 that will ensure the upgrade will succeed
(but may remove packages to do so). The only relevant rocr-runtime
change for this bug would be if libamdhip64-5 were fixed on Trixie and
finally removed entirely from Unstable. If that happens, we could put a
version limit on the Breaks: libamdhip64-5 in libhsa-runtime64-1,
thereby improving the experience for Trixie upgrades and backports.

Sincerely,
Cory Bloor

Hi,


Ultimately that seems like a problem of those rdepends, not of the
depended-on package.

This all doesn't feel like it should block migration of the version in
unstable. Also, it doesn't look like the difference between the version
in unstable and the version in testing is all that big. Is this bug even
tagged with the right version?

Paul