Dear Maintainer,

my backport of llama.cpp/ggml for trixie starts to fail when upgrading
libraries from rocr-runtime from 6.1 to 6.4.

llama.cpp and ggml were recently removed from testing for dependency
reasons, but since this is nevertheless easily reproducible, I need to
suspect that rocr-runtime is broken, therefore filing with severity:
serious to prevent migration to testing.


Steps to reproduce on a vanilla trixie system with an AMD GPU:

// Prerequisites
# cat /etc/apt/sources.list
deb     http://192.168.0.21:9999/debian trixie main contrib non-free-firmware
deb-src http://192.168.0.21:9999/debian trixie main contrib non-free-firmware

deb     http://192.168.0.21:9999/debian unstable main contrib non-free-firmware
deb-src http://192.168.0.21:9999/debian unstable main contrib non-free-firmware

# cat /etc/apt/preferences.d/unstable.pref
Package: *
Pin: release a=unstable
Pin-Priority: 50


// We need some build dependencies from unstable, primarily for #1116585
# apt-get install -y pkg-rocm-tools/unstable libamdhip64-dev/unstable libhiprtc-builtins5/unstable

// Set to whatever test arch you are using, to speed things up
# export ROCM_TARGET_ARCH_FIXED=gfx1030

// Build the package for trixie
# apt-get build-dep -y ggml
# apt-get source ggml && cd ggml-*
# dpkg-buildpackage

// Install ggml with the right backends
# dpkg -i ../libggml0_*.deb ../libggml0-backend-{cpu,hip}_*.deb

// Install llama.cpp from unstable
# apt-get install -y llama.cpp-tools/unstable

// Download a model
# wget https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf

// Known good case produces benchmark statistics
# llama-bench -m gemma-3-1b-it-Q4_K_M.gguf

// Upgrade rocr-runtime
# apt-get install -y libhsa-runtime-dev/unstable libhsa-runtime64-1/unstable libhsakmt1/unstable

// llama-bench now errors out
# llama-bench -m gemma-3-1b-it-Q4_K_M.gguf

KFD does not support xnack mode query.
ROCr must assume xnack is disabled.
free(): invalid pointer
Aborted


Best,
Christian

Thank you for the report.

I was testing with packages built against newer versions of the HIP
Runtime, so I didn't catch this regression. I'd only tested the basics
with rocminfo, which worked fine for this case because it doesn't use
the broken function.

The problem seems to be that upstream changed the ABI of
hsa_agent_iterate_isas by changing the types for the callback function:

Old Callback Data Definition:

     std::pair<hsa_isa_t, bool> comparison_data

New Callback Data Definition:

     struct callBackData {
         std::pair<hsa_isa_t, bool> comparison_data;
         const unsigned int codeGenericV;
     };

Sincerely,
Cory Bloor

Control: reassign -1 libamdhip64-5 5.7.1-6

I was mistaken. I tried to fix the ABI breakage only to find nothing
there. AFAICT, the expected data layout for the callback function is
defined by the caller, not by libhsa-runtime64-1. I created a minimal
reproducer to compare the traced output when linked libamdhip64-5 and
libamdhip64-6.

$ cat main.hip
#include <hip/hip_runtime_api.h>

int main() {
   float *dArr;
   hipMalloc(&dArr, 1);
   return 0;
}

$ clang++-21 main.hip -o hip5
$ apt install -t experimental libamdhip64-dev
$ clang++-21 main.hip -o hip6
$ ltrace -C -e "hsa*" -e "hip*" ./hip5  2> ltrace5.txt
$ ltrace -C -e "hsa*" -e "hip*" ./hip6 2> ltrace5.txt

The ltrace5.txt and ltrace6.txt logs are attached. The output indicates
the crash is in libamdhip64-5.

Enabling logging indicates that this is the fault of libamdhip64-5, as
it incorrectly assumes there cannot be more than one supported ISA for a
given device.

$ AMD_LOG_LEVEL=5 ./hip5
:3:rocdevice.cpp            :442 : 102386455878 us: [pid:127063
tid:0x7fbecb179ac0] Initializing HSA stack.
:1:rocdevice.cpp            :590 : 102386476567 us: [pid:127063
tid:0x7fbecb179ac0] HSA device gfx906 (PCI ID 66af) has 2 ISAs but can
only support a single ISA
:1:rocdevice.cpp            :506 : 102386476578 us: [pid:127063
tid:0x7fbecb179ac0] Error creating new instance of Device.
free(): invalid pointer
Aborted                    AMD_LOG_LEVEL=5 ./hip5

This bad assumption was fixed in libamdhip64-6 6.4.3-1~exp2.

Sincerely,
Cory Bloor

Hi Cory,

Regression tests like these should be testable in our CI, by testing the
reverse dependencies from testing while pinning the new package from
unstable. This is what the official debci does to prevent migration.

For reverse build dependencies, bin:ratt helps.
because there is nothing yet stopping rocr-runtime from being rebuilt
from source with the "wrong" libamdhip64-dev.

I believe either rocr-runtime's B-D of libamdhip64-dev needs to be
bumped to 6.4.3-1~exp2, or libamdhip64-5 needs to be fixed in stable.

Best,
Christian

Hi Christian,

You have the dependency order backwards. The libamdhip64-5 package
depends on librocr-runtime64-1.

I suspect we can fix this in libamdhip64-5, but I'll have to take a
look. I'd be backporting some code from rocdevice.cpp to avoid "Error
creating new instance of Device".

Sincerely,
Cory Bloor

Oh, got it. Thanks for clarifying.

In that case, I think a

  Breaks: libamdhip65-5 (<= $somever)

would be warranted for libhsa-runtime64-1, to prevent the upgrade of the
latter on systems where this would trigger this case.

Where $somever would be the most recent version broken, so 5.7.1-7
unless I'm mistaken.
That might be even better. Obviously my own primary interest is for
ggml, but if this is a general issue, I suspect others could run into it
as well.

If a fix comes out, the Breaks could be adjusted for it.

Best,
Christian

On Thu, 23 Oct 2025 20:15:12 +0200 Christian Kastner <ckk@debian.org> wrote:
 > If a fix comes out, the Breaks could be adjusted for it.

The fix is quite simple [1]. I made the obvious fix in
d/p/0042-support-generic-isas.patch and then checked what upstream did.
Their fix was identical to mine, so I used their patch. I've proposed an
update to testing, although it doesn't sound like it will be accepted
[2]. I suppose I'll propose an update to stable later.

Sincerely,
Cory Bloor

[1]:
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=2;bug=1118212;filename=rocm-hipamd_5.7.1-7%2Bdeb14u1.debdiff;msg=85
[2]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1118212#85