Hello there, thanks for packaging and maintaining llama.cpp in Debian!

I am trying to compare different backends, to see which one works best on
my hardware.

```
$ llama-bench -hf MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF
load_backend: loaded CPU backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 7B Q4_K - Medium         |   4.07 GiB |     7.25 B | CPU        |       6 |           pp512 |         44.84 ± 0.18 |
| llama 7B Q4_K - Medium         |   4.07 GiB |     7.25 B | CPU        |       6 |           tg128 |          6.73 ± 0.03 |

build: Debian (11429)

# aptitude install libggml0-backend-vulkan

$ llama-bench -hf MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-vulkan.so
load_backend: loaded CPU backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 7B Q4_K - Medium         |   4.07 GiB |     7.25 B | Vulkan     |  -1 |           pp512 |         85.80 ± 0.25 |
| llama 7B Q4_K - Medium         |   4.07 GiB |     7.25 B | Vulkan     |  -1 |           tg128 |          6.96 ± 0.00 |

build: Debian (11429)

# aptitude install libggml0-backend-hip

$ llama-bench -hf MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 13977 MiB):
  Device 0: AMD Radeon Graphics, gfx90c:xnack+ (0x90c), VMM: no, Wave Size: 64, VRAM: 13977 MiB
load_backend: loaded ROCm backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-hip.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-vulkan.so
load_backend: loaded CPU backend from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
ROCm error: invalid device function
  current device: 0, in function ggml_cuda_kernel_launch at /build/reproducible-path/ggml-0.27.0/src/ggml-cuda/common.cuh:1720
  hipGetLastError()
/build/reproducible-path/ggml-0.27.0/src/ggml-cuda/ggml-cuda.cu:109: ROCm error
[New LWP 219286 (id 2)]
[New LWP 219277 (id 3)]
[New LWP 219276 (id 4)]
[New LWP 219274 (id 5)]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.debian.net>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
[Switching to thread 1 (Thread 0x7f2bf5530440 (LWP 219273))]
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
⚠ warning: 56   ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x00007f2bf566d9d4 in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:49
⚠ warning: 49   ./nptl/cancellation.c: No such file or directory
#2  0x00007f2bf566da1d in __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x00007f2bf56d8b97 in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
⚠ warning: 30   ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x00007f2bf5547963 in ?? () from /usr/lib/x86_64-linux-gnu/libggml-base.so.0
#5  0x00007f2bf5547b2e in ggml_abort () from /usr/lib/x86_64-linux-gnu/libggml-base.so.0
#6  0x00007f2ba8390ec2 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-hip.so
#7  0x00007f2ba84b3d3a in ?? () from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-hip.so
#8  0x00007f2ba84b2733 in ggml_cuda_op_rms_norm_fused(ggml_backend_cuda_context&, ggml_tensor*, ggml_tensor*) () from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-hip.so
#9  0x00007f2ba839d0f6 in ?? () from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-hip.so
#10 0x00007f2ba8397357 in ?? () from /usr/lib/x86_64-linux-gnu/ggml/backends0/libggml-hip.so
#11 0x00007f2bf5561866 in ggml_backend_sched_graph_compute_async () from /usr/lib/x86_64-linux-gnu/libggml-base.so.0
#12 0x00007f2bf4df0f80 in llama_context::graph_compute(ggml_cgraph*, bool) () from /usr/lib/x86_64-linux-gnu/libllama.so.0
#13 0x00007f2bf4df416a in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /usr/lib/x86_64-linux-gnu/libllama.so.0
#14 0x00007f2bf4df9a8a in llama_context::decode(llama_batch_ext const&) () from /usr/lib/x86_64-linux-gnu/libllama.so.0
#15 0x00007f2bf5af3e6c in ?? () from /usr/lib/x86_64-linux-gnu/llama/libllama-bench-impl.so
#16 0x00007f2bf5b04b68 in llama_bench(int, char**) () from /usr/lib/x86_64-linux-gnu/llama/libllama-bench-impl.so
#17 0x00007f2bf5605fba in __libc_start_call_main (main=main@entry=0x56504cc97230, argc=argc@entry=3, argv=argv@entry=0x7ffe7555eea8) at ../sysdeps/nptl/libc_start_call_main.h:59
⚠ warning: 59   ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#18 0x00007f2bf56060d7 in __libc_start_main_impl (main=0x56504cc97230, argc=3, argv=0x7ffe7555eea8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe7555ee98) at ../csu/libc-start.c:360
⚠ warning: 360  ../csu/libc-start.c: No such file or directory
#19 0x000056504cc97261 in ?? ()
[Inferior 1 (process 219273) detached]
Aborted                    llama-bench -hf MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF
```

As you can see, `llama-bench` crashed when trying to use the HIP backend.
Why?
Is there any package that I am missing?
Shouldn't it be among the dependencies of package `libggml0-backend-hip`?

What did I fail to understand?

My hardware is:

```
$ lscpu | grep 'Model name'
Model name:                              AMD Ryzen 5 5500GT with Radeon Graphics
```

Please help me to make it work, thanks for your time!


P.S.: Is it expected (on my hardware) that the prompt-processing
      performance (`pp512`) for the Vulkan backend is almost double the
      CPU backend one, while the token-generation performance (`tg128`)
      is only very slightly better?

control: severity -1 wishlist

Hi Francesco,
ROCm way of saying "the library you're trying to use was not built for
your GPU." The gfx90c architecture is not currently on the list of GPUs
supported by ROCm math libraries in Unstable or Forky, because the
"fallback architecture" patch we were using to enable gfx90c in ROCm 5
could not easily be carried over into ROCm 6 or later. This was
mentioned briefly in the NEWS for rocm-hipamd 6.4.3-1 [1]. There is a
suggested workaround on the Supported GPU wiki page [2] (which also
reminds me that the wiki has not yet been updated for Forky or current
Unstable). For posterity, I'll repeat the suggestion here: try setting
`export HSA_OVERRIDE_GFX_VERSION=9.0.0` in your environment to use the
code objects built for gfx900 on your gfx90c GPU.

I'm hoping we'll be able to get gfx90c working without needing those
sorts of hacks for Forky, but I cannot make any hard promises. All I can
say is that I will do my best. However, I'd be surprised if the HIP
backend performs better than the vulkan implementation on gfx90c even if
I get it working. I don't think upstream has done much rocblas kernel
tuning for gfx90c (if any), but tuned kernels are vital for achieving
optimal performance on the ROCm BLAS stack.

Sincerely,
Cory Bloor

[1]:
https://salsa.debian.org/rocm-team/rocm-hipamd/-/blob/debian/6.4.3-1/debian/NEWS?ref_type=tags
[2]:
https://salsa.debian.org/rocm-team/community/team-project/-/wikis/Supported-GPU-list#gfx90c