* Package name    : ollama
  Version         : 0.5.7-1
  Upstream Author : Ollama
* URL             : https://github.com/ollama/ollama
* License         : Expat
  Programming Lang: Go
  Description     : large language model tools

 Ollama: Get up and running with large language models.

https://salsa.debian.org/go-team/packages/ollama
https://salsa.debian.org/jas/ollama/-/pipelines

/Simon

@ckk is planning to package llama.cpp within debian deep learning
team (debian-ai@l.d.o). Maybe you want to discuss with the team
whether you want to deal with the embedded copy of llama.cpp inside
ollama source tree?

I did not look into how ollama enables ROCm and CUDA support, but
that's also something the team care about. Do you want to enable
any of them?

Hi,

llama.cpp will land this weekend. I'm almost finished; I still need to
tweak some corner cases. But I also need to run some final tests.

Funnily enough, this package is trickier than one might initially
believe. The library needs to be private; multiple versions need to be
build (one or more for each backend); the library abstraction isn't that
strict; and it requires non-free data to properly test and benchmark.

I've solved most of these problems, so perhaps my approach can help once
it's done.

Best,
Christian

Christian Kastner <ckk@debian.org> writes:

Hello.  That is really good to know!

Do you have any general thoughts around packaging ollama?

I started looking at the llama.cpp code that is inside ollama and I have
many concerns.  Just writing the debian/copyright file didn't feel fun.

Ollama intentionally patch upstream llama before use, see:

https://salsa.debian.org/jas/ollama/-/blob/debian/sid/llama/README.md?ref_type=heads#vendoring
https://salsa.debian.org/jas/ollama/-/tree/debian/sid/llama/patches?ref_type=heads

I didn't review any of them yet.  Any thoughts?  Could some be
upstreamed, or made tunable somehow so that ollama could get what it
wants from llama.cpp but not disturb other users?

Do you think it is possible for ollama to use a llama.cpp packaged
outside of ollama?  The entire package is complex and I haven't
familiarized myself with it.  My perception is that it builds llama.cpp
in a way that fits ollama and integrate it into the ollama Go build.
Finding some way to revert or parametrize that integration would be
nice, so that ollama could use the Debian-packaged llama.cpp code.  But
I'm not sure how to do that, given the patches above and generally how
this is all integrated together.  And different release schedules.

I am very happy if you or others from the debian-ai team can help on
ollama, or join the debian-ai to maintain it there.  I'm certainly no
llama expert and the more I look into this package, the more I think
such knowledge will be useful.  I put it into the Go team because that's
what I'm familiar with, and because ollama is written in Go.  But the
llama relationship has to be resolved somehow.

/Simon

Hi Simon,

Are you still working on packaging Ollama?

I have been looking at Ollama, Jan.ai, LocalAI (https://localai.io/)
and GPT4All (nomic.ai/gpt4all). It would be nice to have at least one
of these available in Debian so people can run them natively without
AppImages and whatnot, and with the assurance of at least the
traceable supply chain that Debian offers.

Like others, Ollama has llama.cpp as the main backend, which is in
Debian but never entered testing
(https://tracker.debian.org/pkg/llama.cpp) because its dependency ggml
never entered testing (https://tracker.debian.org/pkg/ggml) due to
hard dependencies on AMD ROCm libraries (even though many would end up
running llama.cpp specifically on CPU and only a minority own AMD
GPUs.

Related, I also noticed these:

#1092958 ITP: llm-ollama -- LLM plugin providing access to models
running on an Ollama server

#1108419 RFP: alpaca -- An Ollama client made with GTK4 and Adwaita

Hi all,

FWIW, the ROCm blocker is actually because of the documentation build,
blocked by some dependency 4 layers below:

  rocblas -> doxysphinx -> sphinx-needs -> python-memray -> libunwind

The underlying cause in libunwind appears to have been fixed, but I now
see a regression on riscv64...

Alternatively, the rocblas/hipblas maintainers could temporarily disable
the documentation build.

Best,
Christian

Hello!

Maybe there could be for the start some CPU & Vulkan enabled version? Support
for ROCm can be definitely enabled later.

Thank you for working on this!

David

Hello,

May I also suggest, the packages being partitioned into a base ollama
package and then separate packages for the GPU support (ollama-cuda,
ollama-rocm, ollama-vulkan) - or something along those lines.

The whole thing being about 5GiB when one only uses part of it feels a bit
odd, to say the least...

Best,
Daniel