Support Ollama LLM model pulling

[devtobi]

I want to suggest a new step

• Which tool is this about? Where is its repository?
  I want topgrade to support ollama, a tool for managing LLMs locally, see https://github.com/ollama/ollama.
  Topgrade should invoke model pulling to update all locally installed models.

• Which operating systems are supported by this tool?
  Windows, macOS, Linux

• What should Topgrade do to figure out if the tool needs to be invoked?
  if ollama is found in path it should be invoked, this also has to work when ollama was installed through package managers like homebrew.

• Which exact commands should Topgrade run?
  I am currently running through a custom command with ollama list | awk 'NR>1 && !/reviewer/ {system(\"ollama pull \"$1)}'
  This allows to pull all new model files for currently installed models.
  I am not sure if this command works OS independent.

• Does it have a --dry-run option? i.e., print what should be done and exit
  No dry run option available

• Does it need the user to confirm the execution? And does it provide a --yes
  option to skip this step?
  No user confirmation needed

More information

To test this a model should be downloaded on the local machine first, e.g. via ollama run qwen2.5-coder:0.5b.
When topgrade invokes ollama it should show an output similar to

pulling manifest
pulling 170370233dd5... 100% ▕████████████████████████████████████████████████████▏ 4.1 GB
pulling 72d6f08a42f6... 100% ▕████████████████████████████████████████████████████▏ 624 MB
pulling 43070e2d4e53... 100% ▕████████████████████████████████████████████████████▏  11 KB
pulling c43332387573... 100% ▕████████████████████████████████████████████████████▏   67 B
pulling ed11eda7790d... 100% ▕████████████████████████████████████████████████████▏   30 B
pulling 7c658f9561e5... 100% ▕████████████████████████████████████████████████████▏  564 B
verifying sha256 digest
writing manifest
success

[deadeyesky]

This would be a sick feature.

[SteveLauC]

I will try implementing it recently.

[SteveLauC]

Noticed that there is a tag along with the model name, which is kinda similar to docker/podman images, could this tag be a version number? If so, what will $ ollama pull <model_name> do? Update it to latest or do nothing

$ ollama list
NAME                ID              SIZE      MODIFIED      
moondream:latest    55fc3abd3867    1.7 GB    5 minutes ago  

[devtobi]

@SteveLauC If you run the pull command without adding a tag, it will always pull the latest tagged image besides the other tags you already have on your system.
However since ollama list includes the image tags in the output, my command I mentioned above should preserve only pulling the latest versions of the respective tagged images.
This is desired because the models have different sizes in parameters e.g. 2b, 7b and are made for different purposes e.g. instruct, chat. So always updating to the latest tag is not what you want in this case (except if you explicitely installed latestvia ollama pull my-model or ollama pull my-model:latest)

[SteveLauC]

Thanks for the explanation, I will pull the model with a tag included so that Topgrade won't do anything that users do not want to do.

[SteveLauC]

Implemented in #1001, anyone wants to give it a test? I can provide Linux/x86_64 and macOS/arm64 binaries. For the macOS build, I am not sure if it can be executed on other Macs, it seems that the binary needs to be signed.

[SteveLauC]

Here is what it looks like on my machine:

$ ./target/debug/topgrade --only ollama

── 18:51:48 - Sudo ─────────────────────────────────────────────────────────────

── 18:51:48 - Ollama ───────────────────────────────────────────────────────────
Pulling model 'gemma2:2b'
pulling manifest 
pulling 7462734796d6... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏ 1.6 GB                         
pulling e0a42594d802... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏  358 B                         
pulling 097a36493f71... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏ 8.4 KB                         
pulling 2490e7468436... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏   65 B                         
pulling e18ad7af7efb... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏  487 B                         
verifying sha256 digest 
writing manifest 
success 
Pulling model 'moondream:latest'
pulling manifest 
pulling e554c6b9de01... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏ 828 MB                         
pulling 4cc1cb3660d8... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏ 909 MB                         
pulling c71d239df917... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏  11 KB                         
pulling 4b021a3b4b4a... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏   77 B                         
pulling 9468773bdc1f... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏   65 B                         
pulling ba5fbb481ada... 100% ▕██████████████████████████████████████████████████████████████████████████████████████▏  562 B                         
verifying sha256 digest 
writing manifest 
success 

── 18:51:51 - Summary ──────────────────────────────────────────────────────────
Ollama: OK

[devtobi]

Looks great :) Whats the quickest way for me to test? I can test the Mac ARM64 side. 😄 Should I just checkout your branch and build myself?

[SteveLauC]

If you have rustup installed, yeah, you can build it yourself:

$ git clone https://github.com/SteveLauC/topgrade.git
$ cd topgrade
$ cargo build
$ ./target/debug/topgrade --only ollama

If not, I can build it for you and upload it here: https://github.com/SteveLauC/topgrade/releases

[devtobi]

My local environment is currently not set up for Rust development, so I would be happy to have a provided binary from your side. ;) We will see if it works out with the signing problem. ;)

[SteveLauC]

There you go: https://github.com/SteveLauC/topgrade/releases/tag/pr1001

[devtobi]

Thanks for sharing. Everything works like a charm with the binary you provided if only using pulled models. 😄 I get the same output as you shared above.

However we have an issue, when models are built locally. Just like with Docker and Dockerfile, ollama allows building custom models using Modelfile.

The problem now is, that the command ollama list also outputs the locally built models and thus topgrade tries to pull them.
This leads to an error like no manifest file found and the new "Ollama" step of topgrade exits with a fail.

Unfortunately ollama list does not provide a flag to only list pulled model. As far as I saw it ollama saves its models in a specific user directory. Maybe we could find out which models are pulled/created locally this way.

However I am not 100% sure if the effort is justified. Most users most likely only use downloaded models. So this might be kind of an edge case I described here.

We still need to make sure models are further pulled down after a "pull error" occured. I don't know if topgrade stops the executing immediatly after an error occurred or not. In my case the failing model pull was executed last, so I couldn't reproduce this edge case.