Machine Learning Spin

[J3rr1ck]

Hey all, I'm curious if there's been any discussion around an ML/AI Spin of UBlue, the Nvidia base includes the needed libs for ML/AI workloads so for that it would just be a matter of using a lighter desktop (budgie would be a great option) and cuda pytorch + possibly jupyter

I don't have a rocm gpu to test what would be needed for AMD but would be more than happy to Start on an nvidia version :)

[mister-drgn]

(Speaking as an interested observer, not a ublue developer...)
(Also, apologies if parts of this are obvious to you. Looking back over your question, it looks like you may already know a lot of this.)

In principle, it should be pretty easy to do this by adding a few libraries to an existing nvidia image. However, I'm not sure you'd want take this approach. Installing your ML libraries into an image makes sense if you know they won't change--you'll always be using the same libraries. But the ML field is changing incredibly quickly, meaning there are likely new libraries you'll find yourself wanting to add. At the same time, if you're trying to run someone else's code that might be a couple years older or more, you could find yourself wanting to use an older version of the libraries. So you really want a greater degree of flexibility than an immutable system would afford.

I would suggest taking one of two approaches.

1. If you do all your work in python, you can install python libraries locally in your home directory. You can also use python virtual environments to set up multiple python instances, each with their own versions of various ML-related libraries. So you'd create one environment for each project, depending on the needs of the project.

2. You could take things further and do your work in containers, using docker or podman. Then you can install exactly what you need in each container, not just python libraries but also anything else required, for example cudnn, the cuda neural network library.

Personally I use approach 2. I'm currently running Fedora (not immutable) but doing all my dev work in ubuntu-based containers.

That said, you could imagine setting up a UBlue image to support conducting ML research & development in containers. I may actually do something like that. I currently have a setup with Cinnamon DE where I can move around in a workspace grid, and different workspaces are associated with running the terminal in different containers. It's still pretty new, but I think it's going to be a cool way of doing work moving forward.

[J3rr1ck]

on this note IIRC the nvidia images actually include all of the needed support of the bat for cuda work and I actually do my work on a bluefin-nvidia install since it has nvidia-container kit, took a bit of jiggering of configs to get cuda allocating correctly but it's realistically there.
if anyone lurking has a ROCm GPU I'd love to get insights into what would be needed for the main images to run ML/AI with a rocm backend.
Circling back to point two while this is ideal and how most experts are working I think that it's actually a harder task for some newbies to get started with I'd think ideally for an immutable ML spin what would probably work out best is some sort of prebaked cuda/rocm distrobox/podman containers with a generic linux (fed or ubu) container + cuda and anaconda+jupyter?

[mister-drgn]

A newcomer-friendly setup does make a lot of sense. One issue with prebaked ML container images is where are they stored? If you're using using podman for rootless containers, that stores images to the user's home directory, which rpm-ostree doesn't touch, so the system can't come with those pre-installed. What you might want is basically:

1. A base image for the system (likely something like the current nvidia ones would suffice, as you said...I don't have any experience with rocm gpus).
2. A set of images for beginner-friendly ML applications. These could all be hosted at the same location as the base image. I wouldn't be surprised if there are existing options out there. Personally, I always start with nvidia's official cuda images and build from there (EDIT: I see you do as well).

And then users could use distrobox (or their preferred container approach, but distrobox is nice and convenient) to launch containers from 2.

Random thing about distrobox:
The current ublue images install the latest distrobox release, which I don't really like. Since that release, distrobox has added a --nvidia flag that does some kind of magic to give you gpu access inside your containers. I'm not sure how it works, but it does, which is nice because I've had issues getting nvidia-container-toolkit working with podman (though I use it with docker). Since there hasn't been a release since that flag was added, the only way to get it is to clone the project from git, which is easy to do because you don't have to build anything--it's just shell scripts.

[castrojo]

I wouldn't be surprised if there are existing options out there.

Yeah I think that trying to put all that stuff on an image won't scale, but if we got good at finding the containers that people are using in production and streamlined their usage on our images it would be a valuable thing for users. Even though right now it'd be just a just deploy-ml-blah or whatever for now.

[castrojo]

Even if all we did was start collecting premade yaml configs so people can just podman deploy stuff it'd probably be a useful companion to the nvidia images, and we'd be able to figure out what makes sense on an image depending on how people use it!

[J3rr1ck]

I started putting together a list of recipes that I use regularly on GitHub: https://github.com/J3rr1ck/uBlue-AI-OCI.
I'll add some more later today, and I'm open to receiving Pull Requests to make the list even more complete.

[castrojo]

Just thinking aloud a bit more since I'm not an expert in AI/ML. I was doing searches for some images to see what's out there:

• Amazon's: https://github.com/aws/deep-learning-containers/blob/master/available_images.md
• Google: https://cloud.google.com/deep-learning-containers/docs/choosing-container

And there's probably piles from other vendors. I'm wondering if we could grab a popular stack and do a "hello world" that would be useful for folks, maybe the play should be curation of known-working stacks on top of the images we have?

[mister-drgn]

@J3rr1ck might have a better idea of what people want, since I just I kinda of live in my own world--I'm an AI researcher but not an ML person, so I'm usually just looking for containers to run other people's systems. But I tend to use nvidia's containers when possible, and then add onto them whatever is needed. They have a massive selection that makes finding the right thing difficult sometimes, but here, for example, are their images for pytorch and tensorflow, the two most popular ML libraries:

https://catalog.ngc.nvidia.com/orgs/nvidia/collections/nvidia_dlfw

Obviously they only provide containers for running on nvidia gpu's. For other things, something like google's containers might be fine.

[J3rr1ck]

The Nvidia containers are great starting points for Nvidia but are extremely minimal and are missing many common utils, I know most firms have their own internal starting point images that almost always include newer python and a handful of other utils. AMD is kinda a while different beast I'm not super familiar with, I can definitely check back in on this after doing some reading.