Make bitsandbytes portable (with express goal of supporting Apple Silicon/ARM64)

[rickardp]

See issue #252 for discussion

• C++ compiles on ARM64 in non-SIMD mode
• C++ compiles on ARM64 with SIMD (Neon) instructions
• Do not fail unit tests if no CUDA device present
• Make Python library portable to non-CUDA
• Unit tests run on devices present
• Make unit tests work regardless of CUDA present
• GitHub workflow to build wheels for all supported platforms (I've done this in https://github.com/rickardp/splitstream/blob/master/.github/workflows/python-package.yml, some inspiration will be taken from there although this is using a separately built shared lib and will be done differently)
• Replace use of pthread with canonical C++ std::thread (while also avoiding arcane C malloc and using type-safe C++ primitives)
• Switch to CMake to reliably build on all platforms.

[rickardp]

https://github.com/rickardp/bitsandbytes/actions/runs/4719555280

The GH actions added in this PR now builds the library and wheels for the following platforms

• Linux (x64 + ARM64) - CUDA+CPU
• Windows x64 - CUDA + CPU
• macOS (x64 + ARM64) - CPU

i think it could be a good point limit this PR to add this form of portability, and then fork out with different efforts depending on interest

• add more CUDA versions (and h/w archs) to matrix
• Add MPS for Apple platforms (obviously, I am keen on this)
• Add CPU implementations for more of the CUDA ops (maybe mainly for baselining/testing - running CUDA stuff is expensive on cloud hardware and not feasible for an open source project)

[TimDettmers]

I just want to leave a comment here for the community. I am not sure why I closed this. For most issues that I closed I left a comment to why. There was significantly work invested in this and it is a great contribution! Sorry, that I close it like this — it must have been very frustrating.

We will try to add Apple silicon support soon along with AMD and Intel support. We need to figure out next steps first though. It will likely take some weeks, but we will get there. This is one of the highest-priority features that we have and we start working on this soon.

[rickardp]

No worries! It is good to hear back from you that portability is a goal and priority. At the time there was a lot of active talk and PRs floating around, even some ideas on forking, in anticipation of an "official" answer from the maintainers. Good to avoid the fragmentation.

In one of these issues there were quite a number of people expressing interest in helping out. If you get the basics running you might be able to make good use of the community here to progress faster.

[Titus-von-Koeller]

@rickardp The closing of this PR was an accident it seems. We're really sorry about that. I don't find any active Apple Silicon PRs. What's the status? We would be happy to help.

[Titus-von-Koeller]

Ah, now I see, this got replaced by #947. I also gotta gather info about forks. If you can point me to anything important, let me know. It would be good to avoid fragmentation. BNB now has an official maintenance team and is sponsored by Hugging Face. Therefore, we're quite certain that such things wont go unaddressed for so long in the future.

[rickardp]

Ah, now I see, this got replaced by #947. I also gotta gather info about forks. If you can point me to anything important, let me know. It would be good to avoid fragmentation. BNB now has an official maintenance team and is sponsored by Hugging Face. Therefore, we're quite certain that such things wont go unaddressed for so long in the future.

Yes, I tried to save portability changes in #947, but I would say that #949 is the successor to this one. The idea was to make it possible to easily parallelize the porting effort by ensuring automated test and cross platform builds using GH actions. This way it would be easy to verify correctness of new kernels.

It is good that this project now have a dedicated mantenance team. There were some talks about forking a while back due to the lack of response from maintainers earlier, but AFAIK those efforts are no longer active. I think the discussion were in the now closed PRs and issues (like #485) and were continued on the HF Discord. I had to attend to other stuff after a while so I might have missed out on some discussions.

[matthewdouglas]

I think the discussion were in the now closed PRs and issues (like #485) and were continued on the HF Discord.

@rickardp Is there a particular channel for this on HF Discord? I'm interested in getting more involved.

[rickardp]

I think the discussion were in the now closed PRs and issues (like #485) and were continued on the HF Discord.

@rickardp Is there a particular channel for this on HF Discord? I'm interested in getting more involved.

Now that this repo is more actively maintained, maybe better to keep the discussions here? @Titus-von-Koeller @TimDettmers have you considered enabling GitHub discussions on this repo?

[Titus-von-Koeller]

Yes, thanks for the hint! I agree, it would be better to keep these discussions here. I'll search the HF Discord to see what I find and look into GH Discussions.

There's so much willingness to contribute in the community around this library: really, really cool! I'll work hard to enable people and change the circumstances to make it easier to enable + accept contributions. I'm on bitsandbytes full-time now and @younesbelkada will also contribute a significant fraction of his time. We also have HF as an official sponsor now, yay 🤗

Feel free to contact me (also through messaging, e.g. Slack or Discord) if you have any needs or ideas regarding BNB.

Thanks again for your active contributions and sorry for any delays + frustrations you may have experienced!

[Titus-von-Koeller]

Ok, we enabled Discussions and I started one on the cross-platform topic.

#990

[younesbelkada]

Thanks @rickardp for all your work on this ! Thanks @Titus-von-Koeller for your message 💪
Could you send us an email we can reach you out so that we can more closely collaborate on this through Slack ? 🙏 Thanks !