Conda-forge package of NNPOps

[dominicrufa]

@peastman , I've had some difficulty cmaking and installing this repo (ultimately with success). would it be possible to make this conda-installable so we can have a consistent working version with openmm-torch and OpenMM 7.5.1 maybe?

[peastman]

I think that ought to be possible. I don't know of any reason it wouldn't be.

[dominicrufa]

alright. is there anything that is blocking re: this? it would help if we can get this in asap.

[peastman]

So far as I know, the only thing blocking it is time. Can you implement it? I don't expect to be able to any time soon. @raimis created the Python module, so he might be able to give advice on what needs to be done.

[jchodera]

@peastman: We can do this, but we'll restructure this repo to conform to some familiar structure, including:

• Making this an installable Python package
• Adding continuous integration
• Adding standard documentation framework
• Building a conda-forge package via a feedstock

@ijpulidos will reach out to you to schedule a time for us (with @mikemhenry) to chat and figure out what needs to be done here. Hopefully @raimis can make it too, but given the differences in time zones, it may be difficult to plan for that.

[jchodera]

Take a look at PR #15 for inspiration

[jchodera]

We might also use this opportunity to restructure based on MolSSI cookiecutter

[raimis]

@mikemhenry we have a proper build system (#33). Could you try to make a feedstock to see if more changes are needed?

[raimis]

For an actual release, I still have to finish #13, but it is already good for an alpha release.

[mikemhenry]

Sounds good! I will get working on the feedstock!

[mikemhenry]

@raimis Can I publish a 0.0.1 release? I'll need to have a release in order to make the feedstock.

[raimis]

@peastman has to do that. I don't have the permission.

[mikemhenry]

@peastman can you give me permissions to create releases?

In the meantime, I can use my fork with a release on it to play around with the recipe and can just swap URLs before merging.

[peastman]

@peastman has to do that. I don't have the permission.

You should have permission now.

[mikemhenry]

@peastman I don't

Just to check, this is what it looks like on a repo that I can make a release for:

I also tried using the cli to make a release and I got a 404 from the api endpoint, so I don't think I've got permission.

[mikemhenry]

@raimis
What OS do we want to support? Linux and MacOSX? Or windows as well? Also I saw the environment yml pins python 3.9, does it only work with python 3.9 or can we support 3.7, 3.8, 3.9? (assuming the dependency resolver can find the packages needed).

[raimis]

We need it to be compatible with OpenMM-Torch (https://github.com/conda-forge/openmm-torch-feedstock). So just copy what OpenMM-Torch supports.

[raimis]

I have finished #13. So we have everything merged regarding the ANI models.

@mikemhenry how is it going with the packages?

[jchodera]

Fantastic! 🎉 Thanks so much @raimis!

@mikemhenry : Have you managed to connect up directly with @raimis to see if we need to restructure the repo any to help make packaging easier (e.g. make it look more like the MolSSI cookiecutter with a setup.py, etc)?

[raimis]

There is a bit difference between:

• Pure Python package
• C++/CUDA library with Python API

Cookiecutter and setuptools are good for the first case, but they aren't for the second one. setuptools does not have a builtin support of CUDA. PyTorch-Geometric uses the PyTorch extension (https://pytorch.org/docs/stable/cpp_extension.html) of setuptools for that, but it looks overengineered.

The easiest solution is just to use CMake, which has a good support of CUDA and we have plenty of experience using it for OpenMM byself and other projects.

[mikemhenry]

Good! I've got a feedstock that I'm working with, just working on making it build on multiple platforms

[raimis]

@mikemhenry for the moment we need just Linux-64, the others can wait.

[raimis]

@mikemhenry I have tagged the first release (https://github.com/openmm/NNPOps/releases/tag/v0.0). Use it to build the packages.

[mikemhenry]

Nothing, but using {{ compiler('cxx') }} for the compilers lets conda-forge figure out which package it needs to pull in based on the OS + makes things less fragile when migrations happen/versions change, but now I'm just focusing on getting it on the forge ASAP and we can later figure that out.

[mikemhenry]

@raimis Does it make sense to have a CPU build of this package?

[raimis]

In general yes, but it isn't a priority.

The source will have to be modified to enable a conditional building of the CUDA part.

[mikemhenry]

@raimis
How hard would it be to modify the source so CUDA is conditionally built?
Because of the way conda-forge works, it has been hard to get GPU builds to work in the staging-recipe repository:

conda-forge/staged-recipes#16257 (comment)

I think if set things up so we can just build a CPU version, then we can get a CPU build on conda-forge, then we can get the GPU build added by working in the feedstock repository instead.

[raimis]

@mikemhenry OK, I'll add a CPU-only option.

[raimis]

@mikemhenry I have added an option for a CPU-only build (#48):

$ cmake .. \
        -DENABLE_CUDA=OFF \
        -DTorch_DIR=$CONDA_PREFIX/lib/python3.9/site-packages/torch/share/cmake/Torch \
        -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX

After #41 is merged, I will make a release of NNPOps 0.2. If you need a tag faster, I can make a release candidate.

[mikemhenry]

A release candidate would be great! But I could also merge the PR into my fork for getting the staging recipe on conda-forge and then change it later as well, whatever works best for you! Thanks!

[raimis]

Try with your fork, it will be faster.

[raimis]

@mikemhenry the CPU-only option (#48) has been merged!

[raimis]

@mikemhenry what is the situation with this?

[mikemhenry]

Working on the CPU build now!

[mikemhenry]

Quick update: Worked with @jaimergp yesterday and fixed a few blockers, still working through some issues but progress is being made.

[jchodera]

@mikemhenry : For the sake of all the rest of us, would it be possible for you to summarize your learnings and the remaining steps here? It's not possible for us to learn how to overcome these blockers in the future for other projects otherwise.

[mikemhenry]

Yes! The big tl;dr is that building CUDA/GPU packages is really tough in the staged-recipies repository, since we have little control over the docker images available. For example, as of this post, only cuda 10.2 and centos6 are viable (but we can get centos7 on the non-cuda image) in the staged-repcies repository. So the approach is to try and build a minimal working package (for us that meant setting up a CPU only build) to get into to a feedstock repo, where then we will have much more control over the images for building. I'm going to work on improving the conda-forge documentation so that this will be easier for everyone going forward, another difficultly is that many of the workarounds needed are scattered as github comments across different repositories, so my plan is to consolidate them into some sort of "tips and tricks" section on the GPU build documentation.

[mikemhenry]

This command will install the package. I only uploaded cuda 11.2 builds, python 3.7, 3.8, 3.9 to the channel but once we get it on conda-forge proper we will have a full matrix of python, cuda, any pytorch versions

mamba create -n nnpops-cf-test -c mmh nnpops

[mikemhenry]

I've updated the nnpops package in my channel to 0.2
https://anaconda.org/mmh/nnpops/files
I've now got a mix of different cuda/python packages built. Still waiting on a review from the conda-forge people but conda update -c mmh nnpops should pull in the latest version now.

[mikemhenry]

Resolved with
conda-forge/staged-recipes#18189

[mikemhenry]

Feedstock here:
https://github.com/conda-forge/nnpops-feedstock

[raimis]

Great news! @mikemhenry thank you for your tremendous effort.

[mikemhenry]

👐 thank you for your patience!!! I was naive in thinking it would be easy!

[raimis]

https://anaconda.org/conda-forge/nnpops