SCENIC+ fails to install due to pybigtools build failure

[dmalzl]

Hey there,

not really a bug but a failure to install SCENIC+ following the install instructions. Specifically using Python 3.11 does not allow me to install pybigtools 0.1.2 which fails to build due to some third party library. Changing to Python 3.10 fixed this issue but then other incompatibilities popped up. I don't know why but I am simply not able to install it the way it is described and with the current requirements (already tried a couple of different rust versions also with no success). However, installing pybigtools 0.1.3 worked. Thus, is there a possibility to update the pybigtools version to this or some newer version (the newest version also does not work for me unfortunately). Is there any experience with why this is?

Thanks

[dmalzl]

furthermore, the longer I spend trying to get this to work the more I realize that many of the things in the requirements.txt file is just dumped there via installation through the actual requirements (at least it seems so from the comment lines). Although, I appreciate this in the light of reproducibility, I would suggest to only keep the stuff that is really a requirement to run SCENIC+ there and let the rest be handles through the requirments of the requirements. This would simplify the requirements file and make the installation more flexible if something down the requirements tree changes

[SeppeDeWinter]

Hi @dmalzl

The requirements.txt is indeed designed to create a reproducible environment. This file is built based on this file https://github.com/aertslab/scenicplus/blob/main/requirements.in .

Can you try to install SCENIC+ based on the requirements.in file?

All the best,

Seppe

[dmalzl]

Hi @SeppeDeWinter,

Thanks for the quick reply. I now came around to test your suggestion and still the same problems persist. Here is what I did

replacing the requrements.txt file with the requirements.in file and running pip install . again

similar to before the install fails on building pybigtools, which is due to failing to build ring 0.17.8 (as far as the error log tells me). However, investigating the matter I found that the problem might be with the subversion of Python 3.11. By default conda seems to install 3.11.10 which seems to introduce some inconsistencies. Explicitly using v3.11.0 seems to fix the problem. However there were also a couple of other changes I had to make / prerequisites I had to fulfil to get everything to run. Specifically this concerns the installation of the pysam and pybedtools package which seem to require htslib and bedtools to be installed previously as well as the pybigtools requiring a running distribution of Rust >= 1.63. This is not entirely obvious from the install instructions and I think should be mentioned in the docs like: make sure to have rust >= 1.63, samtools and bedtools installed and on the path before running pip install . i.e. adding

conda install -c bioconda -c conda-forge bedtools samtools 'rust>=1.63'

or equivalently

conda create -n python=3.11.0 bedtools samtools 'rust>=1.63' -c bioconda -c conda-forge

to the install instructions before running pip install .

TL;DR: It seems that there are incompatibilities between python 3.11.10 and Rust which results in some build errors (most possibly some openssl thing but not sure). Explicitly requiring python 3.11.0 + prerequisites installation results in a smooth installation.

Suggested improvement: Add the above command to the install instructions to ensure everything is as it should be before running pip install . (I guess this also should work with the current requirements.txt file)

[daccachejoe]

Thanks @dmalzl for breaking this down for others also trying to install. I have tried to install using your instructions and have encountered errors even with rust > 1.63, most of which are likely due to the default set up on my HPC. If anyone else following these instructions run into these errors, this is how I resolved them.

If you encounter the error

Building wheels for collected packages: scenicplus, annoy, loomxpy, pyscenic, pycistarget, pycistopic
  Building wheel for scenicplus (pyproject.toml) ... done
  Created wheel for scenicplus: filename=scenicplus-1.0a1-py3-none-any.whl size=143466 sha256=4a1d40571fe42b7ea47daf391f01e7a551cf9a1aa18486498a01ce5f0ed85e6d
  Stored in directory: /tmp/pip-ephem-wheel-cache-tfz_lggd/wheels/33/c4/80/b435988243bcf2f4b5d4f1ccefb802452c93ee43235b5da23a
  Building wheel for annoy (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      scenicplus-joe/lib/python3.11/site-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!

then change setuptools with pip install setuptools==58.2.0.

Then if you receive and error related to gcc, such as gcc: error: unrecognized command line option ‘-std=c++14’ then make sure your gcc version is > 4.9 Stack Overflow. On my HPC, the simplest way to fix this was to module load gcc/8.3.0 since user installation is not friendly According to this SO which I didn't question further.

After those two fixes, the current version of scenicplus 1.0a1 is installed successfully.

[ghuls]

Recent versions of pip should install binary wheels by default and should not try to compile from source.

pip install --prefer-binary 'pybigtools==0.1.2'

unless you are on a platform that does not have precompiled wheels:
https://pypi.org/project/pybigtools/0.1.2/#files

Which operating system, Python version and CPU architecture are you on?

[dmalzl]

I am on CentOS7 and most of the CPUs if not all are Intel

[ghuls]

Indeed the manywheel build of pybigtools does not support CentOS7.