[BUG] CELLxGENE not installable - tried conda+pip and docker

[smoe]

Hello,

trying with docker, I ran into

$ cd freshlycheckedoutcellxgenegitrepository
$ docker build . -t cellxgene
...
762.0   Downloading s3fs-2022.1.0-py3-none-any.whl (25 kB)
762.5 Collecting scipy>=1.3.0
762.5   Downloading scipy-1.5.4-cp36-cp36m-manylinux2014_aarch64.whl (24.5 MB)
841.8 Collecting tiledb>=0.5.3
841.9   Downloading tiledb-0.28.0.tar.gz (338 kB)
843.1   Installing build dependencies: started
846.6   Installing build dependencies: finished with status 'done'
846.6   Getting requirements to build wheel: started
846.8   Getting requirements to build wheel: finished with status 'done'
846.8 ERROR: Some build dependencies for tiledb>=0.5.3 from https://files.pythonhosted.org/packages/e9/f8/8c33cfd16b02f9874c39ce3ea26e3658739702dd3cd543343a34a1f723b1/tiledb-0.28.0.tar.gz#sha256=b9be057e0f06bc5692dd1755f6774844e44003234e546e1c1ec82300d6ab7284 (from cellxgene) conflict with the backend dependencies: setuptools==59.6.0 is incompatible with setuptools>=64.
------
Dockerfile:6
--------------------
   5 |     
   6 | >>> RUN apt-get update && \
   7 | >>>     apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev python3-requests python3-aiohttp && \
   8 | >>>     python3 -m pip install --upgrade pip && \
   9 | >>>     pip3 install cellxgene
  10 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update &&     apt-get install -y build-essential libxml2-dev python3-dev python3-pip zlib1g-dev python3-requests python3-aiohttp &&     python3 -m pip install --upgrade pip &&     pip3 install cellxgene" did not complete successfully: exit code: 1

With conda I admittedly did not use pip but pipx and then the installation itself runs fine, just when starting cellxgene, I get

mc00023497:~ u005069$ .local/bin/cellxgene 
Traceback (most recent call last):
  File "/Users/u005069/.local/bin/cellxgene", line 5, in <module>
    from server.cli.cli import cli
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/cli/cli.py", line 3, in <module>
    from .annotate import annotate
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/cli/annotate.py", line 12, in <module>
    import pandas as pd
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/pandas/compat/__init__.py", line 18, in <module>
    from pandas.compat.numpy import (
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module>
    from pandas.util.version import Version
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/pandas/util/__init__.py", line 2, in <module>
    from pandas.util._decorators import (  # noqa:F401
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

This is some already well-documented issue in many projects and CELLxGENE is not to blame for it, except for its dependency constraints.

Can you please direct me on what I should try next? Like indeed use pip and a system-wide install?

[smoe]

See also #2635 .

[smoe]

This fixes the numpy.dtype problem:

pipx install cellxgene --pip-args="numpy==1.26.4"

just now the removal of the imp package kicks in:

$ ~/.local/bin/cellxgene 
Traceback (most recent call last):
  File "/Users/u005069/.local/bin/cellxgene", line 5, in <module>
    from server.cli.cli import cli
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/cli/cli.py", line 4, in <module>
    from .launch import launch
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/cli/launch.py", line 12, in <module>
    from server.app.app import Server
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/app/app.py", line 16, in <module>
    import server.common.rest as common_rest
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/common/rest.py", line 26, in <module>
    from server.common.fbs.matrix import decode_matrix_fbs
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/server/common/fbs/matrix.py", line 5, in <module>
    from flatbuffers import Builder
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/flatbuffers/__init__.py", line 15, in <module>
    from .builder import Builder
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/flatbuffers/builder.py", line 15, in <module>
    from . import number_types as N
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/flatbuffers/number_types.py", line 18, in <module>
    from . import packer
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/flatbuffers/packer.py", line 22, in <module>
    from . import compat
  File "/Users/u005069/.local/pipx/venvs/cellxgene/lib/python3.12/site-packages/flatbuffers/compat.py", line 19, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'

[smoe]

The "imp" module was last supported in Python 3.9. And pipx only installs a Python version that is available on your machine, so you first need to install Python of said version. I used brew (not conda):

$ brew install [email protected]

followed by

pipx install cellxgene --python python3.9 --pip-args="numpy==1.26.4"

and if now there are issues then these likely warrant another tread:

$ ~/.local/bin/cellxgene launch example-dataset/pbmc3k.h5ad --title pbmc3k
[cellxgene] Starting the CLI...
[cellxgene] Loading data from pbmc3k.h5ad.
[cellxgene] Warning: Moving element from .uns['neighbors']['distances'] to .obsp['distances'].

This is where adjacency matrices should go now. 
[cellxgene] Warning: Moving element from .uns['neighbors']['connectivities'] to .obsp['connectivities'].

This is where adjacency matrices should go now. 
[cellxgene] Launching! Please go to http://localhost:5005 in your browser.
[cellxgene] Type CTRL-C at any time to exit.