matplotlib error in colab

[caseywdunn]

For this block of code:

gene_indices = C.gene_scores.sort_values(by="unique_count_values",ascending=False).head(10).index
C.PlotGeneCounts(gene_indices)

I get the following error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/IPython/core/formatters.py](https://localhost:8080/#) in __call__(self, obj)
    332                 pass
    333             else:
--> 334                 return printer(obj)
    335             # Finally look for special method names
    336             method = get_real_method(obj, self.print_method)

12 frames
[/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_svg.py](https://localhost:8080/#) in <module>()
     16 import matplotlib as mpl
     17 from matplotlib import _api, cbook, font_manager as fm
---> 18 from matplotlib.backend_bases import (
     19      _Backend, _check_savefig_extra_args, FigureCanvasBase, FigureManagerBase,
     20      RendererBase)

ImportError: cannot import name '_check_savefig_extra_args' from 'matplotlib.backend_bases' (/usr/local/lib/python3.7/dist-packages/matplotlib/backend_bases.py)

[caseywdunn]

Got matplotlib version in a new empty colab (not the countland colab):

import matplotlib
print(matplotlib.__version__)
3.2.2

Got the version in the countland colab,

print(matplotlib.__version__)
3.2.2

So it is not that countland is clobbering a working version with a newer version that doesn't play well with jupyter.

[shchurch]

!pip uninstall matplotlib
!python -m pip install --upgrade pip
!pip install matplotlib

Using the above and then restarting the colab can resolve the problem, but doesn't seem like a permanent solution

[shchurch]

But the above solution results in another error for the second code block on marker genes:
All numbers are identical in mannwhitneyu
This doesn't come up when running it from a jupyter notebook locally, need to find out why the error is thrown here

[caseywdunn]

Taking a closer look at the matplotlib error. Took a closer look at messages after first chunk. Matplotlib is getting uninstalled and then reinstalled, and it says that the runtime needs to be restarted to use it.

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting countland
  Cloning https://github.com/shchurch/countland.git to /tmp/pip-install-36ony_dh/countland_35c23e7c9aa8405bacd1151492a19ed5
  Running command git clone -q https://github.com/shchurch/countland.git /tmp/pip-install-36ony_dh/countland_35c23e7c9aa8405bacd1151492a19ed5
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting IMA@ git+https://github.com/shchurch/integer_matrix_approximation.git#subdirectory=IMA-python
  Cloning https://github.com/shchurch/integer_matrix_approximation.git to /tmp/pip-install-36ony_dh/ima_66bd8bdae14843fba56e574ed600995a
  Running command git clone -q https://github.com/shchurch/integer_matrix_approximation.git /tmp/pip-install-36ony_dh/ima_66bd8bdae14843fba56e574ed600995a
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from countland) (1.3.5)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from countland) (3.2.2)
Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from countland) (0.0)
Requirement already satisfied: statsmodels in /usr/local/lib/python3.7/dist-packages (from countland) (0.10.2)
Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from countland) (0.11.2)
Collecting glmpca
  Downloading glmpca-0.1.0-py3-none-any.whl (11 kB)
Requirement already satisfied: numpy<1.22,>=1.18 in /usr/local/lib/python3.7/dist-packages (from countland) (1.21.6)
Collecting scanpy
  Downloading scanpy-1.9.1-py3-none-any.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 32.7 MB/s 
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from countland) (1.4.1)
Collecting imgaug<0.2.7,>=0.2.5
  Downloading imgaug-0.2.6.tar.gz (631 kB)
     |████████████████████████████████| 631 kB 61.9 MB/s 
Requirement already satisfied: scikit-image>=0.11.0 in /usr/local/lib/python3.7/dist-packages (from imgaug<0.2.7,>=0.2.5->countland) (0.18.3)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from imgaug<0.2.7,>=0.2.5->countland) (1.15.0)
Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image>=0.11.0->imgaug<0.2.7,>=0.2.5->countland) (2.4.1)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image>=0.11.0->imgaug<0.2.7,>=0.2.5->countland) (2021.11.2)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image>=0.11.0->imgaug<0.2.7,>=0.2.5->countland) (2.6.3)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image>=0.11.0->imgaug<0.2.7,>=0.2.5->countland) (1.3.0)
Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image>=0.11.0->imgaug<0.2.7,>=0.2.5->countland) (7.1.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->countland) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->countland) (1.4.2)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->countland) (0.11.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->countland) (2.8.2)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->countland) (4.2.0)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->countland) (2022.1)
Requirement already satisfied: patsy in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (0.5.2)
Collecting umap-learn>=0.3.10
  Downloading umap-learn-0.5.3.tar.gz (88 kB)
     |████████████████████████████████| 88 kB 8.0 MB/s 
Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (1.1.0)
Collecting session-info
  Downloading session_info-1.0.0.tar.gz (24 kB)
Requirement already satisfied: importlib_metadata>=0.7 in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (4.11.3)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (4.64.0)
Requirement already satisfied: scikit-learn>=0.22 in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (1.0.2)
Collecting matplotlib
  Downloading matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2 MB)
     |████████████████████████████████| 11.2 MB 67.0 MB/s 
Requirement already satisfied: h5py>=3 in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (3.1.0)
Requirement already satisfied: numba>=0.41.0 in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (0.51.2)
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (21.3)
Requirement already satisfied: natsort in /usr/local/lib/python3.7/dist-packages (from scanpy->countland) (5.5.0)
Collecting anndata>=0.7.4
  Downloading anndata-0.8.0-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 5.9 MB/s 
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=3->scanpy->countland) (1.5.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib_metadata>=0.7->scanpy->countland) (3.8.0)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.33.3-py3-none-any.whl (930 kB)
     |████████████████████████████████| 930 kB 55.5 MB/s 
Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.41.0->scanpy->countland) (0.34.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.41.0->scanpy->countland) (57.4.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.22->scanpy->countland) (3.1.0)
Collecting pynndescent>=0.5
  Downloading pynndescent-0.5.7.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 50.5 MB/s 
Collecting stdlib_list
  Downloading stdlib_list-0.8.0-py3-none-any.whl (63 kB)
     |████████████████████████████████| 63 kB 2.7 MB/s 
Building wheels for collected packages: countland, IMA, imgaug, umap-learn, pynndescent, session-info
  Building wheel for countland (PEP 517) ... done
  Created wheel for countland: filename=countland-0.1-py3-none-any.whl size=9326 sha256=ba35527233580199a03c54276b27a49a619cdf5aac97371a01639d736460ed25
  Stored in directory: /tmp/pip-ephem-wheel-cache-oadbrmw9/wheels/6e/c7/7e/7c9c1292f1602661e2a234cc5addddcad03d47b17fa45448d0
  Building wheel for IMA (PEP 517) ... done
  Created wheel for IMA: filename=IMA-0.1-py3-none-any.whl size=3941 sha256=25e92e5fae06c34062632f9fdb95103431d168340f777e06616a8a89d58c6d4c
  Stored in directory: /tmp/pip-ephem-wheel-cache-oadbrmw9/wheels/94/02/e5/2616436e1a9ee0f05e96932c9730b7d85932126f31304d2d65
  Building wheel for imgaug (setup.py) ... done
  Created wheel for imgaug: filename=imgaug-0.2.6-py3-none-any.whl size=654017 sha256=9cebe58138ff63a920766035ef73acdb8d7c72119bd7241f9b63bdcfa08f7a8c
  Stored in directory: /root/.cache/pip/wheels/89/72/98/3ebfdba1069a9a8eaaa7ae7265cfd67d63ef0197aaee2e5f9c
  Building wheel for umap-learn (setup.py) ... done
  Created wheel for umap-learn: filename=umap_learn-0.5.3-py3-none-any.whl size=82829 sha256=23d5b0c1637a8a3170d6f69317b176a150c68a64cf719fc1ca21312d18faf10e
  Stored in directory: /root/.cache/pip/wheels/b3/52/a5/1fd9e3e76a7ab34f134c07469cd6f16e27ef3a37aeff1fe821
  Building wheel for pynndescent (setup.py) ... done
  Created wheel for pynndescent: filename=pynndescent-0.5.7-py3-none-any.whl size=54286 sha256=87b3ab88f3e893a4641ccd2f492c6a90f4467d3e8130a9ff3077f72e15dd0258
  Stored in directory: /root/.cache/pip/wheels/7f/2a/f8/7bd5dcec71bd5c669f6f574db3113513696b98f3f9b51f496c
  Building wheel for session-info (setup.py) ... done
  Created wheel for session-info: filename=session_info-1.0.0-py3-none-any.whl size=8048 sha256=8813b2e00b337d843d35967d5b5daf0552c7999f674fb344f3b6257f7d4c2fe4
  Stored in directory: /root/.cache/pip/wheels/bd/ad/14/6a42359351a18337a8683854cfbba99dd782271f2d1767f87f
Successfully built countland IMA imgaug umap-learn pynndescent session-info
Installing collected packages: fonttools, stdlib-list, pynndescent, matplotlib, umap-learn, session-info, anndata, scanpy, imgaug, IMA, glmpca, countland
  Attempting uninstall: matplotlib
    Found existing installation: matplotlib 3.2.2
    Uninstalling matplotlib-3.2.2:
      Successfully uninstalled matplotlib-3.2.2
  Attempting uninstall: imgaug
    Found existing installation: imgaug 0.2.9
    Uninstalling imgaug-0.2.9:
      Successfully uninstalled imgaug-0.2.9
Successfully installed IMA-0.1 anndata-0.8.0 countland-0.1 fonttools-4.33.3 glmpca-0.1.0 imgaug-0.2.6 matplotlib-3.5.2 pynndescent-0.5.7 scanpy-1.9.1 session-info-1.0.0 stdlib-list-0.8.0 umap-learn-0.5.3
WARNING: The following packages were previously imported in this runtime:
  [matplotlib,mpl_toolkits]
You must restart the runtime in order to use newly installed versions.
Cloning into 'countland'...
remote: Enumerating objects: 582, done.
remote: Counting objects: 100% (83/83), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 582 (delta 37), reused 62 (delta 25), pack-reused 499
Receiving objects: 100% (582/582), 115.00 MiB | 34.59 MiB/s, done.
Resolving deltas: 100% (304/304), done.

[caseywdunn]

The key bit from above is:

Successfully installed IMA-0.1 anndata-0.8.0 countland-0.1 fonttools-4.33.3 glmpca-0.1.0 imgaug-0.2.6 matplotlib-3.5.2 pynndescent-0.5.7 scanpy-1.9.1 session-info-1.0.0 stdlib-list-0.8.0 umap-learn-0.5.3
WARNING: The following packages were previously imported in this runtime:
  [matplotlib,mpl_toolkits]
You must restart the runtime in order to use newly installed versions.

In a clean colab I just verified that matplotlib is imported, even before you do anything, by running alonly the following:

import sys
print(sys.modules.keys())

[caseywdunn]

It isn't clear why pip is reinstalling matplotlib when the dependency is already satisfied. It is almost as if --force-reinstall is set in pip install https://pip.pypa.io/en/stable/cli/pip_install/

[caseywdunn]

Tried this, but it doesn't scrub everything:

import sys
matplotlib_imports = [s for s in sys.modules.keys() if "matplotlib" in s]
for m in matplotlib_imports:
  del m

[caseywdunn]

Oddly, this is among the output of that first chunk -

Requirement already satisfied: matplotlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from countland) (3.2.2)

So the requirement may be coming from a dependency rather than countland itself.

[caseywdunn]

For now we will just alert the user to the fact that they need to restart the runtime after running the first chunk. But we need a better solution eventually.