[QST] Broken package after installing modin

[adrienpacifico]

How do I get modin and cudf working in the same conda virtual environment?

I installed rapids through conda by using the rapids release selector. Not certain, but I think I used:

conda create -n rapids-0.16 -c rapidsai -c nvidia -c conda-forge \ -c defaults rapids=0.16 python=3.8 cudatoolkit=10.1

Context

I broke cudf after installing modin via pip (I wanted to do performance comparisons).

import cudf

leads to

ModuleNotFoundError: No module named 'pyarrow._cuda'

Full trace

--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) in ----> 1 import cudf 2 import pandas as pd 3 import pandarallel 4 import numpy as np 5 cudf.set_allocator("managed")

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/init.py in
9 import rmm
10
---> 11 from cudf import core, datasets, testing
12 from cudf._version import get_versions
13 from cudf.core import (

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/core/init.py in
1 # Copyright (c) 2018-2019, NVIDIA CORPORATION.
----> 2 from cudf.core import buffer, column
3 from cudf.core.buffer import Buffer
4 from cudf.core.dataframe import DataFrame, from_pandas, merge
5 from cudf.core.index import (

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/core/column/init.py in
1 # Copyright (c) 2020, NVIDIA CORPORATION.
2
----> 3 from cudf.core.column.categorical import CategoricalColumn
4 from cudf.core.column.column import (
5 ColumnBase,

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/core/column/categorical.py in
8
9 import cudf
---> 10 from cudf import _lib as libcudf
11 from cudf._lib.transform import bools_to_mask
12 from cudf.core.buffer import Buffer

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/_lib/init.py in
2 import numpy as np
3
----> 4 from . import (
5 avro,
6 binaryop,

cudf/_lib/avro.pyx in init cudf._lib.avro()

cudf/_lib/table.pyx in init cudf._lib.table()

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/core/column_accessor.py in
5
6 import cudf
----> 7 from cudf.utils.utils import (
8 OrderedColumnDict,
9 cached_property,

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/cudf/utils/utils.py in
9 import pyarrow as pa
10 from numba import njit
---> 11 from pyarrow.cuda import CudaBuffer as arrowCudaBuffer
12
13 import rmm

~/anaconda3/envs/rapids_test/lib/python3.7/site-packages/pyarrow/cuda.py in
20 from future import absolute_import
21
---> 22 from pyarrow._cuda import (Context, IpcMemHandle, CudaBuffer,
23 HostBuffer, BufferReader, BufferWriter,
24 new_host_buffer,

ModuleNotFoundError: No module named 'pyarrow._cuda'

What should I do?

I tried :

pip uninstall pyarrow  
pip install pyarrow=0.15.0

following rapidsai-community/notebooks-contrib#313 but it did not seems to solve my problem.

ps; let me know if you think this question is more suited for SO.

[kkraus14]

cudf 0.16 requires pyarrow 1.0.1. If Modin requires an older version then unfortunately they're incompatible to be in the same environment.

[kkraus14]

Would suggest installing modin via conda to prevent pip from breaking the environment. There's packages on conda-forge here: https://anaconda.org/conda-forge/modin

[adrienpacifico]

Oh great!
I did not know I could install it through conda-forge as modin official doc did not document conda to install it.

Thank you for that excellent answer!