Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

jameslamb · 2024-09-04T22:53:08Z

Description

Contributes to rapidsai/build-planning#40

This PR adds support for Python 3.12.

Notes for Reviewers

This is part of ongoing work to add Python 3.12 support across RAPIDS.
It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213.

A follow-up PR will revert back to pointing at the branch-24.10 branch of shared-workflows once all
RAPIDS repos have added Python 3.12 support.

Other changes required to add that support:

pinning proj back to 9.3.x, reverting cuproj: remove pin on 'proj' in conda packages #1449 (Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453 (comment))
requiring geopandas >= 1.0 (Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453 (comment))

This will fail until all dependencies have been updates to Python 3.12

CI here is expected to fail until all of this project's upstream dependencies support Python 3.12.

This can be merged whenever all CI jobs are passing.

jameslamb · 2024-09-06T13:37:05Z

Some changes to the required version of gdal:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides gdal 1.11.4 needed by fiona-1.6.3-np110py27_0

Working on it.

jameslamb · 2024-09-06T22:20:46Z

😬 😬 😬 I think being behind on fmt and spdlog (rapidsai/build-planning#56 (comment)) is finally catching up to us here.

I haven't 100% figured this out yet, but posting my thoughts so far.

Short Summary

Here's what I think is happening:

several geospatial dependencies require gdal / libgdal at runtime
libgdal depends on tiledb
tiledb first introduced a runtime dependency on fmt and spdlog in tiledb==2.20.1 (Use upstream spdlog. conda-forge/tiledb-feedstock#254)
librmm is a run dependency of cuspatial (directly and transitively, via cudf)
librmm has these runtime constraints: fmt >=10.1.1,<11, spdlog >=1.12.0,<1.13
the newest version of tiledb on conda-forge compatible with those is tiledb==2.22.0
with python=3.12, something is preventing conda from falling back to an old enough gdal / libgdal that it can also install tiledb<=2.22.0

If I'm right, options I think we have:

upgrade fmt and spdlog across RAPIDS (update fmt and spdlog across RAPIDS build-planning#56)
remove run / run_exports dependencies on librmm in RAPIDS conda packages (Standardize run exports (and make analogous choices for pip packaging) for header-only libraries build-planning#92)
- that'd at least remove a source of fmt / spdlog run dependencies and improve the success rate for runtime / test-time solves like this... but not sure what other implication it'd have

Other Notes

How to reproduce (and then fiddle with) the exact errors from CI

commands to run (click me)

On an x86_64 machine with CUDA

docker run \
  --rm \
  --gpus 1 \
  --env RAPIDS_BUILD_TYPE=pull-request \
  --env RAPIDS_REPOSITORY=rapidsai/cuspatial \
  --env RAPIDS_REF_NAME=pull-request/1453 \
  -v $(pwd):/opt/work \
  -w /opt/work \
  -it rapidsai/ci-conda:cuda12.5.1-ubuntu22.04-py3.12 \
  bash

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

# guarantee that exactly the CI artifacts are used
# and that 'mamba install' doesn't silently fall back to
# other packages from rapidsai-nightly / rapidsai
BUILD_SUFFIX="240906_g4361f9ed_42"

mamba create \
  --name delete-me \
  -c conda-forge \
  -c "${CPP_CHANNEL}" \
  -c "${PYTHON_CHANNEL}" \
  --dry-run \
  "cuproj==24.10.00a42=cuda12_py312_${BUILD_SUFFIX}" \
  "cuspatial==24.10.00a42=cuda12_py312_${BUILD_SUFFIX}" \
  "libcuspatial==24.10.00a42=cuda12_${BUILD_SUFFIX}" \
  "geopandas>=0.11.0" \
  "python=3.12"

full solver output (click me)

The following packages are incompatible
├─ cuproj ==24.10.00a42 cuda12_py312_240906_g4361f9ed_42 is installable and it requires
│  ├─ librmm >=24.10.0a.32,<24.11.0a0 , which requires
│  │  ├─ fmt >=10.1.1,<11 , which can be installed;
│  │  └─ spdlog >=1.12.0,<1.13  with the potential options
│  │     ├─ spdlog 1.12.0 would require
│  │     │  └─ fmt >=9.1.0,<10.0a0 , which conflicts with any installable versions previously reported;
│  │     └─ spdlog 1.12.0, which can be installed;
│  └─ proj >=9.4.1,<9.5.0a0 , which can be installed;
├─ cuspatial ==24.10.00a42 cuda12_py312_240906_g4361f9ed_42 is not installable because it requires
│  ├─ geopandas >=0.11.0  with the potential options
│  │  ├─ geopandas [0.11.0|0.11.1|0.12.0] would require
│  │  │  └─ geopandas-base [0.11.0 pyha770c72_0|0.11.1 pyha770c72_0|0.12.0 pyha770c72_0], which requires
│  │  │     └─ shapely >=1.7,<2  with the potential options
│  │  │        ├─ shapely 1.7.0 would require
│  │  │        │  └─ python >=2.7,<2.8.0a0 , which can be installed;
│  │  │        ├─ shapely 1.7.0 would require
│  │  │        │  └─ python_abi 2.7.* *_cp27mu, which can be installed;
│  │  │        ├─ shapely [1.7.0|1.7.1] would require
│  │  │        │  └─ python_abi 3.6.* *_cp36m, which can be installed;
│  │  │        ├─ shapely 1.7.0 would require
│  │  │        │  └─ python >=3.6,<3.7.0a0 , which can be installed;
│  │  │        ├─ shapely 1.7.0 would require
│  │  │        │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│  │  │        ├─ shapely [1.7.0|1.7.1|...|1.8.5] would require
│  │  │        │  └─ python_abi 3.7.* *_cp37m, which can be installed;
│  │  │        ├─ shapely 1.7.0 would require
│  │  │        │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  │  │        ├─ shapely [1.7.0|1.7.1|...|1.8.5] would require
│  │  │        │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  │  │        ├─ shapely 1.7.1 would require
│  │  │        │  └─ python_abi 3.6 *_pypy36_pp73, which can be installed;
│  │  │        ├─ shapely [1.7.1|1.8.0] would require
│  │  │        │  └─ python_abi 3.7 *_pypy37_pp73, which can be installed;
│  │  │        ├─ shapely [1.7.1|1.8.0|...|1.8.5] would require
│  │  │        │  └─ python_abi 3.9.* *_cp39, which can be installed;
│  │  │        ├─ shapely [1.8.0|1.8.1|...|1.8.5] would require
│  │  │        │  └─ python_abi 3.10.* *_cp310, which can be installed;
│  │  │        ├─ shapely [1.8.0|1.8.1|...|1.8.5] would require
│  │  │        │  └─ python_abi 3.8 *_pypy38_pp73, which can be installed;
│  │  │        ├─ shapely [1.8.0|1.8.1|...|1.8.5] would require
│  │  │        │  └─ python_abi 3.9 *_pypy39_pp73, which can be installed;
│  │  │        └─ shapely 1.8.5 would require
│  │  │           └─ python_abi 3.11.* *_cp311, which can be installed;
│  │  ├─ geopandas [0.12.1|0.12.2|0.13.0|0.13.1|0.13.2] would require
│  │  │  └─ fiona with the potential options
│  │  │     ├─ fiona [1.6.3|1.6.4] would require
│  │  │     │  └─ gdal 1.11.4 , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona [1.7.0|1.8.11|...|1.8.9.post2] would require
│  │  │     │  └─ python [2.7* |>=2.7,<2.8.0a0 ], which can be installed;
│  │  │     ├─ fiona [1.7.0|1.7.1|...|1.7.9] would require
│  │  │     │  └─ gdal 2.1.* , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona 1.7.0 would require
│  │  │     │  └─ python 3.4* , which can be installed;
│  │  │     ├─ fiona 1.7.0 would require
│  │  │     │  └─ python 3.5* , which can be installed;
│  │  │     ├─ fiona [1.7.11|1.7.12|1.7.13] would require
│  │  │     │  └─ gdal 2.2.* , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona 1.7.13 would require
│  │  │     │  └─ gdal >=2.2.2,<2.3.0a0 , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona 1.7.13 would require
│  │  │     │  └─ gdal >=2.2.4,<2.3.0a0 , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona [1.7.8|1.7.9] would require
│  │  │     │  └─ libgdal 2.1.4 , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona [1.8.0|1.8.1|1.8.2|1.8.3|1.8.4] would require
│  │  │     │  └─ gdal >=2.3.2,<2.4.0a0 , which does not exist (perhaps a missing channel);
│  │  │     ├─ fiona [1.8.11|1.8.12|...|1.8.9.post2] would require
│  │  │     │  └─ python >=3.6,<3.7.0a0 , which can be installed;
│  │  │     ├─ fiona [1.8.11|1.8.12|...|1.8.9.post2] would require
│  │  │     │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│  │  │     ├─ fiona [1.8.11|1.8.12|1.8.13|1.8.9.post2] would require
│  │  │     │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.8.20] would require
│  │  │     │  └─ python_abi 3.6.* *_cp36m, which can be installed;
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.8.22] would require
│  │  │     │  └─ python_abi 3.7.* *_cp37m, which can be installed;
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.9.6] would require
│  │  │     │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  │  │     ├─ fiona [1.8.17|1.8.18|...|1.9.6] would require
│  │  │     │  └─ python_abi 3.9.* *_cp39, which can be installed;
│  │  │     ├─ fiona [1.8.20|1.8.21|...|1.9.6] would require
│  │  │     │  └─ python_abi 3.10.* *_cp310, which can be installed;
│  │  │     ├─ fiona [1.8.22|1.9.0|...|1.9.6] would require
│  │  │     │  └─ python_abi 3.11.* *_cp311, which can be installed;
│  │  │     ├─ fiona 1.10.0 would require
│  │  │     │  ├─ libgdal >=3.9.2,<3.10.0a0 , which requires
│  │  │     │  │  ├─ libgdal-tiledb 3.9.2.* , which requires
│  │  │     │  │  │  └─ tiledb >=2.25.0,<2.26.0a0  but there are no viable options
│  │  │     │  │  │     ├─ tiledb [2.23.0|2.23.1|...|2.25.0] would require
│  │  │     │  │  │     │  ├─ fmt >=10.2.1,<11.0a0 , which can be installed;
│  │  │     │  │  │     │  └─ spdlog >=1.13.0,<1.14.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │  │  │     ├─ tiledb [2.24.2|2.25.0] would require
│  │  │     │  │  │     │  └─ fmt >=11.0.2,<12.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │  │  │     └─ tiledb [2.24.2|2.25.0] would require
│  │  │     │  │  │        └─ fmt >=11.0.1,<12.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │  │  └─ tiledb >=2.25.0,<2.26.0a0 , which cannot be installed (as previously explained);
│  │  │     │  └─ python_abi 3.10.* *_cp310, which can be installed;
│  │  │     ├─ fiona 1.10.0 would require
│  │  │     │  ├─ libgdal >=3.9.2,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │     │  └─ python_abi 3.11.* *_cp311, which can be installed;
│  │  │     ├─ fiona 1.10.0 would require
│  │  │     │  └─ libgdal >=3.9.2,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0 would require
│  │  │     │  ├─ libgdal >=3.9.2,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │     │  └─ python_abi 3.9.* *_cp39, which can be installed;
│  │  │     ├─ fiona [1.9.4|1.9.5] would require
│  │  │     │  └─ libgdal >=3.7.2,<3.8.0a0  but there are no viable options
│  │  │     │     ├─ libgdal [3.7.2|3.7.3|3.8.0|3.8.1] would require
│  │  │     │     │  └─ proj >=9.3.0,<9.3.1.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │     ├─ libgdal 3.7.2 would require
│  │  │     │     │  └─ proj >=9.2.1,<9.2.2.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │     ├─ libgdal [3.7.3|3.8.1|3.8.2|3.8.3|3.8.4] would require
│  │  │     │     │  └─ proj >=9.3.1,<9.3.2.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │     ├─ libgdal [3.7.3|3.8.4|3.8.5|3.9.0] would require
│  │  │     │     │  └─ proj >=9.4.0,<9.4.1.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │     └─ libgdal [3.7.3|3.8.5|3.9.0] would require
│  │  │     │        ├─ proj >=9.4.0,<9.4.1.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │        └─ tiledb >=2.23.0,<2.24.0a0 , which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.5 would require
│  │  │     │  └─ libgdal >=3.8.0,<3.9.0a0  but there are no viable options
│  │  │     │     ├─ libgdal [3.7.2|3.7.3|3.8.0|3.8.1], which cannot be installed (as previously explained);
│  │  │     │     ├─ libgdal [3.7.3|3.8.1|3.8.2|3.8.3|3.8.4], which cannot be installed (as previously explained);
│  │  │     │     ├─ libgdal [3.7.3|3.8.4|3.8.5|3.9.0], which cannot be installed (as previously explained);
│  │  │     │     ├─ libgdal [3.7.3|3.8.5|3.9.0], which cannot be installed (as previously explained);
│  │  │     │     ├─ libgdal [3.8.5|3.9.0|3.9.1] would require
│  │  │     │     │  ├─ libgdal-tiledb 3.9.1.*  but there are no viable options
│  │  │     │     │  │  ├─ libgdal-tiledb [3.9.1|3.9.2], which cannot be installed (as previously explained);
│  │  │     │     │  │  └─ libgdal-tiledb 3.9.1 would require
│  │  │     │     │  │     └─ tiledb >=2.24.2,<2.25.0a0  but there are no viable options
│  │  │     │     │  │        ├─ tiledb [2.23.0|2.23.1|...|2.25.0], which cannot be installed (as previously explained);
│  │  │     │     │  │        ├─ tiledb [2.24.2|2.25.0], which cannot be installed (as previously explained);
│  │  │     │     │  │        ├─ tiledb [2.24.2|2.25.0], which cannot be installed (as previously explained);
│  │  │     │     │  │        └─ tiledb 2.24.2 would require
│  │  │     │     │  │           ├─ fmt >=10.2.1,<11.0a0 , which can be installed;
│  │  │     │     │  │           └─ spdlog >=1.14.1,<1.15.0a0  but there are no viable options
│  │  │     │     │  │              ├─ spdlog 1.14.1 conflicts with any installable versions previously reported;
│  │  │     │     │  │              └─ spdlog 1.14.1 would require
│  │  │     │     │  │                 └─ fmt >=11.0.1,<12.0a0 , which conflicts with any installable versions previously reported;
│  │  │     │     │  └─ tiledb [>=2.24.0,<2.25.0a0 |>=2.24.1,<2.25.0a0 |>=2.24.2,<2.25.0a0 ], which cannot be installed (as previously explained);
│  │  │     │     ├─ libgdal [3.8.5|3.9.2], which cannot be installed (as previously explained);
│  │  │     │     └─ libgdal [3.8.5|3.9.0] would require
│  │  │     │        └─ tiledb >=2.23.0,<2.24.0a0 , which cannot be installed (as previously explained);
│  │  │     ├─ fiona [1.9.5|1.9.6] would require
│  │  │     │  └─ libgdal [>=3.8.2,<3.9.0a0 |>=3.8.4,<3.9.0a0 ], which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6 would require
│  │  │     │  ├─ libgdal >=3.9.1,<3.10.0a0  but there are no viable options
│  │  │     │  │  ├─ libgdal [3.8.5|3.9.0|3.9.1], which cannot be installed (as previously explained);
│  │  │     │  │  ├─ libgdal [3.8.5|3.9.2], which cannot be installed (as previously explained);
│  │  │     │  │  └─ libgdal 3.9.1 would require
│  │  │     │  │     └─ libgdal-tiledb 3.9.1.* , which cannot be installed (as previously explained);
│  │  │     │  └─ python_abi 3.10.* *_cp310, which can be installed;
│  │  │     ├─ fiona 1.9.6 would require
│  │  │     │  ├─ libgdal >=3.9.1,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │     │  └─ python_abi 3.11.* *_cp311, which can be installed;
│  │  │     ├─ fiona 1.9.6 would require
│  │  │     │  └─ libgdal [>=3.8.5,<3.9.0a0 |>=3.9.0,<3.10.0a0 ], which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6 would require
│  │  │     │  └─ libgdal >=3.9.1,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6 would require
│  │  │     │  ├─ libgdal >=3.9.1,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │     │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  │  │     └─ fiona 1.9.6 would require
│  │  │        ├─ libgdal >=3.9.1,<3.10.0a0 , which cannot be installed (as previously explained);
│  │  │        └─ python_abi 3.9.* *_cp39, which can be installed;
│  │  ├─ geopandas 0.13.2 would require
│  │  │  └─ fiona >=1.8.19  with the potential options
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.8.20], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.8.22], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.17|1.8.18|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.20|1.8.21|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.22|1.9.0|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona [1.9.4|1.9.5], which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.5, which cannot be installed (as previously explained);
│  │  │     ├─ fiona [1.9.5|1.9.6], which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     └─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  ├─ geopandas [0.14.0|0.14.1|0.14.2|0.14.3|0.14.4] would require
│  │  │  └─ fiona >=1.8.21  with the potential options
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.8.22], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.13|1.8.15|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.17|1.8.18|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.20|1.8.21|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona [1.8.22|1.9.0|...|1.9.6], which can be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.10.0, which cannot be installed (as previously explained);
│  │  │     ├─ fiona [1.9.4|1.9.5], which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.5, which cannot be installed (as previously explained);
│  │  │     ├─ fiona [1.9.5|1.9.6], which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     ├─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  │     └─ fiona 1.9.6, which cannot be installed (as previously explained);
│  │  └─ geopandas [1.0.0|1.0.1] would require
│  │     └─ pyogrio >=0.7.2  with the potential options
│  │        ├─ pyogrio [0.7.2|0.8.0|0.9.0] would require
│  │        │  └─ python_abi 3.10.* *_cp310, which can be installed;
│  │        ├─ pyogrio [0.7.2|0.8.0|0.9.0] would require
│  │        │  └─ python_abi 3.11.* *_cp311, which can be installed;
│  │        ├─ pyogrio 0.7.2 would require
│  │        │  └─ libgdal >=3.8.0,<3.9.0a0 , which cannot be installed (as previously explained);
│  │        ├─ pyogrio 0.7.2 would require
│  │        │  └─ libgdal >=3.7.2,<3.8.0a0 , which cannot be installed (as previously explained);
│  │        ├─ pyogrio [0.7.2|0.8.0|0.9.0] would require
│  │        │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  │        ├─ pyogrio [0.7.2|0.8.0|0.9.0] would require
│  │        │  └─ python_abi 3.9.* *_cp39, which can be installed;
│  │        ├─ pyogrio [0.8.0|0.9.0] would require
│  │        │  └─ libgdal [>=3.8.5,<3.9.0a0 |>=3.9.0,<3.10.0a0 ], which cannot be installed (as previously explained);
│  │        ├─ pyogrio 0.9.0 would require
│  │        │  └─ libgdal >=3.9.2,<3.10.0a0 , which cannot be installed (as previously explained);
│  │        └─ pyogrio 0.9.0 would require
│  │           └─ python >=3.13.0rc1,<3.14.0a0 , which requires
│  │              └─ _python_rc, which does not exist (perhaps a missing channel);
│  └─ python >=3.12,<3.13.0a0 , which requires
│     └─ python_abi 3.12.* *_cp312, which conflicts with any installable versions previously reported;
└─ libcuspatial ==24.10.00a42 cuda12_240906_240906_g4361f9ed_42 does not exist (perhaps a typo or a missing channel).

Other notes

geopandas dropped its fiona dependency (on conda-forge) in v1.0.0: geopandas v1.0.0 conda-forge/geopandas-feedstock#116
tiledb has run dependencies on fmt and spdlog, and has participated in the recent migrations for those on conda-forge.

bdice · 2024-09-07T15:30:52Z

Yup, I agree with your assessment. Here's a screenshot that also points to spdlog/fmt issues:

This is from the command

mamba create -n test --dry-run -c rapidsai-nightly -c conda-forge cuspatial==24.10.00a* proj==9.4.1 cuda-version==12.5 python==3.11

Our other choice is to pin cuproj to use proj>=9.3.0,<9.3.1.0a0. That's what I'm going to propose for this PR since it confines the problem to the cuspatial/cuproj repository.

We don't want to let this block all Python 3.12 progress, so 0e25119 is probably our best bet in the short term. Once we complete the Python 3.12 migration we can go back to fix librmm exports and spdlog/fmt upgrades.

bdice · 2024-09-09T12:59:35Z

The proj pinning helped. I fixed an issue with the new C++ wheel workflows which weren't using shared-workflows with Python 3.12. This should pass CI now.

jameslamb · 2024-09-09T14:26:27Z

Ah thank you so much! I hadn't been able to isolate proj as something we could pin back to be able to use older gdal, tiledb, etc.

With that change, you can now see conda able to fall back to older versions of those things:

fmt                       10.2.1               h00ab1b0_0    conda-forge
...
gdal                      3.8.1           py312ha82c271_3    conda-forge
...
spdlog                    1.12.0               hd2e6256_2    conda-forge
...
tiledb                    2.18.2               h99f50a1_1    conda-forge

(build link)

Now we just have one other issue... the version of fiona this project builds from source in its CI fails to compile with Python 3.12.

      fiona/schema.c: In function ‘__Pyx_PyIndex_AsSsize_t’:
      fiona/schema.c:5193:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
       5193 |     const digit* digits = ((PyLongObject*)b)->ob_digit;

(build link)

fiona==1.10.0 has Python 3.12 wheels available (https://pypi.org/project/fiona/#files), so we could try moving the pin to that. We'll still have to build from source in CI, I think, because I don't see aarch64 wheels there.

HOWEVER... first, I'm going to try just moving cuspatial to geopandas>=1.0. It was temporarily pinned in tests to geopandas<1 in #1400 (comment), but I think that we may be able to remove that ceiling now that #1403 has been addressed.

This helps becuase geopandas dropped its fiona dependency in the 1.x series: conda-forge/geopandas-feedstock#116

jameslamb · 2024-09-09T15:07:37Z

README.md

@@ -145,42 +145,6 @@ pip install cuspatial-cu12 --extra-index-url=https://pypi.nvidia.com
 pip install cuspatial-cu11 --extra-index-url=https://pypi.nvidia.com
 ```

-#### Troubleshooting Fiona/GDAL versions


Starting a threaded conversation for this one.

Proposal

pin to geopandas >= 1.0.0 so we can drop this project's reliance on fiona entirely

commit: 9295c3f

Benefits

No more need to install specific versions of fiona based on the compatible GDAL version provided by your operating system's package manager (see these deleted notes in the README).

Risks

geopandas==1.0.0 is only a few months old... it was released in June 2024, so it's only been out about 3 months (https://pypi.org/project/geopandas/1.0.0/). Being so new, it might have other dependency constraints that make it harder for users on older systems to use cuspatial.

Notes

In practice, most of this project's tests have been running against geopandas>=1.0.0 for a few months now. Choose any random successful pr workflow from the last 3 months at https://github.com/rapidsai/cuspatial/actions/workflows/pr.yaml?query=is%3Asuccess and look in the wheel-tests-* or conda-python-tests* jobs. You'll see geopandas>=1.0.0 getting pulled in. For example, here's one from July 8th (chosen randomly): https://github.com/rapidsai/cuspatial/actions/runs/9850995522/job/27197475692#step:7:348

more context on why this is coming up now as part of the Python 3.12 work: #1453 (comment)

(cc @mroeschke )

I don't have the strongest opinion on geopandas support, but given your breakdown, +1 for requiring >= 1.0.0

Thanks very much!

Ok, I'm going to merge this as-is then. @harrism @-ing you for awareness... if you have any concerns about this, let me know and I'd be happy to address them in a follow-up PR.

bdice

All changes look good to me!

jameslamb · 2024-09-09T16:57:31Z

/merge

Add support for Python 3.12.

89617dd

jameslamb added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels Sep 4, 2024

github-actions bot added conda Related to conda and conda configuration Python Related to Python code labels Sep 4, 2024

jameslamb changed the title ~~Add support for Python 3.12~~ WIP: Add support for Python 3.12 Sep 4, 2024

Merge branch 'branch-24.10' into python-3.12

9bb0ae5

bdice mentioned this pull request Sep 6, 2024

re-enable mixing libcuspatial wheels with libcudf conda packages #1456

Merged

3 tasks

Merge branch 'branch-24.10' into python-3.12

4361f9e

bdice mentioned this pull request Sep 7, 2024

Add script to test nightly environments are solvable and using recent nightlies. rapidsai/integration#690

Merged

2 tasks

bdice added 3 commits September 7, 2024 10:32

Pin proj to >=9.3.0,<9.3.1.0a0

0e25119

Pin more instances of proj.

1a5cd18

Update branches for new C++ wheels workflows.

eaac568

bump geopandas to 1.x, drop fiona

9295c3f

github-actions bot added the ci label Sep 9, 2024

jameslamb commented Sep 9, 2024

View reviewed changes

Merge branch 'branch-24.10' into python-3.12

b4d51cf

jameslamb marked this pull request as ready for review September 9, 2024 15:41

jameslamb requested review from a team as code owners September 9, 2024 15:41

jameslamb requested review from AyodeAwe and bdice and removed request for AyodeAwe September 9, 2024 15:41

jameslamb changed the title ~~WIP: Add support for Python 3.12~~ Add support for Python 3.12 Sep 9, 2024

jameslamb changed the title ~~Add support for Python 3.12~~ Add support for Python 3.12, require geopandas>=1.0 Sep 9, 2024

jameslamb changed the title ~~Add support for Python 3.12, require geopandas>=1.0~~ Add support for Python 3.12, ping proj back to 9.3.x, require geopandas>=1.0 Sep 9, 2024

bdice approved these changes Sep 9, 2024

View reviewed changes

rapids-bot bot merged commit 2a30f5f into rapidsai:branch-24.10 Sep 9, 2024
75 checks passed

jameslamb deleted the python-3.12 branch September 9, 2024 16:57

jameslamb changed the title ~~Add support for Python 3.12, ping proj back to 9.3.x, require geopandas>=1.0~~ Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 Sep 9, 2024

jameslamb mentioned this pull request Sep 9, 2024

Add support for Python 3.12, drop pyppeteer dependency rapidsai/cuxfilter#625

Merged

This was referenced Sep 10, 2024

S3 support rapidsai/kvikio#426

Closed

Track conda-forge migrations with automated tooling rapidsai/build-planning#100

Open

This was referenced Sep 10, 2024

Add support for Python 3.12 rapidsai/build-planning#40

Closed

Add support for Python 3.12 rapidsai/integration#719

Merged

bdice mentioned this pull request Sep 11, 2024

Remove run-export of librmm. rapidsai/rmm#1673

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

jameslamb commented Sep 4, 2024 •

edited

Loading

jameslamb commented Sep 6, 2024

jameslamb commented Sep 6, 2024

bdice commented Sep 7, 2024 •

edited

Loading

bdice commented Sep 9, 2024

jameslamb commented Sep 9, 2024

jameslamb Sep 9, 2024 •

edited by bdice

Loading

jameslamb Sep 9, 2024

mroeschke Sep 9, 2024

jameslamb Sep 9, 2024

bdice left a comment

jameslamb commented Sep 9, 2024

Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

Conversation

jameslamb commented Sep 4, 2024 • edited Loading

Description

Notes for Reviewers

This will fail until all dependencies have been updates to Python 3.12

jameslamb commented Sep 6, 2024

jameslamb commented Sep 6, 2024

Short Summary

Other Notes

How to reproduce (and then fiddle with) the exact errors from CI

Other notes

bdice commented Sep 7, 2024 • edited Loading

bdice commented Sep 9, 2024

jameslamb commented Sep 9, 2024

jameslamb Sep 9, 2024 • edited by bdice Loading

Choose a reason for hiding this comment

jameslamb Sep 9, 2024

Choose a reason for hiding this comment

mroeschke Sep 9, 2024

Choose a reason for hiding this comment

jameslamb Sep 9, 2024

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

jameslamb commented Sep 9, 2024

jameslamb commented Sep 4, 2024 •

edited

Loading

bdice commented Sep 7, 2024 •

edited

Loading

jameslamb Sep 9, 2024 •

edited by bdice

Loading