Switch away from Poetry for build (to setuptools?)

[mfisher87]

@MattF-NSIDC : Little experience with poetry, prefer setuptools

@jhkennedy : Has #Reasons, happy to elaborate if the group is amenable to switching

@jrbourbeau : "I want to be able to pip install -e ." (NOTE: Modern pip can find dependencies that are specified somewhere other than project.dependencies, so pip install -e . does work with new enough version of pip. We think it's looking at [build-system] section to discover poetry dependencies.)

@betolink : "less is more"

[danielfromearth]

My two cents, but not trying to throw a wrench in this if y'all are already all for it...
I really like poetry. I've found it easier for package and version management in other projects. And I've had no issue using pip install with poetry packages before.

[jrbourbeau]

FWIW when I try pip install -e . with the latest pip=23.3.1 release I get

 Running setup.py develop for earthaccess
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [1 lines of output]
        ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
    ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

Full traceback:

(test2) ➜  earthaccess git:(remove-binder-comment) ✗ pip install -e .
Obtaining file:///Users/james/projects/nsidc/earthaccess
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting fsspec>=2022.1 (from earthaccess==0.8.1)
  Downloading fsspec-2023.12.1-py3-none-any.whl.metadata (6.8 kB)
Collecting multimethod>=1.8 (from earthaccess==0.8.1)
  Using cached multimethod-1.10-py3-none-any.whl.metadata (8.2 kB)
Collecting pqdm>=0.1 (from earthaccess==0.8.1)
  Using cached pqdm-0.2.0-py2.py3-none-any.whl (6.8 kB)
Collecting python-cmr>=0.7 (from earthaccess==0.8.1)
  Using cached python_cmr-0.9.0-py3-none-any.whl.metadata (9.4 kB)
Collecting requests<3.0.0,>=2.26 (from earthaccess==0.8.1)
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting s3fs<2024,>=2021.11 (from earthaccess==0.8.1)
  Downloading s3fs-2023.12.1-py3-none-any.whl.metadata (1.6 kB)
Collecting tinynetrc<2.0.0,>=1.3.1 (from earthaccess==0.8.1)
  Using cached tinynetrc-1.3.1-py2.py3-none-any.whl (3.9 kB)
Collecting bounded-pool-executor (from pqdm>=0.1->earthaccess==0.8.1)
  Using cached bounded_pool_executor-0.0.3-py3-none-any.whl (3.4 kB)
Collecting tqdm (from pqdm>=0.1->earthaccess==0.8.1)
  Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
Collecting typing-extensions (from pqdm>=0.1->earthaccess==0.8.1)
  Using cached typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0.0,>=2.26->earthaccess==0.8.1)
  Using cached charset_normalizer-3.3.2-cp311-cp311-macosx_11_0_arm64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests<3.0.0,>=2.26->earthaccess==0.8.1)
  Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0,>=2.26->earthaccess==0.8.1)
  Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests<3.0.0,>=2.26->earthaccess==0.8.1)
  Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Collecting aiobotocore<3.0.0,>=2.5.4 (from s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached aiobotocore-2.8.0-py3-none-any.whl.metadata (20 kB)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1 (from s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Downloading aiohttp-3.9.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (7.4 kB)
Collecting botocore<1.33.2,>=1.32.4 (from aiobotocore<3.0.0,>=2.5.4->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached botocore-1.33.1-py3-none-any.whl.metadata (6.1 kB)
Collecting wrapt<2.0.0,>=1.10.10 (from aiobotocore<3.0.0,>=2.5.4->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Downloading wrapt-1.16.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (6.6 kB)
Collecting aioitertools<1.0.0,>=0.5.1 (from aiobotocore<3.0.0,>=2.5.4->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached aioitertools-0.11.0-py3-none-any.whl (23 kB)
Collecting attrs>=17.3.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Downloading multidict-6.0.4-cp311-cp311-macosx_11_0_arm64.whl (29 kB)
Collecting yarl<2.0,>=1.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Downloading yarl-1.9.3-cp311-cp311-macosx_11_0_arm64.whl.metadata (28 kB)
Collecting frozenlist>=1.1.1 (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Downloading frozenlist-1.4.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (5.2 kB)
Collecting aiosignal>=1.1.2 (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from botocore<1.33.2,>=1.32.4->aiobotocore<3.0.0,>=2.5.4->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.33.2,>=1.32.4->aiobotocore<3.0.0,>=2.5.4->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0,>=2.26->earthaccess==0.8.1)
  Using cached urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.33.2,>=1.32.4->aiobotocore<3.0.0,>=2.5.4->s3fs<2024,>=2021.11->earthaccess==0.8.1)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading fsspec-2023.12.1-py3-none-any.whl (168 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.9/168.9 kB 3.9 MB/s eta 0:00:00
Using cached multimethod-1.10-py3-none-any.whl (9.9 kB)
Using cached python_cmr-0.9.0-py3-none-any.whl (11 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Downloading s3fs-2023.12.1-py3-none-any.whl (28 kB)
Using cached aiobotocore-2.8.0-py3-none-any.whl (75 kB)
Downloading aiohttp-3.9.1-cp311-cp311-macosx_11_0_arm64.whl (386 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 386.8/386.8 kB 5.9 MB/s eta 0:00:00
Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp311-cp311-macosx_11_0_arm64.whl (118 kB)
Using cached idna-3.6-py3-none-any.whl (61 kB)
Using cached tqdm-4.66.1-py3-none-any.whl (78 kB)
Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Using cached botocore-1.33.1-py3-none-any.whl (11.6 MB)
Using cached urllib3-2.0.7-py3-none-any.whl (124 kB)
Downloading frozenlist-1.4.0-cp311-cp311-macosx_11_0_arm64.whl (46 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.7/46.7 kB 5.0 MB/s eta 0:00:00
Downloading wrapt-1.16.0-cp311-cp311-macosx_11_0_arm64.whl (38 kB)
Downloading yarl-1.9.3-cp311-cp311-macosx_11_0_arm64.whl (80 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.5/80.5 kB 7.0 MB/s eta 0:00:00
Installing collected packages: tinynetrc, bounded-pool-executor, wrapt, urllib3, typing-extensions, tqdm, six, multimethod, multidict, jmespath, idna, fsspec, frozenlist, charset-normalizer, certifi, attrs, aioitertools, yarl, requests, python-dateutil, pqdm, aiosignal, python-cmr, botocore, aiohttp, aiobotocore, s3fs, earthaccess
  Running setup.py develop for earthaccess
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [1 lines of output]
        ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
    ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

[MattF-NSIDC]

Interesting. I wonder if the below is true IFF setuptools is actually installed?

(NOTE: Modern pip can find dependencies that are specified somewhere other than project.dependencies, so pip install -e . does work with new enough version of pip. We think it's looking at [build-system] section to discover poetry dependencies.)

[danielfromearth]

@jrbourbeau — to clarify — are you trying to use a local, edited, version of a dependency? Something like what's described in this poetry issue?

[jhkennedy]

@danielfromearth we solved @jrbourbeau issue here:
#402 (comment)

We needed to update the build backend and remove the dummy setup.py file as pip has behavioral changes if that file exists or not.

[danielfromearth]

oh, that's great! It's nice to have only pyproject.toml and not both that and setup.py

[chuckwondo]

I'd suggest considering uv. It might not quite be ready for prime-time, but I strongly feel this is worth a look. It's by the same folks that created ruff, which we are already using here.

[chuckwondo]

As a follow-on, I know some people love poetry and others hate it. It's difficult to find any sort of tool or library that is nearly universally appreciated, but poetry might be in the class of tools where more folks are at the extremes of the spectrum than more centrally located.

For those who are not so much in love with poetry, a major factor is that it can be excruciatingly slow to resolve dependencies (as is the case for some other tools as well, such as pipenv). Here's an example that I just ran into on this project:

(earthaccess-dev) $ poetry add importlib-resources
Using version ^6.3.2 for importlib-resources

Updating dependencies
Resolving dependencies... (134.2s)
...(snip)...
  - Updating mkdocs-jupyter (0.22.0 /home/conda/feedstock_root/build_artifacts/mkdocs-jupyter_1663979448056/work -> 0.22.0)

Writing lock file
(earthaccess-dev) 2m55s $

That took almost 3 minutes (134 seconds for resolution alone, plus another ~40 seconds after that) to add a seemly innocuous library that has only 1 transitive dependency (zipp, which has no further dependencies), and that's not even as bad as I've encountered elsewhere, where it can take many 10s of minutes to resolve dependencies.

[chuckwondo]

Along with uv, rye is another option to consider. Both are implemented in Rust and are blazingly fast at resolving dependencies.

As far as I've looked, rye appears to be a bit more straightforward than uv, however, the creator of rye has joined forces with Astral (creators of uv), so it's not clear yet how things will evolve (i.e., whether rye will be subsumed by uv, or if it will remain a distinct tool), so it may behoove us to keep an eye on their progression in the near future before committing to one or the other (or to some other tool).

[jhkennedy]

Just noting that uv has --resolution=lowest, which will install the lowest compatible version of dependencies, which would mean we wouldn't need to maintain a separate minimum-bounds environment as noted in #497 .

https://github.com/astral-sh/uv?tab=readme-ov-file#resolution-strategy

[mfisher87]

Just noting that uv has --resolution=lowest, which will install the lowest compatible version of dependencies, which would mean we wouldn't need to maintain a separate minimum-bounds environment

🤩

[itcarroll]

Boosting this issue, which could be said to be blocking #520, because I'm not sure how to approach resolving environments for CI. We need python-cmr>=0.10.0 now, and the current approach gets it from conda-forge (where it does not yet exist). I assume we want to let poetry or uv manage this testing environment rather than separately maintaining a conda environment.yml?

[chuckwondo]

Boosting this issue, which could be said to be blocking #520, because I'm not sure how to approach resolving environments for CI. We need python-cmr>=0.10.0 now, and the current approach gets it from conda-forge (where it does not yet exist). I assume we want to let poetry or uv manage this testing environment rather than separately maintaining a conda environment.yml?

To get around the build failure for that issue, perhaps just move python-cmr to a pip install within ci/environment-mindeps.yaml. Specifically, replace this line:

  - python-cmr=0.9.0

with these lines:

  - pip
  - pip:
    - python-cmr==0.10.0

[mfisher87]

Open an issue like this one if you'd like to become a maintainer over on the conda-forge feedstock :) It would be really useful to have a couple more folks helping out there I think!

conda-forge/python-cmr-feedstock#7

[mfisher87]

Kinda related #614