Replace poetry with pip, build with hatchling, dynamically calculate minimum dependencies with uv

[mfisher87]

"Dependency groups" are a poetry-specific feature that's different from PEP621 "optional dependencies", aka "extras". To make things compatible with other ways of dependency management, switch to PEP621 optional dependencies, but specified using the Poetry configuration aliases to retain support for poetry. While the Poetry "dependency groups" functionality may be more aimed at development workflows, it's not compatible with other tools and in my experience the dominant convention in the ecosystem is to use optional dependencies. As @itcarroll points out below, Hatch supports a similar concept called "environment dependencies" and specifies them with a different tool-specific configuration key. PDM also calls it something different, "dev dependencies", and uses another unique config table. 😓

I really don't like the way poetry requires optional dependencies to be specified. It requires duplication.

Thoughts? I don't want to add all this duplication to our codebase. Can we go forward with replacing Poetry with good ol' setuptools or maybe hatchling for builds and switch the optional dependency groups to the standard format? Anything PEP621 compatible would be a step forward IMO. I can take that on as part of this PR. The PEP621 example shows our use case, test dependencies, implemented in the conventional way.

Closes #734
Closes #619
Closes #374

---

📚 Documentation preview 📚: https://earthaccess--733.org.readthedocs.build/en/733/

TODO:

• Update docs

[mfisher87]

this will fail if the lockfile hasn't been updated, or if there's a typo in a package name in poetry config.

[itcarroll]

While the Poetry "dependency groups" functionality may be more aimed at development workflows, it's not compatible with other tools and in my experience the dominant convention in the ecosystem is to use optional dependencies.

I have been exploring hatch w/ hatchling, and it is both PEP compliant and also (like Poetry) avoids using extras (a.k.a optional-dependencies) for managing development environments. Information about extras is built into the distribution metadata and primarily provides extra packages users might want (i.e. xarray[io] if you want netCDF4, etc.). Hatch (and Poetry) allow separate ways of providing packages for and setting up development environments; information not included in distribution metadata. I haven't been compelled to duplicate package lists for simple cases with Hatch. Note that the current potential downside with Hatch is a lack of lock files.

[mfisher87]

Just to clarify, I'm for including these optional dependency groups in the package metadata, as that's in my experience a very common method and is compatible with all environment / package managers, and as far as I know there is no standard way to specify dependency groups that do not go in the package metadata.

I'd like to avoid pros and cons of environment / package managers here (i.e. Hatch, Poetry), as we can isolate that concern from our build tooling and configuration. Regardless of whether we decide down the road to document / encourage Hatch or Poetry or something else for package management, if our build configuration is PEP621 compliant, contributors can choose any tool.

[mfisher87]

To make things more confusing, PDM has "dev dependencies", their own uniquely named version of Poetry's "dependency groups" and Hatch's "environment dependencies". Wow! The "same" thing three times with three names and three config keys.

I'd really like to treat this Python environment management feature as non-standard and avoid it until there's a unified interface, which I'm sure there's already an open PEP for 😆

[mfisher87]

Found it: https://peps.python.org/pep-0735/

[chuckwondo]

@mfisher87, sorry this has remained unreviewed for so long! LGTM!

I'm approving, but do you need to update any "contributing" instructions regarding commands that a contributor needs to run to get their environment setup with dev/test deps?

[mfisher87]

Thanks, Chuck! Good point, I do imagine I missed some stuff there.

TODO:

• Ensure docs are up-to-date with dependency management changes

[mfisher87]

@chuckwondo @danielfromearth I committed the changes we were working on in the call (24a5191), and clearly they are no longer poetry compatible :) I didn't have time to debug yet!

[danielfromearth]

So, where do we want to go from here? If there's no way to remove the redundant dependency specifications, then get this working with another manager, like hatch, and then drop the poetry usage?

[chuckwondo]

Any reason this isn't exposed as a nox command?

Seems like pushing as much into nox as possible would further simplify things, and even make it easier to change things in the future (if desired) without requiring a contributor to do anything differently (i.e., I see "nox" as the interface and the stuff in the noxfile as the "implementation", such that "nox" encapsulates things, allowing use to change the implementation as necessary, without affecting the nox user).

[mfisher87]

I haven't done or seen that done before -- I assumed that environment activation would not be possible to do within a nox command because the command doesn't execute in the current shell?

[chuckwondo]

Sorry, yes, I was looking only at the venv creation. Activation, of course, must be done explicitly by the user, so that's one bit that "leaks" through to the user, which cannot be encapsulated/hidden by nox (or make).

I'm suggesting we put as much as possible behind nox, so that as much as possible, the user uses nox as the one way of doing things.

[mfisher87]

I don't know if I agree that we should position nox as the "one way" of doing things. I definitely think it's valuable as a quickstart to common tasks, though. E.g.: "I don't know how to install a dev environment but I would be embarassed if I pushed my simple error message change and it didn't pass tests".

That's getting in to the philosophy of what task runners are for and not for, so let's take that on separately.

[chuckwondo]

Can we also expose this as a nox command instead?

[mfisher87]

I'm not sure! What would this look like? It doesn't really map to how I normally use and think about nox. In my mental model I create "sessions" as independent and stateless things, e.g. running typechecking nox creates and can cache an environment with mypy and all the runtime dependencies, and for linting, nox can depend on an environment that solely contains ruff.

Maybe this is more common than I think, but using nox to do setup just doesn't seem natural to me :)

[chuckwondo]

I consider nox analogous to make, both of which I consider being able to wrap "complex" commands within simpler commands. So, rather than having to type the pip command above, someone might instead be able to type nox install and be done with it, without fat-fingering arguments.

To take this further, nox (and make et al.) can eliminate duplication. Another suggestion I have is to modify the github workflows to also use nox rather than the "bare" commands. This way, both a contributor and github are using the same commands (both using nox). Granted, a contributor might not run all of the things that github would, but by having both use nox (in our case), there's no need to update docs or workflows to match changes to the noxfile (for existing commands), thus eliminating (or greatly reducing) the likelihood of docs and files getting out of sync.

However, to expedite landing this PR, we can open a separate issue to address this.

[mfisher87]

I agree that this should move into a separate PR. How we think about tasks runners, philosophically, is complicated :) This page in the scientific python library development guide provides some suggestions: https://learn.scientific-python.org/development/guides/tasks/#task-runners

[itcarroll]

Fully approve, with a couple minor comments.

I haven't yet groked when the uv lock file is created or what knows to use it (i would probably have to dive into nox or uv or maybe hatchling docs to understand), but the great thing is I don't have to know because it's on autopilot now.

[itcarroll]

Why do we both have nox included here and instruct contributors to install nox with pipx (or brew, etc.)?

[mfisher87]

What do you think? df578ed

[itcarroll]

Better also remove the make test step from docs/contributing/index.md.

[mfisher87]

NICE FIND! How did I miss that!?

[mfisher87]

0923beb ?

[jhkennedy]

🎉 🎉 Looks great! 🎉 🎉

[mfisher87]

Thanks all for your help! 🙇