Refactor development guide

[jhkennedy]

This PR is largely prompted by recent conversations in our hackweeks, #858, and some discussions in the Openscapes slack (e.g., https://openscapes.slack.com/archives/C05TMK269HA/p1730312039229589).

Basically:

• it wasn't clear in our dev guide why you might choose a conda environment, a venv environment, or just use pipx to run nox
• pipx can be as much of a barrier as installing conda/mamba, especially on Windows, so it doesn't feel like a "quick start"
• how to run tests, generally, outside of the pipx discussion wasn't discussed, so the value of nox wasn't clear
• We changed the conda env name but didn't really note it in the docs, leading to some friction

This PR should:

• provide a note that our conda env name has changed and what to do about it
• provide some guidance on which style of dev env (or not) to use
• better explains how to use nox
• provide some guidance on working in an IDE

I think, overall, this should make things more approachable for contributors, but I suspect this proposal may generate some discussion, so it'd be good to get a few reviewer's eyes on this.

I'm happy to make changes -- this is just a starting point IMO.

Note: I do lean heavily into mkdocs syntax, so you may want to view the preview linked below or checkout this branch and build the docs locally while reviewing this PR.

Pull Request (PR) draft checklist - click to expand

• Please review our
  contributing documentation
  before getting started.
• Populate a descriptive title. For example, instead of "Updated README.md", use a
  title such as "Add testing details to the contributor section of the README".
  Example PRs: #763
• Populate the body of the pull request with:
  • A clear description of the change you are proposing.
  • Links to any issues resolved by this PR with text in the PR description, for
    example closes #1. See
    GitHub docs - Linking a pull request to an issue.
• Update CHANGELOG.md with details about your change in a section titled
  ## Unreleased. If such a section does not exist, please create one. Follow
  Common Changelog for your additions.
  Example PRs: #763
• Update the documentation and/or the README.md with details of changes to the
  earthaccess interface, if any. Consider new environment variables, function names,
  decorators, etc.

Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
once these requirements are fulfilled. Don't worry if you see any test failures in
GitHub at this point!

Pull Request (PR) merge checklist - click to expand

Please do your best to complete these requirements! If you need help with any of these
requirements, you can ping the @nsidc/earthaccess-support team in a comment and we
will help you out!

• Add unit tests for any new features.
• Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
  Request containing "pre-commit.ci autofix" to automate this.
• Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
• Get at least one approving review.

---

📚 Documentation preview 📚: https://earthaccess--874.org.readthedocs.build/en/874/

[github-actions]

👈 Launch a binder notebook on this branch for commit bc6399e

I will automatically update this comment whenever this PR is modified

👈 Launch a binder notebook on this branch for commit 4f1eb58

👈 Launch a binder notebook on this branch for commit 2612c25

👈 Launch a binder notebook on this branch for commit 01b3fc3

👈 Launch a binder notebook on this branch for commit bb02093

👈 Launch a binder notebook on this branch for commit 15b81c3

👈 Launch a binder notebook on this branch for commit b959a62

👈 Launch a binder notebook on this branch for commit 057c8e1

👈 Launch a binder notebook on this branch for commit ebd739b

👈 Launch a binder notebook on this branch for commit 14c2634

[jhkennedy]

Some discussion related to this here:
#872 (comment)

[jhkennedy]

Also, I opened a related discussion here about what we install into our development environments by default:
#875

[andypbarrett]

@jhkennedy I think these changes help a lot.

A few suggestions.

In your comment above you say

pipx can be as much of a barrier as installing conda/mamba, especially on Windows, so it doesn't feel like a "quick start"

I wonder if we should add a note to the pipx tab to say that something along the lines of "Installing pipx on Windows can be a pain, so we recommend using mamba/conda"

Also, should we add a section between the "Development Environment Setup" and "Running Tests" about development workflow and branching? I don't think we can push to main so this seems like a missing step.

[mfisher87]

Looking great! Sorry it took me two weeks to get to this.

I suggested some nitpicks and pushed a few commits; the whitespace changes were difficult to make work with the suggestion syntax. Now that I think about it I could have probably increased the size of the suggestion code fence from triple backticks to 6x backticks and that would have worked. 🤷

I feel the CSS change is really important for being able to discern the limits of a tabbed block.

[jhkennedy]

In your comment above you say

pipx can be as much of a barrier as installing conda/mamba, especially on Windows, so it doesn't feel like a "quick start"

I wonder if we should add a note to the pipx tab to say that something along the lines of "Installing pipx on Windows can be a pain, so we recommend using mamba/conda"

@andypbarrett I have this in the top level "Development environment setup" section:

Do you want it duplicated in the pipx+nox section?

---

Also, should we add a section between the "Development Environment Setup" and "Running Tests" about development workflow and branching? I don't think we can push to main so this seems like a missing step.

Technically, we are telling users to fork so they can push to main, but it's still better to use a feature branch, so I've added:

[mfisher87]

LGTM with nitpick fix :)

[jhkennedy]

@chuckwondo also provided the quoted feedback 👇 in slack, which I've responded to:

• I'm concerned that your changes are leading us back down the path of having too many options and potentially creating confusion again as to how someone should set themselves up for development/contributions, which was a major issue with the poetry-based setup. It's not quite as bad as the old situation, but this feels like it's heading that direction.
• One option that is omitted, and perhaps a means for addressing my concern above, is the use of uv, which should be the most straightforward case since we are now using uv instead of poetry. Further, we can put environment creation (either pip-based via uv, or conda-based) into our nox file, users can still choose pip or conda, but simply run a nox session to create the environment, which should further simply the steps.

I understand the desire to remain flexible, but it's a fine line between flexibility and confusion, so perhaps we can simplify to something along these lines, just as an outline:

• Install nox
• Create environment (either nox -s venv or nox -s conda, or something along those lines, where we put all of the work into the noxfile, so users have much less to do)
• Activate environment (either source .venv/bin/activate or conda activate earthaccess)

I think this is all fair criticism. I really dislike telling users what tools they need to use, and if we were to pick one tool, I'd still vote for mamba over uv/pip/pipx since we're fundamentally serving geospatial users and a lot of things are still better installed via conda-forge on that side of the house (it's getting better though). I might be outvoted here for pip or uv, though.

I think pip/uv and conda/mamba are both the big two options to provide, and I personally would drop pipx+nox long before the other two as I don't see how it does anything better and it is less flexible (won't work for integrated tools that don't understand how nox does things). Similarly, as much as I like uv, I don't want to lean too far into the non-universal parts of it, or else we find ourselves again too tightly coupled to a specific tool (admittedly less of a concern with uv as they seem very keen to work with the python community and within community standards).

Further, we can put environment creation (either pip-based via uv, or conda-based) into our nox file, users can still choose pip or conda, but simply run a nox session to create the environment, which should further simply the steps.

I've seen this example in nox's cookbook, but I do not like it as it hides too much and is brittle across OSs, especially Windows.

I suggest we also drop mention of mamba since conda uses the libmamba solver by default nowadays.

If we're going to pick one, either conda or mamba, I think we should do exactly the opposite -- point users to mambaforge miniforge, in particular, as it is better for the OS community and avoids potential licensing gotchas with the recent anaconda.org changes. I also still find mamba significantly faster than conda using the libmamba solver.

---

@andypbarrett @mfisher87 I think I've addressed all your feedback, but do you want to weigh in on anything @chuckwondo brought up?

[mfisher87]

I really like Pixi's model for tasks.

My reason for preferring to prioritize pip / PyPI for development is because I find it easier to reason about this:

flowchart LR

  PyPI --> dev
  dev --> PyPI
  PyPI --> conda-forge

Loading

than this:

flowchart LR

  conda-forge --> dev
  dev --> PyPI
  PyPI --> conda-forge

Loading

Perhaps it would simplify things if we broke the development guide into two pieces:

1. I want to do something that doesn't require a full development environment, for example updating docs or fixing a bug that seems very straightforward. Links to some instructions to install Nox and links to a page which explains things you can do with Nox for this project.

and

2. I want a full development environment, for example to add a feature or run a debugger against the code. Includes full pip/mamba instructions. Also links to the page which explains things you can do with Nox for this project.

Further, we can put environment creation (either pip-based via uv, or conda-based) into our nox file, users can still choose pip or conda, but simply run a nox session to create the environment, which should further simply the steps.

I've seen this example in nox's cookbook, but I do not like it as it hides too much and is brittle across OSs, especially Windows.

Agree!

I suggest we also drop mention of mamba since conda uses the libmamba solver by default nowadays.

If we're going to pick one, either conda or mamba, I think we should do exactly the opposite -- point users to mambaforge, in particular, as it is better for the OS community and avoids potential licensing gotchas with the recent anaconda.org changes. I also still find mamba significantly faster than conda using the libmamba solver.

Agreed, with one nitpick: mambaforge isn't a thing anymore, mamba is now included with miniforge. Solver aside, I believe mamba still has a download parallelization advantage over conda. That said, conda vs mamba is still confusing to newcomers. It may be worth a note admonition explaining conda and mamba are drop-in compatible with each other, and we are documenting mamba for licensing / community trust reasons.

[jhkennedy]

My reason for preferring to prioritize pip / PyPI for development is because I find it easier to reason about

Okay, I like those charts! ⭐ That might have convinced me to default to pip/PyPI if we're going to document just one way.

I want to do something that doesn't require a full development environment, for example updating docs or fixing a bug that seems very straightforward...

Unfortunately, installing nox isn't that straightforward, and we've gotten a bit of feedback that pipx can be hard to set up on Windows. So, all our routes require a dev environment right now, and it's not always clear which path is the "easy" one.

🤔 Maybe we should lean into something like GitHub Codespaces for the easy path instead. A "push button, make changes" approach for docs and whatnot. (TBH, I do quite a bit in the plain GitHub editor and let CI/CD handle the testing/etc.)

Agreed, with one nitpick: mambaforge isn't a thing anymore, mamba is now included with miniforge.

Yes, typo in my comment -- it's right in the dev guide at least!

That said, conda vs mamba is still confusing to newcomers. It may be worth a note admonition explaining conda and mamba are drop-in compatible with each other, and we are documenting mamba for licensing / community trust reasons.

Like this? 😉 We could add the "for community trust reasons" but that might invite more questions than it answers.

[mfisher87]

Okay, I like those charts! ⭐ That might have convinced me to default to pip/PyPI if we're going to document just one way.

Woohoo visualizations! I struggle to put this in to words convincingly, so I'm very glad this helped :)

🤔 Maybe we should lean into something like GitHub Codespaces for the easy path instead. A "push button, make changes" approach for docs and whatnot. (TBH, I do quite a bit in the plain GitHub editor and let CI/CD handle the testing/etc.)

I think this would be amazing if we can get realtime docs previews working in codespaces.

Like this? 😉 We could add the "for community trust reasons" but that might invite more questions than it answers.

Thanks, what you have looks great :) I agree on "inviting questions". Maybe if we made "community trust reasons" a link to a blog post about their licensing changes, that would provide context for those that want to learn more.

[betolink]

I like this, as in not imposing a single workflow to setup a dev environment.