From 469350ea555351cde8fe2c319fd690e5c50a0637 Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Mon, 4 Nov 2024 11:27:34 -0900 Subject: [PATCH 01/11] Refactor dev guide This should: * provide a note that our counda env name has changed * provide some guidance on which style of dev env (or not) to use * provide some guidance on working in an IDE * better explains how to use nox --- CHANGELOG.md | 2 + docs/contributing/development.md | 199 ++++++++++++++++++++++--------- 2 files changed, 144 insertions(+), 57 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e33af427..937eb3bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ### Changed +- Refactored our development guide to clarify development environment setup and how to run tests + ([@jhkennedy](https://github.com/jhkennedy)) - Use built-in `assert` statements instead of `unittest` assertions in integration tests ([#743](https://github.com/nsidc/earthaccess/issues/743)) ([@chuckwondo](https://github.com/chuckwondo)) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 1bd52211..3b7ef3a5 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -11,22 +11,97 @@ In order to develop new features or fix bugs etc. we need to set up a virtual environment and install the library locally. -## Quickstart development +## Development environment setup -The fastest way to start with development is to use nox. If you don't have nox, -you can use `pipx run nox` to run it without installing, or `pipx install nox`. -If you don't have pipx (pip for applications), then you can install with -`pip install pipx` (the only case were installing an application with regular -pip is reasonable). If you use macOS, then pipx and nox are both in brew, use -`brew install pipx nox`. -To use, run `nox` without any arguments. This will run the type check and unit -test "sessions" (tasks) using your local (and active) Python version. -Nox handles everything for you, including setting up a temporary virtual -environment for each run. +There are a few options for setting up a development environment; you can use Python's `venv`, use `conda`/`mamba`, or _not_ +manage one yourself and use `pipx` to run tests and build docs with `nox`. -You can see all available sessions with `nox --list`: +* If you're a Windows user, you'll likely want to set up an environment yourself with `conda`/`mamba`. +* If you're working in a JupyterHub, you'll likely want to set up an environment yourself with `conda`/`mamba`. +* If you're using an IDE like VS Code or PyCharm, you'll likely want to set up an environment yourself with `venv` or `conda`/`mamba`. +* If you're using a plain text editor and don't know how to or want to manage a virtual environment, you'll likely want to start with `pipx`. + +=== "`venv`" + + `venv` is a virtual environment manager that's built into Python. + + Create and activate the development environment with: + + ```bash + python -m venv .venv + source .venv/bin/activate + ``` + + Then install `earthaccess` into the environment in editable mode with the optional development dependencies: + + ```bash + python -m pip install --editable ".[dev,test,docs]" + ``` + + +=== "`conda`/`mamba`" + + `conda` and `mamba` are open-source package and environment managers that are language and platform agnostic. + `mamba` is a newer and faster re-implementation of `conda` -- you can use either `conda` or `mamba` + in the commands below. See the installation instructions here (we reccomend using Miniforge): + + . + + Once you have `conda`/`mamba` installed, you can create and activate the development environment with: + + ```bash + mamba env update -f environment.yml + mamba activate earthaccess + ``` + + This will update (or create if missing) the `earthaccess` environment and active it. The `earthaccess` package will + be installed into the environment in editable mode with the optional development dependencies. + + !!! note + + On Sept. 23, 2024, the name of the conda environment changed from `earthaccess-dev` to `earthacess` to align with + community best practices. If you have an `earthaccess-dev` environment, we reccomend deleting it and creating a new one. + From the repository root, you can do that with these commands: + + ```bash + mamba env update -f environment.yml + mamba activate earthaccess + mamba env remove -n earthaccess-dev + ``` + +=== "`pipx`+`nox`" + + `pipx` is a tool to help you install and run end-user applications written in Python and `nox` is Python application + that automates testing in multiple Python environments. That means, we can let `pipx` and `nox` manage the `earthaccess` + development environment(s) entirely without having to set one up locally. See the `pipx` installation instructions here: + . + + Once you have `pipx` installed, you can either run `nox` without installing it via: + ```bash + pipx run nox [NOX_ARGS] + ``` + or intall `nox` into an isolated environment and run it with + ```bash + pipx install nox + nox [NOX_ARGS] + ``` + `nox` handles everything for you, including setting up a temporary virtual environment for each test session. + +Now that you're development environment is set up, you can run the tests. + +## Running tests +We recommend using `nox` to run the various test "sessions" (tasks) provided by `earthaccess`. To use, run `nox` without +any arguments: +```bash +nox +``` + +This will run the type check and unit test sessions using your local (and active) Python +version. `nox` handles everything for you, including setting up a temporary virtual environment for each run. + +You can see all available sessions with `nox --list`: ``` $ nox --list Sessions defined in earthaccess/noxfile.py: @@ -41,8 +116,7 @@ Sessions defined in earthaccess/noxfile.py: sessions marked with * are selected, sessions marked with - are skipped. ``` -You can also run individual tasks (_sessions_ in `nox` parlance, hence the `-s` -option below), like so: +You can also run individual sessions like so: ```bash nox -s integration-tests @@ -54,57 +128,68 @@ and pass options to the underlying session like: nox -s integration-tests -- [ARGS] ``` -!!! tip - - In order to run integration tests locally, you must set the - environment variables `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD` to your - username and password, respectively, of your - [NASA Earthdata](https://urs.earthdata.nasa.gov/) account (registration is - free). - - +!!! info "Important" -## Manual development environment setup + In order to run integration tests locally, you must set the environment variables `EARTHDATA_USERNAME` and + `EARTHDATA_PASSWORD` to the username and password of your [NASA Earthdata](https://urs.earthdata.nasa.gov/) + account, respectively (registration is free). -While `nox` is the fastest way to get started, you will likely need a full -development environment for making code contributions, for example to test in a -REPL, or to resolve references in your favorite IDE. This development -environment also includes `nox`. You can create it with `venv`, `conda`, or `mamba`. +### IDE setup -=== "`venv`" - - `venv` is a virtual environment manager that's built into Python. +Integrated development environments (IDEs) like VS Code and PyCharm provide powerful refactoring, testing, and +debugging integrations, but they typically don't understand "task runners" like `nox` and won't know how to launch +debugging or testing sessions connected to the provided integrations. - Create and activate the development environment with: - - ```bash - python -m venv .venv - source .venv/bin/activate - ``` - - Then install `earthaccess` into the environment in editable mode with the optional development dependencies: - - ```bash - python -m pip install --editable ".[dev,test,docs]" - ``` - - -=== "`conda`/`mamba`" +Fortunately, if you've set up a development environment you should be able to call the underlying testing tools +(e.g., `mypy` and `pytest`) directly, or run them via your IDE integrations. To understand how `nox` is running the +underlying tools in each test session you can read the `noxfile.py` in the repository root, or, run all the test directly +in your development environment like: +```bash +nox -fb none --no-install +``` +This will force `nox` to not use an environment backend (will just use the active environment) and not attempt to install +any packages. When `nox` runs, it will describe the commands it executes: +``` +$ nox -fb none --no-install +nox > Running session typecheck +nox > mypy +Success: no issues found in 35 source files +nox > Session typecheck was successful. +nox > Running session tests +nox > pytest tests/unit -rxXs +========================================== test session starts ========================================== +... +==================================== 43 passed, 1 xfailed in 24.01s ===================================== +nox > Session tests was successful. +nox > Ran multiple sessions: +nox > * typecheck: success +nox > * tests: success +``` +Note these lines in particular: +``` +nox > Running session typecheck +nox > mypy +nox > Running session tests +nox > pytest tests/unit -rxXs +``` +So to reproduce the typecheck session all you have to do is run `mypy` in your development environment. Similarly, reproducing +the unit tests is running `pytest test/unit -rxXs`. - `conda` and `mamba` are open-source package and environment managers that are language and platform agnostic. - `mamba` is a newer and faster re-implementation of `conda` -- you can use either `conda` or `mamba` - in the commands below. +Since we're not doing any complicated configuration or setting complicated arguments to pytest, simply hitting the "play" button +for a pytest in your IDE should work once you've configured it to use your development environment. - Create and activate the development environment with: +!!! info "Important" + Currently, our integration tests are *flakey* and a small number of random failures are expected. When the integration + test suite runs, it may retun a status code of 99 if the failure rate was less than an "acceptable" threshold. Since + any non-zero status code is considered an error, your console and/or IDE wll consider this a failure by default. + `nox`, however, knows about this special status code and will report a success. To get pytest or your IDE to match + this behavior, you can modify the special status code to be zero with the `EARTHACCESS_ALLOWABLE_FAILURE_STATUS_CODE` + evnironment variable: ```bash - mamba env update -f environment.yml - mamba activate earthaccess + export EARTHACCESS_ALLOWABLE_FAILURE_STATUS_CODE=0 ``` - This will update (or create if missing) the `earthaccess` environment and active it. The `earthaccess` package will - be installed into the environment in editable mode with the optional development dependencies. - ## Managing Dependencies If you need to add a new dependency, edit `pyproject.toml` and insert the @@ -134,14 +219,14 @@ Since `python-cmr` exposes the `cmr` package, the stubs appear under `stubs/cmr` To work on documentation locally, we provide a script that will automatically re-render the docs when you make changes: ``` -nox -s serve_docs +nox -s serve-docs ``` MkDocs does not support incremental rebuilds and will execute every Jupyter Notebook every time it builds a new version of the site, which can be quite slow. To speed up the build, you can pass MkDocs these options: ``` -nox -s serve_docs -- --dirty --no-strict +nox -s serve-docs -- --dirty --no-strict ``` !!! warning From bc6399e5073c3d86a0f20493d05d7d8f399ca279 Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Thu, 7 Nov 2024 14:01:17 -0900 Subject: [PATCH 02/11] typo --- docs/contributing/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 3b7ef3a5..bc202c86 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -88,7 +88,7 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. ``` `nox` handles everything for you, including setting up a temporary virtual environment for each test session. -Now that you're development environment is set up, you can run the tests. +Now that your development environment is set up, you can run the tests. ## Running tests From 4f1eb5834e2c6d91d85a80c49e405f6f903001df Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Thu, 7 Nov 2024 14:24:36 -0900 Subject: [PATCH 03/11] codespell --- docs/contributing/development.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index bc202c86..567ecd15 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -44,7 +44,7 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. `conda` and `mamba` are open-source package and environment managers that are language and platform agnostic. `mamba` is a newer and faster re-implementation of `conda` -- you can use either `conda` or `mamba` - in the commands below. See the installation instructions here (we reccomend using Miniforge): + in the commands below. See the installation instructions here (we recommend using Miniforge): . @@ -61,7 +61,7 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. !!! note On Sept. 23, 2024, the name of the conda environment changed from `earthaccess-dev` to `earthacess` to align with - community best practices. If you have an `earthaccess-dev` environment, we reccomend deleting it and creating a new one. + community best practices. If you have an `earthaccess-dev` environment, we recommend deleting it and creating a new one. From the repository root, you can do that with these commands: ```bash @@ -81,7 +81,7 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. ```bash pipx run nox [NOX_ARGS] ``` - or intall `nox` into an isolated environment and run it with + or install `nox` into an isolated environment and run it with ```bash pipx install nox nox [NOX_ARGS] @@ -181,8 +181,8 @@ for a pytest in your IDE should work once you've configured it to use your devel !!! info "Important" Currently, our integration tests are *flakey* and a small number of random failures are expected. When the integration - test suite runs, it may retun a status code of 99 if the failure rate was less than an "acceptable" threshold. Since - any non-zero status code is considered an error, your console and/or IDE wll consider this a failure by default. + test suite runs, it may return a status code of 99 if the failure rate was less than an "acceptable" threshold. Since + any non-zero status code is considered an error, your console and/or IDE will consider this a failure by default. `nox`, however, knows about this special status code and will report a success. To get pytest or your IDE to match this behavior, you can modify the special status code to be zero with the `EARTHACCESS_ALLOWABLE_FAILURE_STATUS_CODE` evnironment variable: From 2612c254b5df4a3bb4dc52293551842981460d53 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 23 Nov 2024 11:03:18 -0700 Subject: [PATCH 04/11] docs: Make the scope of tabbed blocks more discernable --- docs/css/styles.css | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/css/styles.css b/docs/css/styles.css index 43aa0abf..cec552d2 100644 --- a/docs/css/styles.css +++ b/docs/css/styles.css @@ -1,2 +1,9 @@ .details .open { } + +.tabbed-content { + padding: 5px; + border: 1px lightgrey; + border-radius: 3px; + border-style: dashed; +} From 01b3fc323642e6fd1320b4f3bd106b0861516c66 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 23 Nov 2024 11:05:36 -0700 Subject: [PATCH 05/11] docs: Add whitespace in pipx tab --- docs/contributing/development.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 567ecd15..fdfd5520 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -78,14 +78,18 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. . Once you have `pipx` installed, you can either run `nox` without installing it via: + ```bash pipx run nox [NOX_ARGS] ``` + or install `nox` into an isolated environment and run it with + ```bash pipx install nox nox [NOX_ARGS] ``` + `nox` handles everything for you, including setting up a temporary virtual environment for each test session. Now that your development environment is set up, you can run the tests. From bb02093969497e40325462252653a4ec21300763 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 23 Nov 2024 11:13:19 -0700 Subject: [PATCH 06/11] docs: Add whitespace in IDE instructions --- docs/contributing/development.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index fdfd5520..86e2fbe6 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -148,11 +148,14 @@ Fortunately, if you've set up a development environment you should be able to ca (e.g., `mypy` and `pytest`) directly, or run them via your IDE integrations. To understand how `nox` is running the underlying tools in each test session you can read the `noxfile.py` in the repository root, or, run all the test directly in your development environment like: + ```bash nox -fb none --no-install ``` + This will force `nox` to not use an environment backend (will just use the active environment) and not attempt to install any packages. When `nox` runs, it will describe the commands it executes: + ``` $ nox -fb none --no-install nox > Running session typecheck @@ -169,13 +172,16 @@ nox > Ran multiple sessions: nox > * typecheck: success nox > * tests: success ``` + Note these lines in particular: + ``` nox > Running session typecheck nox > mypy nox > Running session tests nox > pytest tests/unit -rxXs ``` + So to reproduce the typecheck session all you have to do is run `mypy` in your development environment. Similarly, reproducing the unit tests is running `pytest test/unit -rxXs`. From 15b81c32c8b4240c035e9b2d3d91dea89ed74d07 Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Mon, 25 Nov 2024 16:58:10 -0900 Subject: [PATCH 07/11] Apply suggestions from code review Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com> --- docs/contributing/development.md | 26 +++++++++++--------------- docs/css/styles.css | 2 +- 2 files changed, 12 insertions(+), 16 deletions(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 86e2fbe6..bafd48f9 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -44,21 +44,19 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. `conda` and `mamba` are open-source package and environment managers that are language and platform agnostic. `mamba` is a newer and faster re-implementation of `conda` -- you can use either `conda` or `mamba` - in the commands below. See the installation instructions here (we recommend using Miniforge): - - . - - Once you have `conda`/`mamba` installed, you can create and activate the development environment with: + in the commands below. + + After you have followed [the `conda`/`mamba` installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html) (we recommend Miniforge), you can create and activate the development environment with: ```bash mamba env update -f environment.yml mamba activate earthaccess ``` - This will update (or create if missing) the `earthaccess` environment and active it. The `earthaccess` package will + This will ensure the `earthaccess` environment exists and is up-to-date, and active it. The `earthaccess` package will be installed into the environment in editable mode with the optional development dependencies. - !!! note + ??? note "2024-09-23: Conda environment name changed from `earthaccess-dev` -> `earthaccess`" On Sept. 23, 2024, the name of the conda environment changed from `earthaccess-dev` to `earthacess` to align with community best practices. If you have an `earthaccess-dev` environment, we recommend deleting it and creating a new one. @@ -73,31 +71,29 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. === "`pipx`+`nox`" `pipx` is a tool to help you install and run end-user applications written in Python and `nox` is Python application - that automates testing in multiple Python environments. That means, we can let `pipx` and `nox` manage the `earthaccess` - development environment(s) entirely without having to set one up locally. See the `pipx` installation instructions here: - . - - Once you have `pipx` installed, you can either run `nox` without installing it via: + that automates testing in multiple Python environments. By using `pipx` to install `nox` and using `nox` to run common development tasks, some users can avoid the need to set up and manage a full development environment. + Once you have [installed `pipx` following the official instructions](https://github.com/pypa/pipx?tab=readme-ov-file#install-pipx), you can either run `nox` without installing it via: ```bash pipx run nox [NOX_ARGS] ``` - or install `nox` into an isolated environment and run it with + or install `nox` into an isolated environment and run it with: ```bash pipx install nox nox [NOX_ARGS] ``` - `nox` handles everything for you, including setting up a temporary virtual environment for each test session. + `nox` handles everything for you, including setting up a temporary virtual environment for each task (e.g. running tests, building docs, etc.) you need to run. Now that your development environment is set up, you can run the tests. ## Running tests -We recommend using `nox` to run the various test "sessions" (tasks) provided by `earthaccess`. To use, run `nox` without +We recommend using `nox` to run the various "sessions" (`nox`'s term for tasks) provided by `earthaccess`. To use, run `nox` without any arguments: + ```bash nox ``` diff --git a/docs/css/styles.css b/docs/css/styles.css index cec552d2..cbc0ab2a 100644 --- a/docs/css/styles.css +++ b/docs/css/styles.css @@ -1,7 +1,7 @@ .details .open { } -.tabbed-content { +.tabbed-set { padding: 5px; border: 1px lightgrey; border-radius: 3px; From b959a6264ffc5e124ef443c9c2f86b3e27fd320f Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Mon, 25 Nov 2024 17:50:15 -0900 Subject: [PATCH 08/11] match API docs table style --- docs/css/styles.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/css/styles.css b/docs/css/styles.css index cbc0ab2a..dbb65baa 100644 --- a/docs/css/styles.css +++ b/docs/css/styles.css @@ -4,6 +4,6 @@ .tabbed-set { padding: 5px; border: 1px lightgrey; - border-radius: 3px; - border-style: dashed; + border-radius: 2px; + border-style: solid; } From 057c8e19394381748e4ba050cacc1822859b6cf3 Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Mon, 25 Nov 2024 17:59:27 -0900 Subject: [PATCH 09/11] Discuss using a feature branch --- docs/contributing/development.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index bafd48f9..c1f9fb35 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -7,8 +7,13 @@ ```bash git clone git@github.com:{my-username}/earthaccess ``` +1. We recommend using a feature when developing new features, fixing bugs, updating docs, etc. It's best to give the branch +a descriptive name For example, if you're updating the contributing docs, you might create a branch called `update-contributing-docs`: + ```bash + git switch -c update-contributing-docs + ``` -In order to develop new features or fix bugs etc. we need to set up a virtual +Next, In order to develop new features or fix bugs etc., you'll need to set up a virtual environment and install the library locally. ## Development environment setup From ebd739be7ae381607fc9f8d9d912dd77fee43ea2 Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Mon, 25 Nov 2024 18:04:16 -0900 Subject: [PATCH 10/11] Update docs/contributing/development.md Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com> --- docs/contributing/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index c1f9fb35..0df0787f 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -7,7 +7,7 @@ ```bash git clone git@github.com:{my-username}/earthaccess ``` -1. We recommend using a feature when developing new features, fixing bugs, updating docs, etc. It's best to give the branch +1. We recommend creating a feature branch when developing new features, fixing bugs, updating docs, etc. It's best to give the branch a descriptive name For example, if you're updating the contributing docs, you might create a branch called `update-contributing-docs`: ```bash git switch -c update-contributing-docs From 14c2634c4ed60c7b385f2b780309dcdbd8a9637d Mon Sep 17 00:00:00 2001 From: Joseph H Kennedy Date: Mon, 25 Nov 2024 18:10:22 -0900 Subject: [PATCH 11/11] whitespace --- docs/contributing/development.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 0df0787f..aa97c41a 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -44,13 +44,12 @@ manage one yourself and use `pipx` to run tests and build docs with `nox`. python -m pip install --editable ".[dev,test,docs]" ``` - === "`conda`/`mamba`" `conda` and `mamba` are open-source package and environment managers that are language and platform agnostic. `mamba` is a newer and faster re-implementation of `conda` -- you can use either `conda` or `mamba` in the commands below. - + After you have followed [the `conda`/`mamba` installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html) (we recommend Miniforge), you can create and activate the development environment with: ```bash