From ee0a9e5212b90732a8eab58095a81f3fee02c845 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Tue, 28 May 2024 14:37:25 -0400 Subject: [PATCH 01/35] recategorize changes into "Common Changelog" groups, format headings, and use imperative mood According to --- CHANGELOG.md | 426 +++++++++++++++++++++++++++------------------------ 1 file changed, 229 insertions(+), 197 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aabad758..219275a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,224 +2,256 @@ ## [Unreleased] -* Changes +### Changed +- [#555](https://github.com/nsidc/earthaccess/issues/555): YAML formatting is + now performed with `yamlfmt` instead of `prettier`. +- [#511](https://github.com/nsidc/earthaccess/issues/511): Replace `print` + calls with `logging` calls where appropriate and add T20 Ruff rule. +- [#508](https://github.com/nsidc/earthaccess/issues/508): Correct and + enhance static type hints for functions and methods that make CMR queries + or handle CMR query results. +- [#562](https://github.com/nsidc/earthaccess/issues/562): The destination + path is now created prior to direct S3 downloads, if it doesn't already + exist. + +### Added +- [#483](https://github.com/nsidc/earthaccess/issues/483): Now using + [Search After](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#search-after) + for collection and granule searches to support deep-paging through large + result sets. +- [#421](https://github.com/nsidc/earthaccess/issues/421): Enable queries to + Earthdata User Acceptance Testing (UAT) system for authenticated accounts. - * [#421](https://github.com/nsidc/earthaccess/issues/421): Removed the +### Removed +- **Breaking:**: [#421](https://github.com/nsidc/earthaccess/issues/421): Remove the `get_user_profile` method and the `email_address` and `profile` attributes from the `Auth` class. Calling the EDL API to get user profile information is not intended for library access and is not necessary for this library's intended use cases. - * [#555](https://github.com/nsidc/earthaccess/issues/555): YAML formatting is - now performed with `yamlfmt` instead of `prettier`. - * [#511](https://github.com/nsidc/earthaccess/issues/511): Replaced `print` - calls with `logging` calls where appropriate and added T20 Ruff rule. -* Enhancements +## [0.9.0] - 2024-02-28 - * [#483](https://github.com/nsidc/earthaccess/issues/483): Now using - [Search After](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#search-after) - for collection and granule searches to support deep-paging through large - result sets. - * [#508](https://github.com/nsidc/earthaccess/issues/508): Corrected and - enhanced static type hints for functions and methods that make CMR queries - or handle CMR query results. - * [#421](https://github.com/nsidc/earthaccess/issues/421): Enabled queries to - Earthdata User Acceptance Testing (UAT) system for authenticated accounts. - * [#562](https://github.com/nsidc/earthaccess/issues/562): The destination - path is now created prior to direct S3 downloads, if it doesn't already - exist. +### Added +- Address #427 by adding parameters to collection query +- Add user-agent in the request to track usage, closes #436 + +### Fixed +- Fix #439 by implementing more trusted domains in the SessionWithRedirection +- Fix #438 by using an authenticated session for hits() -## [v0.9.0] 2024-02-28 - -* Bug fixes: - * fixed #439 by implementing more trusted domains in the SessionWithRedirection - * fixed #438 by using an authenticated session for hits() -* Enhancements: - * addressing #427 by adding parameters to collection query - * added user-agent in the request to track usage, closes #436 - -## [v0.8.2] 2023-12-06 -* Bug fixes: - * Enable AWS check with IMDSv2 - * Add region to running in AWS check - * Handle opening multi-file granules -* Maintenance: - * Add CI tests with minimum supported versions - * Update poetry lockfile - * Add `python-dateutil` as a direct dependency - * Remove binder PR comments - * Add YAML formatting (prettier) - -## [v0.8.1] 2023-12-01 -* New Features: - * Add `kerchunk` metadata consolidation utility. -* Enhancements: - * Handle S3 credential expiration more gracefully. -* Maintenanece: - * Use dependabot to update Github Actions. - * Consolidate dependabot updates. - * Switch to `ruff` for formatting. - -## [v0.8.0] 2023-11-29 -* Bug fixes: - * Fix zero granules being reported for restricted datasets. -* Enhancements: - * earthaccess will `raise` errors instead of `print`ing them in more cases. - * `daac` and `provider` parameters are now normalized to uppercase, since lowercase +## [0.8.2] - 2023-12-06 + +### Changed +- Update poetry lockfile +- Use YAML formatting (prettier) + +### Added +- Add CI tests with minimum supported versions +- Add `python-dateutil` as a direct dependency + +### Removed +- Remove binder PR comments + +### Fixed +- Enable AWS check with IMDSv2 +- Add region to running in AWS check +- Handle opening multi-file granules + +## [0.8.1] - 2023-12-01 + +### Changed +- Handle S3 credential expiration more gracefully. +- Use dependabot to update Github Actions. +- Consolidate dependabot updates. +- Switch to `ruff` for formatting. + +### Added +- Add `kerchunk` metadata consolidation utility. + +## [0.8.0] - 2023-11-29 + +### Changed +- earthaccess will `raise` errors instead of `print`ing them in more cases. +- `daac` and `provider` parameters are now normalized to uppercase, since lowercase characters are never valid. -## [v0.7.1] 2023-11-08 -* Bug Fixes: - * Treat granules without `RelatedUrls` as not cloud-hosted. +### Fixed +- Fix zero granules being reported for restricted datasets. -## [v0.7.0] 2023-10-31 -* Bug Fixes: - * Fix spelling mistake in `access` variable assignment (`direc` -> `direct`) - in `earthaccess.store._get_granules`. - * Pass `threads` arg to `_open_urls_https` in - `earthaccess.store._open_urls`, replacing the hard-coded value of 8. - * Return S3 data links by default when in region. -* Enhancements: - * `earthaccess.download` now accepts a single granule as input in addition to a list of granules. - * `earthaccess.download` now returns fully qualified local file paths. -* New Features: - * Earthaccess will now automatically search for Earthdata authentication. ``earthaccess.login()`` +## [0.7.1] - 2023-11-08 + +### Fixed +- Treat granules without `RelatedUrls` as not cloud-hosted. + +## [0.7.0] - 2023-10-31 + +### Changed +- `earthaccess.download` now accepts a single granule as input in addition to a list of granules. +- `earthaccess.download` now returns fully qualified local file paths. + +### Added +- Earthaccess will now automatically search for Earthdata authentication. ``earthaccess.login()`` still works as before, but is no longer required if you have a ``~/.netrc`` file for have set ``EARTHDATA_USERNAME`` and ``EARTHDATA_PASSWORD`` environment variables. - * Add `earthaccess.auth_environ()` utility for getting Earthdata authentication environment variables. - -## [v0.6.0] 2023-09-20 -* bug fixes: - * earthaccess.search_datasets() and earthaccess.search_data() can find restricted datasets #296 - * distributed serialization fixed for EarthAccessFile #301 and #276 -* new features: - * earthaccess.get_s3fs_session() can use the results to find the right set of S3 credentials - -## [v0.5.3] 2023-08-01 -* bug fixes: - * granule's size() returned zero - * Added exception handling for fsspec sessions, thanks to @jrbourbeau -* CI changes: - * integration tests are now only run when we push to main (after a merge) - * unit tests run for any branch and opened PR - -## [v0.5.2] 2023-04-21 -* bug fixes: - * Fixing #230 by removing Benedict as the dict handler, thanks to @psarka! - * S3 credential endpoints are tried with tokens and basic auth until all the DAACs accept the same auth - -* Core dependencies: - * Removed Benedict as the default dict for JSON coming from CMR. - -## [v0.5.1] 2023-03-20 - -* bug fixes: - * get_s3_credentials() only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 - * including tests for all DAAC S3 endpoints - * Notebooks updated to use the new top level API - * removed magic from dependencies (not available in windows and not used but just in tests) - -* CI changes: - * documentation for readthedocs fixed by including poetry as the default tool - * injected new secrets to test Auth using the icepyx convention (EARTHDATA_USERNAME) -* New Features - * we can get the user's profile with auth.user_profile which includes the user email - * added LAAD as a supported DAAC -## [v0.5.0] 2023-02-23 - -* bug fixes: - * @JessicaS11 fixed a bug where the Auth class was invoked without the proper parameters - * if a user specifies the netrc strategy and there is no netrc an exception is raised - * S3 URLs broke the Store class when opened outside AWS - * Opening files using URLs was not working properly on AWS, thanks to @amfriesz for reporting it! -* CI changes: - * documentation is now only built for the main, dev and documentation branches - * noteboks are executed every time the documentation gets published! -* New features: - * we can now use the top level API to get S3 credentials, authenticated fsspec and requests sessions! - * ASF direct access for Sentinel1 products is now available - -## [v0.4.7] 2022-12-11 - -* bug fixes: - * fixed open() for direct access - * python-magic is a test dependency so moved to the dev section. - * Minor edits in the README - -## [v0.4.6] 2022-12-08 - -* Features: - * search collections by DOI - * new API documentation and simplified notation to access data -* CI changes: - * only run the publish workflow after a release on github - -## [v0.4.1] 2022-11-02 - -* improved documentation: - * reimplemented python_cmr methods for docstring compatibility - * added types to method signatures -* Using `CMR-Search-After` see #145 - -* CI changes: - * Poetry is installed using the new script - * Dependabot alerts to monthly - -* Added GES_DISC S3 endpoint - -## [v0.4.0] 2022-08-17 - -* `store` - * uses fsspec s3fs for in cloud access and https sessions for out of region access - * we can open files with fsspec in and out of region (stream into xarray) -* `auth` - * we can persist our credentials into a `.netrc` file - -* Documentation - * added store, auth to docs and updated mkdocs congif - - -## [v0.3.0] 2022-04-28 - -- Fixed bug with CMR tokens -- dropped python 3.7 support -- updated python-cmr to NASA fork -- added documentation for readthedocs and github -- verifying git tag and poetry version are the same before publish to pypi +- Add `earthaccess.auth_environ()` utility for getting Earthdata authentication environment variables. + +### Fixed +- Fix spelling mistake in `access` variable assignment (`direc` -> `direct`) + in `earthaccess.store._get_granules`. +- Pass `threads` arg to `_open_urls_https` in + `earthaccess.store._open_urls`, replacing the hard-coded value of 8. +- Return S3 data links by default when in region. + +## [0.6.0] - 2023-09-20 + +### Added +- earthaccess.get_s3fs_session() can use the results to find the right set of S3 credentials + +### Fixed +- earthaccess.search_datasets() and earthaccess.search_data() can find restricted datasets #296 +- distributed serialization fixed for EarthAccessFile #301 and #276 + +## [0.5.3] - 2023-08-01 + +### Changed +- For CI, integration tests are now only run when we push to main (after a merge) +- For CI, unit tests are run for any branch and opened PR + +### Fixed +- granule's size() returned zero +- Add exception handling for fsspec sessions, thanks to @jrbourbeau + +## [0.5.2] - 2023-04-21 + +### Removed +- Remove Benedict (core dependency) as the default dict for JSON coming from CMR. + +### Fixed +- Fix #230 by removing Benedict as the dict handler, thanks to @psarka! +- S3 credential endpoints are tried with tokens and basic auth until all the DAACs accept the same auth + +## [0.5.1] - 2023-03-20 + +### Changed +- For CI, documentation for readthedocs fixed by including poetry as the default tool +- For CI, injected new secrets to test Auth using the icepyx convention (EARTHDATA_USERNAME) + +### Added +- Add ability to get the user's profile with auth.user_profile which includes the user email +- Add LAAD as a supported DAAC + +### Fixed +- get_s3_credentials() only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 +- including tests for all DAAC S3 endpoints +- Notebooks updated to use the new top level API +- removed magic from dependencies (not available in windows and not used but just in tests) + +## [0.5.0] - 2023-02-23 + +### Changed +- For CI, documentation is now only built for the main, dev and documentation branches +- For CI, noteboks are executed every time the documentation gets published! + +### Added +- Add ability to use the top level API to get S3 credentials, authenticated fsspec and requests sessions! +- Make available ASF direct access for Sentinel1 products + +### Fixed +- @JessicaS11 fixed a bug where the Auth class was invoked without the proper parameters +- if a user specifies the netrc strategy and there is no netrc an exception is raised +- S3 URLs broke the Store class when opened outside AWS +- Opening files using URLs was not working properly on AWS, thanks to @amfriesz for reporting it! + +## [0.4.7] - 2022-12-11 + +### Fixed +- Fix open() for direct access +- Move python-magic to the dev section because it is a test dependency +- Make minor edits in the README + +## [0.4.6] - 2022-12-08 + +### Changed +- For CI, only run the publish workflow after a release on GitHub + +### Added +- Add feature to search collections by DOI +- Add new API documentation and simplify notation to access data + +## [0.4.1] - 2022-11-02 + +### Changed +- For CI, install Poetry using the new script +- For CI, change dependabot alerts to monthly +- Improve documentation by reimplementing python_cmr methods for docstring compatibility +- Use `CMR-Search-After` see #145 + +### Added +- Add GES_DISC S3 endpoint +- Improve documentation by adding types to method signatures + +## [0.4.0] - 2022-08-17 + +### Added +- Add store, auth to docs and update mkdocs congif +- For `auth`, add ability to persist credentials into a `.netrc` file +- For `store`, use fsspec s3fs for in cloud access and https sessions for out of region access +- For `store`, can open files with fsspec in and out of region (stream into xarray) + +## [0.3.0] - 2022-04-28 + +### Changed +- Update python-cmr to NASA fork +- Drop unused `pydantic` dependency - Auth can refresh CMR tokens -- Dropped unused `pydantic` dependency -- Added missing `python-datutil` dependency +- Verify git tag and poetry version are the same before publish to pypi + +### Added +- Add documentation for readthedocs and GitHub + +### Removed +- **Breaking**: Dropped python 3.7 support -## [v0.2.2] 2022-03-23 -- Bug fixes to store to download multi-file granules +### Fixed +- Fix bug with CMR tokens +- Add missing `python-datutil` dependency + +## [0.2.2] - 2022-03-23 + +### Fixed +- Fix store to download multi-file granules - Fix granule formatting -## [v0.2.1] 2022-03-19 -- Renamed Accessor to Store -- relaxed dependency requirements +## [0.2.1] - 2022-03-19 + +### Changed +- Rename Accessor to Store +- Relax dependency requirements - Store can download plain links if they are on prem -## [v0.1.0-beta.1] - 2021-09-21 +## [0.1.0-beta.1] - 2021-09-21 + +_Conception!_ -- Conception! +### Added - Add basic classes to interact with NASA CMR, EDL and cloud access. - Basic object formatting. [Unreleased]: https://github.com/nsidc/earthaccess/compare/v0.9.0...HEAD -[v0.8.2]: https://github.com/nsidc/earthaccess/releases/tag/v0.8.2 -[v0.8.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.8.1 -[v0.8.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.8.0 -[v0.7.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.7.1 -[v0.7.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.7.0 -[v0.6.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.6.0 -[v0.5.3]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.3 -[v0.5.2]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.2 -[v0.5.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.1 -[v0.5.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.0 -[v0.4.7]: https://github.com/nsidc/earthaccess/releases/tag/v0.4.7 -[v0.4.6]: https://github.com/nsidc/earthaccess/releases/tag/v0.4.6 -[v0.4.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.4.1 -[v0.3.0]: https://github.com/betolink/earthaccess/releases/tag/v0.3.0 -[v0.2.2]: https://github.com/betolink/earthaccess/releases/tag/v0.2.2 -[v0.2.1]: https://github.com/betolink/earthaccess/releases/tag/v0.2.1 -[v0.1.0-beta.1]: https://github.com/betolink/earthaccess/releases/tag/v0.1.0-beta.1 +[0.8.2]: https://github.com/nsidc/earthaccess/releases/tag/v0.8.2 +[0.8.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.8.1 +[0.8.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.8.0 +[0.7.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.7.1 +[0.7.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.7.0 +[0.6.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.6.0 +[0.5.3]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.3 +[0.5.2]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.2 +[0.5.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.1 +[0.5.0]: https://github.com/nsidc/earthaccess/releases/tag/v0.5.0 +[0.4.7]: https://github.com/nsidc/earthaccess/releases/tag/v0.4.7 +[0.4.6]: https://github.com/nsidc/earthaccess/releases/tag/v0.4.6 +[0.4.1]: https://github.com/nsidc/earthaccess/releases/tag/v0.4.1 +[0.3.0]: https://github.com/betolink/earthaccess/releases/tag/v0.3.0 +[0.2.2]: https://github.com/betolink/earthaccess/releases/tag/v0.2.2 +[0.2.1]: https://github.com/betolink/earthaccess/releases/tag/v0.2.1 +[0.1.0-beta.1]: https://github.com/betolink/earthaccess/releases/tag/v0.1.0-beta.1 From 7fba5a3edc06e96749a7c96a18ca4c15a413ae48 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Wed, 29 May 2024 12:26:33 -0400 Subject: [PATCH 02/35] typo --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 219275a6..8ed932bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,7 +23,7 @@ Earthdata User Acceptance Testing (UAT) system for authenticated accounts. ### Removed -- **Breaking:**: [#421](https://github.com/nsidc/earthaccess/issues/421): Remove the +- **Breaking**: [#421](https://github.com/nsidc/earthaccess/issues/421): Remove the `get_user_profile` method and the `email_address` and `profile` attributes from the `Auth` class. Calling the EDL API to get user profile information is not intended for library access and is not necessary for this library's @@ -150,7 +150,7 @@ ### Changed - For CI, documentation is now only built for the main, dev and documentation branches -- For CI, noteboks are executed every time the documentation gets published! +- For CI, notebooks are executed every time the documentation gets published! ### Added - Add ability to use the top level API to get S3 credentials, authenticated fsspec and requests sessions! @@ -193,8 +193,8 @@ ## [0.4.0] - 2022-08-17 ### Added -- Add store, auth to docs and update mkdocs congif -- For `auth`, add ability to persist credentials into a `.netrc` file +- Add store, auth to docs and update mkdocs config +- For `auth`, add the ability to persist credentials into a `.netrc` file - For `store`, use fsspec s3fs for in cloud access and https sessions for out of region access - For `store`, can open files with fsspec in and out of region (stream into xarray) From 8556e11686aba6235046bc934ec318cce8f030fc Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 25 Jun 2024 12:33:56 -0600 Subject: [PATCH 03/35] Init PR template --- .github/pull_request_template.md | 36 ++++++++++++++++++++++++++++++++ docs/contributing/index.md | 19 +++-------------- 2 files changed, 39 insertions(+), 16 deletions(-) create mode 100644 .github/pull_request_template.md diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..6394a631 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,36 @@ +
PR draft checklist + +Please start by opening this Pull Request as a "draft". You can do this by clicking the +arrow on the right side of the green "Create pull request" button. + +- [ ] Please review our + [contributing documentation](https://earthaccess.readthedocs.io/en/latest/contributing/) + before getting started. +- [ ] Ensure an issue exists representing the problem being solved in this PR. +- [ ] Link any issues resolved by this PR with text in the PR description, for example + `closes #1`. +- [ ] Update `CHANGELOG.md` with details about your change in a section titled + `## Unreleased`. If such a section does not exist, please create one. +- [ ] Update the documentation and/or the `README.md` with details of changes to the + interface. This includes new environment variables, function names, decorators, + etc. + +Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub +once these requirements are fulfilled. Don't worry if you see any test failures in +GitHub at this point! + +
+ +
PR merge checklist + +Please do your best to complete these requirements! If you need help with any of these +requirements, you can ping the `@nsidc/earthaccess-support` team in a comment and we +will help you out! + +- [ ] Add unit tests for any new features. +- [ ] Apply formatting and linting autofixes. You can add a GitHub comment in this Pull + Request containing "pre-commit.ci autofix" to automate this. +- [ ] Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass. +- [ ] Get at least one approving review. + +
diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 3d149637..b0f24a1b 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -66,21 +66,8 @@ We have included type stubs for the untyped `python-cmr` library, which we intend to eventually upstream. Since `python-cmr` exposes the `cmr` package, the stubs appear under `stubs/cmr`. -### Requirements to merge code (Pull Request Process) - -- you must include test coverage -- you must update the documentation -- you must format and lint - ## Pull Request process -1. Ensure you include test coverage for all changes -1. Ensure your code is formatted properly following this document -1. Update the documentation and the `README.md` with details of changes to the - interface, this includes new environment variables, function names, - decorators, etc. -1. Update `CHANGELOG.md` with details about your change in a section titled - `Unreleased`. If one does not exist, please create one. -1. You may merge the Pull Request once you have the sign-off of another - developer, or if you do not have permission to do that, you may request the - reviewer to merge it for you. +Fork the repository using the "Fork" button on the [repository +homepage](https://github.com/nsidc/earthaccess), make your change on the fork, then open +a pull request from your fork and follow the instructions populated in the text box. From 47a1366f797ed1dc6deecf3e22fed9224f49e06c Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 25 Jun 2024 12:49:07 -0600 Subject: [PATCH 04/35] Clarify the instructional text in the PR template --- .github/pull_request_template.md | 42 +++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 6394a631..5b2df6f3 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,27 +1,45 @@ -
PR draft checklist - -Please start by opening this Pull Request as a "draft". You can do this by clicking the -arrow on the right side of the green "Create pull request" button. - + + + + +
Pull Request (PR) draft checklist - click to expand + +- [ ] Populate a descriptive title. Ensure the title does not look like "Updated + README.md". +- [ ] Populate the body of the pull request with : + - A clear description of the change you are proposing. + - Links to any issues resolved by this PR with text in the PR description, for + example "closes #1". - [ ] Please review our [contributing documentation](https://earthaccess.readthedocs.io/en/latest/contributing/) before getting started. - [ ] Ensure an issue exists representing the problem being solved in this PR. -- [ ] Link any issues resolved by this PR with text in the PR description, for example - `closes #1`. - [ ] Update `CHANGELOG.md` with details about your change in a section titled `## Unreleased`. If such a section does not exist, please create one. - [ ] Update the documentation and/or the `README.md` with details of changes to the interface. This includes new environment variables, function names, decorators, etc. - -Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub -once these requirements are fulfilled. Don't worry if you see any test failures in -GitHub at this point! +- [ ] Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub + once these requirements are fulfilled. Don't worry if you see any test failures in + GitHub at this point!
-
PR merge checklist +
Pull Request (PR) merge checklist - click to expand Please do your best to complete these requirements! If you need help with any of these requirements, you can ping the `@nsidc/earthaccess-support` team in a comment and we From f529c35cc18434c2223c12207a45b61e6eb3658b Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 25 Jun 2024 12:49:28 -0600 Subject: [PATCH 05/35] Fix Unreleased section in CHANGELOG to be compatible with our bumpversion config --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c5ee95ac..a0cc813c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## Unreleased ### Changed From 9df202217b6dad88eac8a60630d6f71bef3310cd Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sun, 30 Jun 2024 09:17:59 -0600 Subject: [PATCH 06/35] Enable integration test workflow to communicate failure --- scripts/integration-test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/integration-test.sh b/scripts/integration-test.sh index 2d55064f..1c3d3195 100755 --- a/scripts/integration-test.sh +++ b/scripts/integration-test.sh @@ -3,5 +3,5 @@ set -e set -x -pytest --cov=earthaccess --cov=tests/integration --cov-report=term-missing ${@} -s --tb=native --log-cli-level=INFO +pytest tests/integration --cov=earthaccess --cov=tests/integration --cov-report=term-missing ${@} -s --tb=native --log-cli-level=INFO bash ./scripts/lint.sh From 467b3d7b2c0f7f5b7e4088f067699e28c1953e4e Mon Sep 17 00:00:00 2001 From: Sherwin-14 Date: Wed, 3 Jul 2024 18:49:29 +0530 Subject: [PATCH 07/35] Fleshed out contributing docs to include pre-commit,style,build info --- docs/contributing/index.md | 54 +++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 15 deletions(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 3d149637..72494f8a 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -6,7 +6,7 @@ with the community and maintainers via [a GitHub Discussion](https://github.com/nsidc/earthaccess/discussions), or [any other method](our-meet-ups.md). -Please note that we have a [code of conduct](./code-of-conduct.md). Please follow it in all of your interactions with the project. +Please note that we have a [code of conduct](/CODE_OF_CONDUCT.md). Please follow it in all of your interactions with the project. ## First Steps to contribute @@ -43,6 +43,28 @@ Once you've completed these steps, you are ready to submit your contribution. Once you've completed these steps, you are ready to submit your contribution. + +### ...to contribute to docs + +#### Documentation Style + +To ensure that our code is well-documented and easy to understand, we use [Google-style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) to document all functions, classes, and methods in this library + +#### Getting Help with Setup and Configuration + +To work on docs locally follow the steps below: + +- To setup the documentation, navigate to the scripts directory +- Run the `scripts/build-docs.sh` script to build the documentation +- Serve the documentation locally, using the `scripts/docs-live.sh` script + +If you encounter any issues while setting up the documentation using the provided steps, please refer to our [tutorial]( https://www.youtube.com/watch?v=mNjlMZ4F3So) for additional guidance. + +#### Caveats and Considerations + +Our mkdocs setup has a known limitation: the hot reloader won't auto-reload when changing docstrings. To see updates, manually rebuild and re-serve docs. We're working to improve the dev experience and appreciate your patience. + + ## Submitting your contribution - Run all unit tests successfully in your branch @@ -53,18 +75,20 @@ Once you've completed these steps, you are ready to submit your contribution. ### Please format and lint as you go -```bash -make format lint -``` -We attempt to provide comprehensive type annotations within this repository. If -you do not provide fully annotated functions or methods, the `lint` command will -fail. Over time, we plan to increase type-checking strictness in order to -ensure more precise, beneficial type annotations. +#### Usage of Pre-Commit + +To maintain code quality, we use pre-commit for automated checks before committing changes. Install and configure pre-commit to ensure high-quality contributions + +To set up pre-commit follow these steps: + +- `pip install pre-commit` for Windows and for Linux refer to your distribution's package index +- `pre-commit install` to enable it to run automatically +- `pre-commit run -a` to run it manually + +We attempt to provide comprehensive type annotations within this repository. If you do not provide fully annotated functions or methods, the `lint` command will fail. Over time, we plan to increase type-checking strictness in order to ensure more precise, beneficial type annotations. -We have included type stubs for the untyped `python-cmr` library, which we -intend to eventually upstream. Since `python-cmr` exposes the `cmr` package, -the stubs appear under `stubs/cmr`. +We have included type stubs for the untyped `python-cmr` library, which we intend to eventually upstream. Since `python-cmr` exposes the `cmr` package, the stubs appear under `stubs/cmr`. ### Requirements to merge code (Pull Request Process) @@ -75,12 +99,12 @@ the stubs appear under `stubs/cmr`. ## Pull Request process 1. Ensure you include test coverage for all changes -1. Ensure your code is formatted properly following this document -1. Update the documentation and the `README.md` with details of changes to the +2. Ensure your code is formatted properly following this document +3. Update the documentation and the `README.md` with details of changes to the interface, this includes new environment variables, function names, decorators, etc. -1. Update `CHANGELOG.md` with details about your change in a section titled +4. Update `CHANGELOG.md` with details about your change in a section titled `Unreleased`. If one does not exist, please create one. -1. You may merge the Pull Request once you have the sign-off of another +5. You may merge the Pull Request once you have the sign-off of another developer, or if you do not have permission to do that, you may request the reviewer to merge it for you. From a2d9cc576a42aeb22fcefc226d9d6ca29704e476 Mon Sep 17 00:00:00 2001 From: Sherwin-14 Date: Thu, 4 Jul 2024 13:37:14 +0530 Subject: [PATCH 08/35] Made the necessary changes like adding proper headings, and grammar suggestions --- docs/contributing/index.md | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 72494f8a..c5c40455 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -19,7 +19,7 @@ Please note that we have a [code of conduct](/CODE_OF_CONDUCT.md). Please follow From here, you might want to fix and issue or bug, or add a new feature. Jump to the relevant section to proceed. -### ...to fix an issue or bug +### Fixing an Issue or Bug - Create a GitHub issue with a [minimal reproducible example](https://en.wikipedia.org/wiki/Minimal_reproducible_example), @@ -33,7 +33,7 @@ relevant section to proceed. Once you've completed these steps, you are ready to submit your contribution. -### ...to contribute a new feature +### Contributing a New Feature - Create an issue and discuss the feature's scope and its fit for this package with the team - Create a branch for your new feature in your fork @@ -44,15 +44,15 @@ Once you've completed these steps, you are ready to submit your contribution. Once you've completed these steps, you are ready to submit your contribution. -### ...to contribute to docs +### Contributing to Documentation -#### Documentation Style +#### Documentation Style -To ensure that our code is well-documented and easy to understand, we use [Google-style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) to document all functions, classes, and methods in this library +To ensure that our code is well-documented and easy to understand, we use [Google-style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) to document all functions, classes, and methods in this library. #### Getting Help with Setup and Configuration -To work on docs locally follow the steps below: +To work on documentation locally, follow these steps: - To setup the documentation, navigate to the scripts directory - Run the `scripts/build-docs.sh` script to build the documentation @@ -62,7 +62,7 @@ If you encounter any issues while setting up the documentation using the provide #### Caveats and Considerations -Our mkdocs setup has a known limitation: the hot reloader won't auto-reload when changing docstrings. To see updates, manually rebuild and re-serve docs. We're working to improve the dev experience and appreciate your patience. +Our mkdocs setup has a known limitation: the hot reloader won't auto-reload when changing docstrings. To see updates, manually rebuild and re-serve docs. We're working to improve the developer experience and appreciate your patience. ## Submitting your contribution @@ -78,9 +78,9 @@ Our mkdocs setup has a known limitation: the hot reloader won't auto-reload when #### Usage of Pre-Commit -To maintain code quality, we use pre-commit for automated checks before committing changes. Install and configure pre-commit to ensure high-quality contributions +To maintain code quality, we use pre-commit for automated checks before committing changes. Install and configure pre-commit to ensure high-quality contributions. -To set up pre-commit follow these steps: +To set up pre-commit, follow these steps: - `pip install pre-commit` for Windows and for Linux refer to your distribution's package index - `pre-commit install` to enable it to run automatically @@ -99,12 +99,11 @@ We have included type stubs for the untyped `python-cmr` library, which we inten ## Pull Request process 1. Ensure you include test coverage for all changes -2. Ensure your code is formatted properly following this document -3. Update the documentation and the `README.md` with details of changes to the - interface, this includes new environment variables, function names, - decorators, etc. -4. Update `CHANGELOG.md` with details about your change in a section titled +1. Ensure your code is formatted properly following this document +1. Update the documentation and the `README.md` with details of changes to the + interface, this includes new environment variables, function names,decorators, etc. +1. Update `CHANGELOG.md` with details about your change in a section titled `Unreleased`. If one does not exist, please create one. -5. You may merge the Pull Request once you have the sign-off of another +1. You may merge the Pull Request once you have the sign-off of another developer, or if you do not have permission to do that, you may request the reviewer to merge it for you. From c148a081c58ede5b56a7aab31fa4a19e630d3c04 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Fri, 5 Jul 2024 18:04:32 -0400 Subject: [PATCH 09/35] add links to issues/PRs/commits and authors for changelog entries (in-progress, about half-way through) --- CHANGELOG.md | 90 ++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ed932bc..3564d2ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,7 +61,7 @@ ### Changed - Handle S3 credential expiration more gracefully. -- Use dependabot to update Github Actions. +- Use dependabot to update GitHub Actions. - Consolidate dependabot updates. - Switch to `ruff` for formatting. @@ -133,101 +133,101 @@ ## [0.5.1] - 2023-03-20 ### Changed -- For CI, documentation for readthedocs fixed by including poetry as the default tool -- For CI, injected new secrets to test Auth using the icepyx convention (EARTHDATA_USERNAME) +- For CI, documentation for readthedocs fixed by including poetry as the default tool ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +- For CI, injected new secrets to test Auth using the icepyx convention (EARTHDATA_USERNAME) ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@JessicaS11**](https://github.com/JessicaS11), [**@betolink**](https://github.com/betolink)) ### Added -- Add ability to get the user's profile with auth.user_profile which includes the user email -- Add LAAD as a supported DAAC +- Add ability to get the user's profile with auth.user_profile which includes the user email ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +- Add LAAD as a supported DAAC ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) ### Fixed -- get_s3_credentials() only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 -- including tests for all DAAC S3 endpoints -- Notebooks updated to use the new top level API -- removed magic from dependencies (not available in windows and not used but just in tests) +- get_s3_credentials() only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink), [**@JessicaS11**](https://github.com/JessicaS11), [**@scottyhq**](https://github.com/scottyhq)) +- including tests for all DAAC S3 endpoints ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +- Notebooks updated to use the new top level API ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +- removed magic from dependencies (not available in windows and not used but just in tests) ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) ## [0.5.0] - 2023-02-23 ### Changed -- For CI, documentation is now only built for the main, dev and documentation branches -- For CI, notebooks are executed every time the documentation gets published! +- For CI, documentation is now only built for the main, dev and documentation branches ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) +- For CI, notebooks are executed every time the documentation gets published! ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink), [**@asteiker**](https://github.com/asteiker)) ### Added -- Add ability to use the top level API to get S3 credentials, authenticated fsspec and requests sessions! -- Make available ASF direct access for Sentinel1 products +- Add ability to use the top level API to get S3 credentials, authenticated fsspec and requests sessions! ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) +- Make available ASF direct access for Sentinel1 products ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) ### Fixed -- @JessicaS11 fixed a bug where the Auth class was invoked without the proper parameters -- if a user specifies the netrc strategy and there is no netrc an exception is raised -- S3 URLs broke the Store class when opened outside AWS -- Opening files using URLs was not working properly on AWS, thanks to @amfriesz for reporting it! +- Fix a bug where the Auth class is invoked without the proper parameters ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@JessicaS11**](https://github.com/JessicaS11)) +- if a user specifies the netrc strategy and there is no netrc an exception is raised ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) +- S3 URLs broke the Store class when opened outside AWS ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) +- Opening files using URLs was not working properly on AWS, thanks to @amfriesz for reporting it! ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink), ([**@amfriesz**](https://github.com/amfriesz))) ## [0.4.7] - 2022-12-11 ### Fixed -- Fix open() for direct access -- Move python-magic to the dev section because it is a test dependency -- Make minor edits in the README +- Fix open() for direct access ([#186](https://github.com/nsidc/earthaccess/pull/186))([**@betolink**](https://github.com/betolink)) +- Move python-magic to the dev section because it is a test dependency ([#186](https://github.com/nsidc/earthaccess/pull/186))([**@betolink**](https://github.com/betolink)) +- Make minor edits in the README ([#186](https://github.com/nsidc/earthaccess/pull/186))([**@betolink**](https://github.com/betolink)) ## [0.4.6] - 2022-12-08 ### Changed -- For CI, only run the publish workflow after a release on GitHub +- For CI, only run the publish workflow after a release on GitHub ([#183](https://github.com/nsidc/earthaccess/pull/183))([**@betolink**](https://github.com/betolink)) ### Added -- Add feature to search collections by DOI -- Add new API documentation and simplify notation to access data +- Add feature to search collections by DOI ([#183](https://github.com/nsidc/earthaccess/pull/183))([**@betolink**](https://github.com/betolink)) +- Add new API documentation and simplify notation to access data ([#183](https://github.com/nsidc/earthaccess/pull/183)) ([**@jroebuck932**](https://github.com/jroebuck932)) ## [0.4.1] - 2022-11-02 ### Changed -- For CI, install Poetry using the new script -- For CI, change dependabot alerts to monthly -- Improve documentation by reimplementing python_cmr methods for docstring compatibility -- Use `CMR-Search-After` see #145 +- For CI, install Poetry using the new script ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) +- For CI, change dependabot alerts to monthly ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) +- Improve documentation by reimplementing python_cmr methods for docstring compatibility ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) +- Use `CMR-Search-After` see #145 ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) ### Added -- Add GES_DISC S3 endpoint -- Improve documentation by adding types to method signatures +- Add GES_DISC S3 endpoint ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) +- Improve documentation by adding types to method signatures ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) ## [0.4.0] - 2022-08-17 ### Added -- Add store, auth to docs and update mkdocs config -- For `auth`, add the ability to persist credentials into a `.netrc` file -- For `store`, use fsspec s3fs for in cloud access and https sessions for out of region access -- For `store`, can open files with fsspec in and out of region (stream into xarray) +- Add store, auth to docs and update mkdocs config ([#119](https://github.com/nsidc/earthaccess/pull/119))([**@betolink**](https://github.com/betolink)) +- For `auth`, add the ability to persist credentials into a `.netrc` file ([#119](https://github.com/nsidc/earthaccess/pull/119))([**@betolink**](https://github.com/betolink)) +- For `store`, use fsspec s3fs for in cloud access and https sessions for out of region access ([#43](https://github.com/nsidc/earthaccess/issues/43))([**@betolink**](https://github.com/betolink)) +- For `store`, can open files with fsspec in and out of region (stream into xarray) ([#41](https://github.com/nsidc/earthaccess/issues/41))([**@betolink**](https://github.com/betolink)) ## [0.3.0] - 2022-04-28 ### Changed -- Update python-cmr to NASA fork -- Drop unused `pydantic` dependency -- Auth can refresh CMR tokens -- Verify git tag and poetry version are the same before publish to pypi +- Update python-cmr to NASA fork ([#75](https://github.com/nsidc/earthaccess/pull/75))([**@jhkennedy**](https://github.com/jhkennedy)) +- Drop unused `pydantic` dependency ([`5761548`](https://github.com/nsidc/earthaccess/pull/75/commits/5761548fcd8ba8733ce4f5ff9b8ce7967c3a8398))([**@jhkennedy**](https://github.com/jhkennedy)) +- Auth can refresh CMR tokens ([#82](https://github.com/nsidc/earthaccess/pull/82))([**@betolink**](https://github.com/betolink)) +- Verify git tag and poetry version are the same before publishing to PyPI ### Added -- Add documentation for readthedocs and GitHub +- Add documentation for readthedocs and GitHub ([#82](https://github.com/nsidc/earthaccess/pull/82))([**@betolink**](https://github.com/betolink)) ### Removed -- **Breaking**: Dropped python 3.7 support +- **Breaking**: Drop python 3.7 support ([#82](https://github.com/nsidc/earthaccess/pull/82))([**@betolink**](https://github.com/betolink)) ### Fixed - Fix bug with CMR tokens -- Add missing `python-datutil` dependency +- Add missing `python-datutil` dependency ([`747e992`](https://github.com/nsidc/earthaccess/pull/75/commits/747e9926a5ab83d75bbf7f17d4c52f24b563147b))([**@jhkennedy**](https://github.com/jhkennedy)) ## [0.2.2] - 2022-03-23 ### Fixed -- Fix store to download multi-file granules -- Fix granule formatting +- Fix store to download multi-file granules ([#73](https://github.com/nsidc/earthaccess/pull/73))([**@betolink**](https://github.com/betolink)) +- Fix granule formatting ([#73](https://github.com/nsidc/earthaccess/pull/73))([**@betolink**](https://github.com/betolink)) ## [0.2.1] - 2022-03-19 ### Changed -- Rename Accessor to Store -- Relax dependency requirements -- Store can download plain links if they are on prem +- Rename Accessor to Store ([`4bd618d`](https://github.com/nsidc/earthaccess/pull/66/commits/4bd618d4d48c3cd256a077fb8329f40df2d5b7ff))([**@betolink**](https://github.com/betolink)) +- Relax dependency requirements ([`c9a5ed6`](https://github.com/nsidc/earthaccess/pull/66/commits/c9a5ed6b917435e7c4ece58485939065fa71cc8f))([**@betolink**](https://github.com/betolink)) +- Store can download plain links if they are on prem ([`92d2919`](https://github.com/nsidc/earthaccess/commit/92d291962e5b72b458c2971eae8a6b813d4bae39))([**@betolink**](https://github.com/betolink)) ## [0.1.0-beta.1] - 2021-09-21 From 79080085d49d64f4b36e7e2c5ae929de65063a59 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 09:44:38 -0600 Subject: [PATCH 10/35] Add `flake8-logging-format` ruleset to catch deprecated `logger.warn` calls Allow f-strings in log calls. --- pyproject.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 0e15f45d..316a4ac7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -144,8 +144,8 @@ line-length = 88 src = ["earthaccess", "stubs", "tests"] [tool.ruff.lint] -extend-select = ["I", "T20", "D"] -ignore = ["D1", "D205", "D401", "D417"] +extend-select = ["I", "T20", "D", "G"] +ignore = ["D1", "D205", "D401", "D417", "G004"] [tool.ruff.lint.pydocstyle] convention = "google" From 5edff3a412792a0e397cc817a033ae62fc1f73e8 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 09:45:15 -0600 Subject: [PATCH 11/35] Replace deprecated calls to `logger.warn()` --- earthaccess/api.py | 4 +++- earthaccess/store.py | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/earthaccess/api.py b/earthaccess/api.py index 8522dbbb..0418d69f 100644 --- a/earthaccess/api.py +++ b/earthaccess/api.py @@ -68,7 +68,9 @@ def search_datasets(count: int = -1, **kwargs: Any) -> List[DataCollection]: ``` """ if not validate.valid_dataset_parameters(**kwargs): - logger.warn("A valid set of parameters is needed to search for datasets on CMR") + logger.warning( + "A valid set of parameters is needed to search for datasets on CMR" + ) return [] if earthaccess.__auth__.authenticated: query = DataCollections(auth=earthaccess.__auth__).parameters(**kwargs) diff --git a/earthaccess/store.py b/earthaccess/store.py index 147e6ce8..f3b1054c 100644 --- a/earthaccess/store.py +++ b/earthaccess/store.py @@ -113,7 +113,7 @@ def __init__(self, auth: Any, pre_authorize: bool = False) -> None: self.set_requests_session(url) else: - logger.warn("The current session is not authenticated with NASA") + logger.warning("The current session is not authenticated with NASA") self.auth = None self.in_region = self._running_in_us_west_2() From be8b40bf90bd0ba9a445021b8e98b85ba61fa1b9 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 10:38:38 -0600 Subject: [PATCH 12/35] Notify the user when integration tests are artificially marked success --- scripts/integration-test.sh | 15 +++++++++++++-- tests/integration/conftest.py | 23 +++++++++++++++++++++-- 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/scripts/integration-test.sh b/scripts/integration-test.sh index 1c3d3195..7a7b6551 100755 --- a/scripts/integration-test.sh +++ b/scripts/integration-test.sh @@ -1,7 +1,18 @@ #!/usr/bin/env bash -set -e set -x +pytest tests/integration --cov=earthaccess --cov=tests/integration --cov-report=term-missing ${@} --capture=no --tb=native --log-cli-level=INFO +RET=$? +set +x + +set -e +# NOTE: 99 is a special return code we selected for this case, it has no other meaning. +if [[ $RET == 99 ]]; then + echo -e "\e[0;31mWARNING: The integration test suite has been permitted to return 0 because the failure rate was less than a hardcoded threshold.\e[0m" + echo "For more details, see conftest.py." + exit 0 +elif [[ $RET != 0 ]]; then + exit $RET +fi -pytest tests/integration --cov=earthaccess --cov=tests/integration --cov-report=term-missing ${@} -s --tb=native --log-cli-level=INFO bash ./scripts/lint.sh diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 8be3de59..3ea79824 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -5,8 +5,27 @@ @pytest.hookimpl() def pytest_sessionfinish(session, exitstatus): - if exitstatus == 0: + """Return exit code 99 if up to N% of tests have failed. + + N = ACCEPTABLE_FAILURE_RATE + + 99 was chosen arbitrarily to avoid conflict with current and future pytest error + codes (https://docs.pytest.org/en/stable/reference/exit-codes.html), and avoid + other exit codes with special meanings + (https://tldp.org/LDP/abs/html/exitcodes.html). + + IMPORTANT: This is calculated against every test collected in the session, so the + ratio will change depending on which tests are executed! E.g. executing integration + tests and unit tests at the same time allows more tests to fail than executing + integration tests alone. + """ + if exitstatus != 1: + # Exit status 1 in PyTest indicates "Tests were collected and run but some of + # the tests failed". In all other cases, for example "an internal error happened + # while executing the tests", or "test execution interrupted by the user", we + # want to defer to original pytest behavior. return + failure_rate = (100.0 * session.testsfailed) / session.testscollected if failure_rate <= ACCEPTABLE_FAILURE_RATE: - session.exitstatus = 0 + session.exitstatus = 99 From 91e559990659fa77e1bc68452f8bbde220e292ee Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 11:19:06 -0600 Subject: [PATCH 13/35] Clarify "descriptive title" task Co-Authored-By: danielfromearth --- .github/pull_request_template.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 5b2df6f3..8971a273 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -18,8 +18,8 @@ the PR as a non-draft, don't worry, you can still change it to a draft using the
Pull Request (PR) draft checklist - click to expand -- [ ] Populate a descriptive title. Ensure the title does not look like "Updated - README.md". +- [ ] Populate a descriptive title. For example, instead of "Updated README.md", use a + title such as "Add testing details to the contributor section of the README". - [ ] Populate the body of the pull request with : - A clear description of the change you are proposing. - Links to any issues resolved by this PR with text in the PR description, for From f058822173cce6cbfa03edb5def895352b6fd1f8 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 11:25:58 -0600 Subject: [PATCH 14/35] Move items that belong at the top of the checklist to the top Co-Authored-By: danielfromearth --- .github/pull_request_template.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 8971a273..40f3938e 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -18,16 +18,16 @@ the PR as a non-draft, don't worry, you can still change it to a draft using the
Pull Request (PR) draft checklist - click to expand +- [ ] Please review our + [contributing documentation](https://earthaccess.readthedocs.io/en/latest/contributing/) + before getting started. +- [ ] Ensure an issue exists representing the problem being solved in this PR. - [ ] Populate a descriptive title. For example, instead of "Updated README.md", use a title such as "Add testing details to the contributor section of the README". - [ ] Populate the body of the pull request with : - A clear description of the change you are proposing. - Links to any issues resolved by this PR with text in the PR description, for example "closes #1". -- [ ] Please review our - [contributing documentation](https://earthaccess.readthedocs.io/en/latest/contributing/) - before getting started. -- [ ] Ensure an issue exists representing the problem being solved in this PR. - [ ] Update `CHANGELOG.md` with details about your change in a section titled `## Unreleased`. If such a section does not exist, please create one. - [ ] Update the documentation and/or the `README.md` with details of changes to the From d5c5aaa8f964df8e129af71f30f3e6390197668b Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 11:32:29 -0600 Subject: [PATCH 15/35] Clarify expectations for maintainer attention Co-Authored-By: Ian Carroll --- .github/pull_request_template.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 40f3938e..2c8ec43c 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -10,9 +10,12 @@ expect you to complete the steps in the "PR draft checklist" below. Maintainers willing and ready to help pick it up from there! Please start by opening this Pull Request as a "draft". You can do this by -clicking the arrow on the right side of the green "Create pull request" button. Follow -the draft checklist below to move the PR out of draft state. If you accidentally created -the PR as a non-draft, don't worry, you can still change it to a draft using the +clicking the arrow on the right side of the green "Create pull request" button. While +your pull request is in "draft" state, maintainers will assume the PR isn't ready for +their attention unless they are specifically summoned using GitHub's @ system. + +Follow the draft checklist below to move the PR out of draft state. If you accidentally +created the PR as a non-draft, don't worry, you can still change it to a draft using the "Convert to draft" button on the right side panel under the "Reviewers" section. --> From 7e34898e2dded37f526532730e320b86cceb59c1 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 11:33:55 -0600 Subject: [PATCH 16/35] Un-checkbox "ready for review" instructions Co-Authored-By: Ian Carroll --- .github/pull_request_template.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 2c8ec43c..71a136f6 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -36,9 +36,10 @@ created the PR as a non-draft, don't worry, you can still change it to a draft u - [ ] Update the documentation and/or the `README.md` with details of changes to the interface. This includes new environment variables, function names, decorators, etc. -- [ ] Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub - once these requirements are fulfilled. Don't worry if you see any test failures in - GitHub at this point! + +Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub +once these requirements are fulfilled. Don't worry if you see any test failures in +GitHub at this point!
From 3cb282b877cdc603872c1b502224a0e6b4bcb47b Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 11:34:13 -0600 Subject: [PATCH 17/35] Clarify draft PR instructions --- .github/pull_request_template.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 71a136f6..60c23104 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -27,15 +27,16 @@ created the PR as a non-draft, don't worry, you can still change it to a draft u - [ ] Ensure an issue exists representing the problem being solved in this PR. - [ ] Populate a descriptive title. For example, instead of "Updated README.md", use a title such as "Add testing details to the contributor section of the README". -- [ ] Populate the body of the pull request with : +- [ ] Populate the body of the pull request with: - A clear description of the change you are proposing. - Links to any issues resolved by this PR with text in the PR description, for - example "closes #1". + example `closes #1`. See + [GitHub docs - Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue). - [ ] Update `CHANGELOG.md` with details about your change in a section titled `## Unreleased`. If such a section does not exist, please create one. - [ ] Update the documentation and/or the `README.md` with details of changes to the - interface. This includes new environment variables, function names, decorators, - etc. + earthaccess interface, if any. Consider new environment variables, function names, + decorators, etc. Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub once these requirements are fulfilled. Don't worry if you see any test failures in From 805c65598437ba81344a733ed4c449d8b760e40c Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 11:43:19 -0600 Subject: [PATCH 18/35] Make PR instructions clearer and more welcoming Co-authored-by: Ian Carroll --- docs/contributing/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index b0f24a1b..fc39f629 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -69,5 +69,6 @@ the stubs appear under `stubs/cmr`. ## Pull Request process Fork the repository using the "Fork" button on the [repository -homepage](https://github.com/nsidc/earthaccess), make your change on the fork, then open -a pull request from your fork and follow the instructions populated in the text box. +homepage](https://github.com/nsidc/earthaccess), create a branch with your changes in the fork, then open +a draft pull request from your fork. Starting a pull request provides additional instructions and requirements, and +there is no harm in starting a draft pull request while still developing. From 77a7e483283c1255b361fdc83363889dc25a1294 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Sat, 6 Jul 2024 12:55:40 -0600 Subject: [PATCH 19/35] Use a constant for pytest's exit status --- tests/integration/conftest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 3ea79824..c2d4a3c2 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -19,7 +19,7 @@ def pytest_sessionfinish(session, exitstatus): tests and unit tests at the same time allows more tests to fail than executing integration tests alone. """ - if exitstatus != 1: + if exitstatus != pytest.ExitCode.TESTS_FAILED: # Exit status 1 in PyTest indicates "Tests were collected and run but some of # the tests failed". In all other cases, for example "an internal error happened # while executing the tests", or "test execution interrupted by the user", we From 728eb17e52738f62dcd08d86093da7fa7bde1a3d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sat, 6 Jul 2024 18:57:36 +0000 Subject: [PATCH 20/35] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/contributing/maintainers-guide.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/contributing/maintainers-guide.md b/docs/contributing/maintainers-guide.md index 1ac2fc79..2d54a354 100644 --- a/docs/contributing/maintainers-guide.md +++ b/docs/contributing/maintainers-guide.md @@ -53,6 +53,3 @@ The GitHub Actions CI services handle the project's building, testing, and manag ## Continuous Documentation [ReadTheDocs](https://readthedocs.org/projects/earthaccess/) is used to generate and host [our documentation website](https://earthaccess.readthedocs.io/) as well as the preview for documentation changes made in pull requests. This service uses a configuration file in the root of the project, `.readthedocs.yml`. - - - From b4e4beb4206b6f8dd5982040ad3fa2f4df77aaba Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Mon, 8 Jul 2024 12:25:05 -0400 Subject: [PATCH 21/35] consolidate redundant entry --- CHANGELOG.md | 70 +++++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3564d2ba..675db999 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,93 +42,95 @@ ## [0.8.2] - 2023-12-06 ### Changed -- Update poetry lockfile -- Use YAML formatting (prettier) +- Update poetry lockfile ([#403](https://github.com/nsidc/earthaccess/pull/403))([**@jrbourbeau**](https://github.com/jrbourbeau)) +- Use YAML formatting (prettier) ([#398](https://github.com/nsidc/earthaccess/pull/398))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Added -- Add CI tests with minimum supported versions -- Add `python-dateutil` as a direct dependency +- Add CI tests with minimum supported versions ([#402](https://github.com/nsidc/earthaccess/pull/402))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@jhkennedy**](https://github.com/jhkennedy)) +- Add `python-dateutil` as a direct dependency ([#397](https://github.com/nsidc/earthaccess/pull/397))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Removed -- Remove binder PR comments +- Remove binder PR comments ([#400](https://github.com/nsidc/earthaccess/pull/400))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Fixed -- Enable AWS check with IMDSv2 -- Add region to running in AWS check -- Handle opening multi-file granules +- Enable AWS check with IMDSv2 ([#391](https://github.com/nsidc/earthaccess/pull/391))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@itcarroll**](https://github.com/itcarroll)) +- Add region to running in AWS check ([#395](https://github.com/nsidc/earthaccess/pull/395))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) +- Handle opening multi-file granules ([#394](https://github.com/nsidc/earthaccess/pull/394))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) ## [0.8.1] - 2023-12-01 ### Changed -- Handle S3 credential expiration more gracefully. -- Use dependabot to update GitHub Actions. -- Consolidate dependabot updates. -- Switch to `ruff` for formatting. +- Handle S3 credential expiration more gracefully ([#354](https://github.com/nsidc/earthaccess/pull/354))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) +- Use dependabot to update GitHub Actions ([#373](https://github.com/nsidc/earthaccess/pull/373))([**@jhkennedy**](https://github.com/jhkennedy)) +- Consolidate dependabot updates ([#380](https://github.com/nsidc/earthaccess/pull/380))([**@mfisher87**](https://github.com/mfisher87)) +- Switch to `ruff` for formatting ([#372](https://github.com/nsidc/earthaccess/pull/372))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Added -- Add `kerchunk` metadata consolidation utility. +- Add `kerchunk` metadata consolidation utility ([#278](https://github.com/nsidc/earthaccess/pull/278)) + ([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), + [**@betolink**](https://github.com/betolink), [**@martindurant**](https://github.com/martindurant), + [**lsterzinger**](https://github.com/lsterzinger), [**mrocklin**](https://github.com/mrocklin)) ## [0.8.0] - 2023-11-29 ### Changed -- earthaccess will `raise` errors instead of `print`ing them in more cases. +- earthaccess will `raise` errors instead of `print`ing them in more cases ([#351](https://github.com/nsidc/earthaccess/pull/351))([**@jrbourbeau**](https://github.com/jrbourbeau)) - `daac` and `provider` parameters are now normalized to uppercase, since lowercase - characters are never valid. + characters are never valid ([#355](https://github.com/nsidc/earthaccess/pull/355))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Fixed -- Fix zero granules being reported for restricted datasets. +- Fix zero granules being reported for restricted datasets ([#358](https://github.com/nsidc/earthaccess/pull/358))([**@danielfromearth**](https://github.com/danielfromearth)) ## [0.7.1] - 2023-11-08 ### Fixed -- Treat granules without `RelatedUrls` as not cloud-hosted. +- Treat granules without `RelatedUrls` as not cloud-hosted ([#339](https://github.com/nsidc/earthaccess/pull/339))([**@mfisher87**](https://github.com/mfisher87)) ## [0.7.0] - 2023-10-31 ### Changed -- `earthaccess.download` now accepts a single granule as input in addition to a list of granules. -- `earthaccess.download` now returns fully qualified local file paths. +- `earthaccess.download` now accepts a single granule as input in addition to a list of granules ([#317](https://github.com/nsidc/earthaccess/pull/317))([**@jrbourbeau**](https://github.com/jrbourbeau)) +- `earthaccess.download` now returns fully qualified local file paths ([#317](https://github.com/nsidc/earthaccess/pull/317))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Added - Earthaccess will now automatically search for Earthdata authentication. ``earthaccess.login()`` still works as before, but is no longer required if you have a ``~/.netrc`` file for have set - ``EARTHDATA_USERNAME`` and ``EARTHDATA_PASSWORD`` environment variables. -- Add `earthaccess.auth_environ()` utility for getting Earthdata authentication environment variables. + ``EARTHDATA_USERNAME`` and ``EARTHDATA_PASSWORD`` environment variables ([#300](https://github.com/nsidc/earthaccess/pull/300))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@mfisher87**](https://github.com/mfisher87)) +- Add `earthaccess.auth_environ()` utility for getting Earthdata authentication environment variables ([#316](https://github.com/nsidc/earthaccess/pull/316))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Fixed - Fix spelling mistake in `access` variable assignment (`direc` -> `direct`) - in `earthaccess.store._get_granules`. + in `earthaccess.store._get_granules` ([#308](https://github.com/nsidc/earthaccess/pull/308))([**@trey-stafford**](https://github.com/trey-stafford)) - Pass `threads` arg to `_open_urls_https` in - `earthaccess.store._open_urls`, replacing the hard-coded value of 8. -- Return S3 data links by default when in region. + `earthaccess.store._open_urls`, replacing the hard-coded value of 8 ([#308](https://github.com/nsidc/earthaccess/pull/308))([**@trey-stafford**](https://github.com/trey-stafford)) +- Return S3 data links by default when in region ([#318](https://github.com/nsidc/earthaccess/pull/318))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@jhkennedy**](https://github.com/jhkennedy)) ## [0.6.0] - 2023-09-20 ### Added -- earthaccess.get_s3fs_session() can use the results to find the right set of S3 credentials +- earthaccess.get_s3fs_session() can use the results to find the right set of S3 credentials ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) ### Fixed -- earthaccess.search_datasets() and earthaccess.search_data() can find restricted datasets #296 -- distributed serialization fixed for EarthAccessFile #301 and #276 +- earthaccess.search_datasets() and earthaccess.search_data() can find restricted datasets ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) +- Fix distributed serialization for EarthAccessFile ([#301](https://github.com/nsidc/earthaccess/pull/301))([**@jrbourbeau**](https://github.com/jrbourbeau)) and ([#276](https://github.com/nsidc/earthaccess/pull/276))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) ## [0.5.3] - 2023-08-01 ### Changed -- For CI, integration tests are now only run when we push to main (after a merge) -- For CI, unit tests are run for any branch and opened PR +- For CI, integration tests are now only run when we push to main (after a merge) ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) +- For CI, unit tests are run for any branch and opened PR ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) ### Fixed -- granule's size() returned zero -- Add exception handling for fsspec sessions, thanks to @jrbourbeau +- Granule's size() returned zero ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) +- Add exception handling for fsspec sessions, thanks to @jrbourbeau ([#249](https://github.com/nsidc/earthaccess/pull/249))([**@jrbourbeau**](https://github.com/jrbourbeau)) ## [0.5.2] - 2023-04-21 ### Removed -- Remove Benedict (core dependency) as the default dict for JSON coming from CMR. +- Remove Benedict (core dependency) as the default dict for JSON coming from CMR ([#229](https://github.com/nsidc/earthaccess/pull/229), [#230](https://github.com/nsidc/earthaccess/issues/230))([**@psarka**](https://github.com/psarka)) ### Fixed -- Fix #230 by removing Benedict as the dict handler, thanks to @psarka! -- S3 credential endpoints are tried with tokens and basic auth until all the DAACs accept the same auth +- S3 credential endpoints are tried with tokens and basic auth until all the DAACs accept the same auth ([#234](https://github.com/nsidc/earthaccess/pull/234))([**@betolink**](https://github.com/betolink)) ## [0.5.1] - 2023-03-20 From b4613a1aac75b71334847bb7f35d1be5aee81847 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Mon, 8 Jul 2024 13:00:13 -0400 Subject: [PATCH 22/35] Add and update remaining links to issues/PRs/commits in the CHANGELOG.md --- CHANGELOG.md | 77 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 675db999..ececce3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,41 +3,58 @@ ## [Unreleased] ### Changed -- [#555](https://github.com/nsidc/earthaccess/issues/555): YAML formatting is - now performed with `yamlfmt` instead of `prettier`. -- [#511](https://github.com/nsidc/earthaccess/issues/511): Replace `print` - calls with `logging` calls where appropriate and add T20 Ruff rule. -- [#508](https://github.com/nsidc/earthaccess/issues/508): Correct and - enhance static type hints for functions and methods that make CMR queries - or handle CMR query results. -- [#562](https://github.com/nsidc/earthaccess/issues/562): The destination - path is now created prior to direct S3 downloads, if it doesn't already - exist. +- Perform YAML formatting with `yamlfmt` instead of + `prettier` ([#555](https://github.com/nsidc/earthaccess/issues/555)) + ([**@chuckwondo**](https://github.com/chuckwondo),[**@mfisher87**](https://github.com/mfisher87)) +- Replace `print` calls with `logging` calls where appropriate and add T20 Ruff rule + ([#511](https://github.com/nsidc/earthaccess/issues/511)) + ([**@botanical**](https://github.com/botanical),[**@chuckwondo**](https://github.com/chuckwondo), + [**@mfisher87**](https://github.com/mfisher87)) +- Correct and enhance static type hints for functions and methods that make CMR queries + or handle CMR query results ([#508](https://github.com/nsidc/earthaccess/issues/508)) + ([**@mfisher87**](https://github.com/mfisher87),[**@jhkennedy**](https://github.com/jhkennedy), + [**@chuckwondo**](https://github.com/chuckwondo),[**@betolink**](https://github.com/betolink)) +- Create destination path prior to direct S3 downloads, if it doesn't already + exist ([#562](https://github.com/nsidc/earthaccess/issues/562)) + ([**@itcarroll**](https://github.com/itcarroll),[**@mfisher87**](https://github.com/mfisher87), + [**@chuckwondo**](https://github.com/chuckwondo)) ### Added -- [#483](https://github.com/nsidc/earthaccess/issues/483): Now using - [Search After](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#search-after) - for collection and granule searches to support deep-paging through large - result sets. -- [#421](https://github.com/nsidc/earthaccess/issues/421): Enable queries to - Earthdata User Acceptance Testing (UAT) system for authenticated accounts. +- Use [Search After](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#search-after) + for collection and granule searches to support deep-paging through large result sets + ([#483](https://github.com/nsidc/earthaccess/issues/483)) + ([**@doug-newman-nasa**](https://github.com/doug-newman-nasa),[**@chuckwondo**](https://github.com/chuckwondo), + [**@mfisher87**](https://github.com/mfisher87),[**@betolink**](https://github.com/betolink)) +- Enable queries to Earthdata User Acceptance Testing (UAT) system for authenticated accounts + ([#421](https://github.com/nsidc/earthaccess/issues/421)) + ([**@danielfromearth**](https://github.com/danielfromearth),[**@mfisher87**](https://github.com/mfisher87), + [**@jhkennedy**](https://github.com/jhkennedy),[**@chuckwondo**](https://github.com/chuckwondo), + [**@betolink**](https://github.com/betolink)) ### Removed -- **Breaking**: [#421](https://github.com/nsidc/earthaccess/issues/421): Remove the +- **Breaking:** Remove the `get_user_profile` method and the `email_address` and `profile` attributes from the `Auth` class. Calling the EDL API to get user profile information is not intended for library access and is not necessary for this library's - intended use cases. + intended use cases. ([#421](https://github.com/nsidc/earthaccess/issues/421)) + ([**@danielfromearth**](https://github.com/danielfromearth),[**@mfisher87**](https://github.com/mfisher87), + [**@jhkennedy**](https://github.com/jhkennedy),[**@chuckwondo**](https://github.com/chuckwondo), + [**@betolink**](https://github.com/betolink)) ## [0.9.0] - 2024-02-28 ### Added -- Address #427 by adding parameters to collection query -- Add user-agent in the request to track usage, closes #436 +- Improve search by adding instrument and project to collection queries + ([#427](https://github.com/nsidc/earthaccess/issues/427)) + ([**@betolink**](https://github.com/betolink),[**@mfisher87**](https://github.com/mfisher87), + [**@jhkennedy**](https://github.com/jhkennedy)) +- Add user-agent in the request to track usage ([#436](https://github.com/nsidc/earthaccess/issues/436)) + ([**@asteiker**](https://github.com/asteiker),[**@@mikedorfman**](https://github.com/@mikedorfman), + [**@betolink**](https://github.com/betolink)) ### Fixed -- Fix #439 by implementing more trusted domains in the SessionWithRedirection -- Fix #438 by using an authenticated session for hits() +- Implement more trusted domains in the SessionWithRedirection ([#439](https://github.com/nsidc/earthaccess/issues/439))([**@cmspeed**](https://github.com/cmspeed), [**@mfisher87**](https://github.com/mfisher87), [**@betolink**](https://github.com/betolink), [**@jhkennedy**](https://github.com/jhkennedy)) +- Use an authenticated session for hits() instead of calling parent's class super() ([#438](https://github.com/nsidc/earthaccess/issues/438))([**@betolink**](https://github.com/betolink), [**@mfisher87**](https://github.com/mfisher87), [**@jhkennedy**](https://github.com/jhkennedy)) ## [0.8.2] - 2023-12-06 @@ -108,10 +125,10 @@ ## [0.6.0] - 2023-09-20 ### Added -- earthaccess.get_s3fs_session() can use the results to find the right set of S3 credentials ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) +- `earthaccess.get_s3fs_session()` can use the results to find the right set of S3 credentials ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) ### Fixed -- earthaccess.search_datasets() and earthaccess.search_data() can find restricted datasets ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) +- `earthaccess.search_datasets()` and `earthaccess.search_data()` can find restricted datasets ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) - Fix distributed serialization for EarthAccessFile ([#301](https://github.com/nsidc/earthaccess/pull/301))([**@jrbourbeau**](https://github.com/jrbourbeau)) and ([#276](https://github.com/nsidc/earthaccess/pull/276))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) ## [0.5.3] - 2023-08-01 @@ -142,11 +159,13 @@ - Add ability to get the user's profile with auth.user_profile which includes the user email ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) - Add LAAD as a supported DAAC ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +### Removed +- Remove magic from dependencies (not available in windows and not used but just in tests) ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) + ### Fixed -- get_s3_credentials() only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink), [**@JessicaS11**](https://github.com/JessicaS11), [**@scottyhq**](https://github.com/scottyhq)) -- including tests for all DAAC S3 endpoints ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) -- Notebooks updated to use the new top level API ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) -- removed magic from dependencies (not available in windows and not used but just in tests) ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +- `get_s3_credentials()` only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink), [**@JessicaS11**](https://github.com/JessicaS11), [**@scottyhq**](https://github.com/scottyhq)) +- Include tests for all DAAC S3 endpoints ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) +- Update notebooks to use the new top level API ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) ## [0.5.0] - 2023-02-23 @@ -160,7 +179,7 @@ ### Fixed - Fix a bug where the Auth class is invoked without the proper parameters ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@JessicaS11**](https://github.com/JessicaS11)) -- if a user specifies the netrc strategy and there is no netrc an exception is raised ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) +- Raise and exception if a user specifies the netrc strategy and there is no netrc ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) - S3 URLs broke the Store class when opened outside AWS ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) - Opening files using URLs was not working properly on AWS, thanks to @amfriesz for reporting it! ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink), ([**@amfriesz**](https://github.com/amfriesz))) From 8bf7d5eabfba3309cadfc19729afee05efd571d3 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Mon, 8 Jul 2024 13:01:48 -0400 Subject: [PATCH 23/35] add single blank line after each header --- CHANGELOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ececce3a..be67504e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] ### Changed + - Perform YAML formatting with `yamlfmt` instead of `prettier` ([#555](https://github.com/nsidc/earthaccess/issues/555)) ([**@chuckwondo**](https://github.com/chuckwondo),[**@mfisher87**](https://github.com/mfisher87)) @@ -20,6 +21,7 @@ [**@chuckwondo**](https://github.com/chuckwondo)) ### Added + - Use [Search After](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#search-after) for collection and granule searches to support deep-paging through large result sets ([#483](https://github.com/nsidc/earthaccess/issues/483)) @@ -32,6 +34,7 @@ [**@betolink**](https://github.com/betolink)) ### Removed + - **Breaking:** Remove the `get_user_profile` method and the `email_address` and `profile` attributes from the `Auth` class. Calling the EDL API to get user profile information @@ -44,6 +47,7 @@ ## [0.9.0] - 2024-02-28 ### Added + - Improve search by adding instrument and project to collection queries ([#427](https://github.com/nsidc/earthaccess/issues/427)) ([**@betolink**](https://github.com/betolink),[**@mfisher87**](https://github.com/mfisher87), @@ -53,23 +57,28 @@ [**@betolink**](https://github.com/betolink)) ### Fixed + - Implement more trusted domains in the SessionWithRedirection ([#439](https://github.com/nsidc/earthaccess/issues/439))([**@cmspeed**](https://github.com/cmspeed), [**@mfisher87**](https://github.com/mfisher87), [**@betolink**](https://github.com/betolink), [**@jhkennedy**](https://github.com/jhkennedy)) - Use an authenticated session for hits() instead of calling parent's class super() ([#438](https://github.com/nsidc/earthaccess/issues/438))([**@betolink**](https://github.com/betolink), [**@mfisher87**](https://github.com/mfisher87), [**@jhkennedy**](https://github.com/jhkennedy)) ## [0.8.2] - 2023-12-06 ### Changed + - Update poetry lockfile ([#403](https://github.com/nsidc/earthaccess/pull/403))([**@jrbourbeau**](https://github.com/jrbourbeau)) - Use YAML formatting (prettier) ([#398](https://github.com/nsidc/earthaccess/pull/398))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Added + - Add CI tests with minimum supported versions ([#402](https://github.com/nsidc/earthaccess/pull/402))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@jhkennedy**](https://github.com/jhkennedy)) - Add `python-dateutil` as a direct dependency ([#397](https://github.com/nsidc/earthaccess/pull/397))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Removed + - Remove binder PR comments ([#400](https://github.com/nsidc/earthaccess/pull/400))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Fixed + - Enable AWS check with IMDSv2 ([#391](https://github.com/nsidc/earthaccess/pull/391))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@itcarroll**](https://github.com/itcarroll)) - Add region to running in AWS check ([#395](https://github.com/nsidc/earthaccess/pull/395))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) - Handle opening multi-file granules ([#394](https://github.com/nsidc/earthaccess/pull/394))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) @@ -77,12 +86,14 @@ ## [0.8.1] - 2023-12-01 ### Changed + - Handle S3 credential expiration more gracefully ([#354](https://github.com/nsidc/earthaccess/pull/354))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) - Use dependabot to update GitHub Actions ([#373](https://github.com/nsidc/earthaccess/pull/373))([**@jhkennedy**](https://github.com/jhkennedy)) - Consolidate dependabot updates ([#380](https://github.com/nsidc/earthaccess/pull/380))([**@mfisher87**](https://github.com/mfisher87)) - Switch to `ruff` for formatting ([#372](https://github.com/nsidc/earthaccess/pull/372))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Added + - Add `kerchunk` metadata consolidation utility ([#278](https://github.com/nsidc/earthaccess/pull/278)) ([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@betolink**](https://github.com/betolink), [**@martindurant**](https://github.com/martindurant), @@ -91,31 +102,37 @@ ## [0.8.0] - 2023-11-29 ### Changed + - earthaccess will `raise` errors instead of `print`ing them in more cases ([#351](https://github.com/nsidc/earthaccess/pull/351))([**@jrbourbeau**](https://github.com/jrbourbeau)) - `daac` and `provider` parameters are now normalized to uppercase, since lowercase characters are never valid ([#355](https://github.com/nsidc/earthaccess/pull/355))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Fixed + - Fix zero granules being reported for restricted datasets ([#358](https://github.com/nsidc/earthaccess/pull/358))([**@danielfromearth**](https://github.com/danielfromearth)) ## [0.7.1] - 2023-11-08 ### Fixed + - Treat granules without `RelatedUrls` as not cloud-hosted ([#339](https://github.com/nsidc/earthaccess/pull/339))([**@mfisher87**](https://github.com/mfisher87)) ## [0.7.0] - 2023-10-31 ### Changed + - `earthaccess.download` now accepts a single granule as input in addition to a list of granules ([#317](https://github.com/nsidc/earthaccess/pull/317))([**@jrbourbeau**](https://github.com/jrbourbeau)) - `earthaccess.download` now returns fully qualified local file paths ([#317](https://github.com/nsidc/earthaccess/pull/317))([**@jrbourbeau**](https://github.com/jrbourbeau)) ### Added + - Earthaccess will now automatically search for Earthdata authentication. ``earthaccess.login()`` still works as before, but is no longer required if you have a ``~/.netrc`` file for have set ``EARTHDATA_USERNAME`` and ``EARTHDATA_PASSWORD`` environment variables ([#300](https://github.com/nsidc/earthaccess/pull/300))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87), [**@mfisher87**](https://github.com/mfisher87)) - Add `earthaccess.auth_environ()` utility for getting Earthdata authentication environment variables ([#316](https://github.com/nsidc/earthaccess/pull/316))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Fixed + - Fix spelling mistake in `access` variable assignment (`direc` -> `direct`) in `earthaccess.store._get_granules` ([#308](https://github.com/nsidc/earthaccess/pull/308))([**@trey-stafford**](https://github.com/trey-stafford)) - Pass `threads` arg to `_open_urls_https` in @@ -125,44 +142,54 @@ ## [0.6.0] - 2023-09-20 ### Added + - `earthaccess.get_s3fs_session()` can use the results to find the right set of S3 credentials ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) ### Fixed + - `earthaccess.search_datasets()` and `earthaccess.search_data()` can find restricted datasets ([#296](https://github.com/nsidc/earthaccess/pull/296))([**@betolink**](https://github.com/betolink)) - Fix distributed serialization for EarthAccessFile ([#301](https://github.com/nsidc/earthaccess/pull/301))([**@jrbourbeau**](https://github.com/jrbourbeau)) and ([#276](https://github.com/nsidc/earthaccess/pull/276))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@betolink**](https://github.com/betolink)) ## [0.5.3] - 2023-08-01 ### Changed + - For CI, integration tests are now only run when we push to main (after a merge) ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) - For CI, unit tests are run for any branch and opened PR ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) ### Fixed + - Granule's size() returned zero ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) - Add exception handling for fsspec sessions, thanks to @jrbourbeau ([#249](https://github.com/nsidc/earthaccess/pull/249))([**@jrbourbeau**](https://github.com/jrbourbeau)) ## [0.5.2] - 2023-04-21 ### Removed + - Remove Benedict (core dependency) as the default dict for JSON coming from CMR ([#229](https://github.com/nsidc/earthaccess/pull/229), [#230](https://github.com/nsidc/earthaccess/issues/230))([**@psarka**](https://github.com/psarka)) ### Fixed + - S3 credential endpoints are tried with tokens and basic auth until all the DAACs accept the same auth ([#234](https://github.com/nsidc/earthaccess/pull/234))([**@betolink**](https://github.com/betolink)) ## [0.5.1] - 2023-03-20 ### Changed + - For CI, documentation for readthedocs fixed by including poetry as the default tool ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) - For CI, injected new secrets to test Auth using the icepyx convention (EARTHDATA_USERNAME) ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@JessicaS11**](https://github.com/JessicaS11), [**@betolink**](https://github.com/betolink)) ### Added + - Add ability to get the user's profile with auth.user_profile which includes the user email ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) - Add LAAD as a supported DAAC ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) ### Removed + - Remove magic from dependencies (not available in windows and not used but just in tests) ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) ### Fixed + - `get_s3_credentials()` only worked when a netrc file was present, bug reported by @scottyhq and @JessicaS11 ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink), [**@JessicaS11**](https://github.com/JessicaS11), [**@scottyhq**](https://github.com/scottyhq)) - Include tests for all DAAC S3 endpoints ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) - Update notebooks to use the new top level API ([#214](https://github.com/nsidc/earthaccess/pull/214))([**@betolink**](https://github.com/betolink)) @@ -170,14 +197,17 @@ ## [0.5.0] - 2023-02-23 ### Changed + - For CI, documentation is now only built for the main, dev and documentation branches ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) - For CI, notebooks are executed every time the documentation gets published! ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink), [**@asteiker**](https://github.com/asteiker)) ### Added + - Add ability to use the top level API to get S3 credentials, authenticated fsspec and requests sessions! ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) - Make available ASF direct access for Sentinel1 products ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) ### Fixed + - Fix a bug where the Auth class is invoked without the proper parameters ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@JessicaS11**](https://github.com/JessicaS11)) - Raise and exception if a user specifies the netrc strategy and there is no netrc ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) - S3 URLs broke the Store class when opened outside AWS ([#202](https://github.com/nsidc/earthaccess/pull/202))([**@betolink**](https://github.com/betolink)) @@ -186,6 +216,7 @@ ## [0.4.7] - 2022-12-11 ### Fixed + - Fix open() for direct access ([#186](https://github.com/nsidc/earthaccess/pull/186))([**@betolink**](https://github.com/betolink)) - Move python-magic to the dev section because it is a test dependency ([#186](https://github.com/nsidc/earthaccess/pull/186))([**@betolink**](https://github.com/betolink)) - Make minor edits in the README ([#186](https://github.com/nsidc/earthaccess/pull/186))([**@betolink**](https://github.com/betolink)) @@ -193,27 +224,32 @@ ## [0.4.6] - 2022-12-08 ### Changed + - For CI, only run the publish workflow after a release on GitHub ([#183](https://github.com/nsidc/earthaccess/pull/183))([**@betolink**](https://github.com/betolink)) ### Added + - Add feature to search collections by DOI ([#183](https://github.com/nsidc/earthaccess/pull/183))([**@betolink**](https://github.com/betolink)) - Add new API documentation and simplify notation to access data ([#183](https://github.com/nsidc/earthaccess/pull/183)) ([**@jroebuck932**](https://github.com/jroebuck932)) ## [0.4.1] - 2022-11-02 ### Changed + - For CI, install Poetry using the new script ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) - For CI, change dependabot alerts to monthly ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) - Improve documentation by reimplementing python_cmr methods for docstring compatibility ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) - Use `CMR-Search-After` see #145 ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) ### Added + - Add GES_DISC S3 endpoint ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) - Improve documentation by adding types to method signatures ([#131](https://github.com/nsidc/earthaccess/pull/131)) ([**@betolink**](https://github.com/betolink)) ## [0.4.0] - 2022-08-17 ### Added + - Add store, auth to docs and update mkdocs config ([#119](https://github.com/nsidc/earthaccess/pull/119))([**@betolink**](https://github.com/betolink)) - For `auth`, add the ability to persist credentials into a `.netrc` file ([#119](https://github.com/nsidc/earthaccess/pull/119))([**@betolink**](https://github.com/betolink)) - For `store`, use fsspec s3fs for in cloud access and https sessions for out of region access ([#43](https://github.com/nsidc/earthaccess/issues/43))([**@betolink**](https://github.com/betolink)) @@ -222,30 +258,36 @@ ## [0.3.0] - 2022-04-28 ### Changed + - Update python-cmr to NASA fork ([#75](https://github.com/nsidc/earthaccess/pull/75))([**@jhkennedy**](https://github.com/jhkennedy)) - Drop unused `pydantic` dependency ([`5761548`](https://github.com/nsidc/earthaccess/pull/75/commits/5761548fcd8ba8733ce4f5ff9b8ce7967c3a8398))([**@jhkennedy**](https://github.com/jhkennedy)) - Auth can refresh CMR tokens ([#82](https://github.com/nsidc/earthaccess/pull/82))([**@betolink**](https://github.com/betolink)) - Verify git tag and poetry version are the same before publishing to PyPI ### Added + - Add documentation for readthedocs and GitHub ([#82](https://github.com/nsidc/earthaccess/pull/82))([**@betolink**](https://github.com/betolink)) ### Removed + - **Breaking**: Drop python 3.7 support ([#82](https://github.com/nsidc/earthaccess/pull/82))([**@betolink**](https://github.com/betolink)) ### Fixed + - Fix bug with CMR tokens - Add missing `python-datutil` dependency ([`747e992`](https://github.com/nsidc/earthaccess/pull/75/commits/747e9926a5ab83d75bbf7f17d4c52f24b563147b))([**@jhkennedy**](https://github.com/jhkennedy)) ## [0.2.2] - 2022-03-23 ### Fixed + - Fix store to download multi-file granules ([#73](https://github.com/nsidc/earthaccess/pull/73))([**@betolink**](https://github.com/betolink)) - Fix granule formatting ([#73](https://github.com/nsidc/earthaccess/pull/73))([**@betolink**](https://github.com/betolink)) ## [0.2.1] - 2022-03-19 ### Changed + - Rename Accessor to Store ([`4bd618d`](https://github.com/nsidc/earthaccess/pull/66/commits/4bd618d4d48c3cd256a077fb8329f40df2d5b7ff))([**@betolink**](https://github.com/betolink)) - Relax dependency requirements ([`c9a5ed6`](https://github.com/nsidc/earthaccess/pull/66/commits/c9a5ed6b917435e7c4ece58485939065fa71cc8f))([**@betolink**](https://github.com/betolink)) - Store can download plain links if they are on prem ([`92d2919`](https://github.com/nsidc/earthaccess/commit/92d291962e5b72b458c2971eae8a6b813d4bae39))([**@betolink**](https://github.com/betolink)) @@ -255,6 +297,7 @@ _Conception!_ ### Added + - Add basic classes to interact with NASA CMR, EDL and cloud access. - Basic object formatting. From 0ec48491f162fc0ae478ea620955f826cf24120b Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 8 Jul 2024 17:16:49 +0000 Subject: [PATCH 24/35] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/contributing/maintainers-guide.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/contributing/maintainers-guide.md b/docs/contributing/maintainers-guide.md index 1ac2fc79..2d54a354 100644 --- a/docs/contributing/maintainers-guide.md +++ b/docs/contributing/maintainers-guide.md @@ -53,6 +53,3 @@ The GitHub Actions CI services handle the project's building, testing, and manag ## Continuous Documentation [ReadTheDocs](https://readthedocs.org/projects/earthaccess/) is used to generate and host [our documentation website](https://earthaccess.readthedocs.io/) as well as the preview for documentation changes made in pull requests. This service uses a configuration file in the root of the project, `.readthedocs.yml`. - - - From a3ce75cf72dbe5d4bacc74c79827a8826634749b Mon Sep 17 00:00:00 2001 From: jbrownrs <36203116+jbrownrs@users.noreply.github.com> Date: Mon, 8 Jul 2024 22:56:56 +0100 Subject: [PATCH 25/35] 427: Fix broken image link in sea level rise tutorial --- CHANGELOG.md | 2 ++ docs/tutorials/SSL.ipynb | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c5ee95ac..2252ba95 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,8 @@ ([#508](https://github.com/nsidc/earthaccess/issues/508)) * Create destination directory prior to direct S3 downloads, if it doesn't already exist ([#562](https://github.com/nsidc/earthaccess/issues/562)) +* Fix broken image link in sea level rise tutorial + (([#427](https://github.com/nsidc/earthaccess/issues/427)) ## [v0.9.0] 2024-02-28 diff --git a/docs/tutorials/SSL.ipynb b/docs/tutorials/SSL.ipynb index 75182d4b..265f1fd7 100644 --- a/docs/tutorials/SSL.ipynb +++ b/docs/tutorials/SSL.ipynb @@ -10,7 +10,7 @@ "\n", "### This notebook is entirely based on Jinbo Wang's [tutorial](https://github.com/betolink/the-coding-club/blob/main/notebooks/Earthdata_webinar_20220727.ipynb)\n", "\n", - "\n", + "\"Sea\n", "\n", "--- \n", "\n", @@ -157,11 +157,11 @@ " * not fully tested with Dask distributed\n", "* Data is gridded\n", " * xarray works better with homogeneous coordinates, working with swath data will be cumbersome.\n", - "* Data is chunked using reasonable large sizes(1MB or more)\n", + "* Data is chunked using reasonable large sizes (1MB or more)\n", " * If our files are chunked in small pieces the access time will be orders of magnitude bigger than just downloading the data and accessing it locally.\n", " \n", - "Opening a year of SSH (SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL1812) data (1.1 GB approx) can take up to 5 minutes streaming the data out of region(not in AWS)\n", - "The reason for this is not that the data transfer is order of magintude slower but due the client libraries not fetching data concurrently and the metadata of the files in HDF is usually not consolidated like in Zaar, hence h5netcdf has to issue a lot of requests to get the info it needs.\n", + "Opening a year of SSH (SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL1812) data (1.1 GB approx) can take up to 5 minutes streaming the data out of region (not in AWS)\n", + "The reason for this is not that the data transfer is order of magnitude slower but due the client libraries not fetching data concurrently and the metadata of the files in HDF is usually not consolidated like in Zaar, hence h5netcdf has to issue a lot of requests to get the info it needs.\n", "\n", "> Note: we are looping through each year and getting the metadata for the first granule in May" ] @@ -195,7 +195,7 @@ "id": "8b63ca2f-c94c-4d4a-a620-a086ee66137f", "metadata": {}, "source": [ - "### What `earthaccess.open()` do?\n", + "### What does `earthaccess.open()` do?\n", "\n", "`earthaccess.open()` takes a list of results from `earthaccess.search_data()` or a list of URLs and creates a list of Python File-like objects that can be used in our code as if the remote files were local. When executed in AWS the file system used is [S3FS](https://github.com/fsspec/s3fs) when we open files outside of AWS we get a regular HTTPS file session. \n" ] From 2c065f0ca73872b7c899f8ceb234be77b0ff29f3 Mon Sep 17 00:00:00 2001 From: jbrownrs <36203116+jbrownrs@users.noreply.github.com> Date: Mon, 8 Jul 2024 23:39:41 +0100 Subject: [PATCH 26/35] 427: Remove blank lines in maintainers-guide.md (pre-commit) --- docs/contributing/maintainers-guide.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/contributing/maintainers-guide.md b/docs/contributing/maintainers-guide.md index 1ac2fc79..2d54a354 100644 --- a/docs/contributing/maintainers-guide.md +++ b/docs/contributing/maintainers-guide.md @@ -53,6 +53,3 @@ The GitHub Actions CI services handle the project's building, testing, and manag ## Continuous Documentation [ReadTheDocs](https://readthedocs.org/projects/earthaccess/) is used to generate and host [our documentation website](https://earthaccess.readthedocs.io/) as well as the preview for documentation changes made in pull requests. This service uses a configuration file in the root of the project, `.readthedocs.yml`. - - - From fdc080cd0df0fe248a7cc8f088988afdf9a514b3 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Tue, 9 Jul 2024 11:10:51 -0400 Subject: [PATCH 27/35] Add a few missing entries to the CHANGELOG.md --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fc09a87..d035ef57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -115,6 +115,7 @@ - earthaccess will `raise` errors instead of `print`ing them in more cases ([#351](https://github.com/nsidc/earthaccess/pull/351))([**@jrbourbeau**](https://github.com/jrbourbeau)) - `daac` and `provider` parameters are now normalized to uppercase, since lowercase characters are never valid ([#355](https://github.com/nsidc/earthaccess/pull/355))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) +- Allow single file URL inputs for `earthaccess.download` ([#347](https://github.com/nsidc/earthaccess/pull/347))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mfisher87**](https://github.com/mfisher87)) ### Fixed @@ -161,10 +162,17 @@ ## [0.5.3] - 2023-08-01 +### Added + +- Add download from onprem how-to ([#265](https://github.com/nsidc/earthaccess/pull/265))([**@andypbarrett**](https://github.com/andypbarrett)) + ### Changed - For CI, integration tests are now only run when we push to main (after a merge) ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) - For CI, unit tests are run for any branch and opened PR ([#267](https://github.com/nsidc/earthaccess/pull/267))([**@betolink**](https://github.com/betolink)) +- Update example in `search_datasets` ([#247](https://github.com/nsidc/earthaccess/pull/247))([**@jrbourbeau**](https://github.com/jrbourbeau)) +- Improve remote cluster performance ([#259](https://github.com/nsidc/earthaccess/pull/259))([**@jrbourbeau**](https://github.com/jrbourbeau), [**@mrocklin**](https://github.com/mrocklin), [**@mfisher87**](https://github.com/mfisher87)) +- Return useful error message for failed download ([#263](https://github.com/nsidc/earthaccess/pull/263))([**@andypbarrett**](https://github.com/andypbarrett)) ### Fixed From ee33da9fe2e7e1729c1baca26ba3aa024774f297 Mon Sep 17 00:00:00 2001 From: danielfromearth Date: Tue, 9 Jul 2024 11:20:55 -0400 Subject: [PATCH 28/35] change changelog to contain change entry about changing the changelog --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index fed43dcf..506d46b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,10 @@ ([#511](https://github.com/nsidc/earthaccess/issues/511)) ([**@botanical**](https://github.com/botanical),[**@chuckwondo**](https://github.com/chuckwondo), [**@mfisher87**](https://github.com/mfisher87)) +- Update `CHANGELOG.md` to follow Common Changelog conventions + ([#584](https://github.com/nsidc/earthaccess/pull/584)) + ([**@danielfromearth**](https://github.com/danielfromearth),[**@chuckwondo**](https://github.com/chuckwondo), + [**@jhkennedy**](https://github.com/jhkennedy),[**@mfisher87**](https://github.com/mfisher87)) ### Added From ea3a9c256a5c60ef323b39ee37bdae9b2fab5a17 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 9 Jul 2024 11:32:49 -0600 Subject: [PATCH 29/35] Adjust wording of subheaders --- docs/contributing/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index c5c40455..b312609d 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -50,7 +50,7 @@ Once you've completed these steps, you are ready to submit your contribution. To ensure that our code is well-documented and easy to understand, we use [Google-style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) to document all functions, classes, and methods in this library. -#### Getting Help with Setup and Configuration +#### Locally rendering the documentation To work on documentation locally, follow these steps: @@ -60,7 +60,7 @@ To work on documentation locally, follow these steps: If you encounter any issues while setting up the documentation using the provided steps, please refer to our [tutorial]( https://www.youtube.com/watch?v=mNjlMZ4F3So) for additional guidance. -#### Caveats and Considerations +##### Caveats and considerations Our mkdocs setup has a known limitation: the hot reloader won't auto-reload when changing docstrings. To see updates, manually rebuild and re-serve docs. We're working to improve the developer experience and appreciate your patience. From b62b5dd8d08d606c43f8eee9f080a6c40a0d2bbe Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 9 Jul 2024 11:33:34 -0600 Subject: [PATCH 30/35] Fix code of conduct link --- docs/contributing/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index b312609d..6a5abf5a 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -6,7 +6,7 @@ with the community and maintainers via [a GitHub Discussion](https://github.com/nsidc/earthaccess/discussions), or [any other method](our-meet-ups.md). -Please note that we have a [code of conduct](/CODE_OF_CONDUCT.md). Please follow it in all of your interactions with the project. +Please note that we have a [code of conduct](./code-of-conduct.md). Please follow it in all of your interactions with the project. ## First Steps to contribute From 59dabee37404403d16214d9086dcb79e66a94e02 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 9 Jul 2024 11:34:17 -0600 Subject: [PATCH 31/35] Make the local documentation workflow more concise --- docs/contributing/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 6a5abf5a..4df59b9b 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -52,11 +52,11 @@ To ensure that our code is well-documented and easy to understand, we use [Googl #### Locally rendering the documentation -To work on documentation locally, follow these steps: +To work on documentation locally, we provide a script that will automatically re-render the docs when you make changes: -- To setup the documentation, navigate to the scripts directory -- Run the `scripts/build-docs.sh` script to build the documentation -- Serve the documentation locally, using the `scripts/docs-live.sh` script +``` +./scripts/docs-live.sh +``` If you encounter any issues while setting up the documentation using the provided steps, please refer to our [tutorial]( https://www.youtube.com/watch?v=mNjlMZ4F3So) for additional guidance. From b324ff8694dff32fc344c88da3b6b6e24edf930c Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 9 Jul 2024 11:34:41 -0600 Subject: [PATCH 32/35] Link to pre-commit install docs --- docs/contributing/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 4df59b9b..2f661f86 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -82,7 +82,7 @@ To maintain code quality, we use pre-commit for automated checks before committi To set up pre-commit, follow these steps: -- `pip install pre-commit` for Windows and for Linux refer to your distribution's package index +- `pip install pre-commit` ([official installation docs](https://pre-commit.com/#install)) - `pre-commit install` to enable it to run automatically - `pre-commit run -a` to run it manually From d8f95114605cf5f9069c8430741f9dce714c2a70 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 9 Jul 2024 11:35:12 -0600 Subject: [PATCH 33/35] Remove instructions referencing removed Make command instructions --- docs/contributing/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 2f661f86..adabeda6 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -86,7 +86,6 @@ To set up pre-commit, follow these steps: - `pre-commit install` to enable it to run automatically - `pre-commit run -a` to run it manually -We attempt to provide comprehensive type annotations within this repository. If you do not provide fully annotated functions or methods, the `lint` command will fail. Over time, we plan to increase type-checking strictness in order to ensure more precise, beneficial type annotations. We have included type stubs for the untyped `python-cmr` library, which we intend to eventually upstream. Since `python-cmr` exposes the `cmr` package, the stubs appear under `stubs/cmr`. From f1176e615cd7a2813dcfbb79aa62585401180fe6 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 9 Jul 2024 17:47:27 +0000 Subject: [PATCH 34/35] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/contributing/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 6ac83fd1..a8b9f624 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -94,4 +94,4 @@ We have included type stubs for the untyped `python-cmr` library, which we inten Fork the repository using the "Fork" button on the [repository homepage](https://github.com/nsidc/earthaccess), create a branch with your changes in the fork, then open a draft pull request from your fork. Starting a pull request provides additional instructions and requirements, and -there is no harm in starting a draft pull request while still developing. \ No newline at end of file +there is no harm in starting a draft pull request while still developing. From a1ee17d351c8beeddb2fe210bb1bf93917da1d25 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 9 Jul 2024 12:41:26 -0600 Subject: [PATCH 35/35] Add note about Common Changelog to PR template --- .github/pull_request_template.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 60c23104..95911c4f 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -33,7 +33,8 @@ created the PR as a non-draft, don't worry, you can still change it to a draft u example `closes #1`. See [GitHub docs - Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue). - [ ] Update `CHANGELOG.md` with details about your change in a section titled - `## Unreleased`. If such a section does not exist, please create one. + `## Unreleased`. If such a section does not exist, please create one. Follow + [Common Changelog](https://common-changelog.org/) for your additions. - [ ] Update the documentation and/or the `README.md` with details of changes to the earthaccess interface, if any. Consider new environment variables, function names, decorators, etc.