Merge branch 'main' of github.com:Sherwin-14/earthaccess

.github/pull_request_template.md

@@ -0,0 +1,60 @@
+<!--
+Replace this text, including the symbols above and below it, with descriptive text about
+the change you are proposing. Please include a reference to any issues addressed or
+resolved in that text, for example "resolves #1".
+-->
+
+<!--
+IMPORTANT: As a contributor, we would like as much help as you can offer, but we only
+expect you to complete the steps in the "PR draft checklist" below. Maintainers are
+willing and ready to help pick it up from there!
+
+Please start by opening this Pull Request as a "draft". You can do this by
+clicking the arrow on the right side of the green "Create pull request" button. While
+your pull request is in "draft" state, maintainers will assume the PR isn't ready for
+their attention unless they are specifically summoned using GitHub's @ system.
+
+Follow the draft checklist below to move the PR out of draft state. If you accidentally
+created the PR as a non-draft, don't worry, you can still change it to a draft using the
+"Convert to draft" button on  the right side panel under the "Reviewers" section.
+-->
+
+<details><summary>Pull Request (PR) draft checklist - click to expand</summary>
+
+- [ ] Please review our
+      [contributing documentation](https://earthaccess.readthedocs.io/en/latest/contributing/)
+      before getting started.
+- [ ] Ensure an issue exists representing the problem being solved in this PR.
+- [ ] Populate a descriptive title. For example, instead of "Updated README.md", use a
+      title such as "Add testing details to the contributor section of the README".
+- [ ] Populate the body of the pull request with:
+    - A clear description of the change you are proposing.
+    - Links to any issues resolved by this PR with text in the PR description, for
+      example `closes #1`. See
+      [GitHub docs - Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
+- [ ] Update `CHANGELOG.md` with details about your change in a section titled
+      `## Unreleased`. If such a section does not exist, please create one. Follow
+      [Common Changelog](https://common-changelog.org/) for your additions.
+- [ ] Update the documentation and/or the `README.md` with details of changes to the
+      earthaccess interface, if any. Consider new environment variables, function names,
+      decorators, etc.
+
+Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
+once these requirements are fulfilled. Don't worry if you see any test failures in
+GitHub at this point!
+
+</details>
+
+<details><summary>Pull Request (PR) merge checklist - click to expand</summary>
+
+Please do your best to complete these requirements! If you need help with any of these
+requirements, you can ping the `@nsidc/earthaccess-support` team in a comment and we
+will help you out!
+
+- [ ] Add unit tests for any new features.
+- [ ] Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
+      Request containing "pre-commit.ci autofix" to automate this.
+- [ ] Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
+- [ ] Get at least one approving review.
+
+</details>

docs/contributing/index.md

@@ -19,7 +19,7 @@ Please note that we have a [code of conduct](./code-of-conduct.md). Please follo
 From here, you might want to fix and issue or bug, or add a new feature.  Jump to the
 relevant section to proceed.
 
-### ...to fix an issue or bug
+### Fixing an Issue or Bug
 
 - Create a GitHub issue with a
   [minimal reproducible example](https://en.wikipedia.org/wiki/Minimal_reproducible_example),
@@ -33,7 +33,7 @@ relevant section to proceed.
 
 Once you've completed these steps, you are ready to submit your contribution.
 
-### ...to contribute a new feature
+### Contributing a New Feature
 
 - Create an issue and discuss the feature's scope and its fit for this package with the team
 - Create a branch for your new feature in your fork
@@ -43,6 +43,28 @@ Once you've completed these steps, you are ready to submit your contribution.
 
 Once you've completed these steps, you are ready to submit your contribution.
 
+
+### Contributing to Documentation
+
+#### Documentation Style
+
+To ensure that our code is well-documented and easy to understand, we use [Google-style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) to document all functions, classes, and methods in this library.
+
+#### Locally rendering the documentation
+
+To work on documentation locally, we provide a script that will automatically re-render the docs when you make changes:
+
+```
+./scripts/docs-live.sh
+```
+
+If you encounter any issues while setting up the documentation using the provided steps, please refer to our [tutorial]( https://www.youtube.com/watch?v=mNjlMZ4F3So) for additional guidance.
+
+##### Caveats and considerations
+
+Our mkdocs setup has a known limitation: the hot reloader won't auto-reload when changing docstrings. To see updates, manually rebuild and re-serve docs. We're working to improve the developer experience and appreciate your patience.
+
+
 ## Submitting your contribution
 
 - Run all unit tests successfully in your branch
@@ -53,34 +75,23 @@ Once you've completed these steps, you are ready to submit your contribution.
 
 ### Please format and lint as you go
 
-```bash
-make format lint
-```
 
-We attempt to provide comprehensive type annotations within this repository.  If
-you do not provide fully annotated functions or methods, the `lint` command will
-fail.  Over time, we plan to increase type-checking strictness in order to
-ensure more precise, beneficial type annotations.
+#### Usage of Pre-Commit
+
+To maintain code quality, we use pre-commit for automated checks before committing changes. Install and configure pre-commit to ensure high-quality contributions.
+
+To set up pre-commit, follow these steps:
 
-We have included type stubs for the untyped `python-cmr` library, which we
-intend to eventually upstream.  Since `python-cmr` exposes the `cmr` package,
-the stubs appear under `stubs/cmr`.
+- `pip install pre-commit` ([official installation docs](https://pre-commit.com/#install))
+- `pre-commit install` to enable it to run automatically
+- `pre-commit run -a`  to run it manually
 
-### Requirements to merge code (Pull Request Process)
 
-- you must include test coverage
-- you must update the documentation
-- you must format and lint
+We have included type stubs for the untyped `python-cmr` library, which we intend to eventually upstream.  Since `python-cmr` exposes the `cmr` package, the stubs appear under `stubs/cmr`.
 
 ## Pull Request process
 
-1. Ensure you include test coverage for all changes
-1. Ensure your code is formatted properly following this document
-1. Update the documentation and the `README.md` with details of changes to the
-   interface, this includes new environment variables, function names,
-   decorators, etc.
-1. Update `CHANGELOG.md` with details about your change in a section titled
-   `Unreleased`. If one does not exist, please create one.
-1. You may merge the Pull Request once you have the sign-off of another
-   developer, or if you do not have permission to do that, you may request the
-   reviewer to merge it for you.
+Fork the repository using the "Fork" button on the [repository
+homepage](https://github.com/nsidc/earthaccess), create a branch with your changes in the fork, then open
+a draft pull request from your fork. Starting a pull request provides additional instructions and requirements, and
+there is no harm in starting a draft pull request while still developing.

docs/contributing/maintainers-guide.md

@@ -53,6 +53,3 @@ The GitHub Actions CI services handle the project's building, testing, and manag
 ## Continuous Documentation
 
 [ReadTheDocs](https://readthedocs.org/projects/earthaccess/) is used to generate and host [our documentation website](https://earthaccess.readthedocs.io/) as well as the preview for documentation changes made in pull requests. This service uses a configuration file in the root of the project, `.readthedocs.yml`.
-
-
-

docs/tutorials/SSL.ipynb

@@ -10,7 +10,7 @@
     "\n",
     "### This notebook is entirely based on Jinbo Wang's [tutorial](https://github.com/betolink/the-coding-club/blob/main/notebooks/Earthdata_webinar_20220727.ipynb)\n",
     "\n",
-    "<img src=\"https://www.nasa.gov/sites/default/files/styles/full_width/public/thumbnails/image/sealevel_main-causes3-16.jpg?itok=XAqGqps1\" width=\"800px\" />\n",
+    "<img alt=\"Sea level rise infographic\" src=\"https://www.nasa.gov/wp-content/uploads/2020/08/sealevel_main-causes3-16.jpg\" width=\"800px\" />\n",
     "\n",
     "--- \n",
     "\n",
@@ -157,11 +157,11 @@
     "  * not fully tested with Dask distributed\n",
     "* Data is gridded\n",
     "  * xarray works better with homogeneous coordinates, working with swath data will be cumbersome.\n",
-    "* Data is chunked using reasonable large sizes(1MB or more)\n",
+    "* Data is chunked using reasonable large sizes (1MB or more)\n",
     "  * If our files are chunked in small pieces the access time will be orders of magnitude bigger than just downloading the data and accessing it locally.\n",
     "  \n",
-    "Opening a year of SSH (SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL1812) data (1.1 GB approx) can take up to 5 minutes streaming the data out of region(not in AWS)\n",
-    "The reason for this is not that the data transfer is order of magintude slower but due the client libraries not fetching data concurrently and the metadata of the files in HDF is usually not consolidated like in Zaar, hence h5netcdf has to issue a lot of requests to get the info it needs.\n",
+    "Opening a year of SSH (SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL1812) data (1.1 GB approx) can take up to 5 minutes streaming the data out of region (not in AWS)\n",
+    "The reason for this is not that the data transfer is order of magnitude slower but due the client libraries not fetching data concurrently and the metadata of the files in HDF is usually not consolidated like in Zaar, hence h5netcdf has to issue a lot of requests to get the info it needs.\n",
     "\n",
     "> Note: we are looping through each year and getting the metadata for the first granule in May"
    ]
@@ -195,7 +195,7 @@
    "id": "8b63ca2f-c94c-4d4a-a620-a086ee66137f",
    "metadata": {},
    "source": [
-    "### What `earthaccess.open()` do?\n",
+    "### What does `earthaccess.open()` do?\n",
     "\n",
     "`earthaccess.open()` takes a list of results from `earthaccess.search_data()` or a list of URLs and creates a list of Python File-like objects that can be used in our code as if the remote files were local. When executed in AWS the file system used is [S3FS](https://github.com/fsspec/s3fs) when we open files outside of AWS we get a regular HTTPS file session. \n"
    ]

earthaccess/api.py

@@ -68,7 +68,9 @@ def search_datasets(count: int = -1, **kwargs: Any) -> List[DataCollection]:
         ```
     """
     if not validate.valid_dataset_parameters(**kwargs):
-        logger.warn("A valid set of parameters is needed to search for datasets on CMR")
+        logger.warning(
+            "A valid set of parameters is needed to search for datasets on CMR"
+        )
         return []
     if earthaccess.__auth__.authenticated:
         query = DataCollections(auth=earthaccess.__auth__).parameters(**kwargs)

earthaccess/store.py

@@ -113,7 +113,7 @@ def __init__(self, auth: Any, pre_authorize: bool = False) -> None:
                     self.set_requests_session(url)
 
         else:
-            logger.warn("The current session is not authenticated with NASA")
+            logger.warning("The current session is not authenticated with NASA")
             self.auth = None
         self.in_region = self._running_in_us_west_2()
 

pyproject.toml

@@ -144,8 +144,8 @@ line-length = 88
 src = ["earthaccess", "stubs", "tests"]
 
 [tool.ruff.lint]
-extend-select = ["I", "T20", "D"]
-ignore = ["D1", "D205", "D401", "D417"]
+extend-select = ["I", "T20", "D", "G"]
+ignore = ["D1", "D205", "D401", "D417", "G004"]
 
 [tool.ruff.lint.pydocstyle]
 convention = "google"

scripts/integration-test.sh

@@ -1,7 +1,18 @@
 #!/usr/bin/env bash
 
-set -e
 set -x
+pytest tests/integration --cov=earthaccess --cov=tests/integration --cov-report=term-missing ${@} --capture=no --tb=native --log-cli-level=INFO
+RET=$?
+set +x
+
+set -e
+# NOTE: 99 is a special return code we selected for this case, it has no other meaning.
+if [[ $RET == 99 ]]; then
+    echo -e "\e[0;31mWARNING: The integration test suite has been permitted to return 0 because the failure rate was less than a hardcoded threshold.\e[0m"
+    echo "For more details, see conftest.py."
+    exit 0
+elif [[ $RET != 0 ]]; then
+    exit $RET
+fi
 
-pytest --cov=earthaccess --cov=tests/integration --cov-report=term-missing ${@} -s --tb=native --log-cli-level=INFO
 bash ./scripts/lint.sh

tests/integration/conftest.py

@@ -5,8 +5,27 @@
 
 @pytest.hookimpl()
 def pytest_sessionfinish(session, exitstatus):
-    if exitstatus == 0:
+    """Return exit code 99 if up to N% of tests have failed.
+
+        N = ACCEPTABLE_FAILURE_RATE
+
+    99 was chosen arbitrarily to avoid conflict with current and future pytest error
+    codes (https://docs.pytest.org/en/stable/reference/exit-codes.html), and avoid
+    other exit codes with special meanings
+    (https://tldp.org/LDP/abs/html/exitcodes.html).
+
+    IMPORTANT: This is calculated against every test collected in the session, so the
+    ratio will change depending on which tests are executed! E.g. executing integration
+    tests and unit tests at the same time allows more tests to fail than executing
+    integration tests alone.
+    """
+    if exitstatus != pytest.ExitCode.TESTS_FAILED:
+        # Exit status 1 in PyTest indicates "Tests were collected and run but some of
+        # the tests failed". In all other cases, for example "an internal error happened
+        # while executing the tests", or "test execution interrupted by the user", we
+        # want to defer to original pytest behavior.
         return
+
     failure_rate = (100.0 * session.testsfailed) / session.testscollected
     if failure_rate <= ACCEPTABLE_FAILURE_RATE:
-        session.exitstatus = 0
+        session.exitstatus = 99