fixing searching for restricted datasets and accessing ASF on demand data from Opera #443

betolink · 2024-02-01T17:32:54Z

Fixes Permissions error when downloading files from OPERA_L2_RTC-S1_V1 dataset on ASF DAAC (but not OPERA_L3_DIST-ALERT-HLS_PROVISIONAL_V0 on LP DAAC) #439 by implementing multiple trusted domains (ASF's) on top of NASA's URS
- This will fix downloading data behind ASF datapool but we still can't open them because aiohttp/fsspec do not support rebuilding the authorization headers on redirects. This should work now
- ```
import earthaccess
earthaccess.login()

results = earthaccess.search_data(
    short_name='OPERA_L2_RTC-S1_V1',
    cloud_hosted=True,
    bounding_box=(-117.33, 35.541, -117.880, 35.991),
    temporal=("2023-10-01", "2023-10-15"),
    count=50
)
files = earthaccess.download(results, "./data")
```
Fixes Restricted datasets can't be used with the API notation #438 by using an authenticated session in hits() instead of calling parent's class super()
Improves search by adding instrument and project to collection queries Broken image in sea level rise tutorial #427
- ```
collections = earthaccess.search_datasets(
    instrument="GEDI"
    cloud_hosted=True
)
```

📚 Documentation preview 📚: https://earthaccess--443.org.readthedocs.build/en/443/

betolink · 2024-02-01T17:41:52Z

earthaccess/results.py

@@ -228,7 +228,9 @@ def __repr__(self) -> str:
        Temporal coverage: {self['umm']['TemporalExtent']}
        Size(MB): {self.size()}
        Data: {data_links}\n\n
-        """.strip().replace("  ", "")
+        """.strip().replace(


Black likes this, Ruff does not, which one is right? @mfisher87

I think we should pick one :) Ruff offers a formatter that re-implements Black, but I've never worked with it before. Personally, I'd turn Ruff formatting off and stick with Black due to my lack of experience, but if I had more time I'd try to learn more and switch to Ruff for everything.

Looks like Ruff prefers the compact and black expands. I think Ruff is actually correct here (obvs. subjective).

This doc is helpful for black vs ruff:
https://github.com/astral-sh/ruff/blob/main/docs/formatter/black.md#implicit-string-concatenations-in-attribute-accesses

I don't think it matters which, but I agree that we should pick one -- I'd prefer ruff overall.

@MattF-NSIDC I'm pretty happy with ruff; and the transition from black to ruff is pretty straight forward as they mostly behave the same.

earthaccess/auth.py

earthaccess/search.py

adding Sentinel 1, (I though Sentinel was working before.) Co-authored-by: Joseph H Kennedy <[email protected]>

Co-authored-by: Joseph H Kennedy <[email protected]>

betolink · 2024-02-02T04:44:34Z

binder/environment-dev.yml

@@ -4,7 +4,7 @@ channels:
 dependencies:
  # This environment bootstraps poetry, the actual dev environment
  # is installed and managed with poetry
-  - python=3.9
+  - python=3.10


This one is only for the local dev environment and actually not necessary as we can just pip install the project locally.... I use it to bootstrap the environment and install some conda libs to test notebooks with (cartopy)

betolink · 2024-02-02T04:59:43Z

Should we include #436 on this PR? since we are touching the EDL session?

jhkennedy · 2024-02-02T05:26:03Z

Should we include #436 on this PR? since we are touching the EDL session?

Yes, that's a good idea

betolink · 2024-02-03T18:45:06Z

earthaccess/auth.py


    def __init__(
        self, username: Optional[str] = None, password: Optional[str] = None
    ) -> None:
        super().__init__()
+        self.headers.update({"User-Agent": user_agent})


earthaccess will now use the User-Agent header to identify requests to EDL. 🎉

earthaccess/auth.py

Co-authored-by: Joseph H Kennedy <[email protected]>

betolink · 2024-02-08T16:03:57Z

Should we merge this before the other open PRs? @jhkennedy @mfisher87

mfisher87 · 2024-02-08T19:04:07Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

mfisher87 · 2024-02-08T19:07:11Z

earthaccess/search.py

+    def instrument(self, instrument: str) -> Type[CollectionQuery]:
+        """Searh datasets by instrument
+
+        ???+ Tip


TIL: https://squidfunk.github.io/mkdocs-material/reference/admonitions/#collapsible-blocks

mfisher87 · 2024-02-08T19:08:24Z

Should we merge this before the other open PRs? @jhkennedy @mfisher87

Logistically, that makes sense to me. I haven't done a proper review here and won't have time in short-term, so I'll leave the "approval" decision to others :)

jhkennedy

I think this looks good! Should work as is, but I added a take-or-leave suggestion

earthaccess/auth.py

Co-authored-by: Joseph H Kennedy <[email protected]>

betolink · 2024-02-08T22:46:21Z

I think this looks good! Should work as is, but I added a take-or-leave suggestion

I think that last commit broke things... 😂 @jhkennedy

jhkennedy · 2024-02-08T22:47:44Z

I think this looks good! Should work as is, but I added a take-or-leave suggestion

I think that last commit broke things... 😂 @jhkennedy

Yes, hense the "Note: imports will also need to be adjusted." 😉

betolink · 2024-02-09T03:18:33Z

I'm using importlib so we avoid importing the library and potentially produce one of those circular import errors, I think it should be good now.

betolink · 2024-02-11T20:06:15Z

going to merge this @jhkennedy

betolink added 2 commits February 1, 2024 11:14

refactoring auth to allow other trusted domains besides NASA's URS

e97e6f0

updated changelog and readme

073e4ac

betolink requested a review from jhkennedy February 1, 2024 17:32

betolink commented Feb 1, 2024

View reviewed changes

fix docstring

acade70

jhkennedy reviewed Feb 2, 2024

View reviewed changes

earthaccess/auth.py Show resolved Hide resolved

earthaccess/search.py Outdated Show resolved Hide resolved

betolink and others added 3 commits February 1, 2024 22:35

Update earthaccess/auth.py

429324a

adding Sentinel 1, (I though Sentinel was working before.) Co-authored-by: Joseph H Kennedy <[email protected]>

Update earthaccess/search.py

b320652

Co-authored-by: Joseph H Kennedy <[email protected]>

making ruff happy

0ead9db

betolink marked this pull request as ready for review February 2, 2024 04:42

betolink commented Feb 2, 2024

View reviewed changes

betolink added 3 commits February 2, 2024 19:52

using user-agent to start tracing client usage

c692069

fix lintint

b94897e

fix lintint

11686dd

betolink commented Feb 3, 2024

View reviewed changes

jhkennedy reviewed Feb 7, 2024

View reviewed changes

earthaccess/auth.py Show resolved Hide resolved

Update earthaccess/auth.py

2b14609

Co-authored-by: Joseph H Kennedy <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

72539dd

for more information, see https://pre-commit.ci

mfisher87 reviewed Feb 8, 2024

View reviewed changes

jhkennedy previously approved these changes Feb 8, 2024

View reviewed changes

earthaccess/auth.py Outdated Show resolved Hide resolved

Update earthaccess/auth.py

19b6be1

Co-authored-by: Joseph H Kennedy <[email protected]>

betolink dismissed jhkennedy’s stale review via 19b6be1 February 8, 2024 22:43

reverting last commit, avoid importing the library

d20f713

betolink requested a review from jhkennedy February 9, 2024 03:17

jhkennedy approved these changes Feb 11, 2024

View reviewed changes

betolink merged commit e187ae0 into main Feb 11, 2024
13 checks passed

chuckwondo mentioned this pull request Apr 8, 2024

Query collections using 'instrument' and 'project' #437

Closed

mfisher87 linked an issue Apr 8, 2024 that may be closed by this pull request

Query collections using 'instrument' and 'project' #437

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixing searching for restricted datasets and accessing ASF on demand data from Opera #443

fixing searching for restricted datasets and accessing ASF on demand data from Opera #443

betolink commented Feb 1, 2024 •

edited by github-actions bot

Loading

betolink Feb 1, 2024

mfisher87 Feb 1, 2024

jhkennedy Feb 2, 2024

betolink Feb 2, 2024

betolink commented Feb 2, 2024

jhkennedy commented Feb 2, 2024

betolink Feb 3, 2024

betolink commented Feb 8, 2024

mfisher87 commented Feb 8, 2024

mfisher87 Feb 8, 2024

mfisher87 commented Feb 8, 2024

jhkennedy left a comment

betolink commented Feb 8, 2024

jhkennedy commented Feb 8, 2024

betolink commented Feb 9, 2024

betolink commented Feb 11, 2024

fixing searching for restricted datasets and accessing ASF on demand data from Opera #443

fixing searching for restricted datasets and accessing ASF on demand data from Opera #443

Conversation

betolink commented Feb 1, 2024 • edited by github-actions bot Loading

betolink Feb 1, 2024

Choose a reason for hiding this comment

mfisher87 Feb 1, 2024

Choose a reason for hiding this comment

jhkennedy Feb 2, 2024

Choose a reason for hiding this comment

betolink Feb 2, 2024

Choose a reason for hiding this comment

betolink commented Feb 2, 2024

jhkennedy commented Feb 2, 2024

betolink Feb 3, 2024

Choose a reason for hiding this comment

betolink commented Feb 8, 2024

mfisher87 commented Feb 8, 2024

mfisher87 Feb 8, 2024

Choose a reason for hiding this comment

mfisher87 commented Feb 8, 2024

jhkennedy left a comment

Choose a reason for hiding this comment

betolink commented Feb 8, 2024

jhkennedy commented Feb 8, 2024

betolink commented Feb 9, 2024

betolink commented Feb 11, 2024

betolink commented Feb 1, 2024 •

edited by github-actions bot

Loading