Skip to content

Commit

Permalink
Added a second test for download
Browse files Browse the repository at this point in the history
  • Loading branch information
Sherwin-14 committed Oct 29, 2024
1 parent 767e119 commit 2d485b3
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 29 deletions.
30 changes: 6 additions & 24 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,30 +68,12 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
[@betolink](https://github.com/betolink))
- Add example PR links to pull request template
([#756](https://github.com/nsidc/earthaccess/issues/756))
[**@Sherwin-14**](https://github.com/Sherwin-14),
[**@mfisher87**](https://github.com/mfisher87)

- Added Contributing Naming Convention document
([#532](https://github.com/nsidc/earthaccess/issues/532))
[**@Sherwin-14**](https://github.com/Sherwin-14),
[**@mfisher87**](https://github.com/mfisher87)

### Fixed

- Removed Broken Link "Introduction to NASA earthaccess"
([#779](https://github.com/nsidc/earthaccess/issues/779))
([**@Sherwin-14**](https://github.com/Sherwin-14))
- Restore automation for tidying notebooks used in documentation
([#788](https://github.com/nsidc/earthaccess/issues/788))
([**@itcarroll**](https://github.com/itcarroll))
- Remove the base class on `EarthAccessFile` to fix method resolution
([#610](https://github.com/nsidc/earthaccess/issues/610))
([**@itcarroll**](https://github.com/itcarroll))
- Fix `earthaccess.download` to not ignore errors by default
([#581](https://github.com/nsidc/earthaccess/issues/581))
([**@Sherwin-14**](https://github.com/Sherwin-14),
[**@chuckwondo**](https://github.com/chuckwondo),
[**@mfisher87**](https://github.com/mfisher87))
([@Sherwin-14](https://github.com/Sherwin-14),
[@mfisher87](https://github.com/mfisher87))
- Add Contributing Naming Convention document
([#532](https://github.com/nsidc/earthaccess/issues/532))
([@Sherwin-14](https://github.com/Sherwin-14),
[@mfisher87](https://github.com/mfisher87))

### Removed

Expand Down
10 changes: 5 additions & 5 deletions earthaccess/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,12 +63,12 @@ def __repr__(self) -> str:
def _open_files(
url_mapping: Mapping[str, Union[DataGranule, None]],
fs: fsspec.AbstractFileSystem,
threads: Optional[int] = 8,
threads: int = 8,
pqdm_kwargs: Optional[Mapping[str, Any]] = None,
) -> List[fsspec.AbstractFileSystem]:
) -> List[fsspec.spec.AbstractBufferedFile]:
def multi_thread_open(data: tuple[str, Optional[DataGranule]]) -> EarthAccessFile:
urls, granule = data
return EarthAccessFile(fs.open(urls), granule) # type: ignore
url, granule = data
return EarthAccessFile(fs.open(url), granule) # type: ignore

pqdm_kwargs = {
"exception_behavior": "immediate",
Expand Down Expand Up @@ -346,7 +346,7 @@ def open(
provider: Optional[str] = None,
pqdm_kwargs: Optional[Mapping[str, Any]] = None,
) -> List[fsspec.spec.AbstractBufferedFile]:
"""Returns a list of fsspec file-like objects that can be used to access files
"""Returns a list of file-like objects that can be used to access files
hosted on S3 or HTTPS by third party libraries like xarray.
Parameters:
Expand Down
25 changes: 25 additions & 0 deletions tests/unit/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,28 @@ def mock_get(*args, **kwargs):

with pytest.raises(Exception, match="Download failed"):
earthaccess.download(results, "/home/download-folder")


def test_download_deferred_failure(monkeypatch):
earthaccess.login()

results = earthaccess.search_data(
short_name="ATL06",
bounding_box=(-10, 20, 10, 50),
temporal=("1999-02", "2019-03"),
count=10,
)

def mock_get(*args, **kwargs):
return [Exception("Download failed")] * len(results)

mock_store = Mock()
monkeypatch.setattr(earthaccess, "__store__", mock_store)
monkeypatch.setattr(mock_store, "get", mock_get)

results = earthaccess.download(
results, "/home/download-folder", None, 8, {"exception_behavior": "deferred"}
)

assert all(isinstance(e, Exception) for e in results)
assert len(results) == 10

0 comments on commit 2d485b3

Please sign in to comment.