Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow collection to be on file system while catalog is in the cloud #351

Open
wachsylon opened this issue Aug 5, 2021 · 1 comment
Open
Labels
bug Issues that present a reasonable conviction there is a reproducible bug.

Comments

@wachsylon
Copy link

When using an esm_collection from file system:

intake.open_esm_datastore('/home/dkrz/k204210/intake-esm/esm-collections/dkrz_era5_disk_grb_fromcloud.json')

with url as catalog_file entry value:

cat /home/dkrz/k204210/intake-esm/esm-collections/dkrz_era5_disk_grb_fromcloud.json
...
  "catalog_file": "https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_era5_disk_grb.csv.gz",

results in

Unable to find: /home/dkrz/k204210/intake-esm/esm-collections/https:/swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_era5_disk_grb.csv.gz

I think there is no reason why this should not work.

@wachsylon wachsylon changed the title Allow collection to be on file while data is in the cloud Allow collection to be on file system while catalog is in the cloud Aug 5, 2021
@andersy005 andersy005 added the bug Issues that present a reasonable conviction there is a reproducible bug. label Aug 5, 2021
@andersy005
Copy link
Member

@wachsylon, I concur with you. This is a bug in _fetch_catalog function, and we should relax some of the assumptions made in this function

def _fetch_catalog(collection_data, esmcol_path, csv_kwargs=None):

I am happy to look into it, but if you have time and are interested in working on it, please feel free to submit a PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues that present a reasonable conviction there is a reproducible bug.
Projects
None yet
Development

No branches or pull requests

2 participants