From 10f799b6540f20434516f57cdd9ef9ea6fe20d18 Mon Sep 17 00:00:00 2001 From: Quarto GHA Workflow Runner Date: Thu, 14 Mar 2024 19:13:53 +0000 Subject: [PATCH] Built site for gh-pages --- .nojekyll | 2 +- LICENSE.html | 672 +- README.md | 2 +- .../Access_COG.html | 675 +- ...ile__Direct_S3_Access_NetCDF4_Example.html | 701 +- ...Single_File__HTTPS_Access_COG_Example.html | 675 +- ...-File_Direct_S3_Access_NetCDF_Example.html | 705 +- appendix/authentication.html | 237 +- appendix/index.html | 235 +- contributing/calendar-project-management.html | 240 +- contributing/community.html | 669 +- contributing/environment.html | 671 +- contributing/index.html | 239 +- contributing/onboarding.html | 237 +- contributing/quarto-website-tutorial.html | 235 +- contributing/setup.html | 241 +- contributing/troubleshooting.html | 239 +- contributing/workflow.html | 245 +- environment-setup/index.html | 239 +- examples/Earthdata-cloud-clinic.html | 424 +- ...DISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html | 3029 +- .../Find_and_Access_HLS_PointBuffer.html | 724 +- .../LPDAAC_ECOSTRESS_LSTE__Kerchunk.html | 305 +- examples/NSIDC/ICESat2-CMR-AWS-S3.html | 257 +- .../NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html | 265 +- ...Access__Direct_S3_Access__ORNL_DAYMET.html | 553 +- examples/ORNL/ORNL_DAYMET__Kerchunk.html | 704 +- .../PODAAC/PODAAC_ECCO_SSH__Kerchunk.html | 723 +- examples/Transform-workflow.html | 304 +- .../Landsat_C2_Search_Access.html | 303 +- examples/index.html | 235 +- external/IS2_cloud_Landsat_integration.html | 33999 ---------------- .../figure-html/cell-32-output-1.png | Bin 256510 -> 0 bytes .../figure-html/cell-44-output-1.png | Bin 458177 -> 0 bytes .../figure-html/cell-45-output-1.png | Bin 389864 -> 0 bytes external/appeears_csv_cloud_access.html | 318 +- external/cof-zarr-reformat.html | 355 +- ...l-15-output-1.png => cell-15-output-2.png} | Bin external/data_access_direct_S3.html | 746 +- external/data_discovery_cmr-stac_api.html | 767 +- external/data_discovery_cmr.html | 728 +- external/harmony_subsetting.html | 763 +- external/nasa_earthdata_authentication.html | 690 +- external/on-prem_cloud.html | 842 +- ...l-27-output-1.png => cell-27-output-2.png} | Bin ...l-33-output-1.png => cell-33-output-2.png} | Bin .../sentinel-6_opendap_access_gridding.html | 878 +- ...l-19-output-1.png => cell-19-output-2.png} | Bin ...l-28-output-1.png => cell-28-output-2.png} | Bin ...l-29-output-1.png => cell-29-output-2.png} | Bin ...l-30-output-1.png => cell-30-output-2.png} | Bin external/setup.html | 980 - external/xarray.html | 738 +- external/zarr-eosdis-store.html | 268 +- external/zarr_access.html | 4686 --- .../figure-html/cell-16-output-1.png | Bin 71465 -> 0 bytes .../figure-html/cell-22-output-2.png | Bin 79761 -> 0 bytes glossary.html | 235 +- ...arthdata_Cloud__Open-Science-Tutorial.html | 752 +- how-tos/access-data/access-cloud-python.html | 407 +- how-tos/access-data/access-local-opendap.html | 239 +- how-tos/access-data/access-local-python.html | 405 +- how-tos/aggregate.html | 671 +- how-tos/find-data/earthdata_search.html | 253 +- how-tos/find-data/find-python.html | 256 +- how-tos/find-data/find-r.html | 237 +- how-tos/find-data/programmatic.html | 675 +- how-tos/how_to_guide_parking_lot.html | 236 +- how-tos/plot.html | 671 +- how-tos/read_data.html | 239 +- how-tos/reformat.html | 671 +- how-tos/reproject_regrid.html | 671 +- how-tos/subset.html | 239 +- how-tos/use_apis.html | 237 +- in-development/CMR-STAC-Search.html | 244 +- .../earthdata-python-r-handoff.html | 239 +- in-development/index.html | 233 +- in-development/matlab.html | 689 +- index.html | 239 +- .../add-folks-to-2i2c-github-teams.html | 253 +- leading-workshops/index.html | 235 +- our-cookbook.html | 235 +- search.json | 5806 +-- site_libs/bootstrap/bootstrap-dark.min.css | 12 +- site_libs/bootstrap/bootstrap-icons.css | 148 +- site_libs/bootstrap/bootstrap-icons.woff | Bin 176200 -> 164168 bytes site_libs/bootstrap/bootstrap.min.css | 12 +- site_libs/bootstrap/bootstrap.min.js | 6 +- site_libs/quarto-html/anchor.min.js | 6 +- site_libs/quarto-html/popper.min.js | 4 +- site_libs/quarto-html/quarto.js | 29 +- site_libs/quarto-nav/quarto-nav.js | 21 +- site_libs/quarto-search/autocomplete.umd.js | 4 +- site_libs/quarto-search/quarto-search.js | 226 +- sitemap.xml | 324 +- tutorials/04_On-Prem_Cloud.html | 407 +- ...l-27-output-1.png => cell-27-output-2.png} | Bin ...l-33-output-1.png => cell-33-output-2.png} | Bin tutorials/Above_Ground_Biomass_Density.html | 299 +- tutorials/Data_stories_template.html | 685 +- ...ta_Cloud__Data_Access_OPeNDAP_Example.html | 282 +- ...l-11-output-1.png => cell-11-output-2.png} | Bin ...anes_Wind_and_Sea_Surface_Temperature.html | 278 +- tutorials/Observing_Seasonal_Ag_Changes.html | 390 +- tutorials/Sea_Level_Rise.html | 775 +- tutorials/earthaccess-demo.html | 281 +- tutorials/fair-workflow-geoweaver-demo.html | 333 +- tutorials/index.html | 237 +- tutorials/matlab.html | 255 +- when-to-cloud.html | 247 +- workshops/index.html | 235 +- workshops/setup.html | 247 +- 112 files changed, 17025 insertions(+), 65728 deletions(-) delete mode 100644 external/IS2_cloud_Landsat_integration.html delete mode 100644 external/IS2_cloud_Landsat_integration_files/figure-html/cell-32-output-1.png delete mode 100644 external/IS2_cloud_Landsat_integration_files/figure-html/cell-44-output-1.png delete mode 100644 external/IS2_cloud_Landsat_integration_files/figure-html/cell-45-output-1.png rename external/cof-zarr-reformat_files/figure-html/{cell-15-output-1.png => cell-15-output-2.png} (100%) rename external/on-prem_cloud_files/figure-html/{cell-27-output-1.png => cell-27-output-2.png} (100%) rename external/on-prem_cloud_files/figure-html/{cell-33-output-1.png => cell-33-output-2.png} (100%) rename external/sentinel-6_opendap_access_gridding_files/figure-html/{cell-19-output-1.png => cell-19-output-2.png} (100%) rename external/sentinel-6_opendap_access_gridding_files/figure-html/{cell-28-output-1.png => cell-28-output-2.png} (100%) rename external/sentinel-6_opendap_access_gridding_files/figure-html/{cell-29-output-1.png => cell-29-output-2.png} (100%) rename external/sentinel-6_opendap_access_gridding_files/figure-html/{cell-30-output-1.png => cell-30-output-2.png} (100%) delete mode 100644 external/setup.html delete mode 100644 external/zarr_access.html delete mode 100644 external/zarr_access_files/figure-html/cell-16-output-1.png delete mode 100644 external/zarr_access_files/figure-html/cell-22-output-2.png rename tutorials/04_On-Prem_Cloud_files/figure-html/{cell-27-output-1.png => cell-27-output-2.png} (100%) rename tutorials/04_On-Prem_Cloud_files/figure-html/{cell-33-output-1.png => cell-33-output-2.png} (100%) rename tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example_files/figure-html/{cell-11-output-1.png => cell-11-output-2.png} (100%) diff --git a/.nojekyll b/.nojekyll index 76546c0f3..466111a38 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -75abb083 \ No newline at end of file +dfd3984c \ No newline at end of file diff --git a/LICENSE.html b/LICENSE.html index 7933af3e0..0e13b7647 100644 --- a/LICENSE.html +++ b/LICENSE.html @@ -2,12 +2,12 @@ - + -EarthData Cloud Cookbook +EarthData Cloud Cookbook – license - + @@ -82,13 +82,7 @@ "collapse-after": 3, "panel-placement": "start", "type": "textbox", - "limit": 50, - "keyboard-shortcut": [ - "f", - "/", - "s" - ], - "show-item-context": false, + "limit": 20, "language": { "search-no-results-text": "No results", "search-matching-documents-text": "matching documents", @@ -97,14 +91,13 @@ "search-more-match-text": "more match in this document", "search-more-matches-text": "more matches in this document", "search-clear-button-title": "Clear", - "search-text-placeholder": "", "search-detached-cancel-button-title": "Cancel", "search-submit-button-title": "Submit", "search-label": "Search" } } - + @@ -118,12 +111,12 @@
-

Summary

Welcome to the NASA Earthdata Cloud Clinic.

@@ -678,7 +669,7 @@

Prerequisites

You’ll also need to be aware that data in NASA’s Earthdata Cloud reside in Amazon Web Services (AWS) Simple Storage Service (S3) buckets. Access is provided via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region. While this compute location is required for direct S3 access, all data in Earthdata Cloud are still freely available via download.

Import Required Packages

-
+
# Suppress warnings
 import warnings
 warnings.simplefilter('ignore')
@@ -702,6 +693,7 @@ 
Import Required P
   from holoviews.operation.datashader import (
+
-
+
@@ -3389,15 +3381,10 @@

Access data

- - 2 2160 18 - + -
- -
  • Lat_bounds
    @@ -3410,18 +3397,7 @@

    Access data

    dask.array<chunksize=(1, 960, 2), meta=np.ndarray>
    - - - - +
    units :
    @@ -3471,15 +3447,11 @@

    Access data

    - - 2 960 18 - +
    -
    • -
  • Time_bounds
    @@ -3492,18 +3464,7 @@

    Access data

    dask.array<chunksize=(1, 2), meta=np.ndarray>
    - - - - +
    comment :
    @@ -3549,15 +3510,11 @@

    Access data

    - - 2 18 - +
    -
    • -
  • SLA
    @@ -3570,18 +3527,7 @@

    Access data

    dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>
    - - - - +
    units :
    @@ -3639,15 +3585,11 @@

    Access data

    - - 2160 960 18 - +
    -
    • -
  • SLA_ERR
    @@ -3660,18 +3602,7 @@

    Access data

    dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>
    - - - - +
    units :
    @@ -3721,14 +3652,12 @@

    Access data

    - - 2160 960 18 - +
    -
    • Longitude
      PandasIndex
      PandasIndex(Index([0.0833333358168602,               0.25, 0.4166666567325592,
+
    • Longitude
      PandasIndex
      PandasIndex(Index([0.0833333358168602,               0.25, 0.4166666567325592,
        0.5833333134651184,               0.75, 0.9166666865348816,
        1.0833333730697632,               1.25, 1.4166666269302368,
        1.5833333730697632,
@@ -3755,19 +3684,18 @@ 
      Access data
      
                '2021-09-03 12:00:00', '2021-09-08 12:00:00',
                '2021-09-13 12:00:00', '2021-09-18 12:00:00',
                '2021-09-23 12:00:00', '2021-09-28 12:00:00'],
-              dtype='datetime64[ns]', name='Time', freq=None))
  • Conventions :
    CF-1.6
    ncei_template_version :
    NCEI_NetCDF_Grid_Template_v2.0
    Institution :
    Jet Propulsion Laboratory
    geospatial_lat_min :
    -79.916664
    geospatial_lat_max :
    79.916664
    geospatial_lon_min :
    0.083333336
    geospatial_lon_max :
    359.91666
    time_coverage_start :
    2021-07-05
    time_coverage_end :
    2021-07-05
    date_created :
    2022-10-30T20:58:16.297288
    title :
    Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).
    short_name :
    SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205
    long_name :
    MEaSUREs Gridded Sea Surface Height Anomalies Version 2205
    summary :
    Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance.
    DOI :
    10.5067/SLREF-CDRV3
    version_number :
    2205
    Data_Pnts_Each_Sat :
    {"16": 743215, "1007": 674076}
    source_version :
    commit 58c7da13c0c0069ae940c33a82bf1544b7d991bf
    SLA_Global_MEAN :
    0.06428374482174487
    SLA_Global_STD :
    0.0905195660534004
    latency :
    final
    • + dtype='datetime64[ns]', name='Time', freq=None))
  • Conventions :
    CF-1.6
    ncei_template_version :
    NCEI_NetCDF_Grid_Template_v2.0
    Institution :
    Jet Propulsion Laboratory
    geospatial_lat_min :
    -79.916664
    geospatial_lat_max :
    79.916664
    geospatial_lon_min :
    0.083333336
    geospatial_lon_max :
    359.91666
    time_coverage_start :
    2021-07-05
    time_coverage_end :
    2021-07-05
    date_created :
    2022-10-30T20:58:16.297288
    title :
    Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).
    short_name :
    SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205
    long_name :
    MEaSUREs Gridded Sea Surface Height Anomalies Version 2205
    summary :
    Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance.
    DOI :
    10.5067/SLREF-CDRV3
    version_number :
    2205
    Data_Pnts_Each_Sat :
    {"16": 743215, "1007": 674076}
    source_version :
    commit 58c7da13c0c0069ae940c33a82bf1544b7d991bf
    SLA_Global_MEAN :
    0.06428374482174487
    SLA_Global_STD :
    0.0905195660534004
    latency :
    final
    • -
    - +
    -

    Subset data

    +

    Subset data

    We can create a subsetted xarray dataset by extracting the SLA variable and slicing the dataset by a smaller area of interest near the state of Baja, Mexico.

    -
    +
    ds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) 
 ds_subset
    -
    +
    @@ -4178,9 +4106,7 @@

    Subset data

    - - 156 120 18 - + @@ -4269,35 +4195,30 @@

    Subset data

    dtype='datetime64[ns]', name='Time', freq=None))
  • units :
    m
    long_name :
    Sea Level Anomaly Estimate
    standard_name :
    sea_surface_height_above_sea_level
    alias :
    sea_surface_height_above_sea_level
    • -

    Use the built-in plotting function of xarray to create a plot of SLA standard deviation over time:

    -
    +
    ds_subset.std('Time').plot(figsize=(10,6), x='Longitude', y='Latitude');
    -
    -
    -

    -
    -
    +

    - +
    -

    2. harmony-py + xarray

    +

    2. harmony-py + xarray

    We will be accessing a subsetted NetCDF-4 file using Transformation Services in the Cloud.

    Using Earthdata Search, we can find datasets that support these services using the “Customizable” filter.

    We will find, request, and open customized data using Harmony, below.

    -

    Define area of interest

    +

    Define area of interest

    First, use geopandas to read in a geojson file containing a polygon feature over the Gulf of Mexico. The geojson file is found in the ~/data directory.

    -
    +
    geojson_path = './data/gulf.json'
 gdf = gpd.read_file(geojson_path) #Return a GeoDataFrame object

    We can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map. We can view this with the * operator.

    -
    +
    base = gv.tile_sources.EsriImagery.opts(width=650, height=500)
 ocean_map = gv.Polygons(gdf).opts(line_color='yellow', line_width=5, color=None)
 base * ocean_map
    @@ -4345,15 +4266,15 @@

    Define area of interest

    -

    Create a subset request

    +

    Create a subset request

    Here we’ll create a request for a spatial subset of data.

    First, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.

    When creating the Client, we need to provide Earthdata Login credentials. This basic line below assumes that we have a .netrc available. See the Earthdata Cloud Cookbook appendix for more information on Earthdata Login and netrc setup.

    -
    +
    harmony_client = Client()

    See the harmony-py documentation for details on how to construct your request.

    -
    +
    sst_short_name="MUR-JPL-L4-GLOB-v4.1"
 
 request = Request(
@@ -4367,9 +4288,9 @@ 
    Create a subset request
    -

    Submit a subset request

    +

    Submit a subset request

    Now that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.

    -
    +
    job_id = harmony_client.submit(request)
 job_id
    @@ -4378,9 +4299,9 @@

    Submit a subset request

    -

    Check request status

    +

    Check request status

    Depending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.

    -
    +
    harmony_client.wait_for_processing(job_id, show_progress=True)
     [ Processing: 100% ] |###################################################| [|]
    @@ -4388,7 +4309,7 @@

    Check request status

    We can view Harmony job response and output URLs Once the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.

    result_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.

    -
    +
    data = harmony_client.result_json(job_id)
 pprint(data)
    @@ -4421,11 +4342,11 @@

    Check request status

    -

    Access data

    +

    Access data

    Just like above, the subsetted outputs produced by Harmony can be accessed directly from the cloud.

    First, we’ll retrieve list of output URLs.

    The result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.

    -
    +
    results = harmony_client.result_urls(job_id, link_type=LinkType.s3)
 urls = list(results)
 url = urls[0]
@@ -4435,14 +4356,14 @@ 
    Access data

    Using aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.

    -
    +
    creds = harmony_client.aws_credentials()
    -

    Open staged files with s3fs and xarray

    +

    Open staged files with s3fs and xarray

    We use the AWS s3fs package to create a file system that can then be read by xarray:

    -
    +
    s3_fs = s3fs.S3FileSystem(
     key=creds['aws_access_key_id'],
     secret=creds['aws_secret_access_key'],
@@ -4451,11 +4372,12 @@ 
    Open staged files wi
 )

    Now that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.

    -
    +
    f = s3_fs.open(url, mode='rb')
 ds = xr.open_dataset(f)
 ds
    +
    @@ -4869,26 +4791,22 @@

    Open staged files wi

    As before, we use the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset. We can see that the data are subsetted to our polygon:

    -
    +
    ds.analysed_sst.plot();
    -
    -
    -

    -
    -
    +

    -

    Additional Resources

    +

    Additional Resources

    -

    Tutorials

    +

    Tutorials

    This clinic was based off of several notebook tutorials including those presented during past workshop events, along with other materials co-created by the NASA Openscapes mentors: * 2021 Earthdata Cloud Hackathon * 2021 AGU Workshop * Accessing and working with ICESat-2 data in the cloud * Analyzing Sea Level Rise Using Earth Data in the Cloud

    -

    Cloud services

    +

    Cloud services

    The examples used in the clinic provide an abbreviated and simplified workflow to explore access and subsetting options available through the Earthdata Cloud. There are several other options that can be used to interact with data in the Earthdata Cloud including:

    • OPeNDAP @@ -4904,7 +4822,7 @@

      Cloud services

    -

    Support

    +

    Support

    - + - +