03-nwm_hindcast_intake.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6b9fb9c6",
   "metadata": {},
   "source": [
    "# Explore the National Water Model Reanalysis v2.1 \n",
    "Explore the NWM Reanalysis (1979-2020) NetCDF files (all 367,439 of them) on AWS as a single xarray dataset! \n",
    "The only new file we created was a JSON file that points to data chunks in the original NetCDF files that is then read with the [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) and [zarr](https://zarr.readthedocs.io/en/stable/) packages. \n",
    "\n",
    "See this [blog post](https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935) for how this works. \n",
    "\n",
    "**Important note on performance**: The data in the original NetCDF files is chunked as the entire spatial domain and a single time step.  Thus reading a time series will be very slow -- to extract a time series at a single location for the entire time period will require reading and uncompressing 8TB of data!   But extraction of a few days or weeks of data will be relatively fast. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb6302df",
   "metadata": {},
   "outputs": [],
   "source": [
    "import fsspec\n",
    "import xarray as xr\n",
    "import intake"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e447c3c1",
   "metadata": {},
   "source": [
    "#### Use Intake to load the consolidated NWM dataset\n",
    "The Intake catalog, the consolidated JSON file it accesses, and the NetCDF files the JSON file references are all on public S3 buckets that do not require an AWS account, so no credentials are required!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10610cfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat = intake.open_catalog('./example_jsons/nwm_catalog.yml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6aca0a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = cat['nwm-reanalysis'].to_dask()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f3cb455",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.streamflow.nbytes/1e12 # How many terabytes is a single variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6448953a",
   "metadata": {},
   "source": [
    "#### Read and plot streamflow for a specific time \n",
    "The local National Weather Service office in Houston observed all-time record daily rainfall accumulations on both August 26 and 27, measured at 14.4 in (370 mm) and 16.08 in (408 mm) respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4385d92",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hvplot.pandas\n",
    "import geoviews as gv\n",
    "from holoviews.operation.datashader import rasterize\n",
    "import cartopy.crs as ccrs\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ee11091",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds1 = ds.sel(time='2017-08-27 18:00:00', method='nearest')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e448685",
   "metadata": {},
   "outputs": [],
   "source": [
    "var = 'streamflow'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df1e5525",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = ds1[var].to_pandas().to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe7d20d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "date_title = pd.to_datetime(ds1.time.values).strftime('%Y-%m-%d %H:%M:%S')\n",
    "date_title = f'{var}: {date_title}'\n",
    "date_title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7289d958",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.assign(latitude=ds.latitude)\n",
    "df = df.assign(longitude=ds.longitude)\n",
    "df.rename(columns={0: var}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f065fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = df.hvplot.points(x='longitude', y='latitude', geo=True,\n",
    "                     c=var, colorbar=True, size=14, label=date_title)\n",
    "g = rasterize(p, aggregator='mean', x_sampling=0.02, \n",
    "                y_sampling=0.02, width=500).opts(tools=['hover'], \n",
    "                aspect='equal', logz=True, cmap='viridis', clim=(1e-2, np.nan))\n",
    "g * gv.tile_sources.OSM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0560c60",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "d853cbf2f35f45a59f79ca5e397d8dd1594080251b0b51418fe33f5fb0138a7a"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}