From d4955906021fc244ee2c8a9ab2942623748f1ffc Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Tue, 25 Jun 2024 16:00:12 -0400 Subject: [PATCH 01/16] add backend intro and how-to diagram --- doc/internals/how-to-add-new-backend.rst | 2 +- doc/user-guide/io.rst | 74 ++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/doc/internals/how-to-add-new-backend.rst b/doc/internals/how-to-add-new-backend.rst index 4352dd3df5b..a979abe34e2 100644 --- a/doc/internals/how-to-add-new-backend.rst +++ b/doc/internals/how-to-add-new-backend.rst @@ -4,7 +4,7 @@ How to add a new backend ------------------------ Adding a new backend for read support to Xarray does not require -to integrate any code in Xarray; all you need to do is: +one to integrate any code in Xarray; all you need to do is: - Create a class that inherits from Xarray :py:class:`~xarray.backends.BackendEntrypoint` and implements the method ``open_dataset`` see :ref:`RST backend_entrypoint` diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index da414bc383e..9699239ab2e 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -19,6 +19,80 @@ format (recommended). np.random.seed(123456) +You can [read different types of files](https://docs.xarray.dev/en/stable/user-guide/io.html) +in `xr.open_dataset` by specifying the engine to be used: + +```python +import xarray as xr +xr.open_dataset("my_file.grib" , engine="cfgrib") +``` + +The "engine" provides a set of instructions that tells xarray how +to read the data and pack them into a `dataset` (or `dataarray`). +These instructions are stored in an underlying "backend". + +Xarray comes with several backends that cover many common data formats. +Many more backends are available via external libraries, or you can write your own. +This diagram aims to help you determine - based on the format of the file you'd like to read - +which type of backend you're using and how to use it. + +The rectangular boxes are clickable for more information. +Following the diagram is detailed information on many popular backends. +You can learn more about using and developing backends in the +`Xarray tutorial JupyterBook _`. + +.. mermaid:: + :alt: Flowchart illustrating how to choose the right backend engine to read your data + + flowchart TD + built-in-eng[Is your data stored in one of these formats? + netCDF4 (**netcdf4**) + netCDF3 (**scipy**) + Zarr (**zarr**) + DODS/OPeNDAP (**pydap**) + HDF5 (**h5netcdf**) + ] + + built-in("`You're in luck! Xarray bundles a backend for this format. + Open data using *xr.open_dataset()*. We recommend + always setting the engine you want to use.`") + + installed-eng["`One of these formats (this list is not inclusive)? + GRIB (**cfgrib**) + TileDB (**tiledb**) + GeoTIFF, JPEG-2000, ESRI-hdf (**rioxarray**, via GDAL) + Sentinel-1 SAFE (**xarray-sentinel**) + ... + `"] + + installed("`Install the package indicated in parentheses + to your Python environment. Restart the kernel + and use *xr.open_dataset(files, engine='rioxarray')*`") + + other("`Ask around to see if someone in your data community + has created an Xarray backend for your data type. + If not, you may need to create your own or consider + exporting your data to a more common format.`") + + built-in-eng -->|Yes| built-in + built-in-eng -->|No| installed-eng + + installed-eng -->|Yes| installed + installed-eng -->|No| other + + click built-in-eng "https://docs.xarray.dev/en/stable/getting-started-guide/faq.html#how-do-i-open-format-x-file-as-an-xarray-dataset" + click installed-eng "https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html#rioxarray" + click other "https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html" + + classDef quesNodefmt fill:#9DEEF4,stroke:#206C89 + class built-in-eng,installed-eng quesNodefmt + + classDef ansNodefmt fill:#FFAA05,stroke:#E37F17 + class built-in,installed,other ansNodefmt + + linkStyle default font-size:20pt,color:#206C89 + + .. _io.netcdf: netCDF From 5a7d659cd68187425382510379d7fb2320a659e6 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Tue, 25 Jun 2024 16:21:05 -0400 Subject: [PATCH 02/16] update what's new --- doc/whats-new.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/whats-new.rst b/doc/whats-new.rst index c3383a5648a..cbf1f11b22d 100644 --- a/doc/whats-new.rst +++ b/doc/whats-new.rst @@ -40,7 +40,9 @@ Bug fixes Documentation ~~~~~~~~~~~~~ -- Adds a flow-chart diagram to help users navigate help resources (`Discussion #8990 `_). +- Adds intro to backend section of docs, including a flow-chart to navigate types of backends (:pull:`9175`). + By `Jessica Scheick `_. +- Adds a flow-chart diagram to help users navigate help resources (`Discussion #8990 `_, :pull:`9147`). By `Jessica Scheick `_. - Improvements to Zarr & chunking docs (:pull:`9139`, :pull:`9140`, :pull:`9132`) By `Maximilian Roos `_ From 05620d36cd6ba94d28077120f61de4c57db99040 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Tue, 25 Jun 2024 16:21:37 -0400 Subject: [PATCH 03/16] fix link style [skip-ci] --- doc/user-guide/io.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 9699239ab2e..8d5d34064a5 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -19,7 +19,7 @@ format (recommended). np.random.seed(123456) -You can [read different types of files](https://docs.xarray.dev/en/stable/user-guide/io.html) +You can `read different types of files `_ in `xr.open_dataset` by specifying the engine to be used: ```python @@ -32,7 +32,7 @@ to read the data and pack them into a `dataset` (or `dataarray`). These instructions are stored in an underlying "backend". Xarray comes with several backends that cover many common data formats. -Many more backends are available via external libraries, or you can write your own. +Many more backends are available via external libraries, or you can `write your own `_. This diagram aims to help you determine - based on the format of the file you'd like to read - which type of backend you're using and how to use it. From bf4ee8eca36601a38b4329180930a63ead840d45 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Tue, 25 Jun 2024 16:38:34 -0400 Subject: [PATCH 04/16] update numpy nan syntax to address docs build fail [skip-ci] --- doc/user-guide/computation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/computation.rst b/doc/user-guide/computation.rst index f99d41be538..d9b8316427a 100644 --- a/doc/user-guide/computation.rst +++ b/doc/user-guide/computation.rst @@ -426,7 +426,7 @@ However, the functions also take missing values in the data into account: .. ipython:: python - data = xr.DataArray([np.NaN, 2, 4]) + data = xr.DataArray([np.nan, 2, 4]) weights = xr.DataArray([8, 1, 1]) data.weighted(weights).mean() From ae473e068ae942d5d36af35a99654c439ff31b38 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Thu, 27 Jun 2024 14:20:07 -0400 Subject: [PATCH 05/16] improve some spacing on diagram --- doc/user-guide/io.rst | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 8d5d34064a5..564f3ea16a4 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -44,25 +44,25 @@ You can learn more about using and developing backends in the .. mermaid:: :alt: Flowchart illustrating how to choose the right backend engine to read your data - flowchart TD - built-in-eng[Is your data stored in one of these formats? - netCDF4 (**netcdf4**) - netCDF3 (**scipy**) - Zarr (**zarr**) - DODS/OPeNDAP (**pydap**) - HDF5 (**h5netcdf**) - ] + flowchart LR + built-in-eng["`Is your data stored in one of these formats? + fa:fa-check-square netCDF4 (**netcdf4**) + fa:fa-check-square netCDF3 (**scipy**) + fa:fa-check-square Zarr (**zarr**) + fa:fa-check-square DODS/OPeNDAP (**pydap**) + fa:fa-check-square HDF5 (**h5netcdf**) + `"] built-in("`You're in luck! Xarray bundles a backend for this format. Open data using *xr.open_dataset()*. We recommend always setting the engine you want to use.`") - installed-eng["`One of these formats (this list is not inclusive)? - GRIB (**cfgrib**) - TileDB (**tiledb**) - GeoTIFF, JPEG-2000, ESRI-hdf (**rioxarray**, via GDAL) - Sentinel-1 SAFE (**xarray-sentinel**) - ... + installed-eng["`One of these formats? + fa:fa-check-square GRIB (**cfgrib**) + fa:fa-check-square TileDB (**tiledb**) + fa:fa-check-square GeoTIFF, JPEG-2000, ESRI-hdf (**rioxarray**, via GDAL) + fa:fa-check-square Sentinel-1 SAFE (**xarray-sentinel**) + fa:fa-check-square ... `"] installed("`Install the package indicated in parentheses @@ -84,10 +84,10 @@ You can learn more about using and developing backends in the click installed-eng "https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html#rioxarray" click other "https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html" - classDef quesNodefmt fill:#9DEEF4,stroke:#206C89 + classDef quesNodefmt fill:#9DEEF4,stroke:#206C89,text-align:left class built-in-eng,installed-eng quesNodefmt - classDef ansNodefmt fill:#FFAA05,stroke:#E37F17 + classDef ansNodefmt fill:#FFAA05,stroke:#E37F17,text-align:left,white-space:nowrap class built-in,installed,other ansNodefmt linkStyle default font-size:20pt,color:#206C89 From ef71bec4414dfeb5dd77eb53c8acc1c4bc3f76f2 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Thu, 27 Jun 2024 15:20:16 -0400 Subject: [PATCH 06/16] fix items not rendering properly due to typos [skip-ci] --- doc/user-guide/computation.rst | 2 +- doc/user-guide/io.rst | 9 ++++----- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/doc/user-guide/computation.rst b/doc/user-guide/computation.rst index d9b8316427a..f99d41be538 100644 --- a/doc/user-guide/computation.rst +++ b/doc/user-guide/computation.rst @@ -426,7 +426,7 @@ However, the functions also take missing values in the data into account: .. ipython:: python - data = xr.DataArray([np.nan, 2, 4]) + data = xr.DataArray([np.NaN, 2, 4]) weights = xr.DataArray([8, 1, 1]) data.weighted(weights).mean() diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 564f3ea16a4..e2e69cec037 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -22,10 +22,9 @@ format (recommended). You can `read different types of files `_ in `xr.open_dataset` by specifying the engine to be used: -```python -import xarray as xr -xr.open_dataset("my_file.grib" , engine="cfgrib") -``` +.. ipython:: python + import xarray as xr + xr.open_dataset("my_file.grib" , engine="cfgrib") The "engine" provides a set of instructions that tells xarray how to read the data and pack them into a `dataset` (or `dataarray`). @@ -39,7 +38,7 @@ which type of backend you're using and how to use it. The rectangular boxes are clickable for more information. Following the diagram is detailed information on many popular backends. You can learn more about using and developing backends in the -`Xarray tutorial JupyterBook _`. +`Xarray tutorial JupyterBook `_. .. mermaid:: :alt: Flowchart illustrating how to choose the right backend engine to read your data From 375894722df501dcb2ce3bd3babcea36598ac239 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 27 Jun 2024 19:21:28 +0000 Subject: [PATCH 07/16] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/user-guide/io.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index e2e69cec037..fcf37bf8e10 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -23,8 +23,10 @@ You can `read different types of files Date: Mon, 1 Jul 2024 18:03:01 -0400 Subject: [PATCH 08/16] update python syntax --- doc/user-guide/io.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index fcf37bf8e10..c2d3bd91d56 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -25,7 +25,6 @@ in `xr.open_dataset` by specifying the engine to be used: .. ipython:: python import xarray as xr - xr.open_dataset("my_file.grib", engine="cfgrib") The "engine" provides a set of instructions that tells xarray how From b9c1e02cea13d6e7b7fcfdde4ff46232a69853f3 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Mon, 1 Jul 2024 18:12:30 -0400 Subject: [PATCH 09/16] use html code blocks to fix spacing [skip-ci] --- doc/user-guide/io.rst | 51 +++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index c2d3bd91d56..6edeb6c41c7 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -45,34 +45,33 @@ You can learn more about using and developing backends in the :alt: Flowchart illustrating how to choose the right backend engine to read your data flowchart LR - built-in-eng["`Is your data stored in one of these formats? - fa:fa-check-square netCDF4 (**netcdf4**) - fa:fa-check-square netCDF3 (**scipy**) - fa:fa-check-square Zarr (**zarr**) - fa:fa-check-square DODS/OPeNDAP (**pydap**) - fa:fa-check-square HDF5 (**h5netcdf**) - `"] - - built-in("`You're in luck! Xarray bundles a backend for this format. - Open data using *xr.open_dataset()*. We recommend - always setting the engine you want to use.`") - - installed-eng["`One of these formats? - fa:fa-check-square GRIB (**cfgrib**) - fa:fa-check-square TileDB (**tiledb**) - fa:fa-check-square GeoTIFF, JPEG-2000, ESRI-hdf (**rioxarray**, via GDAL) - fa:fa-check-square Sentinel-1 SAFE (**xarray-sentinel**) - fa:fa-check-square ... - `"] - - installed("`Install the package indicated in parentheses - to your Python environment. Restart the kernel - and use *xr.open_dataset(files, engine='rioxarray')*`") - - other("`Ask around to see if someone in your data community + built-in-eng["""Is your data stored in one of these formats? + - netCDF4 (netcdf4) + - netCDF3 (scipy) + - Zarr (zarr) + - DODS/OPeNDAP (pydap) + - HDF5 (h5netcdf) + """] + + built-in("""You're in luck! Xarray bundles a backend for this format. + Open data using xr.open_dataset(). We recommend + always setting the engine you want to use.""") + + installed-eng["""One of these formats? + - GRIB (cfgrib) + - TileDB (tiledb) + - GeoTIFF, JPEG-2000, ESRI-hdf (rioxarray, via GDAL) + - Sentinel-1 SAFE (xarray-sentinel) + """] + + installed("""Install the package indicated in parentheses to your + Python environment. Restart the kernel and use + xr.open_dataset(files, engine='rioxarray').""") + + other("""Ask around to see if someone in your data community has created an Xarray backend for your data type. If not, you may need to create your own or consider - exporting your data to a more common format.`") + exporting your data to a more common format.""") built-in-eng -->|Yes| built-in built-in-eng -->|No| installed-eng From 94f4987fe1b3663e5a8b6bbf83c381466786ab8a Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 1 Jul 2024 22:13:22 +0000 Subject: [PATCH 10/16] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/user-guide/io.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 6edeb6c41c7..a4e5df5f6b8 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -25,6 +25,7 @@ in `xr.open_dataset` by specifying the engine to be used: .. ipython:: python import xarray as xr + xr.open_dataset("my_file.grib", engine="cfgrib") The "engine" provides a set of instructions that tells xarray how From 1ee7bc3ba698525b38946b225689626a92523b0b Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Fri, 12 Jul 2024 18:19:33 -0400 Subject: [PATCH 11/16] suppress ipython block output --- doc/user-guide/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index a4e5df5f6b8..569706ae5b6 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -23,7 +23,7 @@ You can `read different types of files Date: Fri, 12 Jul 2024 22:20:06 +0000 Subject: [PATCH 12/16] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/user-guide/io.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 569706ae5b6..12be51b7862 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -24,6 +24,7 @@ in `xr.open_dataset` by specifying the engine to be used: .. ipython:: python :suppress: + import xarray as xr xr.open_dataset("my_file.grib", engine="cfgrib") From 7e2a84b2f6f4394e18d036a8fdb215497c2ddcb2 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Sat, 13 Jul 2024 13:06:23 -0400 Subject: [PATCH 13/16] change to okexcept [skip-ci] --- doc/user-guide/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 12be51b7862..5f4db4cb2a4 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -23,7 +23,7 @@ You can `read different types of files Date: Sat, 13 Jul 2024 14:11:44 -0400 Subject: [PATCH 14/16] re-add supress so error output is ignored [skip-ci] --- doc/user-guide/io.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 5f4db4cb2a4..0cb43b4a485 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -24,6 +24,7 @@ in `xr.open_dataset` by specifying the engine to be used: .. ipython:: python :okexcept: + :supress: import xarray as xr From 376f36057fed66cd6cc20a06f79ff81959307afd Mon Sep 17 00:00:00 2001 From: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> Date: Sat, 13 Jul 2024 13:43:22 -0700 Subject: [PATCH 15/16] Update doc/user-guide/io.rst --- doc/user-guide/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 0cb43b4a485..0e780362a38 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -24,7 +24,7 @@ in `xr.open_dataset` by specifying the engine to be used: .. ipython:: python :okexcept: - :supress: + :suppress: import xarray as xr From 72ce1fd1bb180f6435b4ff02733be3fd53fa6cb5 Mon Sep 17 00:00:00 2001 From: Jessica Scheick Date: Mon, 22 Jul 2024 11:00:09 -0400 Subject: [PATCH 16/16] add per-line links to diagram [skip-ci] --- doc/user-guide/io.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 0e780362a38..85c47334858 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -39,7 +39,7 @@ Many more backends are available via external libraries, or you can `write your This diagram aims to help you determine - based on the format of the file you'd like to read - which type of backend you're using and how to use it. -The rectangular boxes are clickable for more information. +Text and boxes are clickable for more information. Following the diagram is detailed information on many popular backends. You can learn more about using and developing backends in the `Xarray tutorial JupyterBook `_. @@ -61,10 +61,10 @@ You can learn more about using and developing backends in the always setting the engine you want to use.""") installed-eng["""One of these formats? - - GRIB (cfgrib) - - TileDB (tiledb) - - GeoTIFF, JPEG-2000, ESRI-hdf (rioxarray, via GDAL) - - Sentinel-1 SAFE (xarray-sentinel) + - GRIB (cfgrib) + - TileDB (tiledb) + - GeoTIFF, JPEG-2000, ESRI-hdf (rioxarray, via GDAL) + - Sentinel-1 SAFE (xarray-sentinel) """] installed("""Install the package indicated in parentheses to your @@ -83,7 +83,6 @@ You can learn more about using and developing backends in the installed-eng -->|No| other click built-in-eng "https://docs.xarray.dev/en/stable/getting-started-guide/faq.html#how-do-i-open-format-x-file-as-an-xarray-dataset" - click installed-eng "https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html#rioxarray" click other "https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html" classDef quesNodefmt fill:#9DEEF4,stroke:#206C89,text-align:left