Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add backend intro and how-to diagram #9175

Merged
merged 20 commits into from
Jul 22, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/internals/how-to-add-new-backend.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ How to add a new backend
------------------------

Adding a new backend for read support to Xarray does not require
to integrate any code in Xarray; all you need to do is:
one to integrate any code in Xarray; all you need to do is:

- Create a class that inherits from Xarray :py:class:`~xarray.backends.BackendEntrypoint`
and implements the method ``open_dataset`` see :ref:`RST backend_entrypoint`
Expand Down
75 changes: 75 additions & 0 deletions doc/user-guide/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,81 @@ format (recommended).

np.random.seed(123456)

You can `read different types of files <https://docs.xarray.dev/en/stable/user-guide/io.html>`_
in `xr.open_dataset` by specifying the engine to be used:

.. ipython:: python
:suppress:
JessicaS11 marked this conversation as resolved.
Show resolved Hide resolved

import xarray as xr

xr.open_dataset("my_file.grib", engine="cfgrib")
JessicaS11 marked this conversation as resolved.
Show resolved Hide resolved

The "engine" provides a set of instructions that tells xarray how
to read the data and pack them into a `dataset` (or `dataarray`).
These instructions are stored in an underlying "backend".

Xarray comes with several backends that cover many common data formats.
Many more backends are available via external libraries, or you can `write your own <https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html>`_.
This diagram aims to help you determine - based on the format of the file you'd like to read -
which type of backend you're using and how to use it.

The rectangular boxes are clickable for more information.
Following the diagram is detailed information on many popular backends.
You can learn more about using and developing backends in the
`Xarray tutorial JupyterBook <https://tutorial.xarray.dev/advanced/backends/backends.html>`_.

.. mermaid::
:alt: Flowchart illustrating how to choose the right backend engine to read your data

flowchart LR
built-in-eng["""Is your data stored in one of these formats?
- netCDF4 (<code>netcdf4</code>)
- netCDF3 (<code>scipy</code>)
- Zarr (<code>zarr</code>)
- DODS/OPeNDAP (<code>pydap</code>)
- HDF5 (<code>h5netcdf</code>)
"""]

built-in("""You're in luck! Xarray bundles a backend for this format.
Open data using <code>xr.open_dataset()</code>. We recommend
always setting the engine you want to use.""")

installed-eng["""One of these formats?
- GRIB (<code>cfgrib</code>)
- TileDB (<code>tiledb</code>)
- GeoTIFF, JPEG-2000, ESRI-hdf (<code>rioxarray</code>, via GDAL)
- Sentinel-1 SAFE (<code>xarray-sentinel</code>)
"""]

installed("""Install the package indicated in parentheses to your
Python environment. Restart the kernel and use
<code>xr.open_dataset(files, engine='rioxarray')</code>.""")

other("""Ask around to see if someone in your data community
has created an Xarray backend for your data type.
If not, you may need to create your own or consider
exporting your data to a more common format.""")

built-in-eng -->|Yes| built-in
built-in-eng -->|No| installed-eng

installed-eng -->|Yes| installed
installed-eng -->|No| other

click built-in-eng "https://docs.xarray.dev/en/stable/getting-started-guide/faq.html#how-do-i-open-format-x-file-as-an-xarray-dataset"
click installed-eng "https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html#rioxarray"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm looking at https://xray--9175.org.readthedocs.build/en/9175/user-guide/io.html — I think this means that clicking anywhere in One of these formats? links to the RioXarray docs...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - unfortunately you can only add one link per node. My [hesitant] intent was to give users a clickable example, but if you think it's clearer to not have links embedded in the diagram (and either add them as a list after or not at all) I wouldn't argue.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think that's a bit confusing... Any chance it takes markdown links?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't on my first attempt, but it takes html ones!

click other "https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html"

classDef quesNodefmt fill:#9DEEF4,stroke:#206C89,text-align:left
class built-in-eng,installed-eng quesNodefmt

classDef ansNodefmt fill:#FFAA05,stroke:#E37F17,text-align:left,white-space:nowrap
class built-in,installed,other ansNodefmt

linkStyle default font-size:20pt,color:#206C89


.. _io.netcdf:

netCDF
Expand Down
4 changes: 3 additions & 1 deletion doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ Bug fixes
Documentation
~~~~~~~~~~~~~

- Adds a flow-chart diagram to help users navigate help resources (`Discussion #8990 <https://github.com/pydata/xarray/discussions/8990>`_).
- Adds intro to backend section of docs, including a flow-chart to navigate types of backends (:pull:`9175`).
By `Jessica Scheick <https://github.com/jessicas11>`_.
- Adds a flow-chart diagram to help users navigate help resources (`Discussion #8990 <https://github.com/pydata/xarray/discussions/8990>`_, :pull:`9147`).
By `Jessica Scheick <https://github.com/jessicas11>`_.
- Improvements to Zarr & chunking docs (:pull:`9139`, :pull:`9140`, :pull:`9132`)
By `Maximilian Roos <https://github.com/max-sixty>`_.
Expand Down
Loading