Skip to content

Commit

Permalink
update backends tutorials intro (#266)
Browse files Browse the repository at this point in the history
* add draft chart to markdown file

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add mermaid sphinx extension

* revise backend content

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add some color styling

* add sphinx mermaid package to env

* save conflict changes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update conda lock files

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add myst parser to use mermaid lexer with sphinx

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix dash to underscore

* try changing order of extensions

* add source_suffix sphinx config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add curly braces

* remove diagram (moved to docs) and update text

* revert environment changes to main

* add h5netcdf to list of backends

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
JessicaS11 and pre-commit-ci[bot] authored Sep 23, 2024
1 parent 516bcec commit 98b287b
Showing 1 changed file with 29 additions and 24 deletions.
53 changes: 29 additions & 24 deletions advanced/backends/backends.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,49 @@
# Creating new backends
# Reading data using backends

## Introduction

You can [read different type of files](https://docs.xarray.dev/en/stable/user-guide/io.html)) in `xr.open_dataset` by specifying the engine to be used:
You can [read different types of files](https://docs.xarray.dev/en/stable/user-guide/io.html) in `xr.open_dataset` by specifying the engine to be used:

```python
import xarray as xr
xr.open_dataset("my_file.grib" , engine="cfgrib")
```

For each available engine there is an underlying backend, that reads the data and pack them in a dataset.
Navigating Xarray backends can be confusing,
so we recommend checking out [this flow chart](https://docs.xarray.dev/en/stable/user-guide/io.html)
to help you figure out which engine you need and how to use it.

Xarray bundles several backends internally for the following formats:
You can see what backends are currently available in your working environment
with `xarray.backends.list_engines()`.

- netcdf4 - netCDF4
- scipy - netCDF3
- zarr - Zarr
- pydap - DAP
- ...
## Why use the Xarray backend API to write your own backend?

External Backends that use the new backend API (xarray >= v0.18.0) that allows to add support for backend without any change to Xarray
- Your users don't need to learn a new interface; they can use `xr.open_dataset` with the `engine` kwarg.
- With little extra effort you can have lazy loading with Dask. Simply implement a function for reading blocks and Xarray will manage lazy loading with Dask for you.
- It's easy to implement: using the backend API (introduced in v0.18.0), you don't need to integrate any code in Xarray.

- [cfgrib](https://github.com/ecmwf/cfgrib) - GRIB
- [tiledb](https://github.com/TileDB-Inc/TileDB-CF-Py) - TileDB
- [rioxarray](https://corteva.github.io/rioxarray/stable/) - GeoTIFF, JPEG-2000, ESRI-hdr, etc (via GDAL)
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel) - Sentinel-1 SAFE
- ...
## More Information

## Why using the Xarray backend API
See the [documentation](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html) for more details on adding and registering a new backend.

- Your users don't need to learn a new interface that is they can use `xr.open_dataset` with `engine` kwarg.
- With little extra effort you can have lazy loading with Dask. you have to implement a function for reading blocks and Xarray will manage lazy loading with Dask for you
- It's easy to implement: you don't need to integrate any code in Xarray
Follow the tutorials on creating a new backend for binary files.

```{tableofcontents}
## Next
```

See the [documentation](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html) for more.
### Links to internal backends

Follow the tutorial on creating a new backend for binary files.
- [netcdf4](https://pypi.org/project/netCDF4/) - netCDF4
- [scipy](https://scipy.org/) - netCDF3
- [zarr](https://pypi.org/project/zarr/) - Zarr
- [pydap](https://pydap.github.io/pydap/) - Data Access Protocol (DAP/DODS/OPeNDAP)
- [h5netcdf](https://h5netcdf.org/) - hdf5

```{tableofcontents}
### Links to external backends (not comprehensive)

```
- [cfgrib](https://github.com/ecmwf/cfgrib) - GRIB
- [tiledb](https://github.com/TileDB-Inc/TileDB-CF-Py) - TileDB
- [rioxarray](https://corteva.github.io/rioxarray/stable/) - GeoTIFF, JPEG-2000, ESRI-hdr, etc (via GDAL)
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel) - Sentinel-1 SAFE
- ...

0 comments on commit 98b287b

Please sign in to comment.