Relax nanosecond datetime restriction in CF time decoding

doc/api.rst

@@ -1096,6 +1096,17 @@ DataTree methods
 ..    Missing:
 ..    ``open_mfdatatree``
 
+Encoding/Decoding
+=================
+
+Coder objects
+-------------
+
+.. autosummary::
+   :toctree: generated/
+
+   coders.CFDatetimeCoder
+
 Coordinates objects
 ===================
 

doc/internals/index.rst

@@ -26,3 +26,4 @@ The pages in this section are intended for:
    how-to-add-new-backend
    how-to-create-custom-index
    zarr-encoding-spec
+   time-coding

doc/user-guide/time-series.rst

@@ -21,20 +21,31 @@ core functionality.
 Creating datetime64 data
 ------------------------
 
-Xarray uses the numpy dtypes ``datetime64[ns]`` and ``timedelta64[ns]`` to
-represent datetime data, which offer vectorized (if sometimes buggy) operations
-with numpy and smooth integration with pandas.
+Xarray uses the numpy dtypes ``datetime64[unit]`` and ``timedelta64[unit]``
+(where unit is one of "s", "ms", "us" and "ns") to represent datetime
+data, which offer vectorized operations with numpy and smooth integration with pandas.
 
 To convert to or create regular arrays of ``datetime64`` data, we recommend
 using :py:func:`pandas.to_datetime` and :py:func:`pandas.date_range`:
 
 .. ipython:: python
 
     pd.to_datetime(["2000-01-01", "2000-02-02"])
+    pd.DatetimeIndex(
+        ["2000-01-01 00:00:00", "2000-02-02 00:00:00"], dtype="datetime64[s]"
+    )
     pd.date_range("2000-01-01", periods=365)
+    pd.date_range("2000-01-01", periods=365, unit="s")
+
+.. note::
+    Care has to be taken to create the output with the wanted resolution.
+    For :py:func:`pandas.date_range` the ``unit``-kwarg has to be specified
+    and for :py:func:`pandas.to_datetime` the selection of the resolution
+    isn't possible at all. For that :py:class:`pd.DatetimeIndex` can be used
+    directly.
 
 Alternatively, you can supply arrays of Python ``datetime`` objects. These get
-converted automatically when used as arguments in xarray objects:
+converted automatically when used as arguments in xarray objects (with us-resolution):
 
 .. ipython:: python
 
@@ -51,7 +62,7 @@ attribute like ``'days since 2000-01-01'``).
 .. note::
 
    When decoding/encoding datetimes for non-standard calendars or for dates
-   before year 1678 or after year 2262, xarray uses the `cftime`_ library.
+   before [1582-10-15](https://en.wikipedia.org/wiki/Gregorian_calendar), xarray uses the `cftime`_ library by default.
    It was previously packaged with the ``netcdf4-python`` package under the
    name ``netcdftime`` but is now distributed separately. ``cftime`` is an
    :ref:`optional dependency<installing>` of xarray.
@@ -66,17 +77,15 @@ You can manual decode arrays in this form by passing a dataset to
 
     attrs = {"units": "hours since 2000-01-01"}
     ds = xr.Dataset({"time": ("time", [0, 1, 2, 3], attrs)})
+    # Default decoding to 'ns'-resolution
     xr.decode_cf(ds)
+    # Decoding to 's'-resolution
+    coder = xr.coders.CFDatetimeCoder(time_unit="s")
+    xr.decode_cf(ds, decode_times=coder)
 
-One unfortunate limitation of using ``datetime64[ns]`` is that it limits the
-native representation of dates to those that fall between the years 1678 and
-2262. When a netCDF file contains dates outside of these bounds, dates will be
-returned as arrays of :py:class:`cftime.datetime` objects and a :py:class:`~xarray.CFTimeIndex`
-will be used for indexing.  :py:class:`~xarray.CFTimeIndex` enables a subset of
-the indexing functionality of a :py:class:`pandas.DatetimeIndex` and is only
-fully compatible with the standalone version of ``cftime`` (not the version
-packaged with earlier versions ``netCDF4``).  See :ref:`CFTimeIndex` for more
-information.
+From xarray TODO: version the resolution of the dates can be tuned between "s", "ms", "us" and "ns". One limitation of using ``datetime64[ns]`` is that it limits the native representation of dates to those that fall between the years 1678 and 2262, which gets increased significantly with lower resolutions. When a netCDF file contains dates outside of these bounds (or dates < 1582-10-15), dates will be returned as arrays of :py:class:`cftime.datetime` objects and a :py:class:`~xarray.CFTimeIndex` will be used for indexing.
+:py:class:`~xarray.CFTimeIndex` enables a subset of the indexing functionality of a :py:class:`pandas.DatetimeIndex`.
+See :ref:`CFTimeIndex` for more information.
 
 Datetime indexing
 -----------------

doc/user-guide/weather-climate.rst

@@ -10,7 +10,7 @@ Weather and climate data
 
     import xarray as xr
 
-Xarray can leverage metadata that follows the `Climate and Forecast (CF) conventions`_ if present. Examples include :ref:`automatic labelling of plots<plotting>` with descriptive names and units if proper metadata is present and support for non-standard calendars used in climate science through the ``cftime`` module(Explained in the :ref:`CFTimeIndex` section). There are also a number of :ref:`geosciences-focused projects that build on xarray<ecosystem>`.
+Xarray can leverage metadata that follows the `Climate and Forecast (CF) conventions`_ if present. Examples include :ref:`automatic labelling of plots<plotting>` with descriptive names and units if proper metadata is present and support for non-standard calendars used in climate science through the ``cftime`` module (explained in the :ref:`CFTimeIndex` section). There are also a number of :ref:`geosciences-focused projects that build on xarray<ecosystem>`.
 
 .. _Climate and Forecast (CF) conventions: https://cfconventions.org
 
@@ -64,8 +64,7 @@ Through the standalone ``cftime`` library and a custom subclass of
 :py:class:`pandas.Index`, xarray supports a subset of the indexing
 functionality enabled through the standard :py:class:`pandas.DatetimeIndex` for
 dates from non-standard calendars commonly used in climate science or dates
-using a standard calendar, but outside the `nanosecond-precision range`_
-(approximately between years 1678 and 2262).
+using a standard calendar, but outside the `precision range`_ and dates [prior to 1582-10-15](https://en.wikipedia.org/wiki/Gregorian_calendar).
 
 .. note::
 
@@ -75,18 +74,14 @@ using a standard calendar, but outside the `nanosecond-precision range`_
    any of the following are true:
 
    - The dates are from a non-standard calendar
-   - Any dates are outside the nanosecond-precision range.
+   - Any dates are outside the nanosecond-precision range (prior xarray version 2024.11)
+   - Any dates are outside the time span limited by the resolution (from xarray version v2024.11)
 
    Otherwise pandas-compatible dates from a standard calendar will be
-   represented with the ``np.datetime64[ns]`` data type, enabling the use of a
-   :py:class:`pandas.DatetimeIndex` or arrays with dtype ``np.datetime64[ns]``
-   and their full set of associated features.
+   represented with the ``np.datetime64[unit]`` data type (where unit can be one of ["s", "ms", "us", "ns"], enabling the use of a :py:class:`pandas.DatetimeIndex` or arrays with dtype ``np.datetime64[unit]`` and their full set of associated features.
 
    As of pandas version 2.0.0, pandas supports non-nanosecond precision datetime
-   values.  For the time being, xarray still automatically casts datetime values
-   to nanosecond-precision for backwards compatibility with older pandas
-   versions; however, this is something we would like to relax going forward.
-   See :issue:`7493` for more discussion.
+   values. From xarray version 2024.11 the relaxed non-nanosecond precision datetime values will be used.
 
 For example, you can create a DataArray indexed by a time
 coordinate with dates from a no-leap calendar and a
@@ -115,7 +110,7 @@ instance, we can create the same dates and DataArray we created above using:
 Mirroring pandas' method with the same name, :py:meth:`~xarray.infer_freq` allows one to
 infer the sampling frequency of a :py:class:`~xarray.CFTimeIndex` or a 1-D
 :py:class:`~xarray.DataArray` containing cftime objects. It also works transparently with
-``np.datetime64[ns]`` and ``np.timedelta64[ns]`` data.
+``np.datetime64`` and ``np.timedelta64`` data (with "s", "ms", "us" or "ns" resolution).
 
 .. ipython:: python
 
@@ -137,7 +132,7 @@ Conversion between non-standard calendar and to/from pandas DatetimeIndexes is
 facilitated with the :py:meth:`xarray.Dataset.convert_calendar` method (also available as
 :py:meth:`xarray.DataArray.convert_calendar`). Here, like elsewhere in xarray, the ``use_cftime``
 argument controls which datetime backend is used in the output. The default (``None``) is to
-use ``pandas`` when possible, i.e. when the calendar is standard and dates are within 1678 and 2262.
+use ``pandas`` when possible, i.e. when the calendar is ``standard``/``gregorian`` and dates [starting with 1582-10-15]((https://en.wikipedia.org/wiki/Gregorian_calendar)). There is no such restriction when converting to ``proleptic_gregorian`` calendar.
 
 .. ipython:: python
 
@@ -241,6 +236,6 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
 
     da.resample(time="81min", closed="right", label="right", offset="3min").mean()
 
-.. _nanosecond-precision range: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timestamp-limitations
+.. _precision range: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timestamp-limitations
 .. _ISO 8601 standard: https://en.wikipedia.org/wiki/ISO_8601
 .. _partial datetime string indexing: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#partial-string-indexing

doc/whats-new.rst

@@ -27,6 +27,9 @@ New Features
 - Add ``unit`` - keyword argument to :py:func:`date_range` and ``microsecond`` parsing to
   iso8601-parser (:pull:`9885`).
   By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
+- Relax nanosecond datetime restriction in CF time decoding (:issue:`7493`, :pull:`9618`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
+
 
 Breaking changes
 ~~~~~~~~~~~~~~~~
@@ -39,6 +42,10 @@ Breaking changes
 
 Deprecations
 ~~~~~~~~~~~~
+- Time decoding related kwarg ``use_cftime`` is deprecated. Use keyword argument
+  ``decode_times=CFDatetimeCoder(use_cftime=True)`` in the respective functions
+  instead (:pull:`9618`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
 - Finalize deprecation of ``closed`` parameters of :py:func:`cftime_range` and
   :py:func:`date_range` (:pull:`9882`).
   By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.

xarray/__init__.py

@@ -1,6 +1,6 @@
 from importlib.metadata import version as _version
 
-from xarray import groupers, testing, tutorial, ufuncs
+from xarray import coders, groupers, testing, tutorial, ufuncs
 from xarray.backends.api import (
     load_dataarray,
     load_dataset,
@@ -66,6 +66,7 @@
 # `mypy --strict` running in projects that import xarray.
 __all__ = (  # noqa: RUF022
     # Sub-packages
+    "coders",
     "groupers",
     "testing",
     "tutorial",

xarray/backends/api.py

@@ -33,6 +33,7 @@
     _normalize_path,
 )
 from xarray.backends.locks import _get_scheduler
+from xarray.coders import CFDatetimeCoder
 from xarray.core import indexing
 from xarray.core.combine import (
     _infer_concat_order_from_positions,
@@ -481,7 +482,10 @@ def open_dataset(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | Mapping[str, bool] | None = None,
-    decode_times: bool | Mapping[str, bool] | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | Mapping[str, bool] | None = None,
     use_cftime: bool | Mapping[str, bool] | None = None,
     concat_characters: bool | Mapping[str, bool] | None = None,
@@ -543,9 +547,9 @@ def open_dataset(
         be replaced by NA. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
-    decode_times : bool or dict-like, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
         Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
@@ -569,6 +573,8 @@ def open_dataset(
         raise an error. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool or dict-like, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and
@@ -698,7 +704,10 @@ def open_dataarray(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | None = None,
-    decode_times: bool | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | None = None,
     use_cftime: bool | None = None,
     concat_characters: bool | None = None,
@@ -761,9 +770,11 @@ def open_dataarray(
         `missing_value` attribute contains multiple values a warning will be
         issued and all array values matching one of the multiple values will
         be replaced by NA. This keyword may not be supported by all the backends.
-    decode_times : bool, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
+        Pass a mapping, e.g. ``{"my_variable": False}``,
+        to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
     decode_timedelta : bool, optional
         If True, decode variables and coordinates with time units in
@@ -781,6 +792,8 @@ def open_dataarray(
         represented using ``np.datetime64[ns]`` objects.  If False, always
         decode times to ``np.datetime64[ns]`` objects; if this is not possible
         raise an error. This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and
@@ -903,7 +916,10 @@ def open_datatree(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | Mapping[str, bool] | None = None,
-    decode_times: bool | Mapping[str, bool] | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | Mapping[str, bool] | None = None,
     use_cftime: bool | Mapping[str, bool] | None = None,
     concat_characters: bool | Mapping[str, bool] | None = None,
@@ -961,9 +977,9 @@ def open_datatree(
         be replaced by NA. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
-    decode_times : bool or dict-like, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
         Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
@@ -987,6 +1003,8 @@ def open_datatree(
         raise an error. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool or dict-like, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and
@@ -1118,7 +1136,10 @@ def open_groups(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | Mapping[str, bool] | None = None,
-    decode_times: bool | Mapping[str, bool] | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | Mapping[str, bool] | None = None,
     use_cftime: bool | Mapping[str, bool] | None = None,
     concat_characters: bool | Mapping[str, bool] | None = None,
@@ -1180,9 +1201,9 @@ def open_groups(
         be replaced by NA. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
-    decode_times : bool or dict-like, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
         Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
@@ -1206,6 +1227,8 @@ def open_groups(
         raise an error. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool or dict-like, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and

xarray/coders.py

@@ -0,0 +1,10 @@
+"""
+This module provides coder objects that encapsulate the
+"encoding/decoding" process.
+"""
+
+from xarray.coding.times import CFDatetimeCoder
+
+__all__ = [
+    "CFDatetimeCoder",
+]