esm_datastore_v2: A rewrite of esm_datastore with new features + improvements

[andersy005]

TO DO

• Introduce pydantic models that allow us to validate the catalog against a pre-defined schema. This has the potential of reducing code complexity in the existing esm_datastore catalog object.
  • Add pydantic models to facilitate data validation #347
  • Update pydantic models #367
• Public methods to expose (Use ESMCatModel Pydantic model #368)
  • __init__
  • __len__
  • __getitem__
  • __contains__
  • to_dataset_dict()
  • nunique()
  • unique()
  • search()
  • Add Query Model #370
  • Fix catalog serialization #373
  • Fix __repr__ and __repr_html__ #374
  • Ensure multi variable catalogs are parsed properly #375
• Merge intake_esm.source:ESMGroupDataSource and intake_esm.source:ESMDataSource into a single data source intake_esm.source:ESMDataSource: Add ESMDataSource #372
• Add functionality to dictionary output by .to_dataset_dict() #352 proposes creating a dict like object and returning this object as the output of to_dataset_dict(). This object will look and smell like a dictionary but will have additional functionality to facilitate applying operators on all returned datasets and/or a subset of these datasets
• Add Derived Variable functionality
  • Add functionality for derived variables #379
  • Add "query range" to Intake-ESM derived variables #388
  • Add ability to produce derived variables without error only when dependent variables exist #387
• How can we make intake-esm more transparent? #163 proposes providing users with functionality to control how dataset aggregations are done. Currently, the aggregation process is rigid. A rewrite of esm_datastore will allow us to explore exposing aggregation logic to users via a registry of operators/preprocessors.
• Update documentations to reflect new changes

[mgrover1]

@andersy005 what still need to get done here? Are we still missing the docs?

[andersy005]

For this particular issue, there is one pending issue that needs to be addressed

• Integration with xarray datatree #511

Regarding the docs, everything looks good. I plan to organize what we currently have in a separate PR.

For the release, here's a list of outstanding issues that i think are worth addressing before the next release: https://github.com/intake/intake-esm/milestone/11

[mgrover1]

@andersy005 Can we close this now that #511 is good to go?