![](\"Figure_01.png\")
"
+ "![](\"Figure_01.jpg\")
"
]
},
{
@@ -58,8 +58,21 @@
}
},
"source": [
- "![](\"Figure_02.png\")
"
+ "![](\"Figure_02.jpg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1149b078",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "subslide"
+ }
+ },
+ "source": [
+ "![](\"Clouds.jpg\")
\n",
- "* Different Data Sources:\n",
- " * Cloud storage formats (S3/Swift)\n",
- " * ECMWF/ESGF data storage\n",
- " * Tape archives"
+ "## A couple of additional features:\n",
+ "- Users can search their own plugin history and other users results\n",
+ "- Web UI suggests similar results on plugin config\n",
+ "- Full integration of workload managers"
]
},
{
"cell_type": "markdown",
- "id": "39148a8e",
+ "id": "3ca11b56",
"metadata": {
"slideshow": {
- "slide_type": "subslide"
+ "slide_type": "slide"
}
},
"source": [
- "## Containerising the User Plugins\n",
- "![](\"docker.png\")
\n",
- "* Advantages:\n",
- " * Host architecture independent (sort of)\n",
- " * Better reproducibility than anaconda\n",
- " * Scalable solutions (kubernetes)"
+ "## What's next?"
]
},
{
"cell_type": "markdown",
- "id": "e10e36fb",
+ "id": "f589a7c8",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
- "# More features coming up in Warm Wold"
+ "* Freva *has* to be come a RestAPI\n",
+ " * Clients are easy to implement since we only need a couple of REST calls\n",
+ "* We teach freva the Common Workflow Language (cwl) via toil:\n",
+ "```yaml\n",
+ "cwlVersion: v1.0\n",
+ "class: CommandLineTool\n",
+ "hints:\n",
+ " SoftwareRequirement:\n",
+ " packages:\n",
+ " - package: cdo\n",
+ "baseCommand: cdo\n",
+ "```\n",
+ "* We add a system for [ordering/pre-processing data](https://www-regiklim.dkrz.de/plugins/climpact/setup/)\n",
+ "* We add a simple interface for intake\n"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
- "display_name": "Plotting for (plotting)",
+ "display_name": "plotting",
"language": "python",
"name": "plotting"
},
@@ -500,7 +521,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.10.2"
+ "version": "3.10.4"
}
},
"nbformat": 4,
diff --git a/talks/Dkrz_User_Workshop/include.html b/talks/Dkrz_User_Workshop/include.html
index 60928a7..4938461 100644
--- a/talks/Dkrz_User_Workshop/include.html
+++ b/talks/Dkrz_User_Workshop/include.html
@@ -1,102 +1,165 @@
document.write(`
-
-
-
-
-
-
+
Freva - Data search and analysis framework for the Community¶
+
+
-
+
+
+
+
+
Freva - Data search and analysis framework¶
+
-
-
-
+
+
-
+
+
+
+
-
-
-
-
+by: Andrej Fast, Brian Lewis, Christopher Kadow, Etor Lucio Eceiza, Martin Bergemann and many others
+
+
+
+
+
+
+
by: Brian Lewis, Christopher Kadow, Etor Lucio Eceiza, Andrej Fast, Martin Bergemann and many others
+
-
-
+
-
+
+
+
+
+
+
+
Common Problem I: Finding Data
-
+
+
-
-
+
+
+
-
-
+
+
+
-
-
+
+
+
Common Problem II: Use code of others
-
+
-
-
-
-
-
-
-
-
Common Problem III: Reproducibility
-
-
+
+
+
+
+
+
+
-
+
Common Problem III. Remember your own code
+
-
-
-
+
+
+
+
+
+
+
+
- How can we search and access various datasets efficiently? -
- How can we streamline user data analysis tools (reusable and reproducible)? +
- How can be streamline user data analysis tools reusable and reproducible?
-
-
-
-
-
Yet another solution: ¶
+
+
+
+
+
+
+
Yet another solution - Freva framework¶
-
+
- Freva comes with a three layer architecture +
-
-
-
Freva: The free evaluation system framework
-
+
+
-
+
+
+
+
+
+
+
+
+
+
+
+
-
+
+
+
+
+
-
-
+
-
-
-
+
-
+Example I: Searching and reading CMIP6 data¶
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Example I: Searching and reading CMIP6 data¶
+
-
-
-
I. Python call:¶
import freva
+
+
+
+
+
+
+Python call:¶
-
-
-
-
-II. Command line call:¶
⋊ freva databrowser project=cmip6 experiment=historical model='mip*' ensemble='r1i1*' time_frequency=mon variable=tas
+
+
+
+
+
+
+
+
+
+CLI call:¶
⋊ freva databrowser project=cmip6 experiment=historical model='mip*' ensemble='r1i1*' time_frequency=mon variable=tas
+
-
-
-
-III. Web user interface:
+
+
+
+
+
+
+Web UI
-
-
-
-
-
-Example II: Data analysis tool / plugins¶
+
+
+
+
+
+
+
+
+Inspecting metadata
+
+
+
+
+
+
+
+
+
+
+
+
+Example II: Data analysis tool / plugins¶
+
-
-
-
-Create animation of high resolution data¶
-- Where is the data?
-- Where are the tools/libraries? Like video codecs etc
+
+
+
+
+
+
+Calculate climate change signal and uncertainty in Düren¶
+- Where is the data for the ensemble?
+- How to select the municipality area?
- ...
-Solution: Use an existing freva plugin.
+
+
-
-
-
-The animator plugin:
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Inspecting the results
+
+
+
+
+
+
+
+
+
+
+
+
+Setting up data analysis tools¶
+
+
+
+
+
+
+
+
+
+
+
+- Freva plugin API is code agnostic
+- Yet: User defined plugins are defined and within an configured within an python API wrapper:
+
+
+
+
+
-
-
-Plugin command line interface:
-
+
+
+
+
+
+
+
+from evaluation_system.api import plugin
+from evaluation_system.api.parameters import File, ParameterDictionary, SolrField
+class PlottingTool(plugin.PluginAbstract):
+"""Plugin that plots content of selected data."""
+
+
+
+
+
+
+
+
+
+
+
+
+
+# Plugin setup
+ tool_developer = {"name": "Jane Doe", "email": "Jane.Doe@example.com"}
+ __category__ = "support"
+ __short_description__ = "Plot data"
+ __version__ = (2022, 1, 1)
+ __parameters__ = ParameterDictionary(
+ File(
+ name="in_file",
+ file_extension="nc",
+ mandatory=True,
+ help="Select the input netCDF file.",
+ ),
+ SolrField(
+ name="variable",
+ mandatory=False,
+ facet="variable",
+ help=("Select the variables(s) to be plotted"),
+ max_items=1,
+ )
+ )
+
+
+
+
+
+
+
+
+
+
+
+
+
+# Workflow definition of the actual tool
+ def runTool(self, setup: dict[str, str] = {}) -> None:
+ """This method calls the actual plugin."""
+ self.call(
+ f"python src/plot_map.py {setup['in_file']} -v {setup['variable']}"
+ )
+
+
+
+
+
+
+
+
+
+
+
+
⋊ freva plugin -l
Animator: Animate data on lon/lat grids
ClimateChangeProfile: Create climate change signals.
ClimDexCalc: Calculate the ETCCDI climate extreme indices on the
basis of daily temperature and precipitation data using
the ClimDex software provided by PCIC.
-Climpact: Process climate model data for input of impact model
+Climpact: Process climate model data for input of impact model
CWT: Calculate Circulation Weather Type by mean sea level pressure.
EnsemblePlotter: Create and Plot maps of multi-model ensemble means and
standard deviations
-MoviePlotter: Plots 2D lon/lat movies in GIF format
+MoviePlotter: Plots 2D lon/lat movies in GIF format
Papermill: Parametrize a given notebook
+PlottingTool: Plot data
+
-
-
-
-freva-plugin animator variable=pr project=reanalysis experiment=era5 cmap=Blues
+
+
+
+
+
+
+⋊ freva plugin plottingtool --doc
+PlottingTool (v2022.1.1): Plot data
+Options:
+in_file (default: <undefined>) [mandatory]
+ Select the input netCDF file.
+variable (default: <undefined>)
+ Select the variables(s) to be plotted
+
+
+
+
+
+
+
+
+
+
+
+Example III: Users adding their own data¶
+
+
+
+
+
+
+
+
+
+
+import freva
+import xarray as xr
+
+user_data = freva.UserData()
+user_data.add("eur-11b", "/tmp/my_awesome_data/outfile_?.nc",
+ institute="clex", model="UM-RA2T",
+ experiment="Bias-correct")
+dset = xr.open_mfdataset(freva.databrowser(experiment="bias*", experiment="clex"))
+
+
+
+
+
+
+
+
+CLI call:¶
+ ⋊ freva-user-data add eur-11b /tmp/my_awesome_data/outfile_?.nc --institute clex --model UM-RA2T --experiment Bias-correct
+
+
+
+
-
+
+
+
+
+
+
+A couple of additional features:¶
+- Users can search their own plugin history and other users results
+- Web UI suggests similar results on plugin config
+- Full integration of workload managers
+
+
+
+
+
+
+
+
+
+
+
+
+What's next?¶
+
+
+
+
+
+
+
+
+
+
+
+- Freva has to be come a RestAPI
+- Clients are easy to implement since we only need a couple of REST calls
+
+
+- We teach freva the Common Workflow Language (cwl) via toil:
cwlVersion: v1.0
+class: CommandLineTool
+hints:
+ SoftwareRequirement:
+ packages:
+ - package: cdo
+baseCommand: cdo
+
+
+- We add a system for ordering/pre-processing data
+- We add a simple interface for intake
+
+
+
+
+
+