From ceca26df8443da8f5492806547390f5315bb2814 Mon Sep 17 00:00:00 2001 From: James Frost Date: Mon, 23 Oct 2023 14:18:10 +0100 Subject: [PATCH 01/21] Add installation tutorial Also create stub files for other tutorials. --- docs/source/contributing/getting-started.rst | 6 +-- .../getting-started/create-first-recipe.rst | 5 ++ docs/source/getting-started/index.rst | 6 ++- docs/source/getting-started/installation.rst | 47 +++++++++++++++---- docs/source/getting-started/run-recipe.rst | 4 ++ .../getting-started/visualise-recipe.rst | 4 ++ 6 files changed, 58 insertions(+), 14 deletions(-) create mode 100644 docs/source/getting-started/create-first-recipe.rst create mode 100644 docs/source/getting-started/run-recipe.rst create mode 100644 docs/source/getting-started/visualise-recipe.rst diff --git a/docs/source/contributing/getting-started.rst b/docs/source/contributing/getting-started.rst index 514e103e5..a20b8f535 100644 --- a/docs/source/contributing/getting-started.rst +++ b/docs/source/contributing/getting-started.rst @@ -1,7 +1,5 @@ -.. _working_practices_getting_started: - -Getting Started -=============== +Getting Started with Contributing +================================= Before you can get to coding, there are a few steps you need to do. diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst new file mode 100644 index 000000000..ffbbd507d --- /dev/null +++ b/docs/source/getting-started/create-first-recipe.rst @@ -0,0 +1,5 @@ +Creating your first custom recipe +================================= + +.. Tutorial guiding through how to create a simple custom recipe. +.. Should include an overview of the recipe format. diff --git a/docs/source/getting-started/index.rst b/docs/source/getting-started/index.rst index 89bd98da1..9ab2b3b2d 100644 --- a/docs/source/getting-started/index.rst +++ b/docs/source/getting-started/index.rst @@ -2,9 +2,13 @@ Getting Started =============== This section contains tutorials suitable for someone getting started with CSET. -By following them you should learn the basics of using CSET. +By following them you should learn the basics of using CSET. It is recommended +that you do them in order. .. toctree:: :maxdepth: 2 installation + run-recipe + create-first-recipe + visualise-recipe diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst index 18c02686e..44e255dc2 100644 --- a/docs/source/getting-started/installation.rst +++ b/docs/source/getting-started/installation.rst @@ -3,15 +3,44 @@ Installation .. Tutorial saying how to install CSET. For edge cases should link elsewhere. -The recommended way to install CSET is via conda. It is packaged on -`conda-forge`_ and can be installed with a simple ``conda create --name=cset ---channel=conda-forge cset``. This will install CSET into its own conda -environment, which is the recommended way to use it, but it is also possible to -install it into an existing environment. +.. note:: -If you instead want to run a development version that has yet to be released, -the easiest way is via an editable install. You can learn how to do this in the -:ref:`working_practices_getting_started` section of the -:doc:`/contributing/index`. + This page details installing a released version of CSET. If you instead want + to run a development version that has yet to be released, see + :doc:`/contributing/getting-started`. +The recommended way to install CSET is via conda_. It is packaged on +`conda-forge`_ and can be installed from the ``cset`` package. The following +command will install CSET into its own conda environment, which is the +recommended. + +.. code-block:: bash + + conda create --name=cset --channel=conda-forge cset + +To use CSET, you need to activate the conda environment with the ``conda +activate`` command. + +.. code-block:: bash + + conda activate cset + +Once that is done, CSET should be ready to use. This can be verified by running +a simple command. + +.. code-block:: bash + + cset --version + +This command should output the installed version of CSET. This will look +something like ``CSET v0.4.0``. + +.. note:: + + You will have to rerun the ``conda activate cset`` command whenever you use + a new terminal. + +You now have CSET installed, so try another tutorial. + +.. _conda: https://docs.conda.io/en/latest/ .. _conda-forge: https://anaconda.org/conda-forge/cset diff --git a/docs/source/getting-started/run-recipe.rst b/docs/source/getting-started/run-recipe.rst new file mode 100644 index 000000000..a1b9dc17c --- /dev/null +++ b/docs/source/getting-started/run-recipe.rst @@ -0,0 +1,4 @@ +Visualising a recipe graphically +================================ + +.. Tutorial on cset graph. diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst new file mode 100644 index 000000000..27cb1d1c1 --- /dev/null +++ b/docs/source/getting-started/visualise-recipe.rst @@ -0,0 +1,4 @@ +Run a pre-existing recipe +========================= + +.. Tutorial saying how to run a pre-existing recipe From be00951edf58267dc8f42e65933fc87e4a7d670d Mon Sep 17 00:00:00 2001 From: James Frost Date: Thu, 26 Oct 2023 10:29:04 +0100 Subject: [PATCH 02/21] Add abbreviation for IDEs --- docs/source/contributing/getting-started.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/source/contributing/getting-started.rst b/docs/source/contributing/getting-started.rst index a20b8f535..8144946cc 100644 --- a/docs/source/contributing/getting-started.rst +++ b/docs/source/contributing/getting-started.rst @@ -9,9 +9,10 @@ Getting the code Git is a version control software, which makes collaborating on software much easier. If you are new to git have a look at the :doc:`git` page to get started. -To get the code onto your computer, you need to clone it with git. Most IDEs -include the ability to do this in their interfaces, but from the command line it -can be done with one of the following commands: +To get the code onto your computer, you need to clone it with git. Most +:abbr:`IDEs (Integrated Development Environments)` include the ability to do +this in their interfaces, but from the command line it can be done with one of +the following commands: .. code-block:: bash From 6d85e3cd9ae503c1f1218786ca319158fd31bcba Mon Sep 17 00:00:00 2001 From: James Frost Date: Thu, 26 Oct 2023 12:26:52 +0100 Subject: [PATCH 03/21] Reword conda environment installation instructions --- docs/source/getting-started/index.rst | 2 +- docs/source/getting-started/installation.rst | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/source/getting-started/index.rst b/docs/source/getting-started/index.rst index 9ab2b3b2d..ed37bd92a 100644 --- a/docs/source/getting-started/index.rst +++ b/docs/source/getting-started/index.rst @@ -10,5 +10,5 @@ that you do them in order. installation run-recipe - create-first-recipe visualise-recipe + create-first-recipe diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst index 44e255dc2..e971eda60 100644 --- a/docs/source/getting-started/installation.rst +++ b/docs/source/getting-started/installation.rst @@ -10,9 +10,9 @@ Installation :doc:`/contributing/getting-started`. The recommended way to install CSET is via conda_. It is packaged on -`conda-forge`_ and can be installed from the ``cset`` package. The following -command will install CSET into its own conda environment, which is the -recommended. +`conda-forge`_ in the ``cset`` package. The following command will install CSET +into its own conda environment, which is recommended to avoid possible package +conflicts. .. code-block:: bash @@ -25,6 +25,11 @@ activate`` command. conda activate cset +.. note:: + + You will have to rerun the ``conda activate cset`` command whenever you use + a new terminal. + Once that is done, CSET should be ready to use. This can be verified by running a simple command. @@ -35,11 +40,6 @@ a simple command. This command should output the installed version of CSET. This will look something like ``CSET v0.4.0``. -.. note:: - - You will have to rerun the ``conda activate cset`` command whenever you use - a new terminal. - You now have CSET installed, so try another tutorial. .. _conda: https://docs.conda.io/en/latest/ From 8b2763756b3cfbfd47275541ee59909a143c68c0 Mon Sep 17 00:00:00 2001 From: James Frost Date: Thu, 26 Oct 2023 12:28:49 +0100 Subject: [PATCH 04/21] Correct tutorial stubs --- docs/source/getting-started/run-recipe.rst | 15 ++++++++++++--- docs/source/getting-started/visualise-recipe.rst | 6 +++--- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/source/getting-started/run-recipe.rst b/docs/source/getting-started/run-recipe.rst index a1b9dc17c..adf146920 100644 --- a/docs/source/getting-started/run-recipe.rst +++ b/docs/source/getting-started/run-recipe.rst @@ -1,4 +1,13 @@ -Visualising a recipe graphically -================================ +Run a pre-existing recipe +========================= -.. Tutorial on cset graph. +.. Tutorial saying how to run a pre-existing recipe. +.. Covers cset cookbook and cset bake. + +CSET works by running recipes that describe how to transform and and visualise +data. It comes with a collection of pre-written recipes to get you started. In +this tutorial you will use CSET to plot the mean surface air temperature of a +forecast. + +The collection of pre-existing recipes can be retrieved with the CSET cookbook +command. diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index 27cb1d1c1..a1b9dc17c 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -1,4 +1,4 @@ -Run a pre-existing recipe -========================= +Visualising a recipe graphically +================================ -.. Tutorial saying how to run a pre-existing recipe +.. Tutorial on cset graph. From dd400425631ae73eb6888217e9984e00474d003a Mon Sep 17 00:00:00 2001 From: James Frost Date: Tue, 31 Oct 2023 14:58:24 +0000 Subject: [PATCH 05/21] Work on tutorials --- docs/source/getting-started/run-recipe.rst | 40 ++++++++++++++++++- .../getting-started/visualise-recipe.rst | 3 ++ 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/source/getting-started/run-recipe.rst b/docs/source/getting-started/run-recipe.rst index adf146920..b6c567eee 100644 --- a/docs/source/getting-started/run-recipe.rst +++ b/docs/source/getting-started/run-recipe.rst @@ -9,5 +9,41 @@ data. It comes with a collection of pre-written recipes to get you started. In this tutorial you will use CSET to plot the mean surface air temperature of a forecast. -The collection of pre-existing recipes can be retrieved with the CSET cookbook -command. +We will create a basic spatial plot of the mean surface air +temperature. There is a pre-existing recipe for this that can be retrieved with +the CSET cookbook command. + +Try the following: + +.. code-block:: bash + + cset cookbook + +Inside that interface find the "Mean Air Temperature Spatial Plot" recipe, and +click the button to use it. This will write out a ``.yaml`` file to your current +directory containing your selected recipe. + +Now you need to find some data to process. You can `download an example file +here`_, or with the following command. + +.. code-block:: bash + + curl -O https://blob.example.com/example-uk-air-temperature.nc + +Now we are ready to run our recipe. This is where we use the ``cset bake`` +command. This takes the input data file, an output path and the recipe file. The +output should be a directory, but it will be created if it does not exist. + +.. code-block:: bash + + cset bake example-uk-air-temperature.nc output/ mean-air-temp-spacial-plot.yaml + +This will run the recipe and leave its output in the specified output directory. + +The most interesting will be the plot, which you can look at with ``xdg-open +output/plot.svg``. + +You've now successfully run CSET with a pre-existing recipe. In the next +tutorial we will see what is going on inside. + +.. _download an example file here: https://blob.example.com/example-uk-air-temperature.nc diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index a1b9dc17c..253dc90ff 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -2,3 +2,6 @@ Visualising a recipe graphically ================================ .. Tutorial on cset graph. + +In this tutorial we will investigate what is going on inside of a recipe, and +visualise the *operators* inside. From 2990441a9967c62a1c027dedee608de58acc6a77 Mon Sep 17 00:00:00 2001 From: James Frost Date: Thu, 2 Nov 2023 10:03:07 +0000 Subject: [PATCH 06/21] Add link to actual test data --- docs/source/getting-started/run-recipe.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/getting-started/run-recipe.rst b/docs/source/getting-started/run-recipe.rst index b6c567eee..5ee89d82e 100644 --- a/docs/source/getting-started/run-recipe.rst +++ b/docs/source/getting-started/run-recipe.rst @@ -28,7 +28,7 @@ here`_, or with the following command. .. code-block:: bash - curl -O https://blob.example.com/example-uk-air-temperature.nc + curl -LO https://github.com/MetOffice/CSET/raw/main/tests/test_data/air_temp.nc Now we are ready to run our recipe. This is where we use the ``cset bake`` command. This takes the input data file, an output path and the recipe file. The @@ -36,14 +36,14 @@ output should be a directory, but it will be created if it does not exist. .. code-block:: bash - cset bake example-uk-air-temperature.nc output/ mean-air-temp-spacial-plot.yaml + cset bake air_temp.nc output/ mean-air-temp-spacial-plot.yaml This will run the recipe and leave its output in the specified output directory. -The most interesting will be the plot, which you can look at with ``xdg-open -output/plot.svg``. +The most interesting output will be the plot, which you can look at with +``xdg-open output/plot.svg``. You've now successfully run CSET with a pre-existing recipe. In the next tutorial we will see what is going on inside. -.. _download an example file here: https://blob.example.com/example-uk-air-temperature.nc +.. _download an example file here: https://github.com/MetOffice/CSET/raw/main/tests/test_data/air_temp.nc From def1b7d20207b79735d11ebc510bf6cf6ebfbf56 Mon Sep 17 00:00:00 2001 From: James Frost Date: Thu, 2 Nov 2023 13:02:57 +0000 Subject: [PATCH 07/21] Add most of visualisation tutorial --- .../getting-started/visualise-recipe.rst | 34 ++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index 253dc90ff..ba5b6d28a 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -1,7 +1,39 @@ Visualising a recipe graphically ================================ -.. Tutorial on cset graph. +.. Tutorial on cset graph and introduction to the recipe format. In this tutorial we will investigate what is going on inside of a recipe, and visualise the *operators* inside. + +Using the ``cset cookbook`` command find the "Mean Air Temperature Spatial Plot" +recipe, and click the button to use it. This will write out a ``.yaml`` file to +your current directory containing your selected recipe. + +We will now visualise the steps inside the recipe using the ``cset graph`` +command. + +.. code-block:: bash + + cset graph recipe.yaml + +This should open an image of a visualisation of the recipe. Each node is a step, +or an *operator*, which does a single processing task. You can see that later +operators depend on previous ones, and this relationship can be as complicated +as needed. + +.. Include image of visualised recipe. + +.. Discussion of the specific operators shown. + +To see more detail about each individual operator running we can use the +``--details`` flag. This shows the configuration of each operator in the recipe. + +.. code-block:: bash + + cset graph --details recipe.yaml + +.. Discussion of configuration of the shown operators. + +You now know how to visualise a recipe, and a little about the operators it is +made up of. In the next tutorial you will learn to make your own. From a7c78e86d0b75f560798fbd1ce0dfa02c11ac4cc Mon Sep 17 00:00:00 2001 From: James Frost Date: Fri, 3 Nov 2023 14:05:42 +0000 Subject: [PATCH 08/21] Add skeleton recipe annotations --- .../getting-started/create-first-recipe.rst | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index ffbbd507d..2e80c1a67 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -3,3 +3,128 @@ Creating your first custom recipe .. Tutorial guiding through how to create a simple custom recipe. .. Should include an overview of the recipe format. + +.. code-block:: yaml + :emphasize-lines: 1 + + title: Mean Air Temperature Spatial Plot + description: | + Extracts and plots the instantaneous 1.5m air temperature from a file. + + The temperature is averaged across the time coordinate. + + These descriptions are Markdown, so you can do things like **bold**, *italics*, and more. + + steps: + - operator: read.read_cubes + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + + - operator: filters.filter_cubes + constraint: + operator: constraints.combine_constraints + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] + + - operator: collapse.collapse + coordinate: time + method: MEAN + + - operator: plot.spatial_contour_plot + file_path: CSET_OUTPUT_PATH + + - operator: write.write_cube_to_nc + file_path: CSET_OUTPUT_PATH + +The title of the recipe is highlighted. + +.. TODO: Write what should go here. + +.. code-block:: yaml + :emphasize-lines: 2-7 + + title: Mean Air Temperature Spatial Plot + description: | + Extracts and plots the instantaneous 1.5m air temperature from a file. + + The temperature is averaged across the time coordinate. + + These descriptions are Markdown, so you can do things like **bold**, *italics*, and more. + + steps: + - operator: read.read_cubes + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + + - operator: filters.filter_cubes + constraint: + operator: constraints.combine_constraints + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] + + - operator: collapse.collapse + coordinate: time + method: MEAN + + - operator: plot.spatial_contour_plot + file_path: CSET_OUTPUT_PATH + + - operator: write.write_cube_to_nc + file_path: CSET_OUTPUT_PATH + +The description is highlighted. + +.. TODO: Write what should go here. + + + +.. code-block:: yaml + :emphasize-lines: 10-13 + + title: Mean Air Temperature Spatial Plot + description: | + Extracts and plots the 1.5m air temperature from a file. The temperature + is averaged across the time coordinate. + + These descriptions are Markdown, so you can do things like **bold**, + *italics*, and more. + + steps: + - operator: read.read_cubes + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + + - operator: filters.filter_cubes + constraint: + operator: constraints.combine_constraints + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] + + - operator: collapse.collapse + coordinate: time + method: MEAN + + - operator: plot.spatial_contour_plot + file_path: CSET_OUTPUT_PATH + + - operator: write.write_cube_to_nc + file_path: CSET_OUTPUT_PATH + +The first operator is highlighted. + +.. TODO: Write what should go here. From 533e245d123261ae4e80a4c1638a150eb5b16cf9 Mon Sep 17 00:00:00 2001 From: James Frost Date: Fri, 3 Nov 2023 14:42:19 +0000 Subject: [PATCH 09/21] Add recipe for Mean Air Temp Spatial Plot --- .../getting-started/create-first-recipe.rst | 86 +++++++------------ ..._surface_air_temperature_spatial_plot.yaml | 25 ++++++ 2 files changed, 55 insertions(+), 56 deletions(-) create mode 100644 src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index 2e80c1a67..2ce614243 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -7,29 +7,21 @@ Creating your first custom recipe .. code-block:: yaml :emphasize-lines: 1 - title: Mean Air Temperature Spatial Plot + title: Mean Surface Air Temperature Spatial Plot description: | - Extracts and plots the instantaneous 1.5m air temperature from a file. - - The temperature is averaged across the time coordinate. - - These descriptions are Markdown, so you can do things like **bold**, *italics*, and more. + Extracts and plots the 1.5m air temperature from a file. The temperature + is averaged across the time coordinate. steps: - operator: read.read_cubes constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - - - operator: filters.filter_cubes - constraint: - operator: constraints.combine_constraints - constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - cell_methods_constraint: - operator: constraints.generate_cell_methods_constraint - cell_methods: [] + operator: constraints.combine_constraints + stash_constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] - operator: collapse.collapse coordinate: time @@ -46,31 +38,23 @@ The title of the recipe is highlighted. .. TODO: Write what should go here. .. code-block:: yaml - :emphasize-lines: 2-7 + :emphasize-lines: 2-4 - title: Mean Air Temperature Spatial Plot + title: Mean Surface Air Temperature Spatial Plot description: | - Extracts and plots the instantaneous 1.5m air temperature from a file. - - The temperature is averaged across the time coordinate. - - These descriptions are Markdown, so you can do things like **bold**, *italics*, and more. + Extracts and plots the 1.5m air temperature from a file. The temperature + is averaged across the time coordinate. steps: - operator: read.read_cubes constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - - - operator: filters.filter_cubes - constraint: - operator: constraints.combine_constraints - constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - cell_methods_constraint: - operator: constraints.generate_cell_methods_constraint - cell_methods: [] + operator: constraints.combine_constraints + stash_constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] - operator: collapse.collapse coordinate: time @@ -86,34 +70,24 @@ The description is highlighted. .. TODO: Write what should go here. - - .. code-block:: yaml - :emphasize-lines: 10-13 + :emphasize-lines: 7-15 - title: Mean Air Temperature Spatial Plot + title: Mean Surface Air Temperature Spatial Plot description: | Extracts and plots the 1.5m air temperature from a file. The temperature is averaged across the time coordinate. - These descriptions are Markdown, so you can do things like **bold**, - *italics*, and more. - steps: - operator: read.read_cubes constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - - - operator: filters.filter_cubes - constraint: - operator: constraints.combine_constraints - constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - cell_methods_constraint: - operator: constraints.generate_cell_methods_constraint - cell_methods: [] + operator: constraints.combine_constraints + stash_constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] - operator: collapse.collapse coordinate: time diff --git a/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml b/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml new file mode 100644 index 000000000..6498abb2d --- /dev/null +++ b/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml @@ -0,0 +1,25 @@ +title: Mean Surface Air Temperature Spatial Plot +description: | + Extracts and plots the 1.5m air temperature from a file. The temperature + is averaged across the time coordinate. + +steps: + - operator: read.read_cubes + constraint: + operator: constraints.combine_constraints + stash_constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + cell_methods_constraint: + operator: constraints.generate_cell_methods_constraint + cell_methods: [] + + - operator: collapse.collapse + coordinate: time + method: MEAN + + - operator: plot.spatial_contour_plot + file_path: CSET_OUTPUT_PATH + + - operator: write.write_cube_to_nc + file_path: CSET_OUTPUT_PATH From 25d31be3138d3587944f05352f101d320af112a6 Mon Sep 17 00:00:00 2001 From: James Frost Date: Mon, 13 Nov 2023 11:39:52 +0000 Subject: [PATCH 10/21] Rename contribution onboarding Fixes #223 --- docs/source/contributing/getting-started.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/contributing/getting-started.rst b/docs/source/contributing/getting-started.rst index 8144946cc..837b7a39a 100644 --- a/docs/source/contributing/getting-started.rst +++ b/docs/source/contributing/getting-started.rst @@ -1,5 +1,5 @@ -Getting Started with Contributing -================================= +Before you Contribute +===================== Before you can get to coding, there are a few steps you need to do. From 4f552108b076c00a8f1846fa49acb53c04d47e5b Mon Sep 17 00:00:00 2001 From: James Frost Date: Mon, 20 Nov 2023 12:05:08 +0000 Subject: [PATCH 11/21] cset graph rename detailed to details Since its what I keep typing anyway it should just be the option name. --- docs/source/reference/cli.rst | 2 +- src/CSET/__init__.py | 4 ++-- .../recipes/mean_surface_air_temperature_spatial_plot.yaml | 5 +++++ tests/test_cli.py | 2 +- 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/source/reference/cli.rst b/docs/source/reference/cli.rst index cd0decd47..eb907eed1 100644 --- a/docs/source/reference/cli.rst +++ b/docs/source/reference/cli.rst @@ -68,4 +68,4 @@ exactly what ran. -h, --help show this help message and exit -o [OUTPUT_PATH], --output_path [OUTPUT_PATH] file in which to save the graph image, otherwise uses a temporary file. When specified the file is not automatically opened - -d, --detailed include operator arguments in output + -d, --details include operator arguments in output diff --git a/src/CSET/__init__.py b/src/CSET/__init__.py index 8e85c2aa5..5cfa76462 100644 --- a/src/CSET/__init__.py +++ b/src/CSET/__init__.py @@ -71,7 +71,7 @@ def main(): ) parser_graph.add_argument( "-d", - "--detailed", + "--details", action="store_true", help="include operator arguments in output", ) @@ -122,7 +122,7 @@ def _graph_command(args): args.recipe, args.output_path, auto_open=not bool(args.output_path), - detailed=args.detailed, + detailed=args.details, ) diff --git a/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml b/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml index 6498abb2d..4db8af628 100644 --- a/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml +++ b/src/CSET/recipes/mean_surface_air_temperature_spatial_plot.yaml @@ -5,6 +5,11 @@ description: | steps: - operator: read.read_cubes + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + + - operator: filters.filter_cubes constraint: operator: constraints.combine_constraints stash_constraint: diff --git a/tests/test_cli.py b/tests/test_cli.py index d1b1ad6a7..1326db47e 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -106,7 +106,7 @@ def test_graph_details(tmp_path: Path): ( "cset", "graph", - "--detailed", + "--details", "-o", str(output_file), "tests/test_data/noop_recipe.yaml", From 6939d65a12adb8a84b9de7852a49f90bdae6eb53 Mon Sep 17 00:00:00 2001 From: James Frost Date: Mon, 20 Nov 2023 14:44:42 +0000 Subject: [PATCH 12/21] Further work on tutorials The first three are complete. --- .../getting-started/create-first-recipe.rst | 103 +++++++----- .../getting-started/recipe-graph-details.svg | 157 ++++++++++++++++++ docs/source/getting-started/recipe-graph.svg | 151 +++++++++++++++++ docs/source/getting-started/run-recipe.rst | 11 +- .../getting-started/visualise-recipe.rst | 31 +++- 5 files changed, 403 insertions(+), 50 deletions(-) create mode 100644 docs/source/getting-started/recipe-graph-details.svg create mode 100644 docs/source/getting-started/recipe-graph.svg diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index 2ce614243..50ca620cd 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -4,41 +4,26 @@ Creating your first custom recipe .. Tutorial guiding through how to create a simple custom recipe. .. Should include an overview of the recipe format. -.. code-block:: yaml - :emphasize-lines: 1 +In this tutorial we will look at the recipe format CSET uses, and create a new +recipe from scratch. - title: Mean Surface Air Temperature Spatial Plot - description: | - Extracts and plots the 1.5m air temperature from a file. The temperature - is averaged across the time coordinate. +CSET recipes are written in YAML 1.2, a data format used by many pieces of +software. Even if you've never seen it before don't worry, as this tutorial +should cover everything you need to know. - steps: - - operator: read.read_cubes - constraint: - operator: constraints.combine_constraints - stash_constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - cell_methods_constraint: - operator: constraints.generate_cell_methods_constraint - cell_methods: [] +Recipes consist mostly of key-value pairs, which in YAML are denoted with the +key on the left of a colon, and the value on the right. The value can be a +string, a number, or even more key-value pairs by indenting them below the key. +It is important to note that, like in python, indentation is significant in +YAML. Indentation should be done with two spaces. - - operator: collapse.collapse - coordinate: time - method: MEAN +Recipe Metadata +--------------- - - operator: plot.spatial_contour_plot - file_path: CSET_OUTPUT_PATH - - - operator: write.write_cube_to_nc - file_path: CSET_OUTPUT_PATH - -The title of the recipe is highlighted. - -.. TODO: Write what should go here. +The first thing in a recipe file is some metadata about the recipe itself. .. code-block:: yaml - :emphasize-lines: 2-4 + :emphasize-lines: 1-4 title: Mean Surface Air Temperature Spatial Plot description: | @@ -47,6 +32,11 @@ The title of the recipe is highlighted. steps: - operator: read.read_cubes + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + + - operator: filters.filter_cubes constraint: operator: constraints.combine_constraints stash_constraint: @@ -61,17 +51,31 @@ The title of the recipe is highlighted. method: MEAN - operator: plot.spatial_contour_plot - file_path: CSET_OUTPUT_PATH + filename: CSET_OUTPUT_PATH - operator: write.write_cube_to_nc - file_path: CSET_OUTPUT_PATH + filename: CSET_OUTPUT_PATH + +The title is the name of the recipe. It should be sufficiently descriptive so +you can tell what the recipe does without any further context. It shows up in +the output page, the plot title, and a few other places. + +Next we have the description. This is a long-form description of what the recipe +does, considerations around its use, and what science it is based on. The +description is `markdown`_, so some formatting can be used were helpful, papers +can be linked, and so on. A little bit of unusual syntax here is ``description: +|``, with the pipe after the key. This means the indented block that follows is +a multiline string, so you can have as many lines as you want, provided they are +all indented with at least two spaces. -The description is highlighted. -.. TODO: Write what should go here. +Recipe Steps +------------ + +Just as in baking you would follow a recipe step-by-step, so does CSET. .. code-block:: yaml - :emphasize-lines: 7-15 + :emphasize-lines: 6- title: Mean Surface Air Temperature Spatial Plot description: | @@ -80,6 +84,11 @@ The description is highlighted. steps: - operator: read.read_cubes + constraint: + operator: constraints.generate_stash_constraint + stash: m01s03i236 + + - operator: filters.filter_cubes constraint: operator: constraints.combine_constraints stash_constraint: @@ -94,11 +103,29 @@ The description is highlighted. method: MEAN - operator: plot.spatial_contour_plot - file_path: CSET_OUTPUT_PATH + filename: CSET_OUTPUT_PATH - operator: write.write_cube_to_nc - file_path: CSET_OUTPUT_PATH + filename: CSET_OUTPUT_PATH + +The steps of the recipe are all under the ``steps`` key. Each block prefixed +with a ``-`` is a step, and they are run in order. + +Each step has an ``operator`` key, which specifies which operator to use. A +`complete list of operators is in the documentation`_. The other keys in a step +are the arguments that operator takes. Each operator implicitly takes its first +argument from previous step, but this can be overridden by explicitly providing +it. The first step receives the path to the input data as its implicit input. + +The other thing to note is that the arguments of operators can themselves be +operators. This allows nesting operators to use their output as arguments to +other operators. + +.. _complete list of operators is in the documentation: https://metoffice.github.io/CSET/reference/operators + +Making a new recipe +------------------- -The first operator is highlighted. +.. TODO: Write section. -.. TODO: Write what should go here. +.. Do a contour plot of a single timestamp. diff --git a/docs/source/getting-started/recipe-graph-details.svg b/docs/source/getting-started/recipe-graph-details.svg new file mode 100644 index 000000000..58aec39f9 --- /dev/null +++ b/docs/source/getting-started/recipe-graph-details.svg @@ -0,0 +1,157 @@ + + + + + + + + + +0ea71716-f3ff-43be-b23c-43b6d9d1972a + +read.read_cubes + + + +c1be712d-e343-4b0f-956a-7d2427a4c17d + +filters.filter_cubes + + + +0ea71716-f3ff-43be-b23c-43b6d9d1972a->c1be712d-e343-4b0f-956a-7d2427a4c17d + + + + + +7ef64587-30ed-42f2-bdee-376c53d42ffc + +constraints.combine_constraints + + + +0ea71716-f3ff-43be-b23c-43b6d9d1972a->7ef64587-30ed-42f2-bdee-376c53d42ffc + + + + + +22b128ed-58f4-4a98-8ceb-cec0cd02f234 + +constraints.generate_stash_constraint +<stash: m01s03i236> + + + +0ea71716-f3ff-43be-b23c-43b6d9d1972a->22b128ed-58f4-4a98-8ceb-cec0cd02f234 + + + + + +d97c190b-70ba-4c2b-a01b-4a89c9d8dec2 + +constraints.generate_cell_methods_constraint +<cell_methods: []> + + + +0ea71716-f3ff-43be-b23c-43b6d9d1972a->d97c190b-70ba-4c2b-a01b-4a89c9d8dec2 + + + + + +START + +START + + + +START->0ea71716-f3ff-43be-b23c-43b6d9d1972a + + + + + +b6d3536a-27d8-4882-b103-709c18c3eb49 + +constraints.generate_stash_constraint +<stash: m01s03i236> + + + +START->b6d3536a-27d8-4882-b103-709c18c3eb49 + + + + + +b6d3536a-27d8-4882-b103-709c18c3eb49->0ea71716-f3ff-43be-b23c-43b6d9d1972a + + + + + +e1451adc-5e2a-46d6-8950-2e2fa1f0c364 + +collapse.collapse +<coordinate: time> +<method: MEAN> + + + +c1be712d-e343-4b0f-956a-7d2427a4c17d->e1451adc-5e2a-46d6-8950-2e2fa1f0c364 + + + + + +7ef64587-30ed-42f2-bdee-376c53d42ffc->c1be712d-e343-4b0f-956a-7d2427a4c17d + + + + + +22b128ed-58f4-4a98-8ceb-cec0cd02f234->7ef64587-30ed-42f2-bdee-376c53d42ffc + + + + + +d97c190b-70ba-4c2b-a01b-4a89c9d8dec2->7ef64587-30ed-42f2-bdee-376c53d42ffc + + + + + +5f951fe0-1b14-4816-87a4-3faf03dd2eaa + +plot.spatial_contour_plot +<file_path: CSET_OUTPUT_PATH> + + + +e1451adc-5e2a-46d6-8950-2e2fa1f0c364->5f951fe0-1b14-4816-87a4-3faf03dd2eaa + + + + + +4f19f4e1-9084-4ed5-82b4-563c5ebf2251 + +write.write_cube_to_nc +<file_path: CSET_OUTPUT_PATH> + + + +5f951fe0-1b14-4816-87a4-3faf03dd2eaa->4f19f4e1-9084-4ed5-82b4-563c5ebf2251 + + + + + diff --git a/docs/source/getting-started/recipe-graph.svg b/docs/source/getting-started/recipe-graph.svg new file mode 100644 index 000000000..88f1a4cf4 --- /dev/null +++ b/docs/source/getting-started/recipe-graph.svg @@ -0,0 +1,151 @@ + + + + + +Graph visualisation of the a CSET recipe. + + + + +fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d + +read.read_cubes + + + +e006203b-b212-45b8-acb2-d2aba3a91c18 + +filters.filter_cubes + + + +fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d->e006203b-b212-45b8-acb2-d2aba3a91c18 + + + + + +84e39446-1511-4779-adc8-c88fdcbf38d2 + +constraints.combine_constraints + + + +fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d->84e39446-1511-4779-adc8-c88fdcbf38d2 + + + + + +e27f5b60-93b9-4ac8-bc4e-957547716f51 + +constraints.generate_stash_constraint + + + +fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d->e27f5b60-93b9-4ac8-bc4e-957547716f51 + + + + + +888067b4-392d-427b-9f23-172095516377 + +constraints.generate_cell_methods_constraint + + + +fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d->888067b4-392d-427b-9f23-172095516377 + + + + + +START + +START + + + +START->fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d + + + + + +f81b25eb-5d0a-4ece-99bf-65395ba60400 + +constraints.generate_stash_constraint + + + +START->f81b25eb-5d0a-4ece-99bf-65395ba60400 + + + + + +f81b25eb-5d0a-4ece-99bf-65395ba60400->fc1ca1f4-5d5c-4e34-bbeb-632dd8572e4d + + + + + +af2a4107-d432-454e-8a71-1cdb7cfdd464 + +collapse.collapse + + + +e006203b-b212-45b8-acb2-d2aba3a91c18->af2a4107-d432-454e-8a71-1cdb7cfdd464 + + + + + +84e39446-1511-4779-adc8-c88fdcbf38d2->e006203b-b212-45b8-acb2-d2aba3a91c18 + + + + + +e27f5b60-93b9-4ac8-bc4e-957547716f51->84e39446-1511-4779-adc8-c88fdcbf38d2 + + + + + +888067b4-392d-427b-9f23-172095516377->84e39446-1511-4779-adc8-c88fdcbf38d2 + + + + + +ff4975a6-888d-4ff2-9254-8d6e12e789c7 + +plot.spatial_contour_plot + + + +af2a4107-d432-454e-8a71-1cdb7cfdd464->ff4975a6-888d-4ff2-9254-8d6e12e789c7 + + + + + +98b17d40-2838-4f13-b5d1-644f90f253a3 + +write.write_cube_to_nc + + + +ff4975a6-888d-4ff2-9254-8d6e12e789c7->98b17d40-2838-4f13-b5d1-644f90f253a3 + + + + + diff --git a/docs/source/getting-started/run-recipe.rst b/docs/source/getting-started/run-recipe.rst index 5ee89d82e..33469e4b1 100644 --- a/docs/source/getting-started/run-recipe.rst +++ b/docs/source/getting-started/run-recipe.rst @@ -1,8 +1,7 @@ Run a pre-existing recipe ========================= -.. Tutorial saying how to run a pre-existing recipe. -.. Covers cset cookbook and cset bake. +.. Tutorial on running a pre-existing recipe, covering cookbook and bake. CSET works by running recipes that describe how to transform and and visualise data. It comes with a collection of pre-written recipes to get you started. In @@ -19,9 +18,9 @@ Try the following: cset cookbook -Inside that interface find the "Mean Air Temperature Spatial Plot" recipe, and -click the button to use it. This will write out a ``.yaml`` file to your current -directory containing your selected recipe. +This will write out a recipes folder contain recipe ``.yaml`` files to your +current directory. We will use the +``mean_surface_air_temperature_spatial_plot.yaml`` recipe. Now you need to find some data to process. You can `download an example file here`_, or with the following command. @@ -36,7 +35,7 @@ output should be a directory, but it will be created if it does not exist. .. code-block:: bash - cset bake air_temp.nc output/ mean-air-temp-spacial-plot.yaml + cset bake air_temp.nc output/ recipes/mean-air-temp-spacial-plot.yaml This will run the recipe and leave its output in the specified output directory. diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index ba5b6d28a..977e75ae0 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -15,25 +15,44 @@ command. .. code-block:: bash - cset graph recipe.yaml + cset graph recipes/mean-air-temp-spacial-plot.yaml This should open an image of a visualisation of the recipe. Each node is a step, or an *operator*, which does a single processing task. You can see that later operators depend on previous ones, and this relationship can be as complicated as needed. -.. Include image of visualised recipe. - -.. Discussion of the specific operators shown. +.. image:: recipe-graph.svg + :alt: Graph visualisation of a CSET recipe generated by cset graph. To see more detail about each individual operator running we can use the ``--details`` flag. This shows the configuration of each operator in the recipe. .. code-block:: bash - cset graph --details recipe.yaml + cset graph --details recipes/mean-air-temp-spacial-plot.yaml + +.. image:: recipe-graph-details.svg + :alt: Graph visualisation of a CSET recipe generated by cset graph, showing the operator details. + +Now we can see the structure of the recipe we can delve into what each operator +is doing. The ellipses are the operators, and the arrows between them show where +they pass their output to the next operators. + +The first operator in the recipe is ``read.read_cubes``, however it +takes a constraint on a STASH code, which is itself created by another operator, +``constraints.generate_stash_constraint``. + +This operators-running-operators behaviour is further used in the next step, +where the read CubeList is filtered down to a single air temperature cube. There +are two constraints used here, the STASH code, and the cell methods. These are +combined into a single constraint by the ``constraints.combine_constraints`` +operator before being used by the ``filters.filter_cubes`` operator. -.. Discussion of configuration of the shown operators. +Afterwards the cube has its time dimension meaned by the ``collapse.collapse`` +operator, so it becomes two-dimensional. Then it passes to the +``plot.spacial_contour_plot`` and ``write.write_cube_to_nc`` operators to be +plotted and saved. You now know how to visualise a recipe, and a little about the operators it is made up of. In the next tutorial you will learn to make your own. From bba724dbfe838749c48bcd363bbdc06988aa7134 Mon Sep 17 00:00:00 2001 From: James Frost Date: Mon, 20 Nov 2023 15:39:15 +0000 Subject: [PATCH 13/21] WIP tutorial --- docs/source/getting-started/create-first-recipe.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index 50ca620cd..d0772a5fd 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -68,6 +68,7 @@ can be linked, and so on. A little bit of unusual syntax here is ``description: a multiline string, so you can have as many lines as you want, provided they are all indented with at least two spaces. +.. _markdown: https://commonmark.org/help/ Recipe Steps ------------ @@ -128,4 +129,10 @@ Making a new recipe .. TODO: Write section. -.. Do a contour plot of a single timestamp. +.. Convert the above example into a step by step tutorial. Do it sequentially, and have the full recipe at the end. + +We will now create a recipe from scratch. This recipe will plot a specific +timestamp of the example air temperature data. + +Start by opening a new file in your favourite text editor, and save it as +``single_timestep_surface_air_temperature_spacial_plot.yaml``. From cf6342455c53753c7a17a60302f0b27cc987200d Mon Sep 17 00:00:00 2001 From: James Frost Date: Tue, 21 Nov 2023 10:39:11 +0000 Subject: [PATCH 14/21] Finish create-first-recipe tutorial --- .../getting-started/create-first-recipe.rst | 152 ++++++++++++------ 1 file changed, 104 insertions(+), 48 deletions(-) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index d0772a5fd..81ed92d4a 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -4,10 +4,9 @@ Creating your first custom recipe .. Tutorial guiding through how to create a simple custom recipe. .. Should include an overview of the recipe format. -In this tutorial we will look at the recipe format CSET uses, and create a new -recipe from scratch. +In this tutorial we will write a new recipe from scratch. -CSET recipes are written in YAML 1.2, a data format used by many pieces of +CSET recipes are written in `YAML 1.2`_, a data format used by many pieces of software. Even if you've never seen it before don't worry, as this tutorial should cover everything you need to know. @@ -17,24 +16,82 @@ string, a number, or even more key-value pairs by indenting them below the key. It is important to note that, like in python, indentation is significant in YAML. Indentation should be done with two spaces. +.. _YAML 1.2: https://en.wikipedia.org/wiki/YAML + +Making a new recipe +------------------- + +We will now create a recipe from scratch. This recipe will plot a specific +timestamp of the example air temperature data. + +Start by opening a new file in your favourite text editor, and save it somewhere +accessible as ``single_timestep_surface_air_temperature_spacial_plot.yaml``. + Recipe Metadata --------------- -The first thing in a recipe file is some metadata about the recipe itself. +Title +~~~~~ + +The first thing we will add to our recipe file is some metadata about the recipe +itself, starting with its title. + +The title is the name of the recipe. It should be sufficiently descriptive so +you can tell what the recipe does without any further context. It shows up in +the output page, the plot title, and a few other places. .. code-block:: yaml - :emphasize-lines: 1-4 title: Mean Surface Air Temperature Spatial Plot + +Description +~~~~~~~~~~~ + +Following the title we have the description. This is a long-form description of +what the recipe does, considerations around its use, and what science it is +based on. The description is `markdown`_, so some formatting can be used were +helpful, papers can be linked, and so on. A little bit of unusual syntax here is +``description: |``, with the pipe after the key. This means the indented block +that follows is a multiline string, so you can have as many lines as you want, +provided they are all indented with at least two spaces. + +.. _markdown: https://commonmark.org/help/ + +.. code-block:: yaml + description: | Extracts and plots the 1.5m air temperature from a file. The temperature is averaged across the time coordinate. + +Recipe Steps +------------ + +Just as in baking you would follow a recipe step-by-step, so does CSET. The +steps of the recipe are all under the ``steps`` key. Each block prefixed with a +``-`` (which makes a list in YAML) is a step, and they are run in order. + +Each step has an ``operator`` key, which specifies which operator to use. A +`complete list of operators is in the documentation`_, but for this tutorial we +will describe them here. + +.. _complete list of operators is in the documentation: https://metoffice.github.io/CSET/reference/operators + +The first step we want is to read in the model data. This is done with the +``read.read_cubes`` operator. The first step is special, and receives the path +to the input data as its implicit input. + +.. code-block:: yaml + steps: - operator: read.read_cubes - constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 + +Once we have read the data, we need to filter it to just that we want. +``filter.filter_cubes`` is the operator for that. It also has the +additional effect of turning the CubeList returned by ``read.read_cubes`` into a +Cube. + +.. code-block:: yaml - operator: filters.filter_cubes constraint: @@ -46,37 +103,48 @@ The first thing in a recipe file is some metadata about the recipe itself. operator: constraints.generate_cell_methods_constraint cell_methods: [] +Unlike the ``read.read_cubes`` operator, we have many key-value pairs in this +step. The other keys the step are the named arguments that operator takes. Each +operator implicitly takes its first argument from previous step, but this can be +overridden by explicitly providing it. + +The other thing to note is that the arguments of operators can themselves be +operators. This allows nesting operators to use their output as arguments to +other operators. + + +Next we want to reduce the dimensionality of the data ahead of plotting. In this +case we want the mean of the time coordinate. The ``collapse.collapse`` operator +allows us to do this, and takes as parameters the coordinate to collapse, and +the method by which it is done. + +.. code-block:: yaml + - operator: collapse.collapse coordinate: time method: MEAN +Finally we can actually plot the model data, using the +``plot.spatial_contour_plot`` operator, and then save the processed data with +the ``write.write_cube_to_nc`` operator. This finishes up our recipe. + +.. TODO: Remove filename argument. + +.. code-block:: yaml + - operator: plot.spatial_contour_plot filename: CSET_OUTPUT_PATH - operator: write.write_cube_to_nc filename: CSET_OUTPUT_PATH -The title is the name of the recipe. It should be sufficiently descriptive so -you can tell what the recipe does without any further context. It shows up in -the output page, the plot title, and a few other places. - -Next we have the description. This is a long-form description of what the recipe -does, considerations around its use, and what science it is based on. The -description is `markdown`_, so some formatting can be used were helpful, papers -can be linked, and so on. A little bit of unusual syntax here is ``description: -|``, with the pipe after the key. This means the indented block that follows is -a multiline string, so you can have as many lines as you want, provided they are -all indented with at least two spaces. - -.. _markdown: https://commonmark.org/help/ -Recipe Steps ------------- +Complete Recipe +--------------- -Just as in baking you would follow a recipe step-by-step, so does CSET. +After following this far your recipe should look like this: .. code-block:: yaml - :emphasize-lines: 6- title: Mean Surface Air Temperature Spatial Plot description: | @@ -85,9 +153,6 @@ Just as in baking you would follow a recipe step-by-step, so does CSET. steps: - operator: read.read_cubes - constraint: - operator: constraints.generate_stash_constraint - stash: m01s03i236 - operator: filters.filter_cubes constraint: @@ -109,30 +174,21 @@ Just as in baking you would follow a recipe step-by-step, so does CSET. - operator: write.write_cube_to_nc filename: CSET_OUTPUT_PATH -The steps of the recipe are all under the ``steps`` key. Each block prefixed -with a ``-`` is a step, and they are run in order. +Running the Recipe +------------------ -Each step has an ``operator`` key, which specifies which operator to use. A -`complete list of operators is in the documentation`_. The other keys in a step -are the arguments that operator takes. Each operator implicitly takes its first -argument from previous step, but this can be overridden by explicitly providing -it. The first step receives the path to the input data as its implicit input. +We can run this recipe using `the same data`_ as was used for :doc:`the first +tutorial `. -The other thing to note is that the arguments of operators can themselves be -operators. This allows nesting operators to use their output as arguments to -other operators. - -.. _complete list of operators is in the documentation: https://metoffice.github.io/CSET/reference/operators +.. _the same data: https://github.com/MetOffice/CSET/raw/main/tests/test_data/air_temp.nc -Making a new recipe -------------------- +Use ``cset bake`` to run your newly created recipe. -.. TODO: Write section. +.. code-block:: bash -.. Convert the above example into a step by step tutorial. Do it sequentially, and have the full recipe at the end. + cset bake air_temp.nc output/ single_timestep_surface_air_temperature_spacial_plot.yaml -We will now create a recipe from scratch. This recipe will plot a specific -timestamp of the example air temperature data. +You can investigate the created plot and data file in the specified ``output`` +directory. -Start by opening a new file in your favourite text editor, and save it as -``single_timestep_surface_air_temperature_spacial_plot.yaml``. +You've now successfully written and run a custom CSET recipe. From 7cb8a19c0499acf15b896a0e9dd38978461cab82 Mon Sep 17 00:00:00 2001 From: James Frost Date: Tue, 21 Nov 2023 11:05:39 +0000 Subject: [PATCH 15/21] Tutorial editing pass --- .../getting-started/create-first-recipe.rst | 24 +++++++++++++------ docs/source/getting-started/index.rst | 6 ++--- docs/source/getting-started/installation.rst | 13 +++++----- .../getting-started/visualise-recipe.rst | 5 ++-- 4 files changed, 29 insertions(+), 19 deletions(-) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index 81ed92d4a..42609b1de 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -16,6 +16,13 @@ string, a number, or even more key-value pairs by indenting them below the key. It is important to note that, like in python, indentation is significant in YAML. Indentation should be done with two spaces. +.. code-block:: yaml + + # Simple YAML example. + key: value + key2: + key-as-value: value + .. _YAML 1.2: https://en.wikipedia.org/wiki/YAML Making a new recipe @@ -49,13 +56,13 @@ Description Following the title we have the description. This is a long-form description of what the recipe does, considerations around its use, and what science it is -based on. The description is `markdown`_, so some formatting can be used were +based on. The description is `Markdown`_, so some formatting can be used were helpful, papers can be linked, and so on. A little bit of unusual syntax here is ``description: |``, with the pipe after the key. This means the indented block that follows is a multiline string, so you can have as many lines as you want, provided they are all indented with at least two spaces. -.. _markdown: https://commonmark.org/help/ +.. _Markdown: https://commonmark.org/help/ .. code-block:: yaml @@ -93,6 +100,7 @@ Cube. .. code-block:: yaml + # Filter operator - operator: filters.filter_cubes constraint: operator: constraints.combine_constraints @@ -104,9 +112,9 @@ Cube. cell_methods: [] Unlike the ``read.read_cubes`` operator, we have many key-value pairs in this -step. The other keys the step are the named arguments that operator takes. Each -operator implicitly takes its first argument from previous step, but this can be -overridden by explicitly providing it. +step. The other keys in the step are the named arguments that operator takes. +Each operator implicitly takes its first argument from previous step, but this +can be overridden by explicitly providing it. The other thing to note is that the arguments of operators can themselves be operators. This allows nesting operators to use their output as arguments to @@ -120,6 +128,7 @@ the method by which it is done. .. code-block:: yaml + # Collapse operator - operator: collapse.collapse coordinate: time method: MEAN @@ -132,6 +141,7 @@ the ``write.write_cube_to_nc`` operator. This finishes up our recipe. .. code-block:: yaml + # Plotting and writing operators - operator: plot.spatial_contour_plot filename: CSET_OUTPUT_PATH @@ -177,8 +187,8 @@ After following this far your recipe should look like this: Running the Recipe ------------------ -We can run this recipe using `the same data`_ as was used for :doc:`the first -tutorial `. +We can run this recipe using `the same data`_ as was used for the +:doc:`run-recipe` tutorial. .. _the same data: https://github.com/MetOffice/CSET/raw/main/tests/test_data/air_temp.nc diff --git a/docs/source/getting-started/index.rst b/docs/source/getting-started/index.rst index ed37bd92a..f035a6c4c 100644 --- a/docs/source/getting-started/index.rst +++ b/docs/source/getting-started/index.rst @@ -1,9 +1,9 @@ Getting Started =============== -This section contains tutorials suitable for someone getting started with CSET. -By following them you should learn the basics of using CSET. It is recommended -that you do them in order. +This section contains tutorials suitable for getting started with CSET. By +following them you should learn the basics of using CSET. It is recommended that +you do them in order. .. toctree:: :maxdepth: 2 diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst index e971eda60..8f24353cf 100644 --- a/docs/source/getting-started/installation.rst +++ b/docs/source/getting-started/installation.rst @@ -3,12 +3,6 @@ Installation .. Tutorial saying how to install CSET. For edge cases should link elsewhere. -.. note:: - - This page details installing a released version of CSET. If you instead want - to run a development version that has yet to be released, see - :doc:`/contributing/getting-started`. - The recommended way to install CSET is via conda_. It is packaged on `conda-forge`_ in the ``cset`` package. The following command will install CSET into its own conda environment, which is recommended to avoid possible package @@ -42,5 +36,12 @@ something like ``CSET v0.4.0``. You now have CSET installed, so try another tutorial. +.. note:: + + This page details installing a released version of CSET. If you instead want + to run a development version that has yet to be released, see + :doc:`/contributing/getting-started`. + + .. _conda: https://docs.conda.io/en/latest/ .. _conda-forge: https://anaconda.org/conda-forge/cset diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index 977e75ae0..c6c2f2dd3 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -6,9 +6,8 @@ Visualising a recipe graphically In this tutorial we will investigate what is going on inside of a recipe, and visualise the *operators* inside. -Using the ``cset cookbook`` command find the "Mean Air Temperature Spatial Plot" -recipe, and click the button to use it. This will write out a ``.yaml`` file to -your current directory containing your selected recipe. +As in the previous tutorial use the ``cset cookbook`` command to find the "Mean +Air Temperature Spatial Plot" recipe. We will now visualise the steps inside the recipe using the ``cset graph`` command. From 479500a86553c80bcb1801125ce7dbeb0e1171b3 Mon Sep 17 00:00:00 2001 From: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com> Date: Tue, 21 Nov 2023 17:01:06 +0000 Subject: [PATCH 16/21] Update create-first-recipe.rst --- docs/source/getting-started/create-first-recipe.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index 42609b1de..a70e651a9 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -32,7 +32,7 @@ We will now create a recipe from scratch. This recipe will plot a specific timestamp of the example air temperature data. Start by opening a new file in your favourite text editor, and save it somewhere -accessible as ``single_timestep_surface_air_temperature_spacial_plot.yaml``. +accessible as ``single_timestep_surface_air_temperature_spatial_plot.yaml``. Recipe Metadata --------------- From 6c1453f096de367d4c4758ef09ef1a0987dd326d Mon Sep 17 00:00:00 2001 From: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com> Date: Tue, 21 Nov 2023 17:07:36 +0000 Subject: [PATCH 17/21] Update create-first-recipe.rst --- .../getting-started/create-first-recipe.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index a70e651a9..6b26a59f9 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -56,7 +56,7 @@ Description Following the title we have the description. This is a long-form description of what the recipe does, considerations around its use, and what science it is -based on. The description is `Markdown`_, so some formatting can be used were +based on. The description is `Markdown`_, so some formatting can be used where helpful, papers can be linked, and so on. A little bit of unusual syntax here is ``description: |``, with the pipe after the key. This means the indented block that follows is a multiline string, so you can have as many lines as you want, @@ -76,7 +76,7 @@ Recipe Steps Just as in baking you would follow a recipe step-by-step, so does CSET. The steps of the recipe are all under the ``steps`` key. Each block prefixed with a -``-`` (which makes a list in YAML) is a step, and they are run in order. +``-`` (which makes a list in YAML) is a step, and they are run in order from top to bottom. Each step has an ``operator`` key, which specifies which operator to use. A `complete list of operators is in the documentation`_, but for this tutorial we @@ -84,7 +84,7 @@ will describe them here. .. _complete list of operators is in the documentation: https://metoffice.github.io/CSET/reference/operators -The first step we want is to read in the model data. This is done with the +The first step reads in the model data. This is done with the ``read.read_cubes`` operator. The first step is special, and receives the path to the input data as its implicit input. @@ -93,9 +93,9 @@ to the input data as its implicit input. steps: - operator: read.read_cubes -Once we have read the data, we need to filter it to just that we want. -``filter.filter_cubes`` is the operator for that. It also has the -additional effect of turning the CubeList returned by ``read.read_cubes`` into a +Once we have read the data, we need to filter them down to the data we require for our computations. +``filter.filter_cubes`` is the operator for that. It also ensures that the +CubeList returned by ``read.read_cubes`` is turned into a Cube. .. code-block:: yaml @@ -113,16 +113,16 @@ Cube. Unlike the ``read.read_cubes`` operator, we have many key-value pairs in this step. The other keys in the step are the named arguments that operator takes. -Each operator implicitly takes its first argument from previous step, but this +Each operator implicitly takes its first argument from the previous step, but this can be overridden by explicitly providing it. -The other thing to note is that the arguments of operators can themselves be +Note that arguments of operators can themselves be operators. This allows nesting operators to use their output as arguments to other operators. -Next we want to reduce the dimensionality of the data ahead of plotting. In this -case we want the mean of the time coordinate. The ``collapse.collapse`` operator +Next we reduce the dimensionality of the data ahead of plotting. In this +case we chose the mean of the time coordinate. The ``collapse.collapse`` operator allows us to do this, and takes as parameters the coordinate to collapse, and the method by which it is done. @@ -133,7 +133,7 @@ the method by which it is done. coordinate: time method: MEAN -Finally we can actually plot the model data, using the +Finally we plot the model data, using the ``plot.spatial_contour_plot`` operator, and then save the processed data with the ``write.write_cube_to_nc`` operator. This finishes up our recipe. From 3123209afd89c91260441504de799c0be00b6a79 Mon Sep 17 00:00:00 2001 From: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com> Date: Tue, 21 Nov 2023 17:09:26 +0000 Subject: [PATCH 18/21] Update installation.rst --- docs/source/getting-started/installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst index 8f24353cf..a4a198829 100644 --- a/docs/source/getting-started/installation.rst +++ b/docs/source/getting-started/installation.rst @@ -3,7 +3,7 @@ Installation .. Tutorial saying how to install CSET. For edge cases should link elsewhere. -The recommended way to install CSET is via conda_. It is packaged on +For a user and not a developer of CSET the recommended way to install CSET is via conda_. It is packaged on `conda-forge`_ in the ``cset`` package. The following command will install CSET into its own conda environment, which is recommended to avoid possible package conflicts. From 1b3b86deee24f9a25fe9577cee06771f0d5b42c0 Mon Sep 17 00:00:00 2001 From: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com> Date: Tue, 21 Nov 2023 17:11:41 +0000 Subject: [PATCH 19/21] Update run-recipe.rst --- docs/source/getting-started/run-recipe.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/getting-started/run-recipe.rst b/docs/source/getting-started/run-recipe.rst index 33469e4b1..038686c78 100644 --- a/docs/source/getting-started/run-recipe.rst +++ b/docs/source/getting-started/run-recipe.rst @@ -18,7 +18,7 @@ Try the following: cset cookbook -This will write out a recipes folder contain recipe ``.yaml`` files to your +This will write out a recipes folder containing recipe ``.yaml`` files to your current directory. We will use the ``mean_surface_air_temperature_spatial_plot.yaml`` recipe. From 2981d825fe20451c77ed466d0ebcfea862bb8bce Mon Sep 17 00:00:00 2001 From: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com> Date: Tue, 21 Nov 2023 17:15:00 +0000 Subject: [PATCH 20/21] Update visualise-recipe.rst --- docs/source/getting-started/visualise-recipe.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index c6c2f2dd3..22d7d51d2 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -34,8 +34,8 @@ To see more detail about each individual operator running we can use the .. image:: recipe-graph-details.svg :alt: Graph visualisation of a CSET recipe generated by cset graph, showing the operator details. -Now we can see the structure of the recipe we can delve into what each operator -is doing. The ellipses are the operators, and the arrows between them show where +Now we can see the structure of the recipe graphically we can delve into what each operator +is doing. The ellipses represent the operators, and the arrows between them show where they pass their output to the next operators. The first operator in the recipe is ``read.read_cubes``, however it @@ -48,9 +48,9 @@ are two constraints used here, the STASH code, and the cell methods. These are combined into a single constraint by the ``constraints.combine_constraints`` operator before being used by the ``filters.filter_cubes`` operator. -Afterwards the cube has its time dimension meaned by the ``collapse.collapse`` +Afterwards the cube has its time dimension removed by the mean method applied by the ``collapse.collapse`` operator, so it becomes two-dimensional. Then it passes to the -``plot.spacial_contour_plot`` and ``write.write_cube_to_nc`` operators to be +``plot.spatial_contour_plot`` and ``write.write_cube_to_nc`` operators to be plotted and saved. You now know how to visualise a recipe, and a little about the operators it is From bec4b98c90c45b01f35853e4fb0d30e7254ab51a Mon Sep 17 00:00:00 2001 From: James Frost Date: Tue, 21 Nov 2023 17:25:57 +0000 Subject: [PATCH 21/21] Fix some more spelling --- docs/source/getting-started/create-first-recipe.rst | 2 +- docs/source/getting-started/installation.rst | 2 +- docs/source/getting-started/visualise-recipe.rst | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/getting-started/create-first-recipe.rst b/docs/source/getting-started/create-first-recipe.rst index 6b26a59f9..5cb100ec8 100644 --- a/docs/source/getting-started/create-first-recipe.rst +++ b/docs/source/getting-started/create-first-recipe.rst @@ -196,7 +196,7 @@ Use ``cset bake`` to run your newly created recipe. .. code-block:: bash - cset bake air_temp.nc output/ single_timestep_surface_air_temperature_spacial_plot.yaml + cset bake air_temp.nc output/ single_timestep_surface_air_temperature_spatial_plot.yaml You can investigate the created plot and data file in the specified ``output`` directory. diff --git a/docs/source/getting-started/installation.rst b/docs/source/getting-started/installation.rst index a4a198829..2ec992285 100644 --- a/docs/source/getting-started/installation.rst +++ b/docs/source/getting-started/installation.rst @@ -3,7 +3,7 @@ Installation .. Tutorial saying how to install CSET. For edge cases should link elsewhere. -For a user and not a developer of CSET the recommended way to install CSET is via conda_. It is packaged on +For a user of CSET the recommended way to install CSET is via conda_. It is packaged on `conda-forge`_ in the ``cset`` package. The following command will install CSET into its own conda environment, which is recommended to avoid possible package conflicts. diff --git a/docs/source/getting-started/visualise-recipe.rst b/docs/source/getting-started/visualise-recipe.rst index 22d7d51d2..0c7130731 100644 --- a/docs/source/getting-started/visualise-recipe.rst +++ b/docs/source/getting-started/visualise-recipe.rst @@ -14,7 +14,7 @@ command. .. code-block:: bash - cset graph recipes/mean-air-temp-spacial-plot.yaml + cset graph recipes/mean-air-temp-spatial-plot.yaml This should open an image of a visualisation of the recipe. Each node is a step, or an *operator*, which does a single processing task. You can see that later @@ -29,7 +29,7 @@ To see more detail about each individual operator running we can use the .. code-block:: bash - cset graph --details recipes/mean-air-temp-spacial-plot.yaml + cset graph --details recipes/mean-air-temp-spatial-plot.yaml .. image:: recipe-graph-details.svg :alt: Graph visualisation of a CSET recipe generated by cset graph, showing the operator details.