Add age of air diagnostic functionality

[jwarner8]

Adding capability to run the age of air diagnostic on data, linked to issue #269

[github-actions]

[jwarner8]

Tests to do include;

• Test calc distance with some known values.
• Test shape is correctly rejecting if not all equal, also if time coord not hourly.
• Check if time decreasing will fail correctly.
• Check pressure level not found fails correctly.
• check all variables present (need to add this in function directly).

[jwarner8]

.yaml is going to look something like 1. read cubes, 2. filter cubes (u,v,w,p), might need some additional filtering to only get pressure level data, otherwise it might start returning 10m winds and we don't want/use this. Better to do this filtering in dedicated operators than start finding ways around this within the age of air operator itself. Run the age of air diagnostic, and then produce contourf plot; tbd (and data)

[jfrost-mo]

Some vague review comments. I've not properly read through and understood the code yet though. I'll do that a bit later.

[jwarner8]

Some vague review comments. I've not properly read through and understood the code yet though. I'll do that a bit later.

thanks! There is a lot more to add yet, this is mostly initial commits for copy/pasting code into CSET. So best to hold of any review, as code needs a lot of tidying up/documentation, plus all the testing and cylc/yaml additions

[jfrost-mo]

Got a bit eager, but its great to see it coming along. I'll mark this PR as a draft for now.

[jwarner8]

Meta

• Create new template variable, with description. Q: what is sort-key.
• If age of air turned on, create a new variety of options; plev; list, list of pressure levels to run code on. Includevertical; bool, cyclic; bool.
• These variables (plev, includevertical, cyclic) need passing to yaml. Q: how to we get menu's to appear if diagnostic turned on.

YAML

• Will inherit plev, includevertical, cyclic from those selected in the GUI (and thus rose-suite.conf). For the filter cubes, if includevertical is false, we don't need vertical velocity fields. Q: how to make the YAML not look for this STASH if option selected? Can we put logic in YAML? Or would it be better to have two YAML (vertical and no vertical)?
• To plot, we can use standard contour plot and timeseries plot functionality already in CSET.

Cycling

• Will run AOA task on each forecast intialisation, and loop through each plev specified in rose-suite.conf. For example for 2 forecasts, with only ageofair diagnostic turned on, and plev= [250,500,850], run-time would look like;

20240417T0000Z
task: ageofair_250
task: ageofair_500
task: ageofair_850
20240417T1200Z
task: ageofair_250
task: ageofair_500
task: ageofair_850

So all pressure levels run in parallel.

[jwarner8]

Only thing left to do now is build the cylc workflow, tests etc. all done.

[jwarner8]

Now ready for first stage technical review (quite a big changelog)
A TODO is how to visualise the result, 2D plots for each leadtime, but also want a timeseries (collapse latitude,longitude)

[jfrost-mo]

Thanks James, this seems like a really useful operator.

The largest thing is going to be updating the recipe to accommodate the changes we have made in CSET over the past couple months. It would probably be worth rebasing your changes on the latest version of the main branch to make that easier. (git rebase main, then fix a bunch of conflicts...)

I've noted a bunch of specific points, but the general design and approach are sound.

[jfrost-mo]

Note that there are other suggestions that GitHub has hidden under "hidden conversations". You might be better off looking through the Files tab for them.

[jwarner8]

I think the design of this operator needs some consideration with how it will work with the current default of CSET of cycling through valid-time. The operator expects to be run once, for a given initialisation, and passed all the data for that forecast. I don't really know how compatible this is with the current architecture, because each calculation at each forecast time depends on the preceding data.

[jfrost-mo]

For now this can be worked around by running the operator within the collate step. We are currently discussing a longer term change to cycling on data time, which would alleviate this.

[jwarner8]

Update 04/07 - dependency on ability for regrid to handle cubelists (which will make the yaml much nicer as we need to repeat regridding for multiple variables). Links #720 #721

[jwarner8]

Ready for final review. Only thing not implemented is the realise_data, as this throws up AttributeError: 'Cube' object has no attribute 'realise_data'. I am using standard names in the recipe x_wind, y_wind but suggestions of how I implement any existing lookup table welcome. I am also getting low coverage due to calls within multiprocessing - which might be related to how pytest-cov generates the coverage report (https://stackoverflow.com/questions/61143858/how-to-measure-coverage-when-using-multirpocessing-via-pytest).

[daflack]

One minor change in documentation: a bit of legacy documentation was still left over. Otherwise, from a scientific perspective it looks good.

[jwarner8]

Coverage near 100% now, so ready for final check before merge. Thanks all for time reviewing.