Analysis updates #11
Analysis updates #11
Conversation
ceblanton
commented
Aug 19, 2024
•
ceblanton
commented
- Add build and publish graph for analysis scripts that specify them; done
- Generate catalog before running analysis; done
- Remove mdtf as a standalone task; done
- Add build and publish task definitions; not quite working
- Use analysis yaml instead of the analysis/rose-app.conf
But only for analysis scripts that use timeseries.
…ions For now, keep the task definition in flow.cylc but it can be elsewhere.
…ed-interval analysis
…nalysis intervals with [^] syntax
|
do analysis only option turned on |
|
do analysis only option turned off. normal graph, a bit overwhelming 3 years for both |
…lection - turn off some that usually fail - add install and publish scripts for the ray-analysis
- P1Y script freq, not cumulative - P1Y script freq, cumulative - R1 script freq
…own reasons but succeeds on retry
…-script. Remaining rose apps should set the definition themselves.
- move module loads to ppan site file - install analysis virtual envs to a subdirectory - add one more fre analysis env var expected by scripts
- install analysis envs in a subdirectory - parse legacy analysis script options
Non-accumulative timeaverage requests must conform to one of the chunks in order to pass a single time to the script in "in_data_file"
…d Bronx frequency mappings for AM5 scripts
Set them to yr1 and yr2, though they have different meaning in Bronx
in order to choose the most appropriate one for each script.
| @@ -1,5 +1,8 @@ | |||
| #!jinja2 | |||
|
|
|||
| {# fre version should be programatically set, not hard-coded. #} | |||
| {% set FRE_VERSION = 2025.01 %} | |||
There was a problem hiding this comment.
this is a "fine-for-now". In the future, a Jinja2Filter should assign this via import fre; fre.version;
There was a problem hiding this comment.
Great point, thank you. You make it sound quite doable! We should do it right away if it's only three words like that.
| # When the stall timeout is reached, a stalled workflow will exit | ||
| # and remove the /xtmp working directory. | ||
| [[events]] | ||
| stall timeout = P1W |
There was a problem hiding this comment.
The whole notion of this Cylc workflow setting, the stall timer, is a bit opaque and we haven't mentioned it anywhere (yet).
I think we need to add two emailing mechanisms: one to email when the workflow has stalled, with guidance for what to do, and if they don't the workflow will shut down. Then, another email when the workflow actually shutdown (and the workflow workdir is then removed).
site/ppan_test.cylc
Outdated
| {{ EPMT_POSTAMBLE }} | ||
| """ | ||
| [[STAGE-HISTORY]] | ||
| env-script = """ | ||
| {{ EPMT_PREAMBLE }} | ||
| epmt annotate EPMT_JOB_TAGS="exp_name:{{ EXPERIMENT }};exp_fre_mod:canopy;exp_platform:{{ PLATFORM }};exp_target:{{ TARGET }};exp_component:stage-history;exp_component_source:PLACE_HOLDER;exp_time:$CYLC_TASK_CYCLE_POINT;exp_seg_months:{{ HISTORY_SEGMENT }};pp_chunk_a_months:{{ MOS_PER_CHUNK_A }};pp_chunk_b_months:{{ MOS_PER_CHUNK_B }};script_name:$CYLC_TASK_NAME;exp_run_uuid:$CYLC_WORKFLOW_UUID" | ||
| epmt annotate EPMT_JOB_TAGS="exp_name:{{ EXPERIMENT }};exp_fre_mod:{{ FRE_VERSION }};exp_platform:{{ PLATFORM }};exp_target:{{ TARGET }};exp_component:stage-history;exp_component_source:PLACE_HOLDER;exp_time:$CYLC_TASK_CYCLE_POINT;exp_seg_months:{{ HISTORY_SEGMENT }};pp_chunk_a_months:{{ MOS_PER_CHUNK_A }};pp_chunk_b_months:{{ MOS_PER_CHUNK_B }};script_name:$CYLC_TASK_NAME;exp_run_uuid:$CYLC_WORKFLOW_UUID" | ||
| {{ EPMT_POSTAMBLE }} | ||
| """ | ||
| pre-script = module load gcp/2.3 hsm/test |
There was a problem hiding this comment.
this line still causes an error, no?
There was a problem hiding this comment.
Embarrassingly, I don't know, since I think we're using the default job runner.
We should be using the epmt job runner, and fixing anything that doesn't work.
…47.catalogs-before-analysis
