Processor-averaged budgets

[martinjanssens]

In this branch we're implementing an evaluation of budgets of momenta and scalars (already present in modsamptend) on a processor-averaged basis, by extending the sampling routines (modsampling, modsampdata and mostly modsamptend). We want this to do patch-averaged budgets, assuming a processor block coarse grains to a reasonably sized mesoscale patch. Using processor blocks allows us to reuse a lot of processor-averaging code already present.

To make this work, I've made the following changes:

• Since we want to separate horizontal from vertical advection, I've split all the advection routines into horizontal and vertical advection, each followed by a call to samptend. This is quite an intrusive change, but I'd argue it cleans DALES, as the previous advection routines reused a lot of code. The 62 and 52 advection schemes (advec_52.f90 and advec_62.f90) are also made redundant by these changes, since we can now implement them reusing routines from advec_5th, advec_6th and advec_2nd. As far as I can tell running a few tests, there are no changes beyond machine precision to the advection that is calculated (except 52 momentum advection, where there might be a rounding difference - still inconsequential I'd say).
• Added a bunch of switches to the sampling routines, so one does not need to calculate (and allocate space for) all the budgets, and all the terms in the budgets. Specifically, I've added these switches:

  logical :: lsamptendu = .false. !< switch to sample u tendencies
  logical :: lsamptendv = .false. !< switch to sample v tendencies
  logical :: lsamptendw = .false. !< switch to sample w tendencies
  logical :: lsamptendthl = .false. !< switch to sample thl tendencies
  logical :: lsamptendqt = .false. !< switch to sample qt tendencies
  logical :: lsamptendqr = .false. !< switch to sample qr tendencies
  logical :: lsamptendnr = .false. !< switch to sample nr tendencies
  logical :: lprocblock = .false. !< switch to write (so far only tendencies) per processor block instead of domain-averaged
  logical :: ltendleib = .false. !< switch to turn on leibniz tendencies
  logical :: ltenddec = .false. !< switch to get variables needed to scale-decompose (processor-averaged) advective tendencies

• Added a decomposition (activated by ltenddec) of the advection terms for scalar budgets into terms we want to be able to separate (from what I can tell, these add up almost exactly to the exactly evaluated advection terms):
  • For a scalar $\chi$, to decompose its vertical advection $-\frac{1}{\rho_0}\partial_z\left(\rho_0w\chi\right)$ averaged over a processor into contributions from the processor-average $\chi_m$ (or $w_m$) and smaller-scales $\chi'$ ($\chi=\chi_m+\chi'$ anywhere in the processor block), we decompose $(w\chi)_m = w_m\chi_m + (w'\chi')_m$, and evaluate and write to netCDF:
    • block-averaged small-scale vertical scalar fluxes $(w'\chi')_m$ (including subgrid contributions)
    • block-averaged $w_m$ and $\chi_m$
  • To decompose horizontal advection, we evaluate the total fluxes over the processor edges. Over a processor edge (of which each processor has two: a southern and a western), we might define an edge-averaged value of scalar $\chi$ which is associated with the processor scale (let's call this $\chi_e$), and a fluctuation around the edge average associated with a "small" scale ($\chi''$, $\chi=\chi_e+\chi''$ anywhere on the edge). To reconstruct the total flux over the processor edges $(u\chi)_e = u_e\chi_e + (u''\chi'')_e$ and deconstruct it into contributions from small $(u''\chi'')_e$ and processor ($u_e\chi_e$) scales, we now evaluate and write to netCDF:
    • small-scale horizontal scalar fluxes across processor boundaries $(u'\chi')_e$ (excluding subgrid contributions),
    • boundary-averaged velocities $u_e$ on the western edge and $v_e$ on the southern edge and
    • boundary-averaged scalars $\chi_e$ on both western and southern edges.
• To write to netCDF:
  • Have split writing routines so each processor writes to its own netCDF file (samptend.*.*.xxx.nc, analogous to fielddump.*.*.xxx.nc), which we can merge after a simulation should we want
  • To enable processor edge vs processor-averaged writing, added the possibility to get processor's starting edge coordinate along a horizontal dimension to writestat_nc, and treat it as xm/ym, just like we treat finite volumes on the grid scale

This PR runs as expected for shallow convection in a large domain on Fugaku.

Potential, non-tested impacts and future improvements:

• GPU evaluation of the advection routines
• Include further switches to turn off processes which are almost never used (like addons and top)
• Include vertical fluxes of ql, and other variables we want at the block scale, like (cumulative) cloud fraction or ice

[CasparJungbacker]

Did a quick BOMEX on this branch, 2nd order advection runs fine on GPU.

[martinjanssens]

Nice, @CasparJungbacker, thanks for checking so quickly! In the best-case scenario, I've then only upset your syncing/waiting (I'm pretty sure I have), or more obscure advection schemes :)