3D physics diagnostic output, can't output aux3d, 3D convective mass fluxes, cldfra, etc

[aschuh]

Apologize, this is equivalent to a bump of an earlier question (related to inability to get aux variables or cloud mass fluxes to output). I'm struggling with getting any 3D physics output to be returned when added to diag_table. For example, I've tried to add a variety of output, some of which should be being output be default apparently (cloud mass fluxes w/ ldiag3D = TRUE). I can make changes to "gfs_dyn" like tracers (e.g. dust and added co2/ch4 tracers) and they end up in atmnc interpolated Gaussian Grid files. However, no matter what I put in, I can't seem to get the 3D physics variables below to be output, either in RESTART or in atmnc files. I can remove one of the 2d physics vars and it IS removed from the sfcf*.nc file output. Furthermore, the example diag_table in the ufs docs doesn't appear to contain any 3D physics fields. Sorry for being persistent ,but this seems like a basic thing to be able to expect to do.

Can somebody tell me if I'm missing a flag or something? I can't output the aux3d, upd_mf, cldfra, etc variables. Does order of the variables matter here?

20161003.00Z.C96.64bit.non-mono
2016 10 03 00 0 0

"fv3_history", 0, "hours", 1, "hours", "time"
"fv3_history2d", 0, "hours", 1, "hours", "time"

Auxiliary output, only co2 is output below, none of the other variables

"gfs_phys", "aux3d_01", "aux3d_01", "fv3_history", "all", .false., "none", 2
"gfs_phys", "upd_mf", "upd_mf", "fv3_history", "all", .false., "none", 2
"gfs_phys", "dwn_mf", "dwn_mf", "fv3_history", "all", .false., "none", 2
"gfs_phys", "det_mf", "det_mf", "fv3_history", "all", .false., "none", 2
"gfs_dyn", "co2", "co2", "fv3_history", "all", .false., "none", 2
"gfs_phys", "cldfra", "cldfra", "fv3_history", "all", .false., "none", 2

MODEL CONFIGURE FILE:
start_year: 2021
start_month: 03
start_day: 22
start_hour: 06
start_minute: 0
start_second: 0
nhours_fcst: 3
fhrot: 0

dt_atmos: 600
restart_interval: 1 -1
output_1st_tstep_rst: .false.

quilting: .true.
write_groups: 1
write_tasks_per_group: 6
itasks: 1
output_history: .true.
write_dopost: .true.
write_nsflip: .true.
num_files: 2
filename_base: 'atm' 'sfc'
output_grid: gaussian_grid
output_file: 'netcdf'
ideflate: 0
nbits: 0
ichunk3d: 0
jchunk3d: 0
kchunk3d: 0
imo: 384
jmo: 190

output_fh: 0 1 2 3
iau_offset: 0

#####################
PART OF INPUT.NML file:
####################
...
&gfs_physics_nml
fhzero = 1
h2o_phys = .true.
ldiag3d = .true.
qdiag3d = .true.
dtend_select = 'dtendmp', 'dtend_temp_'
naux2d = 0
naux3d = 3
aux3d_time_avg = .false.,.false.,.false.
print_diff_pgr = .false.
fhcyc = 24
...

##################

andrew

[gspetro-NOAA]

Hi Andrew,

Have you tried increasing max_output_fields in the &diag_manager_nml section of input.nml? I'm not convinced that's the problem, since the issue does seem to be specific to 2d physics, but it couldn't hurt to try. Also wondering if you checked out the next section of the docs, which is specific to outputting auxilliary arrays. I'm not an expert on this topic, so it's quite possible neither of these is the issue, but I'm going to see if I can find someone who might know better what to do.

It seems that you are not the only one who has encountered this problem, since @gsketefian posted a question about this in the SRW App repo (Discussion #862), which also doesn't have an answer.

Best,
Gillian Petro

[gspetro-NOAA]

@aschuh It would also be helpful for us to know more about your system and use case. For example, could you let us know:

• The platform or system being used (e.g., Hera, Orion, WCOSS, MacOS, Linux)
• The application (SRW, MRW, etc.) or configuration of the UFS Weather Model (ATM, S2S, etc.) involved
• Compiler + MPI combination being used
• Run directory and code directory (if on an RDHPCS/Level 1 system)

Thanks,
Gillian

[aschuh]

Working on Orion, using ufs-weather-model git pull and then my own forks/branches w/ very minimal changes. Not sure about which application, I typically just use an example out of the Regression Tests folder (RT) to create a sandbox and the reconfigure that. Compiler/MPI stuff is handled by Orion, don't have to do much but point to orion module file. I'm currently running code out of

/work2/noaa/co2/aschuh/ufs-weather-model

Latest run directory is at: /work2/noaa/co2/aschuh/stmp/aschuh/FV3_RT/rt_15823/merra2_thompson_ghg

Having the diag manager embedded in a library that is linked to means it isn't easy to "first order" dig into the code to see where things go awry.

andrew

[dustinswales]

@aschuh Have you looked for the diagnostic tendencies in the sfc*.nc files?

[aschuh]

I typically look at sfcnc, atmnc, and physnc out of RESTART directory. Those files are called "surface" presumably and only appear to contain 2D fields, hence I really didn't expect the 3D physics fields to output to those files. I will double check w/ another run right now, but NO, I don't think they are written to the sfcnc files.

[dustinswales]

Yeah I know it makes no sense. I was banging my head for days trying to find where the tendencies was being written and noticed that the sfcf files were much larger than the atmf files.

[aschuh]

@dustinwales You're RIGHT! My apologies, I hadn't been carefully looking at the sfc*nc files as I seemed to think they would only be 2d fields, which is what they contain by default. Both the dtend as well as the convective mass fluxes are output to those sfc files. Quick question, my real interest is in the convective mass fluxes from the parameterization. If you look above, I'm interpolating a bunch of this stuff which makes it easy to plot, however, I need to work on a mixing scheme based on the mass fluxes and thus need the variables on the native grid for the code debugging. What is best way to quickly move some of these 3D vars to native grid? Presumably being written to RESTART directory. I tried turning off quilting in model_configure but it crashes...