Add radiator from NetCDF file

[mattldawson]

Adds a radiator type radiator_from_netcdf_file_t that loads optical properties for the radiator from a NetCDF file.

The updated documentation can be found here: https://ncar.github.io/tuv-x/branch/develop-3-initial-optics/
It includes a new sub-section of Radiator for the new radiator type's configuration.

closes #3

[carl-drews]

There are no check boxes for Clouds or Aerosols. This means that the HTML values will always be passed into the Fortran executable. There is no option to use the default configuration. I imagine that the new JSON entry should look like this:
radiators[N]:
name: "clouds"
type: "cloud"
netcdf files:
0:
file path: "ToBeCreated.nc"

[mattldawson]

There are no check boxes for Clouds or Aerosols. This means that the HTML values will always be passed into the Fortran executable. There is no option to use the default configuration. I imagine that the new JSON entry should look like this: radiators[N]: name: "clouds" type: "cloud" netcdf files: 0: file path: "ToBeCreated.nc"

Hi @carl-drews - I'm not quite sure I follow what you're asking for, but if it helps, here's a few comments and examples.

• There is no default TUV-x configuration, there is just an example configuration that you can use as a starting point (or not) if it helps
• You can have as many or as few radiators as you want, and TUV-x treats them all the same. You can use the "name" field to give the radiators meaningful names, but TUV-x essentially ignores these (other than to make sure they are unique).

For some examples, if you're only including aerosols, you could do:

"radiators": [
{
  "name": "aerosols",
  "type": "aerosol",
  "optical depths": [2.40e-01, 1.06e-01, 4.56e-02, 1.91e-02, 1.01e-02, 7.63e-03,
                     5.38e-03, 5.00e-03, 5.15e-03, 4.94e-03, 4.82e-03, 4.51e-03,
                     4.74e-03, 4.37e-03, 4.28e-03, 4.03e-03, 3.83e-03, 3.78e-03,
                     3.88e-03, 3.08e-03, 2.26e-03, 1.64e-03, 1.23e-03, 9.45e-04,
                     7.49e-04, 6.30e-04, 5.50e-04, 4.21e-04, 3.22e-04, 2.48e-04,
                     1.90e-04, 1.45e-04, 1.11e-04, 8.51e-05, 6.52e-05, 5.00e-05,
                     3.83e-05, 2.93e-05, 2.25e-05, 1.72e-05, 1.32e-05, 1.01e-05,
                     7.72e-06, 5.91e-06, 4.53e-06, 3.46e-06, 2.66e-06, 2.04e-06,
                     1.56e-06, 1.19e-06, 9.14e-07],
  "single scattering albedo": 0.99,
  "asymmetry factor": 0.61,
  "550 nm optical depth": 0.235,
  "enable diagnostics": false
}
]

or you could use the new "from netcdf file" type for aerosols if you want to specify the optical properties at each vertical level and wavelength band explicitly:

"radiators": [
{
  "name": "aerosols",
  "type": "from netcdf file",
  "file path": "path/to/aerosol_optics.nc"
}
]

You can even include brand-new radiators; e.g., if you want to include clouds and aerosols and hot-air balloons:

"radiators": [
{
  "name": "aerosols",
  "type": "from netcdf file",
  "file path": "path/to/aerosol_optics.nc"
},
{
  "name": "clouds",
  "type": "from netcdf file",
  "file path": "path/to/cloud_optics.nc"
},
{
  "name": "hot air balloons",
  "type": "from netcdf file",
  "file path": "path/to/balloon_optics.nc"
}
]

Hope this helps! Let me know if you want to meet to discuss this further.

[carl-drews]

No Check Box: The Calculator user will always end up with a Cloud radiator. There is no way to "not include clouds."

[carl-drews]

The Fortran executable is giving me a segfault for the new radiator clouds.nc:

cmd = export LD_LIBRARY_PATH="/opt/local/lib64:/opt/local/lib:$LD_LIBRARY_PATH"; /home/drews/TUV-x/tuvx-build/tuv-x/build/tuv-x tuvx_config.json

stdout = Running TUV-x(0.6.0) without OpenMP support


stderr =
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x7f651bebf3ff in ???
#1 0x7f651bf09fbe in ???
#2 0x7f651b7583dc in ???
#3 0x7f651b7585d8 in ???
#4 0x7f651b74678b in ???
#5 0x4f6aa8 in ???
#6 0x4d023f in ???
#7 0x4c6dad in ???
#8 0x445ebb in ???
#9 0x443f6a in ???
#10 0x406f81 in ???
#11 0x4030df in ???
#12 0x4029dc in ???
#13 0x7f651beab554 in ???
#14 0x402a0c in ???
#15 0xffffffffffffffff in ???
/bin/sh: line 1: 212551 Segmentation fault /home/drews/TUV-x/tuvx-build/tuv-x/build/tuv-x tuvx_config.json


Traceback (most recent call last):

Here is a look at the clouds.nc file:
modeling2:/data/TUV-x/stage/20231015-170625--945853> ncdump -h clouds.nc
netcdf clouds {
dimensions:
vertical_level = 119 ;
wavelength = 157 ;
variables:
float optical_depth(wavelength, vertical_level) ;
optical_depth:units = "" ;
float single_scattering_albedo(wavelength, vertical_level) ;
single_scattering_albedo:units = "" ;
float asymmetry_factor(wavelength, vertical_level) ;
asymmetry_factor:units = "" ;
}
modeling2:/data/TUV-x/stage/20231015-170625--945853>

Here are the files in case I cannot attach them
tuvx_config.json
clouds.nc.txt
to this PR:
modeling2:/data/TUV-x/stage/20231015-170625--945853> ls -lh
total 304K
-rw-r--r-- 1 drews support 227K Oct 15 17:06 clouds.nc
lrwxrwxrwx 1 drews support 45 Oct 15 17:06 data -> /home/drews/TUV-x/tuvx-build/tuv-x/build/data/
-rw-r--r-- 1 drews support 75K Oct 15 17:06 tuvx_config.json

[carl-drews]

From: Carl Drews [email protected]
Date: Tue, Oct 17, 2023 at 2:42 PM
Subject: Re: Segmentation fault with clouds
To: Matthew Dawson [email protected]
Cc: Kyle Shores [email protected]

Matt -

Your solution works for clouds.nc. Array dimensions vertical_level and wavelength need to be one element shorter because they refer to levels between the reference markers. Thanks!

I will calculate a new height array for clouds.nc, where the values will be the average of the adjacent heights. That new array will be reference array for Top and Base.

Carl

On Mon, Oct 16, 2023 at 2:44 PM Matthew Dawson <[email protected]> wrote:
Hi Carl,

Yeah, I think I'll try to run in debug mode and with valgrind, if necessary. Hopefully, later today or tomorrow.

Yes, we aim for meaningful error messages, but TUV-x is still pretty new so there are probably a lot of corner-case configuration cases that we don't pick up on yet. We can at least make sure this case gets covered when we figure it out.

Best,
Matt

[mattldawson]

Hi @carl-drews - I believe the problem was a bug in musica-core that prevented TUV-x from returning an error message when the array dimensions in the NetCDF file were incorrect. I think if you rebuild TUV-x, and try to run this configuration again you should get an error related to the array dimensions.

[codecov-commenter]

Codecov Report

All modified lines are covered by tests ✅

❗ No coverage uploaded for pull request base (main@f2d2ec0). Click here to learn what that means.

Additional details and impacted files

@@           Coverage Diff           @@
##             main       #7   +/-   ##
=======================================
  Coverage        ?   77.65%           
=======================================
  Files           ?      109           
  Lines           ?     8638           
  Branches        ?        0           
=======================================
  Hits            ?     6708           
  Misses          ?     1930           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[carl-drews]

From: Carl Drews [email protected]
Date: Wed, Oct 18, 2023 at 6:38 PM
Subject: Re: [NCAR/tuv-x] Add radiator from NetCDF file (PR #7)
To: Matthew Dawson [email protected]

Matt -

Got it!

Modify air profile with density 1.1222e+19 at height 7.77 ground 3.66.

Modify temperature profile with temperature 237.70655 at height 7.77 ground 3.66.

Modify height profile at height 7.77 ground 3.66.

workingDir = /home/drews/TUV-x/

Cloud file /data/TUV-x/stage/20231018-182940--092712/clouds.nc will contain 119 levels and 157 wavelengths.
Fill in clouds from height index 3 up to 11
optical_depth: ('wavelength', 'vertical_level')
single_scattering_albedo: ('wavelength', 'vertical_level')
asymmetry_factor: ('wavelength', 'vertical_level')
Moving to stage directory /data/TUV-x/stage/20231018-182940--092712/

cmd = export LD_LIBRARY_PATH="/opt/local/lib64:/opt/local/lib:$LD_LIBRARY_PATH"; /home/drews/TUV-x/tuvx-build/tuv-x/build/tuv-x tuvx_config.json

stdout = Running TUV-x(0.6.0) without OpenMP support


stderr = ERROR (Musica-230324709): Wrong size container for variable 'optical_depth' in file 'clouds.nc': Expected 119 got 118 for dimension 1
STOP 3


Traceback (most recent call last):
File "tuv-x-remote.py", line 906, in
retValue = main()
File "tuv-x-remote.py", line 821, in main
fpPhoto = netCDF4.Dataset("photolysis_rate_constants.nc", 'r')
File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.init
File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'photolysis_rate_constants.nc'
modeling2:/home/drews/TUV-x>

[carl-drews]

And when the arrays are the correct size, we get TUV-x output:

Moving to stage directory /data/TUV-x/stage/20231018-184608--755188/

cmd = export LD_LIBRARY_PATH="/opt/local/lib64:/opt/local/lib:$LD_LIBRARY_PATH"; /home/drews/TUV-x/tuvx-build/tuv-x/build/tuv-x tuvx_config.json

stdout = Running TUV-x(0.6.0) without OpenMP support


stderr =

Date = 20230320 time = 19:00:00

   0 altitude                                7.77000e+00
   1 wavelength                              1.20700e+02
   2 time                                    1.90000e+01
   3 solar zenith angle                      4.00722e+01
   4 Earth-Sun distance                      9.95857e-01
   5 temperature                             2.37707e+02
   6 O2+hv->O+O                              3.31282e-20
   7 HNO4+hv->HO2+NO2                        7.39032e-06
   8 O3+hv->O2+O(1D)                         2.39979e-05
   9 O3+hv->O2+O(3P)                         4.65101e-04
  10 NO3-(aq)+hv->NO2(aq)+O-                 4.34444e-08
  11 NO3-(aq)+hv->NO2-(aq)+O(3P)             3.16748e-08
  12 NOCl+hv->NO+Cl                          3.58377e-03
  13 CH3Cl+hv->Products                      2.20536e-17
  14 CH3CCl3+hv->Products                    1.66665e-15
  15 CCl2O+hv->Products                      8.11572e-16
  16 CClFO+hv->Products                      5.05780e-16
  17 CH3OOH+hv->CH3O+OH                      6.10504e-06
  18 HOCH2OOH+hv->CH2(OH)O+OH                5.18592e-06
  19 HO2+hv->OH+O              

[carl-drews]

Error message appears when array sizes are incorrect, and radiation output changes when cloud parameters change.