Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TUV-x to CAM #784

Draft
wants to merge 94 commits into
base: cam_development
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 93 commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
452b0fc
add tuv-x to externals
mattldawson Oct 6, 2022
fdf16a3
update externals for tuv-x and json-fortran
mattldawson Oct 10, 2022
9507439
add tuv-x to cam build
mattldawson Oct 12, 2022
6fa3038
compile in MPI support only if needed
mattldawson Oct 13, 2022
2e3b965
Merge pull request #1 from mattldawson/develop-160-tuv-build
K20shores Oct 13, 2022
8403076
draft TUV-x wrapper; link TUV-x libs
mattldawson Oct 18, 2022
dd39794
add MPI/OpenMP support to TUV-x wrapper
mattldawson Oct 19, 2022
9f6b6d2
offset OMP thread for use with array of TUV-x cores
mattldawson Oct 19, 2022
5f1378c
add TUV-x run function
mattldawson Oct 20, 2022
46ff4f3
Merge pull request #2 from mattldawson/develop-162-draft-wrapper
mattldawson Oct 21, 2022
ab18e82
create temporary TUV-x for CAM (#3)
mattldawson Oct 21, 2022
5da13c7
add height and temperature updaters to tuv-x wrapper
mattldawson Nov 1, 2022
7a59241
set up remaining profile updaters
mattldawson Nov 2, 2022
ffb0f5e
send wavelength grid to TUV-x
mattldawson Nov 3, 2022
9cebe0d
add radiator updater for aerosols
mattldawson Nov 4, 2022
3241b31
send CAM height and temperature data to TUV-x
mattldawson Nov 4, 2022
8e100d5
fix height variables sent to tuvx
mattldawson Nov 7, 2022
64f9a3b
send surface albedos to TUV-x
mattldawson Nov 7, 2022
49d323b
send air density to TUV-x
mattldawson Nov 7, 2022
0a0b3ff
send O2 and O3 concentrations to TUV-x
mattldawson Nov 7, 2022
0562e9d
sent solar zenith angle to TUV-x
mattldawson Nov 8, 2022
69a5897
send ET flux to tuv-x
mattldawson Nov 8, 2022
e3cc2ab
use native TUV-x grid for photo calcs
mattldawson Nov 9, 2022
1e05ffe
send aerosol optical properties to tuv-x
mattldawson Nov 10, 2022
80c3479
output whether using online aerosols; include air scale height for ex…
mattldawson Nov 10, 2022
5d89cb7
send above-column O2 and O3 concentrations to TUV-x
mattldawson Nov 10, 2022
422aba7
add diagnostic TUV-x output; set wavelength edges; fix et flux values
mattldawson Nov 11, 2022
147e15a
pass earth sun distance to TUV-x
mattldawson Nov 14, 2022
7854fde
minor clean up
mattldawson Nov 14, 2022
f8be1bb
update externals file
mattldawson Nov 14, 2022
13e030f
Tuvx options (#5)
fvitt Nov 15, 2022
996bd91
update for changes in TUV-x
mattldawson Nov 15, 2022
1011882
merge main to develop-167-tuvx-inputs
mattldawson Nov 15, 2022
ea8a654
draft MOZART TUV-x configuration file
mattldawson Nov 15, 2022
439a0ba
Develop 167 tuvx inputs (#4)
mattldawson Nov 16, 2022
9ded52f
draft MOZART-TS1 TUV-x config file
mattldawson Nov 16, 2022
facccaf
merge develop-add-tuvx into develop-163-mechanism-config
mattldawson Nov 16, 2022
a3b66a0
draft MOZART-TSMLT TUV-x config file
mattldawson Nov 16, 2022
ee5f4fc
draft mapping for tuv-x photo rates
mattldawson Nov 18, 2022
f4fa3dc
mix tuv map message passing
mattldawson Nov 18, 2022
583c4b5
update TUV-x TS1 config
mattldawson Nov 19, 2022
1f433ff
add euv rate calc calls to tuv-x
mattldawson Nov 23, 2022
fd715e1
fix tuvx_active flag use; fix height values near surface
mattldawson Nov 29, 2022
9c3d234
exclude dark columns from photo calculations
mattldawson Nov 29, 2022
a60e90d
fix flux and density calcs
mattldawson Dec 1, 2022
fe1f142
add missing MOZART photo rates
mattldawson Dec 1, 2022
d5c4d84
update for new TUV-x build scripts
mattldawson Dec 14, 2022
26375d9
update MOZART TUV-x config
mattldawson Dec 15, 2022
3d8852c
update TS1 TUV-x config format
mattldawson Dec 15, 2022
364d905
add h2o photo rates to TS1 TUV-x config
mattldawson Dec 15, 2022
d29ea42
add new datasets to TS1 TUV-x config
mattldawson Dec 15, 2022
3c6def8
update TUV-x config format
mattldawson Dec 15, 2022
a7e1030
add remaining TS1 config excluding jno
mattldawson Dec 21, 2022
be7e7f1
add jno stub function
mattldawson Dec 22, 2022
37326c4
finish jno calculation
mattldawson Dec 22, 2022
0608ae3
update TSMLT config file
mattldawson Dec 22, 2022
1b4f238
update tuv-x build flags
mattldawson Dec 24, 2022
790a0e1
Update tuvx_MOZART_TS1.json
mattldawson Dec 24, 2022
4b8ca93
output photo rate diagnostics on CAM vertical grid
mattldawson Jan 11, 2023
a13d6f8
Merge branch 'develop-163-mechanism-config' of https://github.com/mat…
mattldawson Jan 11, 2023
22811e6
fix TUV-x config filees for air radiator; add disable aerosol option …
mattldawson Jan 13, 2023
770f6cc
improve et flux edge calcs; use scale height for 02, 03
mattldawson Jan 20, 2023
620b8d4
fix index offset in photo rate profiles
mattldawson Jan 26, 2023
d69a00a
minor profile calculation changes
mattldawson Jan 27, 2023
1e8d9fa
add thin top layer to TUV-x grid
mattldawson Jan 30, 2023
c47a4c3
adjust top layer density calculations
mattldawson Jan 31, 2023
60e0f54
add in ET mid values
mattldawson Jan 31, 2023
eb69196
update build; adjust O2 profile
mattldawson Feb 3, 2023
1a2204e
update tuvx config files
mattldawson Feb 23, 2023
1a85f0a
use TUV-x from first time step
mattldawson Feb 23, 2023
f8882cd
use LUT for first time step
mattldawson Feb 24, 2023
07fae13
combine TS1 and TSMLT tuv-x configurations
mattldawson Feb 24, 2023
3e1f445
add clouds to TUV-x radiators
mattldawson Feb 24, 2023
6d528a2
Add TUV-x config files and use rates to drive chemistry (#7)
mattldawson Mar 14, 2023
23c8252
switching to a build of musica (#8)
K20shores Apr 4, 2023
f977c85
merge develop-add-tuvx into develop-add-clouds
mattldawson Apr 4, 2023
1919ddd
fix duplicated code from merge; remove first time step filter for usi…
mattldawson Apr 4, 2023
889e146
disable TUV-x until ready for use in CAM
mattldawson Apr 4, 2023
616fcdc
merge cam_development into develop-add-tuvx
mattldawson Apr 4, 2023
561d226
update comments
mattldawson Apr 4, 2023
e0b3ca6
merge cam_development into develop-add-tuvx
mattldawson Apr 4, 2023
c572a68
revert changes to chem-pp code
mattldawson Apr 5, 2023
0087b93
Merge branch 'cam_development' into develop-add-tuvx
mattldawson Oct 6, 2023
ab79052
disable aerosols temporarily
mattldawson Oct 6, 2023
34a48a3
update externals
mattldawson Oct 6, 2023
c90dccb
Merge branch 'cam_development' into develop-add-tuvx
mattldawson Feb 1, 2024
93983f6
update TUV-x
mattldawson Feb 1, 2024
8a8c1af
Develop add heating rates (#9)
mattldawson Mar 21, 2024
3e4c77b
remove working arrays from tuvx wrapper class
mattldawson Mar 22, 2024
a3669a5
clean up some comments
mattldawson Mar 22, 2024
c12d1e4
Address issues with the Intel compiler and aerosol wavelength grid in…
mattldawson Apr 5, 2024
15ba87b
update tuvx build
mattldawson Sep 11, 2024
ace8449
update tuv-x commit
mattldawson Sep 11, 2024
402be6e
change for DEBUG compiler flags
fvitt Sep 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ src/dynamics/fv3/atmos_cubed_sphere
libraries/FMS
libraries/mct
libraries/parallelio
libraries/tuv-x
src/atmos_phys
src/dynamics/mpas/dycore
share
Expand All @@ -23,6 +24,9 @@ src/hemco
buildnmlc
buildcppc

# ignore ide configs
.vscode

# Ignore editor temporaries and backups
*~
.#*
Expand Down
7 changes: 7 additions & 0 deletions Externals_CAM.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,13 @@ sparse = ../.mpas_sparse_checkout
hash = b8c33daa
required = True

[TUV-x]
local_path = libraries/tuv-x
protocol = git
repo_url = https://github.com/NCAR/tuv-x.git
hash = ccda76c0553064a9b7b0eba73162ddeee6d8eaff
required = True

[hemco]
local_path = src/hemco
tag = hemco-cesm1_2_1_hemco3_6_3_cesm
Expand Down
11 changes: 11 additions & 0 deletions bld/namelist_files/namelist_definition.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7352,6 +7352,17 @@ Maximum zenith angle (degrees) used for photolysis.
Default: set by build-namelist.
</entry>

<entry id="tuvx_config_path" type="char*256" input_pathname="abs" category="chemistry"
group="tuvx_opts" valid_values="" >
Filepath of TUV-X configuration specification.
Default: NONE
</entry>

<entry id="tuvx_active" type="logical" category="chemistry"
group="tuvx_opts" valid_values="" >
Switch to turn on TUV-X photolysis.
Default: FALSE
</entry>

<!-- Namelist read by seq_drydep_mod and shared by CAM and CLM -->

Expand Down
141 changes: 140 additions & 1 deletion cime_config/buildlib
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ create the cam library
# pylint: disable=unused-wildcard-import, bad-whitespace, too-many-locals
# pylint: disable=invalid-name
import sys, os, filecmp, shutil, imp

from glob import glob

_CIMEROOT = os.environ.get("CIMEROOT")
if _CIMEROOT is None:
Expand All @@ -21,6 +21,7 @@ from CIME.case import Case
from CIME.utils import run_sub_or_cmd, expect, run_cmd
from CIME.buildlib import parse_input
from CIME.build import get_standard_makefile_args
from CIME.XML.env_build import EnvBuild

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -113,12 +114,150 @@ def _build_cam(caseroot, libroot, bldroot):
logger.info("%s: \n\n output:\n %s \n\n err:\n\n%s\n", cmd, out, err)
expect(rc == 0, "Command %s failed with rc=%s" % (cmd, rc))

###############################################################################
def _run_cmd(command, working_dir):
###############################################################################

rc, out, err = run_cmd(command, from_dir=working_dir, verbose=True)
expect(rc == 0, "Command {} failed with rc={}".format(command, rc))

###############################################################################
def _cmake_default_args(caseroot):
###############################################################################
# Returns a dictionary of CMake variables based on the Macros.cmake file for
# the build.

build = EnvBuild(case_root=caseroot)
with Case(caseroot) as case:
macro_path = os.path.abspath(os.path.join(caseroot, "cmake_macros", ""))
args = "-DCONVERT_TO_MAKE=ON "
args += "-DCASEROOT={} ".format(caseroot)
args += "-DCOMPILER={} ".format(build.get_value("COMPILER"))
args += "-DOS={} ".format(build.get_value("OS"))
args += "-DMACH={} ".format(case.get_value("MACH"))
args += "-DCMAKE_C_COMPILER_WORKS=1 "
args += "-DCMAKE_Fortran_COMPILER_WORKS=1 "
args += "-DCMAKE_CXX_COMPILER_WORKS=1 "
cmd = "cmake {} .".format(args)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattldawson
I believe to get the correct compiler flags you need this line:

       args += "-DDEBUG={} ".format(build.get_value("DEBUG"))

rc, out, err = run_cmd(cmd, combine_output=True, from_dir=macro_path)
expect(rc == 0, "Command {} failed with rc={} out={} err={}".format(cmd, rc, out, err))

arg_dict = {}
for line in out.splitlines():
if ":=" in line:
key, val = line.split(":=")
arg_dict[key.replace('CIME_SET_MAKEFILE_VAR','').strip()] = val.strip()

return arg_dict

###############################################################################
def _build_tuvx(caseroot, libroot, bldroot):
###############################################################################
# Builds the TUV-x library and updates the case variables used to set the
# include paths and linked libraries

build = EnvBuild(case_root=caseroot)
with Case(caseroot) as case:
bldpath = os.path.join(bldroot, "tuv-x")
if not os.path.exists(bldpath):
os.makedirs(bldpath)
srcpath = os.path.abspath(os.path.join(case.get_value("COMP_ROOT_DIR_ATM"), \
"libraries", "tuv-x", ""))
logger.info("Building TUV-x in {} from source in {}\n".format(bldpath, srcpath))

arg_dict = _cmake_default_args(caseroot)
if build.get_value("MPILIB") == "mpi-serial":
cmake_args = "-DCMAKE_Fortran_COMPILER={} ".format(arg_dict["SFC"])
else:
cmake_args = "-DCMAKE_Fortran_COMPILER={} ".format(arg_dict["MPIFC"])
cmake_args += "-DTUVX_ENABLE_MPI:BOOL=TRUE "
if case.get_value("DEBUG"):
cmake_args += "-DCMAKE_BUILD_TYPE=Debug "
else:
cmake_args += "-DCMAKE_BUILD_TYPE=Release "
cmake_args += "-DCMAKE_C_COMPILER_WORKS=1 "
cmake_args += "-DCMAKE_CXX_COMPILER_WORKS=1 "
cmake_args += "-DTUVX_ENABLE_TESTS=OFF "
cmake_args += "-DTUVX_ENABLE_COVERAGE=OFF "
cmake_args += "-DCMAKE_Fortran_FLAGS='{}' ".format(arg_dict["FFLAGS"])
cmake_args += "-DCMAKE_INSTALL_PREFIX='{}' ".format(libroot)
cmake_args += "-DTUVX_INSTALL_INCLUDE_DIR='{}' ".format(_tuvx_include_dir(libroot))
cmake_args += "-DTUVX_INSTALL_MOD_DIR='{}' ".format(_tuvx_include_dir(libroot))
cmake_args += srcpath

_run_cmd("cmake {}".format(cmake_args), bldpath)
_run_cmd(case.get_value("GMAKE"), bldpath)
_run_cmd("{} install".format(case.get_value("GMAKE")), bldpath)

# add TUV-x to include paths
incldir = os.environ.get('USER_INCLDIR')
if incldir is None:
incldir = ''
os.environ['USER_INCLDIR'] = incldir + \
" -I{} ".format(_tuvx_include_dir(libroot))

# create simlink to library in folder CIME expects libraries to be in
dst = os.path.join(libroot, "libtuvx.a")
if os.path.isfile(dst):
os.remove(dst)
os.symlink(_tuvx_lib_path(libroot), dst)

###############################################################################
def _tuvx_include_dir(libroot):
###############################################################################
# Returns the path to the TUV-x include directory

coreinc = os.path.join(libroot, "include", "")
expect(os.path.exists(coreinc), \
"TUV-x include directory not found at {}".format(coreinc))
return coreinc

###############################################################################
def _tuvx_lib_path(libroot):
###############################################################################
# Returns the path to the TUV-x library

corelib = os.path.join(_tuvx_install_dir(libroot), "lib64", "libtuvx.a")
if not os.path.exists(corelib):
corelib = os.path.join(_tuvx_install_dir(libroot), "lib", "libtuvx.a")
expect(os.path.exists(corelib), \
"TUV-x library not found at {}".format(corelib))
return corelib

###############################################################################
def _tuvx_install_dir(libroot):
###############################################################################
# Returns the path to the TUV-x install directory

corepaths = glob(os.path.join(libroot, "tuvx*"))
expect(len(corepaths)>0, \
"TUV-x not found at {}".format(libroot))
expect(len(corepaths)<2, \
"Multiple TUV-x versions found at {}".format(libroot))
expect(os.path.exists(corepaths[0]), \
"TUV-x install directory not found at {}".format(corepaths[0]))
return corepaths[0]

###############################################################################
def _tuvx_package_dir(libroot):
###############################################################################
# Returns the path to the TUV-x CMake package

paths = glob(os.path.join(libroot, "tuvx*", "cmake", "tuvx*" ))
expect(len(paths)>0, \
"TUV-x package not found at {}".format(libroot))
expect(len(paths)<2, \
"Multiple TUV-x versions found at {}".format(libroot))
expect(os.path.exists(paths[0]), \
"TUV-x package directory not found at {}".format(paths[0]))
return paths[0]

###############################################################################


def _main_func():
caseroot, libroot, bldroot = parse_input(sys.argv)
_build_tuvx(caseroot, libroot, bldroot)
_build_cam(caseroot, libroot, bldroot)


Expand Down
11 changes: 11 additions & 0 deletions cime_config/buildnml
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,17 @@ def buildnml(case, caseroot, compname):
if (os.path.isfile(file1)) and (not os.path.isfile(file2)):
shutil.copy(file1,file2)

# Temporary copy of TUV-x data for development
dest_data = os.path.join(rundir, "data")
if os.path.exists(dest_data):
shutil.rmtree(dest_data)
shutil.copytree(os.path.join(srcroot, "libraries", "tuv-x", "data"), \
dest_data)
shutil.copy2(os.path.join(srcroot, "cime_config", "tuvx_MOZART.json"), \
os.path.join(rundir, "tuvx_MOZART.json"))
shutil.copy2(os.path.join(srcroot, "cime_config", "tuvx_MOZART_TS1.json"), \
os.path.join(rundir, "tuvx_MOZART_TS1.json"))
Comment on lines +220 to +223
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two CAM-specific configuration files (each for specific MOZART chemical mechanisms) needed to use TUV-x in CAM. Should these be moved to the CAM dataset repository? @nusbaume @cacraigucar

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handling for config files looks similar to what we originally did for GEOS-Chem. Ultimately we decided to keep the config files in the source code, copy them at case setup time to the case root via cam/cime_config/case.case_setup.py, and then copy from case root to run directory upon submit (via this code in buildnml). This is good if you think your config file template will be tied to source code version and you want users to be able to custom edit it prior to run-time(from case root).


###############################################################################
def _main_func():

Expand Down
12 changes: 12 additions & 0 deletions cime_config/config_component.xml
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,18 @@
</desc>
</entry>

<entry id="CAM_LINKED_LIBS">
<type>char</type>
<valid_values></valid_values>
<default_value>-ltuvx -lstdc++</default_value>
<group>build_component_cam</group>
<file>env_build.xml</file>
<desc>
CAM linked libraries. The libraries are built by CAM's buildlib script and should be included
during linking of the model executable.
</desc>
</entry>

<entry id="CAM_NML_USE_CASE">
<type>char</type>
<valid_values></valid_values>
Expand Down
Loading