Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add KPP Standalone into GEOS-Chem 14.5.1 #2588

Open
wants to merge 42 commits into
base: dev/no-diff-to-benchmark
Choose a base branch
from

Conversation

yantosca
Copy link
Contributor

@yantosca yantosca commented Nov 22, 2024

Name and Institution (Required)

Name: Bob Yantosca
Institution: Harvard + GCST

Describe the update

This PR brings the updates that were formerly in geos-chem PR #2382 and geos-chem PR #2383 into the dev/no-diff-to-benchmark branch. These will ship in 14.5.1.

Expected changes

This is a zero-diff update

Related Github Issue

Obin Sturm and others added 30 commits September 26, 2024 14:44
Signed-off-by: Lizzie Lundgren <[email protected]>
Signed-off-by: Lizzie Lundgren <[email protected]>
run/shared/kpp_standalone_interface.yml
- Fixed typo: "output_dir" -> "output_directory"

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/fullchem_mod.f90
- Use keyword arguments, for clarity

GeosCore/kpp_standalone_interface.F90
- Remove Active_Cell and Active_Cell_Name from KPP_Standalone_YAML
- Add new derived type KPP_Standalone_ActiveCell_Type, which is now
  declared with !$OMP THREADPRIVATE.  This contains the Active_Cell
  and Active_Cell_Name fields.
- Added new variable KPP_Standalone_ActiveCell, based on
  new KPP_Standalone_ActiveCell_Type

NOTE: The KPP_Standalone_YAML is initialized outside of a parallel
loop, so it does not need to be declared !$OMP THREADPRIVATE.  But the
choice of whether an (I,J,L) location corresponds to one of the
"active_cells" (listed in the kpp_standalone_interface.yml file) happens
within a parallelized loop.  Thus we need to move the Active_Cell and
Active_Cell_Name fields out of KPP_Standalone_YAML and into
KPP_Standalone_ActiveCell.

Signed-off-by: Bob Yantosca <[email protected]>
run/shared/kpp_standalone_interface.yml
- Reorganize file so that "settings:" comes first, then "active_cells:",
  and then "locations:"
- Add "settings:activate" YAML tag to toggle the KPP standalone
  interface on or off

GeosCore/kpp_standalone_interface.F90
- Add call to QFYAML_Add_Get to parse the "settings:activate" YAML tag
- Set KPP_Standalone_YAML%SkipIt to .TRUE. if "settings:activate"
  is false.  This will cause the code to ignore saving out the state
  of the model for the KPP standalone even if kpp_standalone_interface.yml
  is present in the run directory.

Signed-off-by: Bob Yantosca <[email protected]>
run/GCClassic/createRunDir.sh
- Copy run/shared/kpp_standalone_interface.yml to fullchem rundirs

run/shared/cleanRunDir.sh
- Skip removing bpch files (and {diag,tracer}info.dat files), we no
  longer generate bpch output
- Add comments and usage examples
- Remove all fort.* files
- Remove OututDir/*.txt files as well (these are KPP standalone
  interface files)

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <[email protected]>
.gitignore
- Also ignore *.exe files

GeosCore/fullchem_mod.F90
- Remove State_Grid from call to Check_ActiveCell

GeosCore/kpp_standalone_interface.F90
- Updated subroutine header comments
- Added cosmetic changes for clarity (mostly making code fit within
  80 characters, for better readability)
- Removed Active_Cell and Active_Cell_name from the
  KPP_Standalone_Interface_Type (should have been done previously)
- Remove State_Grid argument from Check_ActiveCell, this was only
  used for debugging.   Also removed commented out debug prints.
- Use Format statements with write statments where expedient
- Echo a message when the KPP standalone interface is manually disabled
  (i.e. when settings:activate = false)
- Added display of location names and lon/lats at end of the routine
  Config_KPP_Standalone
- In routine Write_Samples, wrap file I/O in an !$OMP CRITICAL block,
  in order to prevent more than one thread from writing to each file.
- Removed RETURN statement from within !$OMP CRITICAL block, this
  is not allowed.
- Bug fix: Write Kpp_Standalone_ActiveCell%Active_Cell_Name to file,
  and not Kpp_Standalone_YAML%Active_Cell_Name (which has been removed).

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kpp_standalone_interface.F90
- Moved to kppsa_interface_mod.F90

GeosCore/kppsa_interface_mod.F90
- Moved from kpp_standalone_interface
- Renamed types, variables and routines using "KppSa" prefix:
  - KPP_Standalone_Interface_Type -> KppSa_Interface_Type
  - KPP_Standalone_ActiveCell_Type -> KppSa_ActiveCell_Type
  - KPP_Standalone_YAML -> KppSa_State
  - KPP_Standalone_ActiveCell -> KppSa_ActiveCell
  - Check_Domain -> KppSa_Check_Domain
  - Check_ActiveCell -> KppSa_Check_ActiveCell
  - Config_KPP_Standalone -> KppSa_Config
  - Write_Samples -> KppSa_Write_Samples
  - Cleanup_KPP_Standalone -> KppSa_Cleanup
- Added KppSa_Check_Time function to determine if it is time to
  write KPP standalone output.  This allows you to only write output
  e.g. at the end of a run instead of for each timestep.
- Added KppSa_State%SkipWriteAtThisTime field, which is used to
  determine if we need to exit a routine early.

GeosCore/fullchem_mod.F90
- "USE Kpp_Standalone_Interface" -> "USE KppSa_Interface_Mod"
- Call renamed routines from kppsa_interface_mod.F90
- Now call KppSa_Check_Domain only if it is the first call to
  DO_FULLCHEM.  This is to avoid repeated computations.
- Now call KppSa_Check_Time to determine if we are in the time window
  when the model state should be archived to disk.
- Updated comments and comment headers

run/shared/kpp_standalone_interface.yml
- Added "start_output_at" to denote starting date time for archiving
  model state
- Added "stop_output_at" to denote ending date time for archiving
  model state
- Updated comments

CHANGELOG.md
GeosCore/CMakeLists.txt
- Updated accordingly

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kppsa_interface_mod.F90
- In routine KppSa_Check_Time:
  - rRmoved leftover debug print statements
- In routine Kpp_Check_ActiveCell
  - Added an IF statement to exit after setting KppSa_State%Active_Cell to .FALSE.
    and  KppSa_Active_Cell_Name to '' if we are outside of the time
    window specified in the kpp_standalone_interface.yml fiel.
  - This will ensure that we only archive model state to disk during
    the specified time window, which helps with computatonal efficiency.
- In routine KppSa_Config:
  - Now write starting & ending date of archival window to log file

Signed-off-by: Bob Yantosca <[email protected]>
run/GCHP/createRunDir.sh
- Added an if statement to copy run/shared/kpp_standalone_interface.yml
  to GCHP fullchem run directories (any option)

Signed-off-by: Bob Yantosca <[email protected]>
Changed the CEDS TMB emission fields to actually emit TMB like they're supposed to, instead of emitting HCOOH. I'm not sure if this fix needs to go into any other HEMCO_Config.rc template files, either in run/GCClassic or in other run/ directories, but hopefully we can make any other necessary changes down the line
KPP/standalone
- Folder which contains source code from the GitHub repository:
  https://github.com/geoschem/KPP-Standalone.  This is a fork of
  the https://github.com/KineticPreProcessor/KPP-Standalone repo
  by Obin Sturm, Mike Long, and Christoph Keller.  The branch
  geoschem-dev is checked out.  This is where we can place
  GEOS-Chem-specific updates without touching the original code
  in the kpp-standalone branch.
  branch is set to geoschem-dev, which is the branch to be used
  for interfacing into the GEOS-Chem model

.gitmodules
- This was updated by running the command:
  git submodule add -b geoschem-dev \
    https://github.com/geoschem/KPP-Standalone \
    KPP/standalone

CHANGELOG.md
- Updated accordingly

KPP/fullchem/Makefile
KPP/fullchem/kpp_standalone*.F90
KPP/fullchem/samples_kpp_standalone/*
- Removed from the KPP/fullchem folder.

KPP/standalone
- Folder containing code from https://github.com/geoschem/KPP-Standalone

Signed-off-by: Bob Yantosca <[email protected]>
KPP/CMakeLists.txt
- Add calls to "add_directory(standalone)" in the fullchem and
  custom "if" blocks.  This will tell CMake to look for the
  CMakeLists.txt file in the KPP/standalone folder, and to build
  the standalone files (libKppStandalone.a) after libKPP.a has
  been built.

Signed-off-by: Bob Yantosca <[email protected]>
Changed the CEDS TMB emission fields to actually emit TMB like they're supposed to, instead of emitting HCOOH.
Changed the CEDS TMB emission fields to actually emit TMB like they're supposed to, instead of emitting HCOOH.
GeosCore/fullchem_mod.F90
- Pass KPP integrator arguments ICNTRL and RCNTRL to KppSa_Write_Samples
  so that we can include it in the KPP standalone output.

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/CMakeLists.txt
- Changed permission from chmod 755 to chmod 644

GeosCore/kppsa_interface_mod.F90
- Now accept ICNTRL and RCNTRL as inputs to routine KppSa_Write_Samples
- Change number of header lines from 48 to 60
- Better separate Meteorological metadata and KPP parameters
  for readability
- Write out ICNTRL as 2 lines of format 10i6
- Write out RCNTRL as 4 lines of format 5f13.6
- Changed format e25.16e3 to es25.16e3, which changes the output from
  e.g. 0.5e1 to 5.0e0, etc., which is more standard
- Write out ATOL after the species concentration with format es10.2e2
- Cosmetic changes (indentation, comments)

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kppsa_interface_mod.F90
- In routine KppSa_Write_Samples:
  - Added a "," in between the concentration and abs tolerance output
  - Simplified FORMAT statements 120 and 130
  - Added FORMAT statements 140 and 150

Signed-off-by: Bob Yantosca <[email protected]>
This merge brings PR #2507 (Fix CEDS TMB emissions, by @kelvinhb)
into the GEOS-Chem 14.6.0 development stream.

This PR fixes errors in the HEMCO_Config.rc.fullchem template files
(for GCClassic and GCHP) pertaining to TMB emissions from the
CEDS global inventory.

Signed-off-by: Bob Yantosca <[email protected]>
CHANGELOG.md
- Move the entry pertaining to PR #2507 (CEDS TMB emissions) out
  of the 14.5.0 section and into the Unreleased section

Signed-off-by: Bob Yantosca <[email protected]>
This merge brings PR #2482 (KPP standalone interface from Obin
Sturm, by @lizziel) into the GEOS-Chem 14.6.0 development stream.

This PR brings the interface for the KPP standalone box model
into GEOS-Chem 14.6.0.  A new kpp_standalone_interface.yml file
is placed into the run directory.  In this YAML file, the user
can select the locations and times for archiving model state
to text files that can be fed into the KPP standalone model.

Also the writing of files is now done in a thread-safe manner.

Signed-off-by: Bob Yantosca <[email protected]>
This merge brings PR #2483 (KPP standalone from Obin Sturm,
by @obin1 and @lizziel) into the GEOS-Chem 14.6.0 development stream.

This PR does the following:

1. Adds submodule https://github.com/geoschem/KPP-Standalone
   in the KPP/standalone folder

2. Updates KPP/CMakeListst.txt to build the KPP standalone
   executable when the GEOS-Chem executable is built
   (currently only for fullchem or custom mechanisms)

Signed-off-by: Bob Yantosca <[email protected]>
This merge brings PR #2482 (KPP standalone interface from Obin
Sturm, by @lizziel) into the GEOS-Chem 14.5.1 development stream.

This PR brings the interface for the KPP standalone box model
into GEOS-Chem 14.6.0.  A new kpp_standalone_interface.yml file
is placed into the run directory.  In this YAML file, the user
can select the locations and times for archiving model state
to text files that can be fed into the KPP standalone model.

Also the writing of files is now done in a thread-safe manner.

Signed-off-by: Bob Yantosca <[email protected]>
This merge brings PR #2483 (KPP standalone from Obin Sturm,
by @obin1 and @lizziel) into the GEOS-Chem 14.5.1 development
stream.

This PR does the following:

1. Adds submodule https://github.com/geoschem/KPP-Standalone
   in the KPP/standalone folder

2. Updates KPP/CMakeListst.txt to build the KPP standalone
   executable when the GEOS-Chem executable is built
   (currently only for fullchem or custom mechanisms)

Signed-off-by: Bob Yantosca <[email protected]>
This commit informs the GEOS-Chem repository about the following
commits that were pushed to the geoschem/KPP-Standalone repo:

cfccbe7 (HEAD -> geoschem-dev, origin/geoschem-dev) Remove build_executable.sh from geoschem-dev branch
337668e Merge kpp-standalone into geoschem-dev (14.5.0 mechanism)

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca added category: Feature Request New feature or request topic: Chemical Mechanisms Related to KPP and/or GEOS-Chem chemistry mechanisms no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations labels Nov 22, 2024
@yantosca yantosca self-assigned this Nov 22, 2024
.gitmodules Outdated
[submodule "KPP/standalone"]
path = KPP/standalone
url = https://github.com/geoschem/KPP-Standalone
branch = geoschem-dev
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should link to a tag here rather than a branch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lizziel. I'll do this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lizziel: In commit 0947b8f we now pull from the GC_Interface tag, which is on the geoschem-dev branch.

[submodule "KPP/standalone"]
        path = KPP/standalone
        url = https://github.com/geoschem/KPP-Standalone
        tag = "GC_Interface"

CHANGELOG.md Outdated
@@ -11,11 +11,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Added `State_Diag%Archive_SatDiagnEdgeCount` field
- Added `State_Diag%Archive_SatDiagnEdge` field
- Added routine `SatDiagn_or_SatDiagnEdge` in `History/history_utils_mod.F90`
- Added KPP standalone interface (archives model state to selected locations)
- Added `https://github/geoschem/KPP-Standalone` as a Git submodule (`geoschem-dev` branch)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use a version rather than a branch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lizziel: The main branch of the KPP-Standalone repo (which is called kpp-standalone) contains the gckpp_*F90 files that were built by KPP. For the interface to GEOS-Chem, I didn't want to carry these, so I created a new branch geoschem-dev (akin to to the gchp-dev branch in the MAPL fork) so that I wouldn't touch the code in the kpp-standalone branch.

I've merged the kpp-standalone branch into the geoschem-dev branch (but removed the gckpp* etc files). That way the kpp_standalone.F90 and kpp_standalone_init.F90 codes are up to date.

Do you have any other recommendations how we can manage this? I was thinking something like how you manage the MAPL repo but am open to other ideas.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog updated in commits 0947b8f and f14b9a4

CHANGELOG.md Outdated

### Changed
- Renamed `Emiss_Carbon_Gases` to `CO2_Production` in `carbon_gases_mod.F90`
- Updated start date and restart file for CO2 and tagCO simulations for consistency with carbon simulations
- Allocated `State_Diag%SatDiagnPEDGE` ffield with vertical dimension `State_Grid%NZ+1`
- Moved PINO3H to be in alphabetical order in `species_database.yml`
- Modified `run/GCClassic/cleanRunDir.sh` to skip removing bpch files, as well as now removing `fort.*` and `OutputDir/*.txt` files
- Moved PINO3H to be in alphabetical order in `species_database.yml`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these changes are part of this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog updated in commits 0947b8f and f14b9a4

! RSTATE(Nhexit), &
! State_Grid, State_Chm, State_Met, &
! Input_Opt, ISTATUS(3), RC, &
! FORCE_WRITE = .TRUE., CELL_NAME = 'root')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to keep this debug code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 0947b8f

# # Also prompts the user before removing diagnostic
# # output files in OutputDir/.
#
# $ ./cleanRunDir.sh 1 # Removes model ouptut files in the run directory,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than use 1 could we use something more descriptive like --force? We should probably also remove restart files to keep only the restart of the current date too, for a true forced rundir clean.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lizzie. The --force argument will work with this script as-is. Basically I test for a non-empty first argument:

#----------------------------------------------------------------------------
# Clean data files in OutputDir.
# These are netCDF files (*.nc) and KPP standalone interface files (*.txt).
#----------------------------------------------------------------------------
if [[ "x${1}" == "x" ]]; then      # User confirmation required
    rm -Iv ./OutputDir/*.nc*
    rm -Iv ./OutputDir/*.txt
else                               # User Confirmation not required
    rm -fv ./OutputDir/*.nc*
    rm -fv ./OutputDir/*.txt*
fi

I've kept the cleanRunDir.sh file as-is but only added the documentation.

I also updated it to remove *.txt files in OutputDir as these would be the KPP interface outputs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lizziel: I would propose to remove the restart files (except for current date) in a future PR. That would involve a little more logic.

# $ ./cleanRunDir.sh 1 # Removes model ouptut files in the run directory,
# # but will remove diagnostic output files without
# # prompting first. USE WITH CAUTION!
#============================================================================
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this new feature to the changelog

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an existing feature but was uncommented. I can add it to the CHANGELOG.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now updated in 0947b8f

.gitmodules
- Now pull the "GC_Interface" tag instead of the geoschem-dev branch

GeosCore/fullchem_mod.F90
- Removed unused code

run/shared/cleanRunDir.sh
- Updated comments in header.  Now use --force (but any non-empty
  argument) will cause files to be removed w/o user confirmation.

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <[email protected]>
CHANGELOG.md
- Now note that we pull the KPP-Standalone repo from the
  "GC_Interface" tag, not a branch.

Signed-off-by: Bob Yantosca <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Feature Request New feature or request no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations topic: Chemical Mechanisms Related to KPP and/or GEOS-Chem chemistry mechanisms
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants