Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement model grid initialization for MPAS dynamical core #257

Merged

Conversation

kuanchihwang
Copy link
Collaborator

@kuanchihwang kuanchihwang commented Apr 1, 2024

This PR implements the model_grid_init subroutine, which is responsible for initializing various model grids (e.g., dynamics, physics) in terms of dynamics decomposition. Additionally, MPAS framework initialization and reading time-invariant (e.g., grid/mesh) variables are also being done in this PR.

Implementation notes:

  • Finish MPAS framework initialization (dyn_mpas_init_phase3).
  • Implement a generic mechanism (dyn_mpas_read_write_stream) for CAM-SIMA to input/output data to/from MPAS dynamical core. This mechanism is the consolidation and generalization of the cam_mpas_read_static [1], cam_mpas_setup_restart [2], cam_mpas_read_restart [3] and cam_mpas_write_restart [4] subroutines in current CAM. Hundreds of lines of code are reduced as a result.
  • Implement a generic mechanism (dyn_mpas_exchange_halo) for CAM-SIMA to exchange the halo layers of an arbitrary decomposed variable in MPAS dynamical core.
  • Implement accessor subroutines (dyn_mpas_get_variable_pointer_* and dyn_mpas_get_variable_value_* families) for users to access the internal states of MPAS dynamical core.

Test steps:

"<path-to>/CAM-SIMA/cime/scripts/create_newcase" --case "<case-name>" --compset FKESSLER --project "<project-id>" --res mpasa480_mpasa480 --run-unsupported
cd "<case-name>"
./case.setup
# Edit `env_build.xml`:
# * Change `CAM_LINKED_LIBS` to only `-lmpas`.
# * Change `DEBUG` to `TRUE`.
# Edit `env_run.xml`:
# * Change `DOUT_S` to `FALSE`.
# Add the following to `user_nl_cam`:
# * `config_block_decomp_file_prefix = '${DIN_LOC_ROOT}/atm/cam/inic/mpas/mpasa480.graph.info.part.'`.
./case.build
./case.submit

Observe log entries similar to the following in atm.log.<job-id>.<date>-<time>:

dyn_debug_print (0): Calling dynconst_init
dyn_debug_print (0): Calling vert_coord_init
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_init_phase3 entered
dyn_mpas_debug_print (0): Number of constituents is 3
dyn_mpas_debug_print (0): Checking PIO file descriptor
dyn_mpas_debug_print (0): Calling mpas_bootstrap_framework_phase1
dyn_mpas_debug_print (0): Calling mpas_bootstrap_framework_phase2
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_init_phase3 completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_read_write_stream entered
dyn_mpas_debug_print (0): Initializing stream
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_init_stream_with_pool entered
dyn_mpas_debug_print (0): Checking PIO file descriptor
dyn_mpas_debug_print (0): Creating "invariant" stream for reading
dyn_mpas_debug_print (0): Adding variables to stream
dyn_mpas_debug_print (0): var_info_list(1) % name = angleEdge
dyn_mpas_debug_print (0): var_info_list(1) % type = real
dyn_mpas_debug_print (0): var_info_list(1) % rank = 1
dyn_mpas_debug_print (0): var_info_list(2) % name = areaCell
dyn_mpas_debug_print (0): var_info_list(2) % type = real
dyn_mpas_debug_print (0): var_info_list(2) % rank = 1
dyn_mpas_debug_print (0): var_info_list(3) % name = areaTriangle
dyn_mpas_debug_print (0): var_info_list(3) % type = real
dyn_mpas_debug_print (0): var_info_list(3) % rank = 1

... (SNIPPED) ...

dyn_mpas_debug_print (0): var_info_list(67) % name = zgrid
dyn_mpas_debug_print (0): var_info_list(67) % type = real
dyn_mpas_debug_print (0): var_info_list(67) % rank = 2
dyn_mpas_debug_print (0): var_info_list(68) % name = zxu
dyn_mpas_debug_print (0): var_info_list(68) % type = real
dyn_mpas_debug_print (0): var_info_list(68) % rank = 2
dyn_mpas_debug_print (0): var_info_list(69) % name = zz
dyn_mpas_debug_print (0): var_info_list(69) % type = real
dyn_mpas_debug_print (0): var_info_list(69) % rank = 2
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_init_stream_with_pool completed
dyn_mpas_debug_print (0): Reading stream
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "angleEdge"
dyn_mpas_debug_print (0): Exchanging halo layers for "angleEdge"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "areaCell"
dyn_mpas_debug_print (0): Exchanging halo layers for "areaCell"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "areaTriangle"
dyn_mpas_debug_print (0): Exchanging halo layers for "areaTriangle"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed

... (SNIPPED) ...

dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "zgrid"
dyn_mpas_debug_print (0): Exchanging halo layers for "zgrid"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "zxu"
dyn_mpas_debug_print (0): Exchanging halo layers for "zxu"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo entered
dyn_mpas_debug_print (0): Inquiring field information for "zz"
dyn_mpas_debug_print (0): Exchanging halo layers for "zz"
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_exchange_halo completed
dyn_mpas_debug_print (0): Closing stream
dyn_mpas_debug_print (0): dyn_mpas_subdriver::dyn_mpas_read_write_stream completed
dyn_debug_print (0): Inquiring local and global mesh dimensions
dyn_debug_print (0): ncells_global    = 2562
dyn_debug_print (0): nedges_global    = 7680
dyn_debug_print (0): nvertices_global = 5120
dyn_debug_print (0): nvertlevels      = 32
dyn_debug_print (0): ncells_max       = 21
dyn_debug_print (0): nedges_max       = 10
dyn_debug_print (0): sphere_radius    =  6.371229E+06
dyn_debug_print (0): Calling init_reference_pressure
dyn_debug_print (0): Reference layer information:
dyn_debug_print (0): ----- | -------------- | --------------
dyn_debug_print (0): Index |     Height (m) | Pressure (hPa)
dyn_debug_print (0): ----- |   41426.919000 |       2.288976
dyn_debug_print (0):     1 |   38524.252750 |       3.700507
dyn_debug_print (0): ----- |   35621.586500 |       5.112039
dyn_debug_print (0):     2 |   33230.541800 |       7.720267
dyn_debug_print (0): ----- |   30839.497100 |      10.328496
dyn_debug_print (0):     3 |   28850.108750 |      14.600401
dyn_debug_print (0): ----- |   26860.720400 |      18.872306
dyn_debug_print (0):     4 |   25327.930900 |      24.561190
dyn_debug_print (0): ----- |   23795.141400 |      30.250074
dyn_debug_print (0):     5 |   22900.719950 |      35.104428
dyn_debug_print (0): ----- |   22006.298500 |      39.958781
dyn_debug_print (0):     6 |   21425.030750 |      43.949775
dyn_debug_print (0): ----- |   20843.763000 |      47.940770
dyn_debug_print (0):     7 |   20281.285500 |      52.586650
dyn_debug_print (0): ----- |   19718.808000 |      57.232531
dyn_debug_print (0):     8 |   19173.205650 |      62.605462
dyn_debug_print (0): ----- |   18627.603300 |      67.978393
dyn_debug_print (0):     9 |   18028.174800 |      75.051291
dyn_debug_print (0): ----- |   17428.746300 |      82.124188
dyn_debug_print (0):    10 |   16913.427200 |      89.370345
dyn_debug_print (0): ----- |   16398.108100 |      96.616502
dyn_debug_print (0):    11 |   15882.886800 |     105.139623
dyn_debug_print (0): ----- |   15367.665500 |     113.662745
dyn_debug_print (0):    12 |   14852.447700 |     123.689543
dyn_debug_print (0): ----- |   14337.229900 |     133.716342
dyn_debug_print (0):    13 |   13821.932800 |     145.514143
dyn_debug_print (0): ----- |   13306.635700 |     157.311944
dyn_debug_print (0):    14 |   12791.361350 |     171.190922
dyn_debug_print (0): ----- |   12276.087000 |     185.069901
dyn_debug_print (0):    15 |   11760.876700 |     201.395649
dyn_debug_print (0): ----- |   11245.666400 |     217.721397
dyn_debug_print (0):    16 |   10726.940100 |     236.885536
dyn_debug_print (0): ----- |   10208.213800 |     256.049675
dyn_debug_print (0):    17 |    9674.235795 |     278.559967
dyn_debug_print (0): ----- |    9140.257790 |     301.070258
dyn_debug_print (0):    18 |    8589.496200 |     327.542135
dyn_debug_print (0): ----- |    8038.734610 |     354.014011
dyn_debug_print (0):    19 |    7470.631760 |     385.146948
dyn_debug_print (0): ----- |    6902.528910 |     416.279885
dyn_debug_print (0):    20 |    6316.673385 |     452.886443
dyn_debug_print (0): ----- |    5730.817860 |     489.493000
dyn_debug_print (0):    21 |    5126.512605 |     532.546026
dyn_debug_print (0): ----- |    4522.207350 |     575.599052
dyn_debug_print (0):    22 |    3987.167550 |     618.653664
dyn_debug_print (0): ----- |    3452.127750 |     661.708275
dyn_debug_print (0):    23 |    3008.722795 |     701.214348
dyn_debug_print (0): ----- |    2565.317840 |     740.720421
dyn_debug_print (0):    24 |    2220.097655 |     774.051773
dyn_debug_print (0): ----- |    1874.877470 |     807.383125
dyn_debug_print (0):    25 |    1632.988910 |     832.153208
dyn_debug_print (0): ----- |    1391.100350 |     856.923292
dyn_debug_print (0):    26 |    1256.271760 |     871.255307
dyn_debug_print (0): ----- |    1121.443170 |     885.587323
dyn_debug_print (0):    27 |     998.067497 |     899.039705
dyn_debug_print (0): ----- |     874.691823 |     912.492088
dyn_debug_print (0):    28 |     763.097833 |     924.943081
dyn_debug_print (0): ----- |     651.503844 |     937.394075
dyn_debug_print (0):    29 |     551.799990 |     948.749828
dyn_debug_print (0): ----- |     452.096135 |     960.105582
dyn_debug_print (0):    30 |     364.453444 |     970.270667
dyn_debug_print (0): ----- |     276.810754 |     980.435752
dyn_debug_print (0):    31 |     201.464120 |     989.313386
dyn_debug_print (0): ----- |     126.117486 |     998.191020
dyn_debug_print (0):    32 |      63.058743 |    1005.720510
dyn_debug_print (0): ----- |       0.000000 |    1013.250000
dyn_debug_print (0): Calling init_physics_grid
Grid: physgrid, ID =  100, lat coord  = lat, lon coord  = lon, unstruct   =  T, block_ind  =  F, zonal_grid =  F
Attribute: area, long name = 'physics column areas'
           dimname = ncol
dyn_debug_print (0): Calling define_cam_grid
dyn_debug_print (0): Registering grid "mpas_cell" with id 101
Grid: mpas_cell, ID =  101, lat coord  = latCell, lon coord  = lonCell, unstruct   =  T, block_ind  =  F, zonal_grid =  F
Attribute: cell_area, long name = 'MPAS cell area'
           dimname = nCells
Attribute: cell_weight, long name = 'MPAS cell weight'
           dimname = nCells
dyn_debug_print (0): Registering grid "cam_cell" with id 102
Grid: cam_cell, ID =  102, lat coord  = lat, lon coord  = lon, unstruct   =  T, block_ind  =  F, zonal_grid =  F
dyn_debug_print (0): Registering grid "mpas_edge" with id 103
Grid: mpas_edge, ID =  103, lat coord  = latEdge, lon coord  = lonEdge, unstruct   =  T, block_ind  =  F, zonal_grid =  F
dyn_debug_print (0): Registering grid "mpas_vertex" with id 104
Grid: mpas_vertex, ID =  104, lat coord  = latVertex, lon coord  = lonVertex, unstruct   =  T, block_ind  =  F, zonal_grid =  F

* Finish MPAS framework initialization.
* Implement a generic mechanism for CAM-SIMA to input/output data to/from
  MPAS dynamical core.
* Implement a generic mechanism for CAM-SIMA to exchange the halo layers of
  an arbitrary decomposed variable in MPAS dynamical core.
* Implement accessor subroutines for users to access the internal states of
  MPAS dynamical core.
They are pointers, not values.
"Look before you leap". Make sure pointers are indeed associated before
dereferencing them. Or runtime errors will ensue.
For developmental and testing purposes of MPAS dynamical core.
Fix the following issues related to standard conformance and/or compiler bugs:

* Different character lengths in array constructor.
* Logicals must be compared with `.eqv.` instead of `==`.
* Deeply nested function calls generate wrong results.
* `index_unique` function generates wrong results when the actual argument is
  a character string array.
Copy link
Collaborator

@nusbaume nusbaume left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some (hopefully minor) change requests, mostly related to adding some additional info to model error messages, especially if users don't have debug-level logging turned on. Otherwise everything else looks good!

Externals_CAM.cfg Show resolved Hide resolved
src/dynamics/mpas/driver/dyn_mpas_subdriver.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/driver/dyn_mpas_subdriver.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/driver/dyn_mpas_subdriver.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/driver/dyn_mpas_subdriver.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/driver/dyn_mpas_subdriver.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/driver/dyn_mpas_subdriver.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/dyn_grid.F90 Show resolved Hide resolved
src/dynamics/mpas/dyn_grid.F90 Outdated Show resolved Hide resolved
src/dynamics/mpas/dyn_grid.F90 Outdated Show resolved Hide resolved
Copy link
Collaborator

@nusbaume nusbaume left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good on my end now. Thanks @kuanchihwang!

Switch to use pointers to conserve memory usage.
@mgduda mgduda self-requested a review April 30, 2024 22:37
Copy link
Collaborator

@mgduda mgduda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@kuanchihwang kuanchihwang merged commit 8acd99e into ESCOMP:development May 1, 2024
6 checks passed
@kuanchihwang kuanchihwang deleted the staging/implement-model-grid-init branch May 1, 2024 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants