Fb smc gpuport

manual/num/space_SMC.tex

@@ -196,6 +196,35 @@ \subsubsection{~Spherical Multiple-Cell (SMC) grid} \label{sub:num_space_SMC}
 combined hybrid and multi-grid parallelization may extend the computer usage
 to over 100 nodes for the 3 Great Lake sub-grids in \emph{mww3\_test\_09}.
 
+Following the ongoing manual porting efforts at the Met Office, a switch has been 
+created for using an initial OpenACC parallelism of the SMC grid. This converts the 
+w3psmcmd.F90 module file and function calls to be able to target a GPU for 
+acceleration. Primarily this has been used with the nvfortran compiler to success 
+after being built on Isambard using the \emph{cmake\_build\_isambard.sh}
+script with Met Office specifications. Contact Chris Bunney 
+(\url{[email protected]}) for details surrounding the Isambard 
+Implementation. 
+
+For optimal performance on GPU there is a range of changes to function calls, 
+array declarations and nested subroutine calls, which are all managed by the 
+switch. Since the GPU will deallocate arrays once they leave scope, local arrays 
+are hoisted to be in the module scope of w3wavemd.F90, and hence resident on 
+the GPU for longer. For GPU parallelism using OpenACC, data transfers and 
+parallelism specifications are applied implicitly where possible. For ease of 
+application, the SMC propagation subroutines are inlined so that the implicit 
+optimisations are correctly defined and maintain valid model output without more 
+intrusive coding changes. 
+
+The current implementation of the GPU switch has some limitations, this switch 
+only targets the multi-resolution grids and has not yet been adapted to the case 
+of \emph{NRLv .EQ. 1}. This expansion would not be difficult but requires further 
+inlining and similar changes to the code, which have not yet been tested properly.
+It is also worth noting that the performance of the current implementation is not
+at its full potential. This is due to the majority of the code still being 
+processed on the CPU, and only a small section actively ported to the GPU. We are 
+viewing the progress so far as a proof of concept and an initial step on determing
+the best way to integrate GPU acceleration into the current parallelisation options.
+
 It is recommended to read the smc\_docs/SMC\_Grid\_Guide.pdf or the
 conference paper \citep{tol:LiS17} at conference web page: 
 http://www.waveworkshop.org/15thWaves/

model/bin/switch_UKMO_GPU

@@ -0,0 +1 @@
+SHRD SMC UNO PR2 RTD FLX0 LN1 ST0 NL1 BT1 IC0 IS0 REF0 DB1 TR0 BS0 WNT1 WNX1 CRT1 CRX1 RWND NOGRB GPU 

model/src/w3initmd.F90

@@ -444,6 +444,21 @@ SUBROUTINE W3INIT ( IMOD, IsMulti, FEXT, MDS, MTRACE, ODAT, FLGRD,  FLGR2, FLGD,
 #endif
 #ifdef W3_UOST
     USE W3UOSTMD, ONLY: UOST_SETGRID
+#endif
+#ifdef W3_GPU
+    USE W3GDATMD, ONLY: NK, NTH, DTH, XFR, ESIN, ECOS, SIG, NX, NY,  &
+         NSEA, SX, SY, MAPSF, FUNO3, FVERG,           &
+         IJKCel, IJKUFc, IJKVFc, NCel, NUFc, NVFc,    &
+         IJKCel3, IJKCel4,                            &
+         IJKVFc5, IJKVFc6,IJKUFc5,IJKUFc6,            &
+         NLvCel, NLvUFc, NLvVFc, NRLv, MRFct,         &
+         DTCFL, CLATS, DTMS, CTRNX, CTRNY
+    USE W3GDATMD, ONLY: NGLO, ANGARC, ARCTC, CLATF
+    USE W3ADATMD, ONLY: CG, WN, U10, CX, CY, ATRNX, ATRNY, ITIME
+    !
+    USE W3IDATMD, ONLY: FLCUR
+    USE W3ODATMD, ONLY: NDSE, NDST, FLBPI, NBI, TBPI0, TBPIN,       &
+         ISBPI, BBPI0, BBPIN
 #endif
     !/
 #ifdef W3_MPI
@@ -1512,6 +1527,23 @@ SUBROUTINE W3INIT ( IMOD, IsMulti, FEXT, MDS, MTRACE, ODAT, FLGRD,  FLGR2, FLGD,
 #endif
     !
     ! 8.  Final MPI set up ----------------------------------------------- /
+#ifdef W3_GPU
+!/LS SMC Grid - GDAT
+!$ACC ENTER DATA COPYIN(NK, NTH, DTH, XFR, ESIN, ECOS, SIG, NX, NY) &
+!$ACC            COPYIN(NSEA, SX, SY, MAPSF, FUNO3, FVERG, IJKCel ) &
+!$ACC            COPYIN(IJKUFc, IJKVFc, NCel, NUFc, NVFc, IJKCel3 ) &
+!$ACC            COPYIN(IJKCel4, IJKVFc5, IJKVFc6,IJKUFc5,IJKUFc6 ) &
+!$ACC            COPYIN(NLvCel, NLvUFc, NLvVFc, NRLv, MRFct, DTCFL) &
+!$ACC            COPYIN(CLATS, DTMS, CTRNX, CTRNY, NGLO, ANGARC   ) &
+!$ACC            COPYIN(ARCTC, CLATF)
+!/LS SMC Grid - ADAT
+!$ACC ENTER DATA COPYIN(CG, WN, U10, CX, CY, ATRNX, ATRNY, ITIME)
+!/LS SMC Grid - IDAT
+!$ACC ENTER DATA COPYIN(FLCUR)
+!/LS SMC Grid - ODAT
+!$ACC ENTER DATA COPYIN(NDSE, NDST, FLBPI, NBI, TBPI0, TBPIN)&
+!$ACC            COPYIN(ISBPI, BBPI0, BBPIN)
+#endif
     !
 #ifdef W3_MPI
     CALL W3MPII ( IMOD )