From 4bca649643e8b4bb0987496d7f9c08c4dd50e709 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 16:37:45 -0500 Subject: [PATCH 001/110] Lassen RHEL8 (#4278) Update the Lassen (LLNL) documentation for their RHEL8 transition and Python support. --- Docs/source/install/hpc/lassen.rst | 192 +++++++++++++----- .../lassen-llnl/install_v100_dependencies.sh | 132 ++++++++++++ Tools/machines/lassen-llnl/install_v100_ml.sh | 61 ++++++ .../{lassen.bsub => lassen_v100.bsub} | 0 .../lassen_v100_warpx.profile.example | 52 +++++ .../lassen-llnl/lassen_warpx.profile.example | 44 ---- .../summit-olcf/summit_warpx.profile.example | 2 +- 7 files changed, 385 insertions(+), 98 deletions(-) create mode 100755 Tools/machines/lassen-llnl/install_v100_dependencies.sh create mode 100755 Tools/machines/lassen-llnl/install_v100_ml.sh rename Tools/machines/lassen-llnl/{lassen.bsub => lassen_v100.bsub} (100%) create mode 100644 Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example delete mode 100644 Tools/machines/lassen-llnl/lassen_warpx.profile.example diff --git a/Docs/source/install/hpc/lassen.rst b/Docs/source/install/hpc/lassen.rst index 1f03a3c117f..785bce3b385 100644 --- a/Docs/source/install/hpc/lassen.rst +++ b/Docs/source/install/hpc/lassen.rst @@ -3,7 +3,7 @@ Lassen (LLNL) ============= -The `Lassen V100 GPU cluster `_ is located at LLNL. +The `Lassen V100 GPU cluster `__ is located at LLNL. Introduction @@ -11,85 +11,168 @@ Introduction If you are new to this system, **please see the following resources**: -* `LLNL user account `_ -* `Lassen user guide `_ -* Batch system: `LSF `_ -* `Production directories `_: +* `LLNL user account `__ (login required) +* `Lassen user guide `__ +* Batch system: `LSF `__ +* `Jupyter service `__ (`documentation `__, login required) +* `Production directories `__: * ``/p/gpfs1/$(whoami)``: personal directory on the parallel filesystem * Note that the ``$HOME`` directory and the ``/usr/workspace/$(whoami)`` space are NFS mounted and *not* suitable for production quality data generation. -Installation ------------- +Login +----- + +.. note:: + + Lassen is currently transitioning to RHEL8. + During this transition, first SSH into lassen and then ``ssh eatoss4`` next to work with the updated RHEL8/TOSS4 nodes. + + Approximately September 2023, the new software environment on these nodes will be the new default. -Use the following commands to download the WarpX source code and switch to the correct branch: + +Preparation +----------- + +Use the following commands to download the WarpX source code: .. code-block:: bash git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx -We use the following modules and environments on the system (``$HOME/lassen_warpx.profile``). +We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx.profile``. +Create it now: -.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_warpx.profile.example - :language: bash - :caption: You can copy this file from ``Tools/machines/lassen-llnl/lassen_warpx.profile.example``. +.. code-block:: bash + + cp $HOME/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example $HOME/lassen_v100_warpx.profile + +.. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example + :language: bash -We recommend to store the above lines in a file, such as ``$HOME/lassen_warpx.profile``, and load it into your shell after a login: +Edit the 2nd line of this script, which sets the ``export proj=""`` variable. +For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx.profile``. +Enter the edit mode by typing ``i`` and edit line 2 to read: .. code-block:: bash - source $HOME/lassen_warpx.profile + export proj="nsldt" + +Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). + +.. important:: + + Now, and as the first step on future logins to lassen, activate these environment settings: -And since Lassen does not yet provide a module for them, install ADIOS2, BLAS++ and LAPACK++: + .. code-block:: bash + + source $HOME/lassen_v100_warpx.profile + +Finally, since lassen does not yet provide software modules for some of our dependencies, install them once: .. code-block:: bash - # c-blosc (I/O compression) - git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc - rm -rf src/c-blosc-lassen-build - cmake -S src/c-blosc -B src/c-blosc-lassen-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/lassen/c-blosc-1.21.1 - cmake --build src/c-blosc-lassen-build --target install --parallel 16 - - # HDF5 - git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git src/hdf5 - rm -rf src/hdf5-lassen-build - cmake -S src/hdf5 -B src/hdf5-lassen-build -DBUILD_TESTING=OFF -DHDF5_ENABLE_PARALLEL=ON -DCMAKE_INSTALL_PREFIX=$HOME/sw/lassen/hdf5-1.14.1.2 - cmake --build src/hdf5-lassen-build --target install --parallel 16 - - # ADIOS2 - git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git src/adios2 - rm -rf src/adios2-lassen-build - cmake -S src/adios2 -B src/adios2-lassen-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/lassen/adios2-2.8.3 - cmake --build src/adios2-lassen-build --target install -j 16 - - # BLAS++ (for PSATD+RZ) - git clone https://github.com/icl-utk-edu/blaspp.git src/blaspp - rm -rf src/blaspp-lassen-build - cmake -S src/blaspp -B src/blaspp-lassen-build -Duse_openmp=ON -Dgpu_backend=cuda -Duse_cmake_find_blas=ON -DBLA_VENDOR=IBMESSL -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/lassen/blaspp-master - cmake --build src/blaspp-lassen-build --target install --parallel 16 - - # LAPACK++ (for PSATD+RZ) - git clone https://github.com/icl-utk-edu/lapackpp.git src/lapackpp - rm -rf src/lapackpp-lassen-build - CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S src/lapackpp -B src/lapackpp-lassen-build -Duse_cmake_find_lapack=ON -DBLA_VENDOR=IBMESSL -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=$HOME/sw/lassen/lapackpp-master -DLAPACK_LIBRARIES=/usr/lib64/liblapack.so - cmake --build src/lapackpp-lassen-build --target install --parallel 16 - -Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following commands to compile: + bash $HOME/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh + source $HOME/sw/lassen/gpu/venvs/warpx-lassen/bin/activate + +.. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies.sh + :language: bash + +.. dropdown:: AI/ML Dependencies (Optional) + :animate: fade-in-slide-down + + If you plan to run AI/ML workflows depending on pyTorch, run the next step as well. + This will take a while and should be skipped if not needed. + + .. code-block:: bash + + runNode bash $HOME/src/warpx/Tools/machines/lassen-llnl/install_v100_ml.sh + + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_ml.sh + :language: bash + + For `optimas dependencies `__ (incl. scikit-learn), plan another hour of build time: + + .. code-block:: bash + + python3 -m pip install -r $HOME/src/warpx/Tools/optimas/requirements.txt + + +.. _building-lassen-compilation: + +Compilation +----------- + +Use the following :ref:`cmake commands ` to compile the application executable: .. code-block:: bash cd $HOME/src/warpx - rm -rf build_lassen - cmake -S . -B build_lassen -DWarpX_COMPUTE=CUDA -DWarpX_DIMS="1;2;RZ;3" -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON - cmake --build build_lassen -j 10 -The other :ref:`general compile-time options ` apply as usual. + cmake -S . -B build_lassen -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_lassen -j 8 + +The WarpX application executables are now in ``$HOME/src/warpx/build_lassen/bin/``. +Additionally, the following commands will install WarpX as a Python module: + +.. code-block:: bash + + rm -rf build_lassen_py + + cmake -S . -B build_lassen_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_lassen_py -j 8 --target pip_install + +Now, you can :ref:`submit lassen compute jobs ` for WarpX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). +Or, you can use the WarpX executables to submit lassen jobs (:ref:`example inputs `). +For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$PROJWORK/$proj/``. + -**That's it!** -WarpX executables for 1D, 2D, RZ and 3D are now in ``build_lassen/bin/`` and :ref:`can be run ` with the respective :ref:`example inputs files `. -Most people execute the binary directly or copy it out to a location in ``/p/gpfs1/$(whoami)``. +.. _building-lassen-update: + +Update WarpX & Dependencies +--------------------------- + +If you already installed WarpX in the past and want to update it, start by getting the latest source code: + +.. code-block:: bash + + cd $HOME/src/warpx + + # read the output of this command - does it look ok? + git status + + # get the latest WarpX source code + git fetch + git pull + + # read the output of these commands - do they look ok? + git status + git log # press q to exit + +And, if needed, + +- :ref:`update the lassen_v100_warpx.profile file `, +- log out and into the system, activate the now updated environment profile as usual, +- :ref:`execute the dependency install scripts `. + +As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_lassen`` and rebuild WarpX. .. _running-cpp-lassen: @@ -146,3 +229,6 @@ Known System Issues .. code-block:: bash export OMPI_MCA_coll_ibm_skip_allgatherv=true + +As part of the same `CORAL acquisition program `__, Lassen is very similar to the design of Summit (OLCF). +Thus, when encountering new issues it is worth checking also the :ref:`known Summit issues and work-arounds `. diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh new file mode 100755 index 00000000000..8c6699c19e7 --- /dev/null +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -0,0 +1,132 @@ +#!/bin/bash +# +# Copyright 2023 The WarpX Community +# +# This file is part of WarpX. +# +# Author: Axel Huebl +# License: BSD-3-Clause-LBNL + +# Exit on first error encountered ############################################# +# +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was perlmutter_gpu_warpx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your lassen_v100_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi + + +# Remove old dependencies ##################################################### +# +SW_DIR="${HOME}/sw/lassen/gpu/" +# better, but needs a request: /usr/WS2/${USER}/ +rm -rf ${SW_DIR} +mkdir -p ${SW_DIR} + +# remove common user mistakes in python, located in .local instead of a venv +python3 -m pip uninstall -qq -y pywarpx +python3 -m pip uninstall -qq -y warpx +python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true + + +# General extra dependencies ################################################## +# + +# tmpfs build directory: avoids issues often seen with $HOME and is faster +build_dir=$(mktemp -d) + +# c-blosc (I/O compression) +if [ -d $HOME/src/c-blosc ] +then + cd $HOME/src/c-blosc + git fetch --prune + git checkout v1.21.1 + cd - +else + git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc +fi +cmake -S $HOME/src/c-blosc -B ${build_dir}/c-blosc-lassen-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 +cmake --build ${build_dir}/c-blosc-lassen-build --target install --parallel 10 + +# HDF5 +if [ -d $HOME/src/hdf5 ] +then + cd $HOME/src/hdf5 + git fetch --prune + git checkout hdf5-1_14_1-2 + cd - +else + git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git $HOME/src/hdf5 +fi +cmake -S $HOME/src/hdf5 -B ${build_dir}/hdf5-lassen-build -DBUILD_TESTING=OFF -DHDF5_ENABLE_PARALLEL=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/hdf5-1.14.1.2 +cmake --build ${build_dir}/hdf5-lassen-build --target install --parallel 10 + +# ADIOS2 +if [ -d $HOME/src/adios2 ] +then + cd $HOME/src/adios2 + git fetch --prune + git checkout v2.8.3 + cd - +else + git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2 +fi +cmake -S $HOME/src/adios2 -B ${build_dir}/adios2-lassen-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake --build ${build_dir}/adios2-lassen-build --target install -j 10 + +# BLAS++ (for PSATD+RZ) +if [ -d $HOME/src/blaspp ] +then + cd $HOME/src/blaspp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp +fi +cmake -S $HOME/src/blaspp -B ${build_dir}/blaspp-lassen-build -Duse_openmp=ON -Dgpu_backend=cuda -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake --build ${build_dir}/blaspp-lassen-build --target install --parallel 10 + +# LAPACK++ (for PSATD+RZ) +if [ -d $HOME/src/lapackpp ] +then + cd $HOME/src/lapackpp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp +fi +CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-lassen-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master -DLAPACK_LIBRARIES=/usr/lib64/liblapack.so +cmake --build ${build_dir}/lapackpp-lassen-build --target install --parallel 10 + +# remove build temporary directory +rm -rf ${build_dir} + + +# Python ###################################################################### +# +python3 -m pip install --upgrade --user virtualenv +rm -rf ${SW_DIR}/venvs/warpx-lassen +python3 -m venv ${SW_DIR}/venvs/warpx-lassen +source ${SW_DIR}/venvs/warpx-lassen/bin/activate +python3 -m pip install --upgrade pip +python3 -m pip cache purge +python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade cython +python3 -m pip install --upgrade numpy +python3 -m pip install --upgrade pandas +python3 -m pip install --upgrade -Ccompile-args="-j10" scipy +python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py +python3 -m pip install --upgrade openpmd-api +python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself +python3 -m pip install --upgrade yt + +# install or update WarpX dependencies such as picmistandard +python3 -m pip install --upgrade -r $HOME/src/warpx/requirements.txt + +# for ML dependencies, see install_v100_ml.sh diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh new file mode 100755 index 00000000000..47f1bf5a2eb --- /dev/null +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# +# Copyright 2023 The WarpX Community +# +# This file is part of WarpX. +# +# Author: Axel Huebl +# License: BSD-3-Clause-LBNL + +# Exit on first error encountered ############################################# +# +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was perlmutter_gpu_warpx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your lassen_v100_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi + + +# Remove old dependencies ##################################################### +# +# remove common user mistakes in python, located in .local instead of a venv +python3 -m pip uninstall -qqq -y torch 2>/dev/null || true + + +# Python ML ################################################################### +# +# for basic python dependencies, see install_v100_dependencies.sh + +# optional: for libEnsemble - WIP: issues with nlopt +# python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt + +# optional: for pytorch +if [ -d ${HOME}/src/pytorch ] +then + cd ${HOME}/src/pytorch + git fetch + git checkout . + git checkout v2.0.1 + cd - +else + git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git /ccs/proj/${proj}/${USER}/src/pytorch +fi +cd ${HOME}/src/pytorch +rm -rf build +python3 -m pip install -r requirements.txt +# patch to avoid compile issues +# https://github.com/pytorch/pytorch/issues/97497#issuecomment-1499069641 +# https://github.com/pytorch/pytorch/pull/98511 +wget -q -O - https://github.com/pytorch/pytorch/pull/98511.patch | git apply +USE_CUDA=1 BLAS=OpenBLAS MAX_JOBS=64 ATEN_AVX512_256=OFF BUILD_TEST=0 python3 setup.py develop +# (optional) If using torch.compile with inductor/triton, install the matching version of triton +#make triton +rm -rf build +cd - + +# optional: optimas dependencies (based on libEnsemble & ax->botorch->gpytorch->pytorch) +# commented because scikit-learn et al. compile > 2 hrs +# please run manually on a login node if needed +#python3 -m pip install -r $HOME/src/warpx/Tools/optimas/requirements.txt diff --git a/Tools/machines/lassen-llnl/lassen.bsub b/Tools/machines/lassen-llnl/lassen_v100.bsub similarity index 100% rename from Tools/machines/lassen-llnl/lassen.bsub rename to Tools/machines/lassen-llnl/lassen_v100.bsub diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example new file mode 100644 index 00000000000..d5f2a633bd3 --- /dev/null +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example @@ -0,0 +1,52 @@ +# please set your project account +#export proj="" # edit this and comment in + +# required dependencies +module load cmake/3.23.1 +module load clang/12.0.1-gcc-8.3.1 +module load cuda/12.0.0 + +# optional: for QED lookup table generation support +module load boost/1.70.0 + +# optional: for openPMD support +export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH + +# optional: for PSATD in RZ geometry support +export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/lapackpp-master:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH + +# optional: for Python bindings +module load python/3.8.2 + +if [ -d "$HOME/sw/lassen/gpu/venvs/warpx-lassen" ] +then + source $HOME/sw/lassen/gpu/venvs/warpx-lassen/bin/activate +fi + +# optional: an alias to request an interactive node for two hours +alias getNode="bsub -G $proj -W 2:00 -nnodes 1 -Is /bin/bash" +# an alias to run a command on a batch node for up to 30min +# usage: runNode +alias runNode="bsub -q debug -P $proj -W 2:00 -nnodes 1 -I" + +# fix system defaults: do not escape $ with a \ on tab completion +shopt -s direxpand + +# optimize CUDA compilation for V100 +export AMREX_CUDA_ARCH=7.0 +export CUDAARCHS=70 + +# compiler environment hints +export CC=$(which clang) +export CXX=$(which clang++) +export FC=$(which gfortran) +export CUDACXX=$(which nvcc) +export CUDAHOSTCXX=$(which clang++) diff --git a/Tools/machines/lassen-llnl/lassen_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_warpx.profile.example deleted file mode 100644 index fb605b53ec2..00000000000 --- a/Tools/machines/lassen-llnl/lassen_warpx.profile.example +++ /dev/null @@ -1,44 +0,0 @@ -# please set your project account -#export proj="" # edit this and comment in - -# required dependencies -module load cmake/3.23.1 -module load clang/12.0.1-gcc-8.3.1 -module load cuda/12.0.0 - -# optional: for QED lookup table generation support -module load boost/1.70.0 - -# optional: for openPMD support -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/c-blosc-1.21.1:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/adios2-2.8.3:$CMAKE_PREFIX_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/adios2-2.8.3/lib64:$LD_LIBRARY_PATH - -# optional: for PSATD in RZ geometry support -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/blaspp-master:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/lapackpp-master:$CMAKE_PREFIX_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/blaspp-master/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/lapackpp-master/lib64:$LD_LIBRARY_PATH - -# optional: for Python bindings -module load python/3.8.2 - -# optional: an alias to request an interactive node for two hours -alias getNode="bsub -G $proj -W 2:00 -nnodes 1 -Is /bin/bash" - -# fix system defaults: do not escape $ with a \ on tab completion -shopt -s direxpand - -# optimize CUDA compilation for V100 -export AMREX_CUDA_ARCH=7.0 -export CUDAARCHS=70 - -# compiler environment hints -export CC=$(which clang) -export CXX=$(which clang++) -export FC=$(which gfortran) -export CUDACXX=$(which nvcc) -export CUDAHOSTCXX=$(which clang++) diff --git a/Tools/machines/summit-olcf/summit_warpx.profile.example b/Tools/machines/summit-olcf/summit_warpx.profile.example index c060102d6e8..5d88bb9aeed 100644 --- a/Tools/machines/summit-olcf/summit_warpx.profile.example +++ b/Tools/machines/summit-olcf/summit_warpx.profile.example @@ -63,7 +63,7 @@ fi # for paralle execution, start on the batch node: jsrun alias getNode="bsub -q debug -P $proj -W 2:00 -nnodes 1 -Is /bin/bash" # an alias to run a command on a batch node for up to 30min -# usage: nrun +# usage: runNode alias runNode="bsub -q debug -P $proj -W 2:00 -nnodes 1 -I" # fix system defaults: do not escape $ with a \ on tab completion From 14db70afd12a995d16aee001e18b2efd07acf688 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 14:51:41 -0700 Subject: [PATCH 002/110] Lassen (LLNL): Numpy==1.22 See https://github.com/numpy/numpy/issues/24673 --- Tools/machines/lassen-llnl/install_v100_dependencies.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index 8c6699c19e7..88a5d482908 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -118,7 +118,8 @@ python3 -m pip install --upgrade pip python3 -m pip cache purge python3 -m pip install --upgrade wheel python3 -m pip install --upgrade cython -python3 -m pip install --upgrade numpy +# see https://github.com/numpy/numpy/issues/24673 +python3 -m pip install --upgrade numpy==1.22 python3 -m pip install --upgrade pandas python3 -m pip install --upgrade -Ccompile-args="-j10" scipy python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py From 0a1e6a0ad1800670ceb291751a6e330c525f8eaa Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 15:05:51 -0700 Subject: [PATCH 003/110] Lassen (LLNL): New SW Directory On workspace --- .../lassen-llnl/install_v100_dependencies.sh | 3 +-- Tools/machines/lassen-llnl/install_v100_ml.sh | 2 +- .../lassen_v100_warpx.profile.example | 25 ++++++++++--------- 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index 88a5d482908..fcf0bb14afd 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -20,8 +20,7 @@ if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in yo # Remove old dependencies ##################################################### # -SW_DIR="${HOME}/sw/lassen/gpu/" -# better, but needs a request: /usr/WS2/${USER}/ +SW_DIR="/usr/workspace/${USER}/lassen/gpu" rm -rf ${SW_DIR} mkdir -p ${SW_DIR} diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh index 47f1bf5a2eb..e736f83ba20 100755 --- a/Tools/machines/lassen-llnl/install_v100_ml.sh +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -40,7 +40,7 @@ then git checkout v2.0.1 cd - else - git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git /ccs/proj/${proj}/${USER}/src/pytorch + git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git ${HOME}/src/pytorch fi cd ${HOME}/src/pytorch rm -rf build diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example index d5f2a633bd3..09808a03ebb 100644 --- a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example @@ -10,25 +10,26 @@ module load cuda/12.0.0 module load boost/1.70.0 # optional: for openPMD support -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH +SW_DIR="/usr/workspace/${USER}/lassen/gpu" +export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=${SW_DIR}/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/adios2-2.8.3/lib64:$LD_LIBRARY_PATH # optional: for PSATD in RZ geometry support -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/blaspp-master:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=$HOME/sw/lassen/gpu/lapackpp-master:$CMAKE_PREFIX_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=$HOME/sw/lassen/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-master:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=${SW_DIR}/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-master/lib64:$LD_LIBRARY_PATH # optional: for Python bindings module load python/3.8.2 -if [ -d "$HOME/sw/lassen/gpu/venvs/warpx-lassen" ] +if [ -d "${SW_DIR}/venvs/warpx-lassen" ] then - source $HOME/sw/lassen/gpu/venvs/warpx-lassen/bin/activate + source ${SW_DIR}/venvs/warpx-lassen/bin/activate fi # optional: an alias to request an interactive node for two hours From 2eb664a66d9b220c402be90c234ef7397f074c07 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 18:22:40 -0500 Subject: [PATCH 004/110] Quartz (LLNL): New Modules, Clang (#4281) Module update on Quartz requires changes. Switching to Clang 14 as the compiler. --- Docs/source/install/hpc/quartz.rst | 8 +++---- .../quartz-llnl/quartz_warpx.profile.example | 23 ++++++++----------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/Docs/source/install/hpc/quartz.rst b/Docs/source/install/hpc/quartz.rst index 8902ec2cf45..95d4d5539c0 100644 --- a/Docs/source/install/hpc/quartz.rst +++ b/Docs/source/install/hpc/quartz.rst @@ -46,15 +46,15 @@ Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following comman .. code-block:: bash cd $HOME/src/warpx - rm -rf build + rm -rf build_quartz - cmake -S . -B build -DWarpX_DIMS="1;2;3" -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON - cmake --build build -j 6 + cmake -S . -B build_quartz -DWarpX_DIMS="1;2;3" -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON + cmake --build build_quartz -j 6 The other :ref:`general compile-time options ` apply as usual. **That's it!** -A 3D WarpX executable is now in ``build/bin/`` and :ref:`can be run ` with a :ref:`3D example inputs file `. +A 3D WarpX executable is now in ``build_quartz/bin/`` and :ref:`can be run ` with a :ref:`3D example inputs file `. Most people execute the binary directly or copy it out to a location in ``/p/lustre1/$(whoami)``. diff --git a/Tools/machines/quartz-llnl/quartz_warpx.profile.example b/Tools/machines/quartz-llnl/quartz_warpx.profile.example index 370e4a601ac..b54a4458658 100644 --- a/Tools/machines/quartz-llnl/quartz_warpx.profile.example +++ b/Tools/machines/quartz-llnl/quartz_warpx.profile.example @@ -2,25 +2,25 @@ #export proj= # required dependencies -module load cmake/3.20.2 -module load intel/2021.4 -module load mvapich2/2.3 +module load cmake/3.23.1 +module load clang/14.0.6-magic +module load mvapich2/2.3.7 # optional: for PSATD support -module load fftw/3.3.8 +module load fftw/3.3.10 # optional: for QED lookup table generation support -module load boost/1.73.0 +module load boost/1.80.0 # optional: for openPMD support # TODO ADIOS2 -module load hdf5-parallel/1.10.2 +module load hdf5-parallel/1.14.0 # optional: for PSATD in RZ geometry support # TODO: blaspp lapackpp # optional: for Python bindings -module load python/3.8.2 +module load python/3.9.12 # optional: an alias to request an interactive node for two hours alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug --pty bash" @@ -29,9 +29,6 @@ alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task shopt -s direxpand # compiler environment hints -export CC=$(which icc) -export CXX=$(which icpc) -export FC=$(which ifort) -# we need a newer libstdc++: -export CFLAGS="-gcc-name=/usr/tce/packages/gcc/gcc-8.3.1/bin/gcc ${CFLAGS}" -export CXXFLAGS="-gxx-name=/usr/tce/packages/gcc/gcc-8.3.1/bin/g++ ${CXXFLAGS}" +export CC=$(which clang) +export CXX=$(which clang++) +export FC=$(which gfortran) From b973b3e03ddb84bac7ca2ef66a455fb693d08bc1 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 16:49:03 -0700 Subject: [PATCH 005/110] Lassen (LLNL): Matplotlib fix Failed on freetype (no module). --- Tools/machines/lassen-llnl/install_v100_dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index fcf0bb14afd..a37f619cbd7 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -123,7 +123,7 @@ python3 -m pip install --upgrade pandas python3 -m pip install --upgrade -Ccompile-args="-j10" scipy python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py python3 -m pip install --upgrade openpmd-api -python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself +MPLLOCALFREETYPE=1 python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself python3 -m pip install --upgrade yt # install or update WarpX dependencies such as picmistandard From 68e77074afc9ba8f95585d06cb07bc11d37203ff Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 18:18:04 -0700 Subject: [PATCH 006/110] Lassen (LLNL): yt fix Avoid an implicit upgrade of matplotlib. --- .../machines/lassen-llnl/install_v100_dependencies.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index a37f619cbd7..ed3f618f431 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -103,9 +103,6 @@ fi CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-lassen-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master -DLAPACK_LIBRARIES=/usr/lib64/liblapack.so cmake --build ${build_dir}/lapackpp-lassen-build --target install --parallel 10 -# remove build temporary directory -rm -rf ${build_dir} - # Python ###################################################################### # @@ -124,9 +121,14 @@ python3 -m pip install --upgrade -Ccompile-args="-j10" scipy python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py python3 -m pip install --upgrade openpmd-api MPLLOCALFREETYPE=1 python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself -python3 -m pip install --upgrade yt +echo "matplotlib==3.2.2" > ${build_dir}/constraints.txt +python3 -m pip install --upgrade -c ${build_dir}/constraints.txt yt # install or update WarpX dependencies such as picmistandard python3 -m pip install --upgrade -r $HOME/src/warpx/requirements.txt # for ML dependencies, see install_v100_ml.sh + + +# remove build temporary directory +rm -rf ${build_dir} From 36b2f9efd6916fa33b2c70e85eae86ee550b1d87 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 8 Sep 2023 21:20:54 -0700 Subject: [PATCH 007/110] Lassen (LLNL): pyTorch w/ GNU Work-around for https://github.com/pytorch/pytorch/issues/108934 --- Tools/machines/lassen-llnl/install_v100_ml.sh | 16 +++++++++------- .../lassen_v100_warpx.profile.example | 2 +- Tools/machines/summit-olcf/install_gpu_ml.sh | 1 + 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh index e736f83ba20..c28f8d21166 100755 --- a/Tools/machines/lassen-llnl/install_v100_ml.sh +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -37,19 +37,21 @@ then cd ${HOME}/src/pytorch git fetch git checkout . - git checkout v2.0.1 + git checkout v2.1.0-rc3 + git submodule update --init --recursive cd - else - git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git ${HOME}/src/pytorch + git clone -b v2.1.0-rc3 --recurse-submodules https://github.com/pytorch/pytorch.git ${HOME}/src/pytorch fi cd ${HOME}/src/pytorch rm -rf build + +# see https://github.com/pytorch/pytorch/issues/108931 +# https://github.com/pytorch/pytorch/pull/108932 +wget -q -O - https://github.com/pytorch/pytorch/pull/108932.patch | git apply + python3 -m pip install -r requirements.txt -# patch to avoid compile issues -# https://github.com/pytorch/pytorch/issues/97497#issuecomment-1499069641 -# https://github.com/pytorch/pytorch/pull/98511 -wget -q -O - https://github.com/pytorch/pytorch/pull/98511.patch | git apply -USE_CUDA=1 BLAS=OpenBLAS MAX_JOBS=64 ATEN_AVX512_256=OFF BUILD_TEST=0 python3 setup.py develop +CXX=g++ CC=gcc USE_CUDA=1 BLAS=OpenBLAS MAX_JOBS=64 ATEN_AVX512_256=OFF BUILD_TEST=0 python3 setup.py develop # (optional) If using torch.compile with inductor/triton, install the matching version of triton #make triton rm -rf build diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example index 09808a03ebb..6bbc6a6a57b 100644 --- a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example @@ -50,4 +50,4 @@ export CC=$(which clang) export CXX=$(which clang++) export FC=$(which gfortran) export CUDACXX=$(which nvcc) -export CUDAHOSTCXX=$(which clang++) +export CUDAHOSTCXX=${CXX} diff --git a/Tools/machines/summit-olcf/install_gpu_ml.sh b/Tools/machines/summit-olcf/install_gpu_ml.sh index ff4eaf1555c..25a8eb27abd 100755 --- a/Tools/machines/summit-olcf/install_gpu_ml.sh +++ b/Tools/machines/summit-olcf/install_gpu_ml.sh @@ -60,6 +60,7 @@ then git fetch git checkout . git checkout v2.0.1 + git submodule update --init --recursive cd - else git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git /ccs/proj/${proj}/${USER}/src/pytorch From 5df916b13e14f588e3561f4d2cb8c461599f01eb Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Sun, 10 Sep 2023 19:13:02 -0700 Subject: [PATCH 008/110] Lassen (LLNL): GNU 11.2.1 (#4283) Switch from Clang 12 to GCC 11 on Lassen (LLNL), due to issues seen in PyTorch and a risk of general vectorization issues in Clang/LLVM for PPC64le in LLVM at this point. --- .../lassen-llnl/install_v100_dependencies.sh | 3 +-- Tools/machines/lassen-llnl/install_v100_ml.sh | 15 +++++++++------ .../lassen-llnl/lassen_v100_warpx.profile.example | 6 +++--- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index ed3f618f431..80913c892b6 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -114,8 +114,7 @@ python3 -m pip install --upgrade pip python3 -m pip cache purge python3 -m pip install --upgrade wheel python3 -m pip install --upgrade cython -# see https://github.com/numpy/numpy/issues/24673 -python3 -m pip install --upgrade numpy==1.22 +python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas python3 -m pip install --upgrade -Ccompile-args="-j10" scipy python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh index c28f8d21166..d3f5cb05241 100755 --- a/Tools/machines/lassen-llnl/install_v100_ml.sh +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -37,21 +37,24 @@ then cd ${HOME}/src/pytorch git fetch git checkout . - git checkout v2.1.0-rc3 + git checkout v2.0.1 git submodule update --init --recursive cd - else - git clone -b v2.1.0-rc3 --recurse-submodules https://github.com/pytorch/pytorch.git ${HOME}/src/pytorch + git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git ${HOME}/src/pytorch fi cd ${HOME}/src/pytorch rm -rf build -# see https://github.com/pytorch/pytorch/issues/108931 -# https://github.com/pytorch/pytorch/pull/108932 -wget -q -O - https://github.com/pytorch/pytorch/pull/108932.patch | git apply +# see https://github.com/pytorch/pytorch/issues/97497#issuecomment-1499069641 +# https://github.com/pytorch/pytorch/pull/98511 +wget -q -O - https://github.com/pytorch/pytorch/pull/98511.patch | git apply python3 -m pip install -r requirements.txt -CXX=g++ CC=gcc USE_CUDA=1 BLAS=OpenBLAS MAX_JOBS=64 ATEN_AVX512_256=OFF BUILD_TEST=0 python3 setup.py develop + +# see https://github.com/pytorch/pytorch/issues/108984#issuecomment-1712938737 +LDFLAGS="-L${CUDA_HOME}/nvidia/targets/ppc64le-linux/lib/" \ +USE_CUDA=1 BLAS=OpenBLAS MAX_JOBS=64 ATEN_AVX512_256=OFF BUILD_TEST=0 python3 setup.py develop # (optional) If using torch.compile with inductor/triton, install the matching version of triton #make triton rm -rf build diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example index 6bbc6a6a57b..dd938e85c11 100644 --- a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example @@ -3,7 +3,7 @@ # required dependencies module load cmake/3.23.1 -module load clang/12.0.1-gcc-8.3.1 +module load gcc/11.2.1 module load cuda/12.0.0 # optional: for QED lookup table generation support @@ -46,8 +46,8 @@ export AMREX_CUDA_ARCH=7.0 export CUDAARCHS=70 # compiler environment hints -export CC=$(which clang) -export CXX=$(which clang++) +export CC=$(which gcc) +export CXX=$(which g++) export FC=$(which gfortran) export CUDACXX=$(which nvcc) export CUDAHOSTCXX=${CXX} From 5c00f09d85a9da10624175995cae7ea0bf9e6add Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Sun, 10 Sep 2023 21:26:38 -0700 Subject: [PATCH 009/110] Doc: Lassen (LLNL) Cleanup --- Docs/source/install/hpc/lassen.rst | 6 ++++-- Tools/machines/lassen-llnl/install_v100_dependencies.sh | 2 +- Tools/machines/lassen-llnl/install_v100_ml.sh | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Docs/source/install/hpc/lassen.rst b/Docs/source/install/hpc/lassen.rst index 785bce3b385..7b5630e0272 100644 --- a/Docs/source/install/hpc/lassen.rst +++ b/Docs/source/install/hpc/lassen.rst @@ -29,9 +29,11 @@ Login Lassen is currently transitioning to RHEL8. During this transition, first SSH into lassen and then ``ssh eatoss4`` next to work with the updated RHEL8/TOSS4 nodes. - Approximately September 2023, the new software environment on these nodes will be the new default. + Approximately October 2023, the new software environment on these nodes will be the new default. +.. _building-lassen-preparation: + Preparation ----------- @@ -79,7 +81,7 @@ Finally, since lassen does not yet provide software modules for some of our depe .. code-block:: bash bash $HOME/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh - source $HOME/sw/lassen/gpu/venvs/warpx-lassen/bin/activate + source /usr/workspace/${USER}/lassen/gpu/venvs/warpx-lassen/bin/activate .. dropdown:: Script Details :color: light diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index 80913c892b6..21650f09ee0 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -14,7 +14,7 @@ set -eu -o pipefail # Check: ###################################################################### # -# Was perlmutter_gpu_warpx.profile sourced and configured correctly? +# Was lassen_v100_warpx.profile sourced and configured correctly? if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your lassen_v100_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh index d3f5cb05241..2ff90adb521 100755 --- a/Tools/machines/lassen-llnl/install_v100_ml.sh +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -14,7 +14,7 @@ set -eu -o pipefail # Check: ###################################################################### # -# Was perlmutter_gpu_warpx.profile sourced and configured correctly? +# Was lassen_v100_warpx.profile sourced and configured correctly? if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your lassen_v100_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi From 2e4d6fdd87615486a7c0da4081b92b2de32af2b5 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Sun, 10 Sep 2023 23:19:14 -0700 Subject: [PATCH 010/110] Quartz (LLNL): PICMI Support (#4284) Document Python (PICMI) and ML support for Quartz (LLNL). --- Docs/source/install/hpc/quartz.rst | 118 ++++++++++++++--- .../quartz-llnl/install_dependencies.sh | 121 ++++++++++++++++++ .../quartz-llnl/quartz_warpx.profile.example | 25 +++- 3 files changed, 245 insertions(+), 19 deletions(-) create mode 100755 Tools/machines/quartz-llnl/install_dependencies.sh diff --git a/Docs/source/install/hpc/quartz.rst b/Docs/source/install/hpc/quartz.rst index 95d4d5539c0..de7c5ace848 100644 --- a/Docs/source/install/hpc/quartz.rst +++ b/Docs/source/install/hpc/quartz.rst @@ -11,52 +11,138 @@ Introduction If you are new to this system, **please see the following resources**: -* `LLNL user account `_ +* `LLNL user account `__ (login required) * `Quartz user guide `_ * Batch system: `Slurm `_ +* `Jupyter service `__ (`documentation `__, login required) * `Production directories `_: * ``/p/lustre1/$(whoami)`` and ``/p/lustre2/$(whoami)``: personal directory on the parallel filesystem * Note that the ``$HOME`` directory and the ``/usr/workspace/$(whoami)`` space are NFS mounted and *not* suitable for production quality data generation. -Installation ------------- +.. _building-quartz-preparation: + +Preparation +----------- -Use the following commands to download the WarpX source code and switch to the correct branch: +Use the following commands to download the WarpX source code: .. code-block:: bash git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx -We use the following modules and environments on the system (``$HOME/quartz_warpx.profile``). +We use system software modules, add environment hints and further dependencies via the file ``$HOME/quartz_warpx.profile``. +Create it now: -.. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz_warpx.profile.example - :language: bash - :caption: You can copy this file from ``Tools/machines/quartz-llnl/quartz_warpx.profile.example``. +.. code-block:: bash + + cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example $HOME/quartz_warpx.profile + +.. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example + :language: bash + +Edit the 2nd line of this script, which sets the ``export proj=""`` variable. +For example, if you are member of the project ``tps``, then run ``vi $HOME/quartz_warpx.profile``. +Enter the edit mode by typing ``i`` and edit line 2 to read: + +.. code-block:: bash -We recommend to store the above lines in a file, such as ``$HOME/quartz_warpx.profile``, and load it into your shell after a login: + export proj="tps" + +Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). + +.. important:: + + Now, and as the first step on future logins to Quartz, activate these environment settings: + + .. code-block:: bash + + source $HOME/quartz_warpx.profile + +Finally, since Quartz does not yet provide software modules for some of our dependencies, install them once: .. code-block:: bash - source $HOME/quartz_warpx.profile + bash $HOME/src/warpx/Tools/machines/quartz-llnl/install_dependencies.sh + source /usr/workspace/${USER}/quartz/venvs/warpx-quartz/bin/activate + +.. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/quartz-llnl/install_dependencies.sh + :language: bash + + +.. _building-quartz-compilation: -Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following commands to compile: +Compilation +----------- + +Use the following :ref:`cmake commands ` to compile the application executable: .. code-block:: bash cd $HOME/src/warpx rm -rf build_quartz - cmake -S . -B build_quartz -DWarpX_DIMS="1;2;3" -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON + cmake -S . -B build_quartz -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" cmake --build build_quartz -j 6 -The other :ref:`general compile-time options ` apply as usual. +The WarpX application executables are now in ``$HOME/src/warpx/build_quartz/bin/``. +Additionally, the following commands will install WarpX as a Python module: + +.. code-block:: bash + + rm -rf build_quartz_py + + cmake -S . -B build_quartz_py -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_quartz_py -j 6 --target pip_install + +Now, you can :ref:`submit Quartz compute jobs ` for WarpX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). +Or, you can use the WarpX executables to submit Quartz jobs (:ref:`example inputs `). +For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$PROJWORK/$proj/``. + + +.. _building-quartz-update: + +Update WarpX & Dependencies +--------------------------- + +If you already installed WarpX in the past and want to update it, start by getting the latest source code: + +.. code-block:: bash + + cd $HOME/src/warpx + + # read the output of this command - does it look ok? + git status + + # get the latest WarpX source code + git fetch + git pull + + # read the output of these commands - do they look ok? + git status + git log # press q to exit + +And, if needed, + +- :ref:`update the quartz_warpx.profile file `, +- log out and into the system, activate the now updated environment profile as usual, +- :ref:`execute the dependency install scripts `. + +As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_quartz`` and rebuild WarpX. -**That's it!** -A 3D WarpX executable is now in ``build_quartz/bin/`` and :ref:`can be run ` with a :ref:`3D example inputs file `. -Most people execute the binary directly or copy it out to a location in ``/p/lustre1/$(whoami)``. +.. _running-cpp-quartz: Running ------- diff --git a/Tools/machines/quartz-llnl/install_dependencies.sh b/Tools/machines/quartz-llnl/install_dependencies.sh new file mode 100755 index 00000000000..c162ac5d390 --- /dev/null +++ b/Tools/machines/quartz-llnl/install_dependencies.sh @@ -0,0 +1,121 @@ +#!/bin/bash +# +# Copyright 2023 The WarpX Community +# +# This file is part of WarpX. +# +# Author: Axel Huebl +# License: BSD-3-Clause-LBNL + +# Exit on first error encountered ############################################# +# +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was quartz_warpx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your quartz_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi + + +# Remove old dependencies ##################################################### +# +SW_DIR="/usr/workspace/${USER}/quartz" +rm -rf ${SW_DIR} +mkdir -p ${SW_DIR} + +# remove common user mistakes in python, located in .local instead of a venv +python3 -m pip uninstall -qq -y pywarpx +python3 -m pip uninstall -qq -y warpx +python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true + + +# General extra dependencies ################################################## +# + +# tmpfs build directory: avoids issues often seen with ${HOME} and is faster +build_dir=$(mktemp -d) + +# c-blosc (I/O compression) +if [ -d ${HOME}/src/c-blosc ] +then + cd ${HOME}/src/c-blosc + git fetch --prune + git checkout v1.21.1 + cd - +else + git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git ${HOME}/src/c-blosc +fi +cmake -S ${HOME}/src/c-blosc -B ${build_dir}/c-blosc-quartz-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 +cmake --build ${build_dir}/c-blosc-quartz-build --target install --parallel 6 + +# ADIOS2 +if [ -d ${HOME}/src/adios2 ] +then + cd ${HOME}/src/adios2 + git fetch --prune + git checkout v2.8.3 + cd - +else + git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git ${HOME}/src/adios2 +fi +cmake -S ${HOME}/src/adios2 -B ${build_dir}/adios2-quartz-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake --build ${build_dir}/adios2-quartz-build --target install -j 6 + +# BLAS++ (for PSATD+RZ) +if [ -d ${HOME}/src/blaspp ] +then + cd ${HOME}/src/blaspp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/blaspp.git ${HOME}/src/blaspp +fi +cmake -S ${HOME}/src/blaspp -B ${build_dir}/blaspp-quartz-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake --build ${build_dir}/blaspp-quartz-build --target install --parallel 6 + +# LAPACK++ (for PSATD+RZ) +if [ -d ${HOME}/src/lapackpp ] +then + cd ${HOME}/src/lapackpp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/lapackpp.git ${HOME}/src/lapackpp +fi +CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-quartz-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master +cmake --build ${build_dir}/lapackpp-quartz-build --target install --parallel 6 + + +# Python ###################################################################### +# +python3 -m pip install --upgrade --user virtualenv +rm -rf ${SW_DIR}/venvs/warpx-quartz +python3 -m venv ${SW_DIR}/venvs/warpx-quartz +source ${SW_DIR}/venvs/warpx-quartz/bin/activate +python3 -m pip install --upgrade pip +python3 -m pip cache purge +python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade cython +python3 -m pip install --upgrade numpy +python3 -m pip install --upgrade pandas +python3 -m pip install --upgrade scipy +python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py +python3 -m pip install --upgrade openpmd-api +python3 -m pip install --upgrade matplotlib +python3 -m pip install --upgrade yt + +# install or update WarpX dependencies such as picmistandard +python3 -m pip install --upgrade -r ${HOME}/src/warpx/requirements.txt + +# ML dependencies +python3 -m pip install --upgrade torch + + +# remove build temporary directory ############################################ +# +rm -rf ${build_dir} diff --git a/Tools/machines/quartz-llnl/quartz_warpx.profile.example b/Tools/machines/quartz-llnl/quartz_warpx.profile.example index b54a4458658..810005bafb7 100644 --- a/Tools/machines/quartz-llnl/quartz_warpx.profile.example +++ b/Tools/machines/quartz-llnl/quartz_warpx.profile.example @@ -1,5 +1,5 @@ # please set your project account -#export proj= +#export proj="" # edit this and comment in # required dependencies module load cmake/3.23.1 @@ -13,21 +13,40 @@ module load fftw/3.3.10 module load boost/1.80.0 # optional: for openPMD support -# TODO ADIOS2 module load hdf5-parallel/1.14.0 +SW_DIR="/usr/workspace/${USER}/quartz" +export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH + # optional: for PSATD in RZ geometry support -# TODO: blaspp lapackpp +export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-master:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=${SW_DIR}/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-master/lib64:$LD_LIBRARY_PATH # optional: for Python bindings module load python/3.9.12 +if [ -d "${SW_DIR}/venvs/warpx-quartz" ] +then + source ${SW_DIR}/venvs/warpx-quartz/bin/activate +fi + # optional: an alias to request an interactive node for two hours alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug --pty bash" +# an alias to run a command on a batch node for up to 30min +# usage: runNode +alias runNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug" # fix system defaults: do not escape $ with a \ on tab completion shopt -s direxpand +# optimize CPU microarchitecture for Intel Xeon E5-2695 v4 +# note: the cc/CC/ftn wrappers below add those +export CXXFLAGS="-march=broadwell" +export CFLAGS="-march=broadwell" + # compiler environment hints export CC=$(which clang) export CXX=$(which clang++) From ac52a762c297712411fec4877b9770e2bf4d4343 Mon Sep 17 00:00:00 2001 From: Arianna Formenti Date: Mon, 11 Sep 2023 10:18:25 -0700 Subject: [PATCH 011/110] add collider-relevant reduced diags (#4024) * added collider reduced diags * added test * fixed xy_ave and xy_std * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed usage of IntVect and removed print * Set up automated CI test To-do: - Fix bug (erroneous arithmetic operation) - Remove unused parameters from input file * Fix warning: set mass only if WARPX_QED * Use amrex::ParticleReal for mass * Update Make.package to fix GNU Make build * reduced_data.value() only once * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Merge `development` into `coll_rel_red_diags` * updated 3d beam test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated 3d test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * `m_write_header` instead of `m_IsNotRestart` * added angles * updated 3d beam test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added 3 particle test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added warning * updated diags names * updated analysis multiple particles * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added positrons to test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update CI test * Reuse input file parser from Tools * Apply suggestions from code review * Apply suggestions from code review * changed order of diags and added docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply WarpX style conventions * Fix CodeQL alerts * assert in RZ and removed fine ref levs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix compilation in 2D * Fix compilation in RZ * Update Docs/source/usage/parameters.rst Co-authored-by: Axel Huebl * Update Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp Co-authored-by: Axel Huebl * Update Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp Co-authored-by: Luca Fedeli * Update Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp Co-authored-by: Luca Fedeli * Update Source/Diagnostics/ReducedDiags/ColliderRelevant.H Co-authored-by: Axel Huebl * Update Source/Diagnostics/ReducedDiags/ColliderRelevant.H Co-authored-by: Axel Huebl * minimized number of kernel launches and parallel ops * updated constructor of ReduceDiags * fixed based on comments * updated docs: if QED not enable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed ReducedDiags constructor * two passes instead of one for robusteness * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bugs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix 1D build (typo `wtot` instead of `w_tot`) * refixed compile in RZ * check io process in chi_ave computation * updated test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Temporary fix: remove IOProcessor * Update Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp Co-authored-by: Axel Huebl * Apply suggestions from code review * Remove `Debug` compilation mode from CI test * CI test analysis: check all particles have same charge * Use `m_beam_name` instead of `species_names` --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Edoardo Zoni Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Co-authored-by: Axel Huebl Co-authored-by: Luca Fedeli --- Docs/source/usage/parameters.rst | 51 ++ .../analysis_multiple_particles.py | 150 +++++ .../inputs_3d_multiple_particles | 130 +++++ .../benchmarks_json/collider_diagnostics.json | 36 ++ Regression/WarpX-tests.ini | 16 + .../Diagnostics/ReducedDiags/CMakeLists.txt | 1 + .../ReducedDiags/ColliderRelevant.H | 61 ++ .../ReducedDiags/ColliderRelevant.cpp | 548 ++++++++++++++++++ Source/Diagnostics/ReducedDiags/Make.package | 1 + .../ReducedDiags/MultiReducedDiags.cpp | 2 + .../Diagnostics/ReducedDiags/ReducedDiags.cpp | 3 +- Tools/Algorithms/stencil.py | 2 + Tools/Parser/__init__.py | 0 .../input_file_parser.py | 0 14 files changed, 999 insertions(+), 2 deletions(-) create mode 100755 Examples/Tests/collider_relevant_diags/analysis_multiple_particles.py create mode 100644 Examples/Tests/collider_relevant_diags/inputs_3d_multiple_particles create mode 100644 Regression/Checksum/benchmarks_json/collider_diagnostics.json create mode 100644 Source/Diagnostics/ReducedDiags/ColliderRelevant.H create mode 100644 Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp create mode 100644 Tools/Parser/__init__.py rename Tools/{Algorithms => Parser}/input_file_parser.py (100%) diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index 4514e6f3fb0..27d7682899d 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -2988,6 +2988,57 @@ Reduced Diagnostics 1 or 0, it is possible to compute the charge on only some part of the embedded boundary. + * ``ColliderRelevant`` + This diagnostics computes properties of two colliding beams that are relevant for particle colliders. + Two species must be specified. Photon species are not supported yet. + It is assumed that the two species propagate and collide along the ``z`` direction. + The output columns (for 3D-XYZ) are the following, where the minimum, average and maximum + are done over the whole species: + + [0]: simulation step (iteration). + + [1]: time (s). + + [2]: time derivative of the luminosity (:math:`m^{-2}s^{-1}`) defined as: + + .. math:: + + \frac{dL}{dt} = 2 c \iiint n_1(x,y,z) n_2(x,y,z) dx dy dz + + where :math:`n_1`, :math:`n_2` are the number densities of the two colliding species. + + [3], [4], [5]: If, QED is enabled, the minimum, average and maximum values of the quantum parameter :math:`\chi` of species 1: + :math:`\chi_{min}`, + :math:`\langle \chi \rangle`, + :math:`\chi_{max}`. + If QED is not enabled, these numbers are not computed. + + [6], [7]: The average and standard deviation of the values of the transverse coordinate :math:`x` (m) of species 1: + :math:`\langle x \rangle`, + :math:`\sqrt{\langle x- \langle x \rangle \rangle^2}`. + + [8], [9]: The average and standard deviation of the values of the transverse coordinate :math:`y` (m) of species 1: + :math:`\langle y \rangle`, + :math:`\sqrt{\langle y- \langle y \rangle \rangle^2}`. + + [10], [11], [12], [13]: The minimum, average, maximum and standard deviation of the angle :math:`\theta_x = \angle (u_x, u_z)` (rad) of species 1: + :math:`{\theta_x}_{min}`, + :math:`\langle \theta_x \rangle`, + :math:`{\theta_x}_{max}`, + :math:`\sqrt{\langle \theta_x- \langle \theta_x \rangle \rangle^2}`. + + [14], [15], [16], [17]: The minimum, average, maximum and standard deviation of the angle :math:`\theta_y = \angle (u_y, u_z)` (rad) of species 1: + :math:`{\theta_y}_{min}`, + :math:`\langle \theta_y \rangle`, + :math:`{\theta_y}_{max}`, + :math:`\sqrt{\langle \theta_y- \langle \theta_y \rangle \rangle^2}`. + + [18], ..., [32]: Analogous quantities for species 2. + + For 2D-XZ, :math:`y`-related quantities are not outputted. + For 1D-Z, :math:`x`-related and :math:`y`-related quantities are not outputted. + RZ geometry is not supported yet. + * ``.intervals`` (`string`) Using the `Intervals Parser`_ syntax, this string defines the timesteps at which reduced diagnostics are written to file. diff --git a/Examples/Tests/collider_relevant_diags/analysis_multiple_particles.py b/Examples/Tests/collider_relevant_diags/analysis_multiple_particles.py new file mode 100755 index 00000000000..b23bb69d52c --- /dev/null +++ b/Examples/Tests/collider_relevant_diags/analysis_multiple_particles.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 + +import os +import sys + +import numpy as np +import openpmd_api as io +import pandas as pd +from scipy.constants import c, e, hbar, m_e + +sys.path.append('../../../../warpx/Regression/Checksum/') +import checksumAPI + +sys.path.append('../../../../warpx/Tools/Parser/') +from input_file_parser import parse_input_file + +E_crit = m_e**2*c**3/(e*hbar) +B_crit = m_e**2*c**2/(e*hbar) + +def chi(ux, uy, uz, Ex, Ey, Ez, Bx, By, Bz): + gamma = np.sqrt(1.+ux**2+uy**2+uz**2) + vx = ux / gamma * c + vy = uy / gamma * c + vz = uz / gamma * c + tmp1x = Ex + vy*Bz - vz*By + tmp1y = Ey - vx*Bz + vz*Bx + tmp1z = Ez + vx*By - vy*Bx + tmp2 = (Ex*vx + Ey*vy + Ez*vz)/c + chi = gamma/E_crit*np.sqrt(tmp1x**2+tmp1y**2+tmp1z**2 - tmp2**2) + return chi + +def dL_dt(): + series = io.Series("diags/diag2/openpmd_%T.h5",io.Access.read_only) + iterations = np.asarray(series.iterations) + lumi = [] + for n,ts in enumerate(iterations): + it = series.iterations[ts] + rho1 = it.meshes["rho_beam_e"] + dV = np.prod(rho1.grid_spacing) + rho1 = it.meshes["rho_beam_e"][io.Mesh_Record_Component.SCALAR].load_chunk() + rho2 = it.meshes["rho_beam_p"][io.Mesh_Record_Component.SCALAR].load_chunk() + beam_e_charge = it.particles["beam_e"]["charge"][io.Mesh_Record_Component.SCALAR].load_chunk() + beam_p_charge = it.particles["beam_p"]["charge"][io.Mesh_Record_Component.SCALAR].load_chunk() + q1 = beam_e_charge[0] + if not np.all(beam_e_charge == q1): + sys.exit('beam_e particles do not have the same charge') + q2 = beam_p_charge[0] + if not np.all(beam_p_charge == q2): + sys.exit('beam_p particles do not have the same charge') + series.flush() + n1 = rho1/q1 + n2 = rho2/q2 + l = 2*np.sum(n1*n2)*dV*c + lumi.append(l) + return lumi + +input_dict = parse_input_file('inputs_3d_multiple_particles') +Ex, Ey, Ez = [float(w) for w in input_dict['particles.E_external_particle']] +Bx, By, Bz = [float(w) for w in input_dict['particles.B_external_particle']] + +CollDiagFname='diags/reducedfiles/ColliderRelevant_beam_e_beam_p.txt' +df = pd.read_csv(CollDiagFname, sep=" ", header=0) + +for species in ['beam_p', 'beam_e']: + + ux1, ux2, ux3 = [float(w) for w in input_dict[f'{species}.multiple_particles_ux']] + uy1, uy2, uy3 = [float(w) for w in input_dict[f'{species}.multiple_particles_uy']] + uz1, uz2, uz3 = [float(w) for w in input_dict[f'{species}.multiple_particles_uz']] + + x = np.array([float(w) for w in input_dict[f'{species}.multiple_particles_pos_x']]) + y = np.array([float(w) for w in input_dict[f'{species}.multiple_particles_pos_y']]) + + w = np.array([float(w) for w in input_dict[f'{species}.multiple_particles_weight']]) + + CHI_ANALYTICAL = np.array([chi(ux1, uy1, uz1, Ex, Ey, Ez, Bx, By, Bz), + chi(ux2, uy2, uz2, Ex, Ey, Ez, Bx, By, Bz), + chi(ux3, uy3, uz3, Ex, Ey, Ez, Bx, By, Bz)]) + THETAX = np.array([np.arctan2(ux1, uz1), np.arctan2(ux2, uz2), np.arctan2(ux3, uz3)]) + THETAY = np.array([np.arctan2(uy1, uz1), np.arctan2(uy2, uz2), np.arctan2(uy3, uz3)]) + + # CHI MAX + fname=f'diags/reducedfiles/ParticleExtrema_{species}.txt' + chimax_pe = np.loadtxt(fname)[:,19] + chimax_cr = df[[col for col in df.columns if f'chi_max_{species}' in col]].to_numpy() + assert np.allclose(np.max(CHI_ANALYTICAL), chimax_cr, rtol=1e-8) + assert np.allclose(chimax_pe, chimax_cr, rtol=1e-8) + + # CHI MIN + fname=f'diags/reducedfiles/ParticleExtrema_{species}.txt' + chimin_pe = np.loadtxt(fname)[:,18] + chimin_cr = df[[col for col in df.columns if f'chi_min_{species}' in col]].to_numpy() + assert np.allclose(np.min(CHI_ANALYTICAL), chimin_cr, rtol=1e-8) + assert np.allclose(chimin_pe, chimin_cr, rtol=1e-8) + + # CHI AVERAGE + chiave_cr = df[[col for col in df.columns if f'chi_ave_{species}' in col]].to_numpy() + assert np.allclose(np.average(CHI_ANALYTICAL, weights=w), chiave_cr, rtol=1e-8) + + # X AVE STD + x_ave_cr = df[[col for col in df.columns if f']x_ave_{species}' in col]].to_numpy() + x_std_cr = df[[col for col in df.columns if f']x_std_{species}' in col]].to_numpy() + x_ave = np.average(x, weights=w) + x_std = np.sqrt(np.average((x-x_ave)**2, weights=w)) + assert np.allclose(x_ave, x_ave_cr, rtol=1e-8) + assert np.allclose(x_std, x_std_cr, rtol=1e-8) + + # Y AVE STD + y_ave_cr = df[[col for col in df.columns if f']y_ave_{species}' in col]].to_numpy() + y_std_cr = df[[col for col in df.columns if f']y_std_{species}' in col]].to_numpy() + y_ave = np.average(y, weights=w) + y_std = np.sqrt(np.average((y-y_ave)**2, weights=w)) + assert np.allclose(y_ave, y_ave_cr, rtol=1e-8) + assert np.allclose(y_std, y_std_cr, rtol=1e-8) + + # THETA X MIN AVE MAX STD + thetax_min_cr = df[[col for col in df.columns if f'theta_x_min_{species}' in col]].to_numpy() + thetax_ave_cr = df[[col for col in df.columns if f'theta_x_ave_{species}' in col]].to_numpy() + thetax_max_cr = df[[col for col in df.columns if f'theta_x_max_{species}' in col]].to_numpy() + thetax_std_cr = df[[col for col in df.columns if f'theta_x_std_{species}' in col]].to_numpy() + thetax_min = np.min(THETAX) + thetax_ave = np.average(THETAX, weights=w) + thetax_max = np.max(THETAX) + thetax_std = np.sqrt(np.average((THETAX-thetax_ave)**2, weights=w)) + assert np.allclose(thetax_min, thetax_min_cr, rtol=1e-8) + assert np.allclose(thetax_ave, thetax_ave_cr, rtol=1e-8) + assert np.allclose(thetax_max, thetax_max_cr, rtol=1e-8) + assert np.allclose(thetax_std, thetax_std_cr, rtol=1e-8) + + # THETA Y MIN AVE MAX STD + thetay_min_cr = df[[col for col in df.columns if f'theta_y_min_{species}' in col]].to_numpy() + thetay_ave_cr = df[[col for col in df.columns if f'theta_y_ave_{species}' in col]].to_numpy() + thetay_max_cr = df[[col for col in df.columns if f'theta_y_max_{species}' in col]].to_numpy() + thetay_std_cr = df[[col for col in df.columns if f'theta_y_std_{species}' in col]].to_numpy() + thetay_min = np.min(THETAY) + thetay_ave = np.average(THETAY, weights=w) + thetay_max = np.max(THETAY) + thetay_std = np.sqrt(np.average((THETAY-thetay_ave)**2, weights=w)) + assert np.allclose(thetay_min, thetay_min_cr, rtol=1e-8) + assert np.allclose(thetay_ave, thetay_ave_cr, rtol=1e-8) + assert np.allclose(thetay_max, thetay_max_cr, rtol=1e-8) + assert np.allclose(thetay_std, thetay_std_cr, rtol=1e-8) + + # dL/dt + dL_dt_cr = df[[col for col in df.columns if 'dL_dt' in col]].to_numpy() + assert np.allclose(dL_dt_cr, dL_dt(), rtol=1e-8) + +# Checksum analysis +plotfile = sys.argv[1] +test_name = os.path.split(os.getcwd())[1] +checksumAPI.evaluate_checksum(test_name, plotfile) diff --git a/Examples/Tests/collider_relevant_diags/inputs_3d_multiple_particles b/Examples/Tests/collider_relevant_diags/inputs_3d_multiple_particles new file mode 100644 index 00000000000..1efc68c33b0 --- /dev/null +++ b/Examples/Tests/collider_relevant_diags/inputs_3d_multiple_particles @@ -0,0 +1,130 @@ +################################# +########## MY CONSTANTS ######### +################################# +my_constants.nx = 8 +my_constants.ny = 8 +my_constants.nz = 8 + +################################# +####### GENERAL PARAMETERS ###### +################################# +max_step = 1 +amr.n_cell = nx ny nz +amr.max_grid_size = 4 +amr.blocking_factor = 4 +amr.max_level = 0 +geometry.dims = 3 +geometry.prob_lo = 0 0 0 +geometry.prob_hi = 8 8 8 +particles.do_tiling = 0 +warpx.use_filter = 0 + +################################# +######## BOUNDARY CONDITION ##### +################################# +boundary.field_lo = periodic periodic periodic +boundary.field_hi = periodic periodic periodic +boundary.particle_lo = periodic periodic periodic +boundary.particle_hi = periodic periodic periodic + +################################# +############ NUMERICS ########### +################################# +algo.maxwell_solver = ckc +warpx.cfl = 0.99 +algo.particle_shape = 1 + +################################# +############ FIELDS ############# +################################# +particles.E_ext_particle_init_style = constant +particles.B_ext_particle_init_style = constant +particles.E_external_particle = 10000. 0. 0. +particles.B_external_particle = 0. 5000. 0. + +################################# +########### PARTICLES ########### +################################# +particles.species_names = pho beam_p beam_e +particles.photon_species = pho + +beam_e.species_type = electron +beam_e.injection_style = MultipleParticles +beam_e.multiple_particles_pos_x = 4.5 3.5 0.5 +beam_e.multiple_particles_pos_y = 4.5 2.5 1.5 +beam_e.multiple_particles_pos_z = 4.5 1.5 1.5 +beam_e.multiple_particles_ux = 0.3 0.2 0.1 +beam_e.multiple_particles_uy = 0.4 -0.3 -0.1 +beam_e.multiple_particles_uz = 0.3 0.1 -10. +beam_e.multiple_particles_weight = 1. 2 3 +beam_e.initialize_self_fields = 0 +beam_e.self_fields_required_precision = 5e-10 +beam_e.do_qed_quantum_sync = 1 +beam_e.qed_quantum_sync_phot_product_species = pho +beam_e.do_not_push = 1 +beam_e.do_not_deposit = 1 + +beam_p.species_type = positron +beam_p.injection_style = MultipleParticles +beam_p.multiple_particles_pos_x = 4.5 3.5 0.5 +beam_p.multiple_particles_pos_y = 4.5 2.5 1.5 +beam_p.multiple_particles_pos_z = 4.5 1.5 1.5 +beam_p.multiple_particles_ux = 0.3 0.2 0.1 +beam_p.multiple_particles_uy = 0.4 -0.3 -0.1 +beam_p.multiple_particles_uz = 0.3 0.1 -10. +beam_p.multiple_particles_weight = 1. 2 3 +beam_p.initialize_self_fields = 0 +beam_p.self_fields_required_precision = 5e-10 +beam_p.do_qed_quantum_sync = 1 +beam_p.qed_quantum_sync_phot_product_species = pho +beam_p.do_not_push = 1 +beam_p.do_not_deposit = 1 + +pho.species_type = photon +pho.injection_style = none + +################################# +############# QED ############### +################################# +qed_qs.photon_creation_energy_threshold = 0. +qed_qs.lookup_table_mode = builtin +qed_qs.chi_min = 1.e-3 +warpx.do_qed_schwinger = 0 + +################################# +######### DIAGNOSTICS ########### +################################# +# FULL +diagnostics.diags_names = diag1 diag2 + +diag1.intervals = 1 +diag1.diag_type = Full +diag1.write_species = 1 +diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho_beam_e rho_beam_p rho +diag1.species = pho beam_e beam_p +diag1.format = plotfile +#diag1.dump_last_timestep = 1 + +diag2.intervals = 1 +diag2.diag_type = Full +diag2.write_species = 1 +diag2.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho_beam_e rho_beam_p rho +diag2.species = pho beam_e beam_p +diag2.format = openpmd +diag2.openpmd_backend = h5 +#diag2.dump_last_timestep = 1 + +# REDUCED +warpx.reduced_diags_names = ParticleExtrema_beam_e ParticleExtrema_beam_p ColliderRelevant_beam_e_beam_p + +ColliderRelevant_beam_e_beam_p.type = ColliderRelevant +ColliderRelevant_beam_e_beam_p.intervals = 1 +ColliderRelevant_beam_e_beam_p.species =beam_e beam_p + +ParticleExtrema_beam_e.type = ParticleExtrema +ParticleExtrema_beam_e.intervals = 1 +ParticleExtrema_beam_e.species = beam_e + +ParticleExtrema_beam_p.type = ParticleExtrema +ParticleExtrema_beam_p.intervals = 1 +ParticleExtrema_beam_p.species = beam_p diff --git a/Regression/Checksum/benchmarks_json/collider_diagnostics.json b/Regression/Checksum/benchmarks_json/collider_diagnostics.json new file mode 100644 index 00000000000..06553a7ec19 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/collider_diagnostics.json @@ -0,0 +1,36 @@ +{ + "lev=0": { + "Bx": 0.0, + "By": 0.0, + "Bz": 0.0, + "Ex": 0.0, + "Ey": 0.0, + "Ez": 0.0, + "jx": 0.0, + "jy": 0.0, + "jz": 0.0, + "rho": 0.0, + "rho_beam_e": 9.613059803999999e-19, + "rho_beam_p": 9.613059803999999e-19 + }, + "beam_e": { + "particle_momentum_x": 1.638554718442694e-22, + "particle_momentum_y": 2.184739624590259e-22, + "particle_momentum_z": 2.8401615119673365e-21, + "particle_opticalDepthQSR": 9.767741201839083e+00, + "particle_position_x": 8.5, + "particle_position_y": 8.5, + "particle_position_z": 7.5, + "particle_weight": 6.0 + }, + "beam_p": { + "particle_momentum_x": 1.638554718442694e-22, + "particle_momentum_y": 2.184739624590259e-22, + "particle_momentum_z": 2.8401615119673365e-21, + "particle_opticalDepthQSR": 8.773870620305825e+00, + "particle_position_x": 8.5, + "particle_position_y": 8.5, + "particle_position_z": 7.5, + "particle_weight": 6.0 + } +} diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 65a1e65e723..ab390fcb01b 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -209,6 +209,22 @@ compileTest = 0 doVis = 0 analysisRoutine = Examples/Tests/btd_rz/analysis_BTD_laser_antenna.py +[collider_diagnostics] +buildDir = . +inputFile = Examples/Tests/collider_relevant_diags/inputs_3d_multiple_particles +runtime_params = warpx.abort_on_warning_threshold=high +dim = 3 +addToCompileString = +cmakeSetupOpts = -DWarpX_DIMS=3 +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +analysisRoutine = Examples/Tests/collider_relevant_diags/analysis_multiple_particles.py + [collisionISO] buildDir = . inputFile = Examples/Tests/collision/inputs_3d_isotropization diff --git a/Source/Diagnostics/ReducedDiags/CMakeLists.txt b/Source/Diagnostics/ReducedDiags/CMakeLists.txt index b2a31267087..e63764bc24f 100644 --- a/Source/Diagnostics/ReducedDiags/CMakeLists.txt +++ b/Source/Diagnostics/ReducedDiags/CMakeLists.txt @@ -3,6 +3,7 @@ foreach(D IN LISTS WarpX_DIMS) target_sources(lib_${SD} PRIVATE BeamRelevant.cpp + ColliderRelevant.cpp FieldEnergy.cpp FieldProbe.cpp FieldProbeParticleContainer.cpp diff --git a/Source/Diagnostics/ReducedDiags/ColliderRelevant.H b/Source/Diagnostics/ReducedDiags/ColliderRelevant.H new file mode 100644 index 00000000000..1998c56812d --- /dev/null +++ b/Source/Diagnostics/ReducedDiags/ColliderRelevant.H @@ -0,0 +1,61 @@ +/* Copyright 2023 The WarpX Community + * + * This file is part of WarpX. + * + * Authors: Arianna Formenti + * License: BSD-3-Clause-LBNL + */ + +#ifndef WARPX_DIAGNOSTICS_REDUCEDDIAGS_COLLIDERRELEVANT_H_ +#define WARPX_DIAGNOSTICS_REDUCEDDIAGS_COLLIDERRELEVANT_H_ + +#include "ReducedDiags.H" + +#include +#include +#include + +/** + * This class contains diagnostics that are relevant to colliders. + */ +class ColliderRelevant : public ReducedDiags +{ +public: + + /** + * constructor + * @param[in] rd_name reduced diags names + */ + ColliderRelevant(std::string rd_name); + + /// name of the two colliding species + std::vector m_beam_name; + + /** + * \brief This function computes collider-relevant diagnostics. + * @param[in] step current time step + * + * [0]step, [1]time, [2]dL/dt, + * for first species: + * [3]chi_min, [4]chi_ave, [5] chi_max, + * [6]x_ave, [7]x_std, + * [8]y_ave, [9]y_std, + * [10]thetax_min, [11]thetax_ave, [12]thetax_max, [13]thetax_std, + * [14]thetay_min, [15]thetay_ave, [16]thetay_max, [17]thetay_std + * same for second species follows. + */ + void ComputeDiags(int step) override final; + +private: + /// auxiliary structure to store headers and indices of the reduced diagnostics + struct aux_header_index + { + std::string header; + int idx; + }; + + /// map to store header texts and indices of the reduced diagnostics + std::map m_headers_indices; +}; + +#endif // WARPX_DIAGNOSTICS_REDUCEDDIAGS_COLLIDERRELEVANT_H_ diff --git a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp new file mode 100644 index 00000000000..1e2c16ac737 --- /dev/null +++ b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp @@ -0,0 +1,548 @@ +/* Copyright 2023 The WarpX Community + * + * This file is part of WarpX. + * + * Authors: Arianna Formenti, Yinjian Zhao + * License: BSD-3-Clause-LBNL + */ +#include "ColliderRelevant.H" + +#include "Diagnostics/ReducedDiags/ReducedDiags.H" +#if (defined WARPX_QED) +# include "Particles/ElementaryProcess/QEDInternals/QedChiFunctions.H" +#endif +#include "Particles/Gather/FieldGather.H" +#include "Particles/Gather/GetExternalFields.H" +#include "Particles/MultiParticleContainer.H" +#include "Particles/Pusher/GetAndSetPosition.H" +#include "Particles/SpeciesPhysicalProperties.H" +#include "Particles/WarpXParticleContainer.H" +#include "Utils/WarpXConst.H" +#include "Utils/TextMsg.H" +#include "WarpX.H" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace amrex; + +ColliderRelevant::ColliderRelevant (std::string rd_name) +: ReducedDiags{std::move(rd_name)} +{ + // read colliding species names - must be 2 + amrex::ParmParse pp_rd_name(m_rd_name); + pp_rd_name.getarr("species", m_beam_name); + + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + m_beam_name.size() == 2u, + "Collider-relevant diagnostics must involve exactly two species"); + + // RZ coordinate is not supported +#if (defined WARPX_DIM_RZ) + WARPX_ABORT_WITH_MESSAGE( + "Collider-relevant diagnostics do not work in RZ geometry."); +#endif + + ablastr::warn_manager::WMRecordWarning( + "Diagnostics", + "The collider-relevant reduced diagnostic is meant for \ + colliding species propagating along the z direction.", + ablastr::warn_manager::WarnPriority::low); + + ablastr::warn_manager::WMRecordWarning( + "Diagnostics", + "The collider-relevant reduced diagnostic only considers the \ + coarsest level of refinement for the calculations involving chi.", + ablastr::warn_manager::WarnPriority::low); + + // get WarpX class object + auto& warpx = WarpX::GetInstance(); + + // get MultiParticleContainer class object + const MultiParticleContainer& mypc = warpx.GetPartContainer(); + + // loop over species + for (int i_s = 0; i_s < 2; ++i_s) + { + // get WarpXParticleContainer class object + const WarpXParticleContainer& myspc = mypc.GetParticleContainerFromName(m_beam_name[i_s]); + + // get charge + amrex::ParticleReal const q = myspc.getCharge(); + + // photon number density is not available yet + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + q!=amrex::Real(0.0), + "Collider-relevant diagnostic does not work for neutral species yet"); + } + + // function to fill a vector with diags names and create corresponding entry in header + std::vector all_diag_names; + auto add_diag = [&,c=0]( + const std::string& name, const std::string& header) mutable { + m_headers_indices[name] = aux_header_index{header, c++}; + all_diag_names.push_back(name); + }; + +#if (defined WARPX_DIM_3D) + add_diag("dL_dt", "dL_dt(m^-2*s^-1)"); +#elif (defined WARPX_DIM_XZ) + add_diag("dL_dt", "dL_dt(m^-1*s^-1)"); +#else + add_diag("dL_dt", "dL_dt(s^-1)"); +#endif + + // loop over species + for (int i_s = 0; i_s < 2; ++i_s) + { + // get WarpXParticleContainer class object + const WarpXParticleContainer& myspc = mypc.GetParticleContainerFromName(m_beam_name[i_s]); + + if (myspc.DoQED()){ + add_diag("chimin_"+m_beam_name[i_s], "chi_min_"+m_beam_name[i_s]+"()"); + add_diag("chiave_"+m_beam_name[i_s], "chi_ave_"+m_beam_name[i_s]+"()"); + add_diag("chimax_"+m_beam_name[i_s], "chi_max_"+m_beam_name[i_s]+"()"); + } +#if (defined WARPX_DIM_3D) + add_diag("x_ave_"+m_beam_name[i_s], "x_ave_"+m_beam_name[i_s]+"(m)"); + add_diag("x_std_"+m_beam_name[i_s], "x_std_"+m_beam_name[i_s]+"(m)"); + add_diag("y_ave_"+m_beam_name[i_s], "y_ave_"+m_beam_name[i_s]+"(m)"); + add_diag("y_std_"+m_beam_name[i_s], "y_std_"+m_beam_name[i_s]+"(m)"); + add_diag("thetax_min_"+m_beam_name[i_s], "theta_x_min_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetax_ave_"+m_beam_name[i_s], "theta_x_ave_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetax_max_"+m_beam_name[i_s], "theta_x_max_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetax_std_"+m_beam_name[i_s], "theta_x_std_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetay_min_"+m_beam_name[i_s], "theta_y_min_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetay_ave_"+m_beam_name[i_s], "theta_y_ave_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetay_max_"+m_beam_name[i_s], "theta_y_max_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetay_std_"+m_beam_name[i_s], "theta_y_std_"+m_beam_name[i_s]+"(rad)"); + +#elif (defined WARPX_DIM_XZ) + add_diag("x_ave_"+m_beam_name[i_s], "x_ave_"+m_beam_name[i_s]+"(m)"); + add_diag("x_std_"+m_beam_name[i_s], "x_std_"+m_beam_name[i_s]+"(m)"); + add_diag("thetax_min_"+m_beam_name[i_s], "theta_x_min_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetax_ave_"+m_beam_name[i_s], "theta_x_ave_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetax_max_"+m_beam_name[i_s], "theta_x_max_"+m_beam_name[i_s]+"(rad)"); + add_diag("thetax_std_"+m_beam_name[i_s], "theta_x_std_"+m_beam_name[i_s]+"(rad)"); +#endif + m_data.resize(all_diag_names.size()); + } + + if (amrex::ParallelDescriptor::IOProcessor()) + { + if ( m_write_header ) + { + // open file + std::ofstream ofs; + ofs.open(m_path + m_rd_name + "." + m_extension, + std::ofstream::out | std::ofstream::app); + // write header row + int off = 0; + ofs << "#"; + ofs << "[" << off++ << "]step()"; + ofs << m_sep; + ofs << "[" << off++ << "]time(s)"; + for (const auto& name : all_diag_names){ + const auto& el = m_headers_indices[name]; + ofs << m_sep << "[" << el.idx + off << "]" << el.header; + } + ofs << std::endl; + // close file + ofs.close(); + } + } +} + +void ColliderRelevant::ComputeDiags (int step) +{ +#if defined(WARPX_DIM_RZ) + amrex::ignore_unused(step); +#else + + // Judge if the diags should be done + if (!m_intervals.contains(step+1)) { return; } + + // get MultiParticleContainer class object + const MultiParticleContainer& mypc = WarpX::GetInstance().GetPartContainer(); + + // get a reference to WarpX instance + auto& warpx = WarpX::GetInstance(); + + // get cell volume + amrex::Geometry const & geom = warpx.Geom(0); + amrex::Real dV = AMREX_D_TERM(geom.CellSize(0), *geom.CellSize(1), *geom.CellSize(2)); + + const auto get_idx = [&](const std::string& name){ + return m_headers_indices.at(name).idx; + }; + + std::array, 2> num_dens; + + // loop over species + for (int i_s = 0; i_s < 2; ++i_s) + { + // get WarpXParticleContainer class object + WarpXParticleContainer& myspc = mypc.GetParticleContainerFromName(m_beam_name[i_s]); + // get charge + amrex::ParticleReal const q = myspc.getCharge(); + + using PType = typename WarpXParticleContainer::SuperParticleType; + + num_dens[i_s] = myspc.GetChargeDensity(0); + num_dens[i_s]->mult(1./q); + +#if defined(WARPX_DIM_1D_Z) + // w_tot + amrex::Real w_tot = ReduceSum( myspc, + [=] AMREX_GPU_HOST_DEVICE (const PType& p) + { + return p.rdata(PIdx::w); + }); + amrex::ParallelDescriptor::ReduceRealSum(w_tot); +#elif defined(WARPX_DIM_XZ) + // w_tot + // x_ave, + // thetax_min, thetax_ave, thetax_max + amrex::ReduceOps reduce_ops; + auto r = amrex::ParticleReduce>( + myspc, + [=] AMREX_GPU_DEVICE(const PType& p) noexcept -> amrex::GpuTuple + { + const amrex::Real w = p.rdata(PIdx::w); + const amrex::Real x = p.pos(0); + const amrex::Real ux = p.rdata(PIdx::ux); + const amrex::Real uz = p.rdata(PIdx::uz); + const amrex::Real thetax = std::atan2(ux, uz); + return {w, w*x, thetax, w*thetax, thetax}; + }, + reduce_ops); + + amrex::Real w_tot = amrex::get<0>(r); + amrex::Real x_ave = amrex::get<1>(r); + amrex::Real thetax_min = amrex::get<2>(r); + amrex::Real thetax_ave = amrex::get<3>(r); + amrex::Real thetax_max = amrex::get<4>(r); + + amrex::ParallelDescriptor::ReduceRealSum({w_tot, x_ave, thetax_ave}); + amrex::ParallelDescriptor::ReduceRealMin({thetax_min}); + amrex::ParallelDescriptor::ReduceRealMax({thetax_max}); + + // x_std, thetax_std + amrex::Real x_std = 0.0_rt; + amrex::Real thetax_std = 0.0_rt; + + if (w_tot > 0.0_rt) + { + x_ave = x_ave / w_tot; + thetax_ave = thetax_ave / w_tot; + + amrex::ReduceOps reduce_ops_std; + auto r_std = amrex::ParticleReduce>( + myspc, + [=] AMREX_GPU_DEVICE(const PType& p) noexcept -> amrex::GpuTuple + { + const amrex::Real w = p.rdata(PIdx::w); + const amrex::Real x = p.pos(0); + const amrex::Real ux = p.rdata(PIdx::ux); + const amrex::Real uz = p.rdata(PIdx::uz); + const amrex::Real thetax = std::atan2(ux, uz); + const amrex::Real tmp1 = (x - x_ave)*(x - x_ave)*w; + const amrex::Real tmp2 = (thetax - thetax_ave)*(thetax - thetax_ave)*w; + return {tmp1, tmp2}; + }, + reduce_ops_std); + + x_std = amrex::get<0>(r_std); + thetax_std = amrex::get<1>(r_std); + + amrex::ParallelDescriptor::ReduceRealSum({x_std, thetax_std}); + + x_std = std::sqrt(x_std / w_tot); + thetax_std = std::sqrt(thetax_std / w_tot); + } + + m_data[get_idx("x_ave_"+m_beam_name[i_s])] = x_ave; + m_data[get_idx("x_std_"+m_beam_name[i_s])] = x_std; + m_data[get_idx("thetax_min_"+m_beam_name[i_s])] = thetax_min; + m_data[get_idx("thetax_ave_"+m_beam_name[i_s])] = thetax_ave; + m_data[get_idx("thetax_max_"+m_beam_name[i_s])] = thetax_max; + m_data[get_idx("thetax_std_"+m_beam_name[i_s])] = thetax_std; +#elif defined(WARPX_DIM_3D) + // w_tot + // x_ave, y_ave, + // thetax_min, thetax_ave, thetax_max + // thetay_min, thetay_ave, thetay_max + amrex::ReduceOps reduce_ops; + auto r = amrex::ParticleReduce>( + myspc, + [=] AMREX_GPU_DEVICE(const PType& p) noexcept -> amrex::GpuTuple + { + const amrex::Real w = p.rdata(PIdx::w); + const amrex::Real x = p.pos(0); + const amrex::Real y = p.pos(1); + const amrex::Real ux = p.rdata(PIdx::ux); + const amrex::Real uy = p.rdata(PIdx::uy); + const amrex::Real uz = p.rdata(PIdx::uz); + const amrex::Real thetax = std::atan2(ux, uz); + const amrex::Real thetay = std::atan2(uy, uz); + return {w, w*x, w*y, + thetax, w*thetax, thetax, + thetay, w*thetay, thetay}; + }, + reduce_ops); + + amrex::Real w_tot = amrex::get<0>(r); + amrex::Real x_ave = amrex::get<1>(r); + amrex::Real y_ave = amrex::get<2>(r); + amrex::Real thetax_min = amrex::get<3>(r); + amrex::Real thetax_ave = amrex::get<4>(r); + amrex::Real thetax_max = amrex::get<5>(r); + amrex::Real thetay_min = amrex::get<6>(r); + amrex::Real thetay_ave = amrex::get<7>(r); + amrex::Real thetay_max = amrex::get<8>(r); + + amrex::ParallelDescriptor::ReduceRealSum({w_tot, x_ave, y_ave, thetax_ave, thetay_ave}); + amrex::ParallelDescriptor::ReduceRealMin({thetax_min, thetay_min}); + amrex::ParallelDescriptor::ReduceRealMax({thetax_max, thetay_max}); + + // x_std, y_std, thetax_std, thetay_std + amrex::Real x_std = 0.0_rt; + amrex::Real y_std = 0.0_rt; + amrex::Real thetax_std = 0.0_rt; + amrex::Real thetay_std = 0.0_rt; + + if (w_tot > 0.0_rt) + { + x_ave = x_ave / w_tot; + y_ave = y_ave / w_tot; + thetax_ave = thetax_ave / w_tot; + thetay_ave = thetay_ave / w_tot; + + amrex::ReduceOps reduce_ops_std; + auto r_std = amrex::ParticleReduce>( + myspc, + [=] AMREX_GPU_DEVICE(const PType& p) noexcept -> amrex::GpuTuple + { + const amrex::Real w = p.rdata(PIdx::w); + const amrex::Real x = p.pos(0); + const amrex::Real ux = p.rdata(PIdx::ux); + const amrex::Real y = p.pos(1); + const amrex::Real uy = p.rdata(PIdx::uy); + const amrex::Real uz = p.rdata(PIdx::uz); + const amrex::Real thetax = std::atan2(ux, uz); + const amrex::Real thetay = std::atan2(uy, uz); + const amrex::Real tmp1 = (x - x_ave)*(x - x_ave)*w; + const amrex::Real tmp2 = (y - y_ave)*(y - y_ave)*w; + const amrex::Real tmp3 = (thetax - thetax_ave)*(thetax - thetax_ave)*w; + const amrex::Real tmp4 = (thetay - thetay_ave)*(thetay - thetay_ave)*w; + return {tmp1, tmp2, tmp3, tmp4}; + }, + reduce_ops_std); + + x_std = amrex::get<0>(r_std); + y_std = amrex::get<1>(r_std); + thetax_std = amrex::get<2>(r_std); + thetay_std = amrex::get<3>(r_std); + + amrex::ParallelDescriptor::ReduceRealSum({x_std, y_std, thetax_std, thetay_std}); + + x_std = std::sqrt(x_std / w_tot); + y_std = std::sqrt(y_std / w_tot); + thetax_std = std::sqrt(thetax_std / w_tot); + thetay_std = std::sqrt(thetay_std / w_tot); + } + + m_data[get_idx("x_ave_"+m_beam_name[i_s])] = x_ave; + m_data[get_idx("x_std_"+m_beam_name[i_s])] = x_std; + m_data[get_idx("y_ave_"+m_beam_name[i_s])] = y_ave; + m_data[get_idx("y_std_"+m_beam_name[i_s])] = y_std; + m_data[get_idx("thetax_min_"+m_beam_name[i_s])] = thetax_min; + m_data[get_idx("thetax_ave_"+m_beam_name[i_s])] = thetax_ave; + m_data[get_idx("thetax_max_"+m_beam_name[i_s])] = thetax_max; + m_data[get_idx("thetax_std_"+m_beam_name[i_s])] = thetax_std; + m_data[get_idx("thetay_min_"+m_beam_name[i_s])] = thetay_min; + m_data[get_idx("thetay_ave_"+m_beam_name[i_s])] = thetay_ave; + m_data[get_idx("thetay_max_"+m_beam_name[i_s])] = thetay_max; + m_data[get_idx("thetay_std_"+m_beam_name[i_s])] = thetay_std; +#endif + +#if (defined WARPX_QED) + // get mass + amrex::ParticleReal m = myspc.getMass(); + const bool is_photon = myspc.AmIA(); + if (is_photon) { + m = PhysConst::m_e; + } + + // compute chimin, chiave and chimax + amrex::Real chimin_f = 0.0_rt; + amrex::Real chimax_f = 0.0_rt; + amrex::Real chiave_f = 0.0_rt; + + if (myspc.DoQED()) + { + // define variables in preparation for field gatheeduce_data.value()ring + const int n_rz_azimuthal_modes = WarpX::n_rz_azimuthal_modes; + const int nox = WarpX::nox; + const bool galerkin_interpolation = WarpX::galerkin_interpolation; + const amrex::IntVect ngEB = warpx.getngEB(); + + // TODO loop over refinement levels: for (int lev = 0; lev <= level_number; ++lev) + const int lev = 0; + + // define variables in preparation for field gathering + const std::array& dx = WarpX::CellSize(std::max(lev, 0)); + const amrex::GpuArray dx_arr = {dx[0], dx[1], dx[2]}; + const amrex::MultiFab & Ex = warpx.getEfield(lev,0); + const amrex::MultiFab & Ey = warpx.getEfield(lev,1); + const amrex::MultiFab & Ez = warpx.getEfield(lev,2); + const amrex::MultiFab & Bx = warpx.getBfield(lev,0); + const amrex::MultiFab & By = warpx.getBfield(lev,1); + const amrex::MultiFab & Bz = warpx.getBfield(lev,2); + + // declare reduce_op + ReduceOps reduce_op; + ReduceData reduce_data(reduce_op); + using ReduceTuple = typename decltype(reduce_data)::Type; + + // Loop over boxes + for (WarpXParIter pti(myspc, lev); pti.isValid(); ++pti) + { + const auto GetPosition = GetParticlePosition(pti); + // get particle arrays + amrex::ParticleReal* const AMREX_RESTRICT ux = pti.GetAttribs()[PIdx::ux].dataPtr(); + amrex::ParticleReal* const AMREX_RESTRICT uy = pti.GetAttribs()[PIdx::uy].dataPtr(); + amrex::ParticleReal* const AMREX_RESTRICT uz = pti.GetAttribs()[PIdx::uz].dataPtr(); + amrex::ParticleReal* const AMREX_RESTRICT w = pti.GetAttribs()[PIdx::w].dataPtr(); + // declare external fields + const int offset = 0; + const auto getExternalEB = GetExternalEBField(pti, offset); + // define variables in preparation for field gathering + amrex::Box box = pti.tilebox(); + box.grow(ngEB); + const amrex::Dim3 lo = amrex::lbound(box); + const std::array& xyzmin = WarpX::LowerCorner(box, lev, 0._rt); + const amrex::GpuArray xyzmin_arr = {xyzmin[0], xyzmin[1], xyzmin[2]}; + const amrex::Array4 & ex_arr = Ex[pti].array(); + const amrex::Array4 & ey_arr = Ey[pti].array(); + const amrex::Array4 & ez_arr = Ez[pti].array(); + const amrex::Array4 & bx_arr = Bx[pti].array(); + const amrex::Array4 & by_arr = By[pti].array(); + const amrex::Array4 & bz_arr = Bz[pti].array(); + const amrex::IndexType ex_type = Ex[pti].box().ixType(); + const amrex::IndexType ey_type = Ey[pti].box().ixType(); + const amrex::IndexType ez_type = Ez[pti].box().ixType(); + const amrex::IndexType bx_type = Bx[pti].box().ixType(); + const amrex::IndexType by_type = By[pti].box().ixType(); + const amrex::IndexType bz_type = Bz[pti].box().ixType(); + + // evaluate reduce_op + reduce_op.eval(pti.numParticles(), reduce_data, + [=] AMREX_GPU_DEVICE (int i) -> ReduceTuple + { + // get external fields + amrex::ParticleReal xp, yp, zp; + GetPosition(i, xp, yp, zp); + amrex::ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; + amrex::ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; + getExternalEB(i, ex, ey, ez, bx, by, bz); + + // gather E and B + doGatherShapeN(xp, yp, zp, + ex, ey, ez, bx, by, bz, + ex_arr, ey_arr, ez_arr, bx_arr, by_arr, bz_arr, + ex_type, ey_type, ez_type, + bx_type, by_type, bz_type, + dx_arr, xyzmin_arr, lo, + n_rz_azimuthal_modes, nox, galerkin_interpolation); + // compute chi + amrex::Real chi = 0.0_rt; + if (is_photon) { + chi = QedUtils::chi_photon(ux[i]*m, uy[i]*m, uz[i]*m, + ex, ey, ez, bx, by, bz); + } else { + chi = QedUtils::chi_ele_pos(ux[i]*m, uy[i]*m, uz[i]*m, + ex, ey, ez, bx, by, bz); + } + return {chi, chi, chi*w[i]}; + }); + } + auto val = reduce_data.value(); + chimin_f = get<0>(val); + chimax_f = get<1>(val); + chiave_f = get<2>(val); + amrex::ParallelDescriptor::ReduceRealMin(chimin_f); + amrex::ParallelDescriptor::ReduceRealMax(chimax_f); + amrex::ParallelDescriptor::ReduceRealSum(chiave_f); + + m_data[get_idx("chimin_"+m_beam_name[i_s])] = chimin_f; + m_data[get_idx("chiave_"+m_beam_name[i_s])] = chiave_f/w_tot; + m_data[get_idx("chimax_"+m_beam_name[i_s])] = chimax_f; + } +#endif + } // end loop over species + + // make density MultiFabs from nodal to cell centered + amrex::BoxArray ba = warpx.boxArray(0); + amrex::DistributionMapping dmap = warpx.DistributionMap(0); + constexpr int ncomp = 1; + constexpr int ngrow = 0; + amrex::MultiFab mf_dst1(ba.convert(amrex::IntVect::TheCellVector()), dmap, ncomp, ngrow); + amrex::MultiFab mf_dst2(ba.convert(amrex::IntVect::TheCellVector()), dmap, ncomp, ngrow); + ablastr::coarsen::sample::Coarsen(mf_dst1, *num_dens[0], 0, 0, ncomp, ngrow); + ablastr::coarsen::sample::Coarsen(mf_dst2, *num_dens[1], 0, 0, ncomp, ngrow); + + // compute luminosity + amrex::Real const n1_dot_n2 = amrex::MultiFab::Dot(mf_dst1, 0, mf_dst2, 0, 1, 0); + amrex::Real const lumi = 2. * PhysConst::c * n1_dot_n2 * dV; + m_data[get_idx("dL_dt")] = lumi; +#endif // not RZ +} diff --git a/Source/Diagnostics/ReducedDiags/Make.package b/Source/Diagnostics/ReducedDiags/Make.package index f265e302d3a..a1f08a24da3 100644 --- a/Source/Diagnostics/ReducedDiags/Make.package +++ b/Source/Diagnostics/ReducedDiags/Make.package @@ -7,6 +7,7 @@ CEXE_sources += FieldProbe.cpp CEXE_sources += FieldProbeParticleContainer.cpp CEXE_sources += FieldMomentum.cpp CEXE_sources += BeamRelevant.cpp +CEXE_sources += ColliderRelevant.cpp CEXE_sources += LoadBalanceCosts.cpp CEXE_sources += LoadBalanceEfficiency.cpp CEXE_sources += ParticleHistogram.cpp diff --git a/Source/Diagnostics/ReducedDiags/MultiReducedDiags.cpp b/Source/Diagnostics/ReducedDiags/MultiReducedDiags.cpp index d895c4caa23..73e63aeb4d3 100644 --- a/Source/Diagnostics/ReducedDiags/MultiReducedDiags.cpp +++ b/Source/Diagnostics/ReducedDiags/MultiReducedDiags.cpp @@ -8,6 +8,7 @@ #include "BeamRelevant.H" #include "ChargeOnEB.H" +#include "ColliderRelevant.H" #include "FieldEnergy.H" #include "FieldMaximum.H" #include "FieldProbe.H" @@ -59,6 +60,7 @@ MultiReducedDiags::MultiReducedDiags () {"FieldReduction", [](CS s){return std::make_unique(s);}}, {"RhoMaximum", [](CS s){return std::make_unique(s);}}, {"BeamRelevant", [](CS s){return std::make_unique(s);}}, + {"ColliderRelevant", [](CS s){return std::make_unique(s);}}, {"LoadBalanceCosts", [](CS s){return std::make_unique(s);}}, {"LoadBalanceEfficiency", [](CS s){return std::make_unique(s);}}, {"ParticleHistogram", [](CS s){return std::make_unique(s);}}, diff --git a/Source/Diagnostics/ReducedDiags/ReducedDiags.cpp b/Source/Diagnostics/ReducedDiags/ReducedDiags.cpp index 77377c7d5ed..758b48b7262 100644 --- a/Source/Diagnostics/ReducedDiags/ReducedDiags.cpp +++ b/Source/Diagnostics/ReducedDiags/ReducedDiags.cpp @@ -24,9 +24,8 @@ using namespace amrex; // constructor ReducedDiags::ReducedDiags (std::string rd_name) + : m_rd_name(std::move(rd_name)) { - m_rd_name = rd_name; - BackwardCompatibility(); const ParmParse pp_rd_name(m_rd_name); diff --git a/Tools/Algorithms/stencil.py b/Tools/Algorithms/stencil.py index cb24a62148b..63bb7f11c2e 100644 --- a/Tools/Algorithms/stencil.py +++ b/Tools/Algorithms/stencil.py @@ -13,7 +13,9 @@ import argparse import os +import sys +sys.path.append('../Parser/') from input_file_parser import parse_input_file import matplotlib.pyplot as plt import numpy as np diff --git a/Tools/Parser/__init__.py b/Tools/Parser/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/Tools/Algorithms/input_file_parser.py b/Tools/Parser/input_file_parser.py similarity index 100% rename from Tools/Algorithms/input_file_parser.py rename to Tools/Parser/input_file_parser.py From 7b4cb7321d8379844597a0ddb560cebf079d23f2 Mon Sep 17 00:00:00 2001 From: David Grote Date: Mon, 11 Sep 2023 18:04:47 -0700 Subject: [PATCH 012/110] PICMI: Add `warpx_intervals` option to BTD (#4288) * Add warpx_intervals option to BTD * Add warpx_intervals to LabFrameParticleDiagnostics --- Python/pywarpx/picmi.py | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index dbdebf452d1..ce20724950d 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -2249,6 +2249,10 @@ class LabFrameFieldDiagnostic(picmistandard.PICMI_LabFrameFieldDiagnostic, warpx_file_prefix: string, optional Passed to .file_prefix + warpx_intervals: integer or string + Selects the snapshots to be made, instead of using "num_snapshots" which + makes all snapshots. "num_snapshots" is ignored. + warpx_file_min_digits: integer, optional Passed to .file_min_digits @@ -2268,6 +2272,7 @@ def init(self, kw): self.format = kw.pop('warpx_format', None) self.openpmd_backend = kw.pop('warpx_openpmd_backend', None) self.file_prefix = kw.pop('warpx_file_prefix', None) + self.intervals = kw.pop('warpx_intervals', None) self.file_min_digits = kw.pop('warpx_file_min_digits', None) self.buffer_size = kw.pop('warpx_buffer_size', None) self.lower_bound = kw.pop('warpx_lower_bound', None) @@ -2286,10 +2291,15 @@ def initialize_inputs(self): self.diagnostic.diag_hi = self.upper_bound self.diagnostic.do_back_transformed_fields = 1 - self.diagnostic.num_snapshots_lab = self.num_snapshots self.diagnostic.dt_snapshots_lab = self.dt_snapshots self.diagnostic.buffer_size = self.buffer_size + # intervals and num_snapshots_lab cannot both be set + if self.intervals is not None: + self.diagnostic.intervals = self.intervals + else: + self.diagnostic.num_snapshots_lab = self.num_snapshots + self.diagnostic.do_back_transformed_particles = self.write_species # --- Use a set to ensure that fields don't get repeated. @@ -2352,6 +2362,10 @@ class LabFrameParticleDiagnostic(picmistandard.PICMI_LabFrameParticleDiagnostic, warpx_file_prefix: string, optional Passed to .file_prefix + warpx_intervals: integer or string + Selects the snapshots to be made, instead of using "num_snapshots" which + makes all snapshots. "num_snapshots" is ignored. + warpx_file_min_digits: integer, optional Passed to .file_min_digits @@ -2365,6 +2379,7 @@ def init(self, kw): self.format = kw.pop('warpx_format', None) self.openpmd_backend = kw.pop('warpx_openpmd_backend', None) self.file_prefix = kw.pop('warpx_file_prefix', None) + self.intervals = kw.pop('warpx_intervals', None) self.file_min_digits = kw.pop('warpx_file_min_digits', None) self.buffer_size = kw.pop('warpx_buffer_size', None) self.write_fields = kw.pop('warpx_write_fields', None) @@ -2379,10 +2394,15 @@ def initialize_inputs(self): self.diagnostic.file_min_digits = self.file_min_digits self.diagnostic.do_back_transformed_particles = 1 - self.diagnostic.num_snapshots_lab = self.num_snapshots self.diagnostic.dt_snapshots_lab = self.dt_snapshots self.diagnostic.buffer_size = self.buffer_size + # intervals and num_snapshots_lab cannot both be set + if self.intervals is not None: + self.diagnostic.intervals = self.intervals + else: + self.diagnostic.num_snapshots_lab = self.num_snapshots + self.diagnostic.do_back_transformed_fields = self.write_fields self.set_write_dir() From 12a695d9fa518ad23a4f0375c0ccbafcabb80473 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 12 Sep 2023 11:34:26 -0700 Subject: [PATCH 013/110] AMReX/PICSAR: Weekly Update (#4293) * AMReX: Weekly Update * PICSAR: Use Tag --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- cmake/dependencies/PICSAR.cmake | 2 +- run_test.sh | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 17b8c259f0a..88037297d35 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 23.09 && cd - + cd ../amrex && git checkout --detach b98bdae9fb67e5d9aafc488de92c53001bd323ec && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index c40dac874df..0e033f186b3 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 23.09 +branch = b98bdae9fb67e5d9aafc488de92c53001bd323ec [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index ab390fcb01b..48364535760 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 23.09 +branch = b98bdae9fb67e5d9aafc488de92c53001bd323ec [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 1e2697a4e50..8d873655fe0 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -257,7 +257,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "23.09" +set(WarpX_amrex_branch "b98bdae9fb67e5d9aafc488de92c53001bd323ec" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/cmake/dependencies/PICSAR.cmake b/cmake/dependencies/PICSAR.cmake index d04269b8876..8400dc06a4b 100644 --- a/cmake/dependencies/PICSAR.cmake +++ b/cmake/dependencies/PICSAR.cmake @@ -106,7 +106,7 @@ if(WarpX_QED) set(WarpX_picsar_repo "https://github.com/ECP-WarpX/picsar.git" CACHE STRING "Repository URI to pull and build PICSAR from if(WarpX_picsar_internal)") - set(WarpX_picsar_branch "aa54e985398c1d575abc7e6737cdbc660a13765f" + set(WarpX_picsar_branch "23.09" CACHE STRING "Repository branch for WarpX_picsar_repo if(WarpX_picsar_internal)") diff --git a/run_test.sh b/run_test.sh index faf4bdd85c4..37b5d370667 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 23.09 && cd - +cd amrex && git checkout --detach b98bdae9fb67e5d9aafc488de92c53001bd323ec && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 51f908bf15e737ca4ac0cc2871de2d2de77e43a7 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 12 Sep 2023 16:14:17 -0700 Subject: [PATCH 014/110] Doc: cupy on Perlmutter (NERSC) (#4289) Document & automate how to install `cupy` on Perlmutter. --- Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh b/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh index a60c3ba7594..7bcc7053974 100755 --- a/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh +++ b/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh @@ -127,6 +127,7 @@ python3 -m pip install --upgrade matplotlib python3 -m pip install --upgrade yt # install or update WarpX dependencies such as picmistandard python3 -m pip install --upgrade -r $HOME/src/warpx/requirements.txt +python3 -m pip install cupy-cuda11x # CUDA 11.7 compatible wheel # optional: for libEnsemble python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt # optional: for optimas (based on libEnsemble & ax->botorch->gpytorch->pytorch) From 15dacea8160503a0bf297063545b2b834a7f4b77 Mon Sep 17 00:00:00 2001 From: Marco Garten Date: Tue, 12 Sep 2023 19:03:00 -0700 Subject: [PATCH 015/110] Revert PR4266 - lower and uppers bounds (#4294) The lower and upper bounds are already present within PICMI. --- Python/pywarpx/picmi.py | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index ce20724950d..056026fbb8f 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -1963,14 +1963,6 @@ class FieldDiagnostic(picmistandard.PICMI_FieldDiagnostic, WarpXDiagnosticBase): warpx_dump_rz_modes: bool, optional Flag whether to dump the data for all RZ modes - - warpx_lower_bound: vector of floats, optional - Lower corner of output fields - that is passed to .lower_bound - - warpx_upper_bound: vector of floats, optional - Upper corner of output fields - that is passed to .upper_bound """ def init(self, kw): @@ -1984,8 +1976,6 @@ def init(self, kw): self.file_prefix = kw.pop('warpx_file_prefix', None) self.file_min_digits = kw.pop('warpx_file_min_digits', None) self.dump_rz_modes = kw.pop('warpx_dump_rz_modes', None) - self.lower_bound = kw.pop('warpx_lower_bound', None) - self.upper_bound = kw.pop('warpx_upper_bound', None) def initialize_inputs(self): From 5f40a2b5304d2a01c3c5676faca8dc5b9714ed4f Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Wed, 13 Sep 2023 09:12:58 -0700 Subject: [PATCH 016/110] Add flag to set GPU-aware MPI from PICMI (#4298) --- Python/pywarpx/picmi.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index 056026fbb8f..ee4422daca6 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -1680,6 +1680,9 @@ class Simulation(picmistandard.PICMI_Simulation): warpx_amrex_the_arena_init_size: long int, optional The amount of memory in bytes to allocate in the Arena. + warpx_amrex_use_gpu_aware_mpi: bool, optional + Whether to use GPU-aware MPI communications + warpx_zmax_plasma_to_compute_max_step: float, optional Sets the simulation run time based on the maximum z value @@ -1739,6 +1742,7 @@ def init(self, kw): self.amr_restart = kw.pop('warpx_amr_restart', None) self.amrex_the_arena_is_managed = kw.pop('warpx_amrex_the_arena_is_managed', None) self.amrex_the_arena_init_size = kw.pop('warpx_amrex_the_arena_init_size', None) + self.amrex_use_gpu_aware_mpi = kw.pop('warpx_amrex_use_gpu_aware_mpi', None) self.zmax_plasma_to_compute_max_step = kw.pop('warpx_zmax_plasma_to_compute_max_step', None) self.compute_max_step_from_btd = kw.pop('warpx_compute_max_step_from_btd', None) @@ -1869,6 +1873,9 @@ def initialize_inputs(self): if self.amrex_the_arena_init_size is not None: pywarpx.amrex.the_arena_init_size = self.amrex_the_arena_init_size + if self.amrex_use_gpu_aware_mpi is not None: + pywarpx.amrex.use_gpu_aware_mpi = self.amrex_use_gpu_aware_mpi + def initialize_warpx(self, mpi_comm=None): if self.warpx_initialized: return From 6c4436e1f9ec0a2fcd40cdf2692beea14074e459 Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Thu, 14 Sep 2023 08:38:12 +0900 Subject: [PATCH 017/110] Clang tidy CI test: add misc-definitions-in-headers check (#4253) * Clang-tidy: add misc-definitions-in-headers check * address issues found with clang-tidy * remove std::cout added for debug purposes * add back newline --- .clang-tidy | 4 +- .../SpectralHankelTransform/BesselRoots.H | 119 +------------- .../SpectralHankelTransform/BesselRoots.cpp | 153 ++++++++++++++++++ .../SpectralHankelTransform/CMakeLists.txt | 1 + .../SpectralHankelTransform/Make.package | 1 + Source/Initialization/InjectorMomentum.H | 1 + Source/Particles/Sorting/CMakeLists.txt | 1 + Source/Particles/Sorting/Make.package | 2 + Source/Particles/Sorting/SortingUtils.H | 13 +- Source/Particles/Sorting/SortingUtils.cpp | 22 +++ 10 files changed, 190 insertions(+), 127 deletions(-) create mode 100644 Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp create mode 100644 Source/Particles/Sorting/SortingUtils.cpp diff --git a/.clang-tidy b/.clang-tidy index 2d495081c8d..5a3deadf304 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -18,6 +18,7 @@ Checks: '-*, google-build-namespaces, google-global-names-in-headers, misc-const-correctness, + misc-definitions-in-headers, misc-misleading-bidirectional, misc-misleading-identifier, misc-misplaced-const, @@ -25,7 +26,6 @@ Checks: '-*, misc-unused-alias-decls, misc-unused-parameters, misc-unused-using-decls, - -misc-definitions-in-headers, modernize-avoid-bind, modernize-concat-nested-namespaces, modernize-deprecated-headers, @@ -82,5 +82,7 @@ Checks: '-*, CheckOptions: - key: modernize-pass-by-value.ValuesOnly value: 'true' +- key: misc-definitions-in-headers.HeaderFileExtensions + value: "H," HeaderFilterRegex: 'Source[a-z_A-Z0-9\/]+\.H$' diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.H b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.H index 64ffc80ac37..2285ab05679 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.H +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.H @@ -32,14 +32,11 @@ ! (www.jpmoreau.fr) ! ------------------------------------------------------------------------ */ -#include "Utils/WarpXConst.H" +#ifndef WARPX_BESSEL_ROOTS_H_ +#define WARPX_BESSEL_ROOTS_H_ #include - -#include - - -void SecantRootFinder(int n, int nitmx, amrex::Real tol, amrex::Real *zeroj, int *ier); +#include /*---------------------------------------------------------------------- * calculate the first nk zeroes of bessel function j(n, x) @@ -56,112 +53,6 @@ void SecantRootFinder(int n, int nitmx, amrex::Real tol, amrex::Real *zeroj, int * abramowitz m. & stegun irene a. * handbook of mathematical functions */ -void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vector &ier) { - using namespace amrex::literals; - - amrex::Real zeroj; - int ierror, ik, k; - - const amrex::Real tol = 1e-14_rt; - const amrex::Real nitmx = 10; - - const amrex::Real c1 = 1.8557571_rt; - const amrex::Real c2 = 1.033150_rt; - const amrex::Real c3 = 0.00397_rt; - const amrex::Real c4 = 0.0908_rt; - const amrex::Real c5 = 0.043_rt; - - const amrex::Real t0 = 4.0_rt*n*n; - const amrex::Real t1 = t0 - 1.0_rt; - const amrex::Real t3 = 4.0_rt*t1*(7.0_rt*t0 - 31.0_rt); - const amrex::Real t5 = 32.0_rt*t1*((83.0_rt*t0 - 982.0_rt)*t0 + 3779.0_rt); - const amrex::Real t7 = 64.0_rt*t1*(((6949.0_rt*t0 - 153855.0_rt)*t0 + 1585743.0_rt)*t0 - 6277237.0_rt); - - roots.resize(nk); - ier.resize(nk); - - // first zero - if (n == 0) { - zeroj = c1 + c2 - c3 - c4 + c5; - SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); - ier[0] = ierror; - roots[0] = zeroj; - ik = 1; - } - else { - // Include the trivial root - ier[0] = 0; - roots[0] = 0.; - const amrex::Real f1 = std::pow(n, (1.0_rt/3.0_rt)); - const amrex::Real f2 = f1*f1*n; - const amrex::Real f3 = f1*n*n; - zeroj = n + c1*f1 + (c2/f1) - (c3/n) - (c4/f2) + (c5/f3); - SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); - ier[1] = ierror; - roots[1] = zeroj; - ik = 2; - } - - // other zeroes - // k counts the nontrivial roots - // ik counts all roots - k = 2; - while (ik < nk) { - - // mac mahon's series for k >> n - const amrex::Real b0 = (k + 0.5_rt*n - 0.25_rt)*MathConst::pi; - const amrex::Real b1 = 8.0_rt*b0; - const amrex::Real b2 = b1*b1; - const amrex::Real b3 = 3.0_rt*b1*b2; - const amrex::Real b5 = 5.0_rt*b3*b2; - const amrex::Real b7 = 7.0_rt*b5*b2; - - zeroj = b0 - (t1/b1) - (t3/b3) - (t5/b5) - (t7/b7); - - const amrex::Real errj = std::abs(jn(n, zeroj)); - - // improve solution using procedure SecantRootFinder - if (errj > tol) SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); - - roots[ik] = zeroj; - ier[ik] = ierror; - - k++; - ik++; - } -} - -void SecantRootFinder(int n, int nitmx, amrex::Real tol, amrex::Real *zeroj, int *ier) { - using namespace amrex::literals; - - amrex::Real p0, p1, q0, q1, dp, p; - amrex::Real c[2]; - - c[0] = 0.95_rt; - c[1] = 0.999_rt; - *ier = 0; - - p = *zeroj; - for (int ntry=0 ; ntry <= 1 ; ntry++) { - p0 = c[ntry]*(*zeroj); +void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vector &ier); - p1 = *zeroj; - q0 = jn(n, p0); - q1 = jn(n, p1); - for (int it=1; it <= nitmx; it++) { - if (q1 == q0) break; - p = p1 - q1*(p1 - p0)/(q1 - q0); - dp = p - p1; - if (it > 1 && std::abs(dp) < tol) { - *zeroj = p; - return; - } - p0 = p1; - q0 = q1; - p1 = p; - q1 = jn(n, p1); - } - } - *ier = 3; - *zeroj = p; -} +#endif // WARPX_BESSEL_ROOTS_H_ diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp new file mode 100644 index 00000000000..210a4baffc7 --- /dev/null +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp @@ -0,0 +1,153 @@ +/* Copyright 2019 David Grote + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +/* ------------------------------------------------------------------------- +! program to calculate the first zeroes (root abscissas) of the first +! kind bessel function of integer order n using the subroutine rootj. +! -------------------------------------------------------------------------- +! sample run: +! +! (calculate the first 10 zeroes of 1st kind bessel function of order 2). +! +! zeroes of bessel function of order: 2 +! +! number of calculated zeroes: 10 +! +! table of root abcissas (5 items per line) +! 5.135622 8.417244 11.619841 14.795952 17.959819 + 21.116997 24.270112 27.420574 30.569204 33.716520 +! +! table of error codes (5 items per line) +! 0 0 0 0 0 +! 0 0 0 0 0 +! +! -------------------------------------------------------------------------- +! reference: from numath library by tuan dang trong in fortran 77 +! [bibli 18]. +! +! c++ release 1.0 by j-p moreau, paris +! (www.jpmoreau.fr) +! ------------------------------------------------------------------------ */ + +#include "BesselRoots.H" + +#include "Utils/WarpXConst.H" + +#include + +namespace{ + + void SecantRootFinder(int n, int nitmx, amrex::Real tol, amrex::Real *zeroj, int *ier) { + using namespace amrex::literals; + + amrex::Real p0, p1, q0, q1, dp, p; + amrex::Real c[2]; + + c[0] = 0.95_rt; + c[1] = 0.999_rt; + *ier = 0; + + p = *zeroj; + for (int ntry=0 ; ntry <= 1 ; ntry++) { + p0 = c[ntry]*(*zeroj); + + p1 = *zeroj; + q0 = jn(n, p0); + q1 = jn(n, p1); + for (int it=1; it <= nitmx; it++) { + if (q1 == q0) break; + p = p1 - q1*(p1 - p0)/(q1 - q0); + dp = p - p1; + if (it > 1 && std::abs(dp) < tol) { + *zeroj = p; + return; + } + p0 = p1; + q0 = q1; + p1 = p; + q1 = jn(n, p1); + } + } + *ier = 3; + *zeroj = p; + } + +} + +void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vector &ier) { + using namespace amrex::literals; + + amrex::Real zeroj; + int ierror, ik, k; + + const amrex::Real tol = 1e-14_rt; + const amrex::Real nitmx = 10; + + const amrex::Real c1 = 1.8557571_rt; + const amrex::Real c2 = 1.033150_rt; + const amrex::Real c3 = 0.00397_rt; + const amrex::Real c4 = 0.0908_rt; + const amrex::Real c5 = 0.043_rt; + + const amrex::Real t0 = 4.0_rt*n*n; + const amrex::Real t1 = t0 - 1.0_rt; + const amrex::Real t3 = 4.0_rt*t1*(7.0_rt*t0 - 31.0_rt); + const amrex::Real t5 = 32.0_rt*t1*((83.0_rt*t0 - 982.0_rt)*t0 + 3779.0_rt); + const amrex::Real t7 = 64.0_rt*t1*(((6949.0_rt*t0 - 153855.0_rt)*t0 + 1585743.0_rt)*t0 - 6277237.0_rt); + + roots.resize(nk); + ier.resize(nk); + + // first zero + if (n == 0) { + zeroj = c1 + c2 - c3 - c4 + c5; + ::SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); + ier[0] = ierror; + roots[0] = zeroj; + ik = 1; + } + else { + // Include the trivial root + ier[0] = 0; + roots[0] = 0.; + const amrex::Real f1 = std::pow(n, (1.0_rt/3.0_rt)); + const amrex::Real f2 = f1*f1*n; + const amrex::Real f3 = f1*n*n; + zeroj = n + c1*f1 + (c2/f1) - (c3/n) - (c4/f2) + (c5/f3); + ::SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); + ier[1] = ierror; + roots[1] = zeroj; + ik = 2; + } + + // other zeroes + // k counts the nontrivial roots + // ik counts all roots + k = 2; + while (ik < nk) { + + // mac mahon's series for k >> n + const amrex::Real b0 = (k + 0.5_rt*n - 0.25_rt)*MathConst::pi; + const amrex::Real b1 = 8.0_rt*b0; + const amrex::Real b2 = b1*b1; + const amrex::Real b3 = 3.0_rt*b1*b2; + const amrex::Real b5 = 5.0_rt*b3*b2; + const amrex::Real b7 = 7.0_rt*b5*b2; + + zeroj = b0 - (t1/b1) - (t3/b3) - (t5/b5) - (t7/b7); + + const amrex::Real errj = std::abs(jn(n, zeroj)); + + // improve solution using procedure SecantRootFinder + if (errj > tol) ::SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); + + roots[ik] = zeroj; + ier[ik] = ierror; + + k++; + ik++; + } +} diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/CMakeLists.txt b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/CMakeLists.txt index a3d2b60bcc4..70c9740367f 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/CMakeLists.txt +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/CMakeLists.txt @@ -1,5 +1,6 @@ target_sources(lib_rz PRIVATE + BesselRoots.cpp SpectralHankelTransformer.cpp HankelTransform.cpp ) diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/Make.package b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/Make.package index 8bb1d7ef7b4..37ca9a93186 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/Make.package +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/Make.package @@ -1,3 +1,4 @@ +CEXE_sources += BesselRoots.cpp CEXE_sources += SpectralHankelTransformer.cpp CEXE_sources += HankelTransform.cpp diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index 87b81381ce9..b0afafa730b 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -94,6 +94,7 @@ namespace { * @param u_th Momentum spread * @param engine Object used to generate random numbers */ + AMREX_FORCE_INLINE AMREX_GPU_HOST_DEVICE amrex::Real generateGaussianFluxDist( amrex::Real u_m, amrex::Real u_th, amrex::RandomEngine const& engine ) { diff --git a/Source/Particles/Sorting/CMakeLists.txt b/Source/Particles/Sorting/CMakeLists.txt index d3c378e43b8..60a156f4649 100644 --- a/Source/Particles/Sorting/CMakeLists.txt +++ b/Source/Particles/Sorting/CMakeLists.txt @@ -3,5 +3,6 @@ foreach(D IN LISTS WarpX_DIMS) target_sources(lib_${SD} PRIVATE Partition.cpp + SortingUtils.cpp ) endforeach() diff --git a/Source/Particles/Sorting/Make.package b/Source/Particles/Sorting/Make.package index 16efc02b89e..e6ad1604dc7 100644 --- a/Source/Particles/Sorting/Make.package +++ b/Source/Particles/Sorting/Make.package @@ -1,2 +1,4 @@ CEXE_sources += Partition.cpp +CEXE_sources += SortingUtils.cpp + VPATH_LOCATIONS += $(WARPX_HOME)/Source/Particles/Sorting diff --git a/Source/Particles/Sorting/SortingUtils.H b/Source/Particles/Sorting/SortingUtils.H index 0bec92d6efc..dd53ad6f610 100644 --- a/Source/Particles/Sorting/SortingUtils.H +++ b/Source/Particles/Sorting/SortingUtils.H @@ -19,18 +19,7 @@ * * \param[inout] v Vector of integers, to be filled by this routine */ -void fillWithConsecutiveIntegers( amrex::Gpu::DeviceVector& v ) -{ -#ifdef AMREX_USE_GPU - // On GPU: Use amrex - auto data = v.data(); - auto N = v.size(); - AMREX_FOR_1D( N, i, {data[i] = i;}); -#else - // On CPU: Use std library - std::iota( v.begin(), v.end(), 0L ); -#endif -} +void fillWithConsecutiveIntegers( amrex::Gpu::DeviceVector& v ); /** \brief Find the indices that would reorder the elements of `predicate` * so that the elements with non-zero value precede the other elements diff --git a/Source/Particles/Sorting/SortingUtils.cpp b/Source/Particles/Sorting/SortingUtils.cpp new file mode 100644 index 00000000000..699119e8e18 --- /dev/null +++ b/Source/Particles/Sorting/SortingUtils.cpp @@ -0,0 +1,22 @@ +/* Copyright 2019-2020 Andrew Myers, Maxence Thevenet, Remi Lehe + * Weiqun Zhang + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ + +#include "SortingUtils.H" + +void fillWithConsecutiveIntegers( amrex::Gpu::DeviceVector& v ) +{ +#ifdef AMREX_USE_GPU + // On GPU: Use amrex + auto data = v.data(); + auto N = v.size(); + AMREX_FOR_1D( N, i, {data[i] = i;}); +#else + // On CPU: Use std library + std::iota( v.begin(), v.end(), 0L ); +#endif +} From ec82914c6753cb3fbb9e20f8c66fc59ae115c080 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 15 Sep 2023 08:18:38 -0700 Subject: [PATCH 018/110] Conda: `make` (#4292) Useful on very fresh Ubuntu/WSL/Docker environments where not even `built-essential` was installed. Also available on all operating systems now, so no downside to list: https://anaconda.org/conda-forge/make/ --- Docs/source/install/dependencies.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Docs/source/install/dependencies.rst b/Docs/source/install/dependencies.rst index a198712cc01..6a8b2093fa3 100644 --- a/Docs/source/install/dependencies.rst +++ b/Docs/source/install/dependencies.rst @@ -75,7 +75,7 @@ Conda (Linux/macOS/Windows) .. code-block:: bash - conda create -n warpx-cpu-mpich-dev -c conda-forge blaspp boost ccache cmake compilers git lapackpp "openpmd-api=*=mpi_mpich*" python numpy pandas scipy yt "fftw=*=mpi_mpich*" pkg-config matplotlib mamba mpich mpi4py ninja pip virtualenv + conda create -n warpx-cpu-mpich-dev -c conda-forge blaspp boost ccache cmake compilers git lapackpp "openpmd-api=*=mpi_mpich*" python make numpy pandas scipy yt "fftw=*=mpi_mpich*" pkg-config matplotlib mamba mpich mpi4py ninja pip virtualenv conda activate warpx-cpu-mpich-dev # compile WarpX with -DWarpX_MPI=ON @@ -85,7 +85,7 @@ Conda (Linux/macOS/Windows) .. code-block:: bash - conda create -n warpx-cpu-dev -c conda-forge blaspp boost ccache cmake compilers git lapackpp openpmd-api python numpy pandas scipy yt fftw pkg-config matplotlib mamba ninja pip virtualenv + conda create -n warpx-cpu-dev -c conda-forge blaspp boost ccache cmake compilers git lapackpp openpmd-api python make numpy pandas scipy yt fftw pkg-config matplotlib mamba ninja pip virtualenv conda activate warpx-cpu-dev # compile WarpX with -DWarpX_MPI=OFF From 366dfe5b8e4bb9173c2dc66095648305483bc417 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 15 Sep 2023 08:19:33 -0700 Subject: [PATCH 019/110] Doc: Conda CUDA Development (#4290) Document how to install the CUDA Toolkit with Conda for development. --- Docs/source/install/dependencies.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Docs/source/install/dependencies.rst b/Docs/source/install/dependencies.rst index 6a8b2093fa3..7bd6c7dc8d5 100644 --- a/Docs/source/install/dependencies.rst +++ b/Docs/source/install/dependencies.rst @@ -107,6 +107,14 @@ For OpenMP support, you will further need: conda install -c conda-forge llvm-openmp +For Nvidia CUDA GPU support, you will need to have `a recent CUDA driver installed `__ or you can lower the CUDA version of `the Nvidia cuda package `__ and `conda-forge to match your drivers `__ and then add these packages: + +.. code-block:: bash + + conda install -c nvidia -c conda-forge cuda cupy + +More info for `CUDA-enabled ML packages `__. + Spack (Linux/macOS) ------------------- From a0a3db1638ac94896924dc2a75b7e7e07072f9c8 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 18 Sep 2023 10:22:38 -0700 Subject: [PATCH 020/110] AMReX: Weekly Update (#4309) --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- run_test.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 88037297d35..0f3adb8593a 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach b98bdae9fb67e5d9aafc488de92c53001bd323ec && cd - + cd ../amrex && git checkout --detach 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index 0e033f186b3..c20dc5ed93c 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = b98bdae9fb67e5d9aafc488de92c53001bd323ec +branch = 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 48364535760..56139d3da17 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = b98bdae9fb67e5d9aafc488de92c53001bd323ec +branch = 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 8d873655fe0..fb80973c96b 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -257,7 +257,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "b98bdae9fb67e5d9aafc488de92c53001bd323ec" +set(WarpX_amrex_branch "48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/run_test.sh b/run_test.sh index 37b5d370667..e24095ee8cf 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach b98bdae9fb67e5d9aafc488de92c53001bd323ec && cd - +cd amrex && git checkout --detach 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 23a896d3c990eb4b7e86ebd8cb1fad723287e5c6 Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Wed, 20 Sep 2023 00:25:55 -0700 Subject: [PATCH 021/110] Prevent NaNs in Coulomb collision module (#4304) * Prevent NaNs in collisions between particles * Avoid another division by 0 t# Please enter the commit message for your changes. Lines starting --- .../Coulomb/UpdateMomentumPerezElastic.H | 343 +++++++++--------- 1 file changed, 176 insertions(+), 167 deletions(-) diff --git a/Source/Particles/Collision/BinaryCollision/Coulomb/UpdateMomentumPerezElastic.H b/Source/Particles/Collision/BinaryCollision/Coulomb/UpdateMomentumPerezElastic.H index 7c2432f6e1b..3101047e211 100644 --- a/Source/Particles/Collision/BinaryCollision/Coulomb/UpdateMomentumPerezElastic.H +++ b/Source/Particles/Collision/BinaryCollision/Coulomb/UpdateMomentumPerezElastic.H @@ -102,184 +102,193 @@ void UpdateMomentumPerezElastic ( T_PR const g1s = ( T_PR(1.0) - vcDv1*inv_c2 )*gc*g1; T_PR const g2s = ( T_PR(1.0) - vcDv2*inv_c2 )*gc*g2; - // Compute the Coulomb log lnLmd - T_PR lnLmd; - if ( L > T_PR(0.0) ) { lnLmd = L; } - else - { - // Compute b0 according to eq (22) from Perez et al., Phys.Plasmas.19.083104 (2012) - // Note: there is a typo in the equation, the last square is incorrect! - // See the SMILEI documentation: https://smileipic.github.io/Smilei/Understand/collisions.html - // and https://github.com/ECP-WarpX/WarpX/files/3799803/main.pdf from GitHub #429 - T_PR const b0 = amrex::Math::abs(q1*q2) * inv_c2 / - (T_PR(4.0)*MathConst::pi*PhysConst::ep0) * gc/mass_g * - ( m1*g1s*m2*g2s/(p1sm*p1sm*inv_c2) + T_PR(1.0) ); - - // Compute the minimal impact parameter - constexpr T_PR hbar_pi = static_cast(PhysConst::hbar*MathConst::pi); - const T_PR bmin = amrex::max(hbar_pi/p1sm, b0); - - // Compute the Coulomb log lnLmd - lnLmd = amrex::max( T_PR(2.0), - T_PR(0.5)*std::log(T_PR(1.0)+lmdD*lmdD/(bmin*bmin)) ); - } - // Compute s - const auto tts = m1*g1s*m2*g2s/(inv_c2*p1sm*p1sm) + T_PR(1.0); - const auto tts2 = tts*tts; - T_PR s = n1*n2/n12 * dt*lnLmd*q1*q1*q2*q2 / - ( T_PR(4.0) * MathConst::pi * PhysConst::ep0 * PhysConst::ep0 * - m1*g1*m2*g2/(inv_c2*inv_c2) ) * gc*p1sm/mass_g * tts2; - - // Compute s' - const auto cbrt_n1 = std::cbrt(n1); - const auto cbrt_n2 = std::cbrt(n2); - const auto coeff = static_cast( - std::pow(4.0*MathConst::pi/3.0,1.0/3.0)); - T_PR const vrel = mass_g*p1sm/(m1*g1s*m2*g2s*gc); - T_PR const sp = coeff * n1*n2/n12 * dt * vrel * (m1+m2) / - amrex::max( m1*cbrt_n1*cbrt_n1, - m2*cbrt_n2*cbrt_n2); - - // Determine s - s = amrex::min(s,sp); - - // Get random numbers - T_PR r = amrex::Random(engine); - - // Compute scattering angle - T_PR cosXs; - T_PR sinXs; - if ( s <= T_PR(0.1) ) - { - while ( true ) + T_PR s = 0; + if (p1sm > std::numeric_limits::min()) { + + // s is non-zero (i.e. particles scatter) only if the relative + // motion between particles is not negligible (p1sm non-zero) + + // Compute the Coulomb log lnLmd first + T_PR lnLmd; + if ( L > T_PR(0.0) ) { lnLmd = L; } + else { - cosXs = T_PR(1.0) + s * std::log(r); - // Avoid the bug when r is too small such that cosXs < -1 - if ( cosXs >= T_PR(-1.0) ) { break; } - r = amrex::Random(engine); + // Compute b0 according to eq (22) from Perez et al., Phys.Plasmas.19.083104 (2012) + // Note: there is a typo in the equation, the last square is incorrect! + // See the SMILEI documentation: https://smileipic.github.io/Smilei/Understand/collisions.html + // and https://github.com/ECP-WarpX/WarpX/files/3799803/main.pdf from GitHub #429 + T_PR const b0 = amrex::Math::abs(q1*q2) * inv_c2 / + (T_PR(4.0)*MathConst::pi*PhysConst::ep0) * gc/mass_g * + ( m1*g1s*m2*g2s/(p1sm*p1sm*inv_c2) + T_PR(1.0) ); + + // Compute the minimal impact parameter + constexpr T_PR hbar_pi = static_cast(PhysConst::hbar*MathConst::pi); + const T_PR bmin = amrex::max(hbar_pi/p1sm, b0); + + // Compute the Coulomb log lnLmd + lnLmd = amrex::max( T_PR(2.0), + T_PR(0.5)*std::log(T_PR(1.0)+lmdD*lmdD/(bmin*bmin)) ); } - } - else if ( s > T_PR(0.1) && s <= T_PR(3.0) ) - { - T_PR const Ainv = static_cast( - 0.0056958 + 0.9560202*s - 0.508139*s*s + - 0.47913906*s*s*s - 0.12788975*s*s*s*s + 0.02389567*s*s*s*s*s); - cosXs = Ainv * std::log( std::exp(T_PR(-1.0)/Ainv) + - T_PR(2.0) * r * std::sinh(T_PR(1.0)/Ainv) ); - } - else if ( s > T_PR(3.0) && s <= T_PR(6.0) ) - { - T_PR const A = T_PR(3.0) * std::exp(-s); - cosXs = T_PR(1.0)/A * std::log( std::exp(-A) + - T_PR(2.0) * r * std::sinh(A) ); - } - else - { - cosXs = T_PR(2.0) * r - T_PR(1.0); - } - sinXs = std::sqrt(T_PR(1.0) - cosXs*cosXs); - - // Get random azimuthal angle - T_PR const phis = amrex::Random(engine) * T_PR(2.0) * MathConst::pi; - T_PR const cosphis = std::cos(phis); - T_PR const sinphis = std::sin(phis); - - // Compute post-collision momenta pfs in COM - T_PR p1fsx; - T_PR p1fsy; - T_PR p1fsz; - // p1sp is the p1s perpendicular - T_PR p1sp = std::sqrt( p1sx*p1sx + p1sy*p1sy ); - // Make sure p1sp is not almost zero - if ( p1sp > std::numeric_limits::min() ) - { - p1fsx = ( p1sx*p1sz/p1sp ) * sinXs*cosphis + - ( p1sy*p1sm/p1sp ) * sinXs*sinphis + - ( p1sx ) * cosXs; - p1fsy = ( p1sy*p1sz/p1sp ) * sinXs*cosphis + - (-p1sx*p1sm/p1sp ) * sinXs*sinphis + - ( p1sy ) * cosXs; - p1fsz = (-p1sp ) * sinXs*cosphis + - ( T_PR(0.0) ) * sinXs*sinphis + - ( p1sz ) * cosXs; - // Note a negative sign is different from - // Eq. (12) in Perez's paper, - // but they are the same due to the random nature of phis. - } - else - { - // If the previous p1sp is almost zero - // x->y y->z z->x - // This set is equivalent to the one in Nanbu's paper - p1sp = std::sqrt( p1sy*p1sy + p1sz*p1sz ); - p1fsy = ( p1sy*p1sx/p1sp ) * sinXs*cosphis + - ( p1sz*p1sm/p1sp ) * sinXs*sinphis + - ( p1sy ) * cosXs; - p1fsz = ( p1sz*p1sx/p1sp ) * sinXs*cosphis + - (-p1sy*p1sm/p1sp ) * sinXs*sinphis + - ( p1sz ) * cosXs; - p1fsx = (-p1sp ) * sinXs*cosphis + - ( T_PR(0.0) ) * sinXs*sinphis + - ( p1sx ) * cosXs; - } - T_PR const p2fsx = -p1fsx; - T_PR const p2fsy = -p1fsy; - T_PR const p2fsz = -p1fsz; + // Compute s + const auto tts = m1*g1s*m2*g2s/(inv_c2*p1sm*p1sm) + T_PR(1.0); + const auto tts2 = tts*tts; + s = n1*n2/n12 * dt*lnLmd*q1*q1*q2*q2 / + ( T_PR(4.0) * MathConst::pi * PhysConst::ep0 * PhysConst::ep0 * + m1*g1*m2*g2/(inv_c2*inv_c2) ) * gc*p1sm/mass_g * tts2; - // Transform from COM to lab frame - T_PR p1fx; T_PR p2fx; - T_PR p1fy; T_PR p2fy; - T_PR p1fz; T_PR p2fz; - if ( vcms > std::numeric_limits::min() ) - { - T_PR const vcDp1fs = vcx*p1fsx + vcy*p1fsy + vcz*p1fsz; - T_PR const vcDp2fs = vcx*p2fsx + vcy*p2fsy + vcz*p2fsz; - /* factor = (gc-1.0)/vcms; Rewrite to avoid subtraction losing precision when gc is close to 1 */ - T_PR const factor = gc*gc*inv_c2/(gc+T_PR(1.0)); - T_PR const factor1 = factor*vcDp1fs + m1*g1s*gc; - T_PR const factor2 = factor*vcDp2fs + m2*g2s*gc; - p1fx = p1fsx + vcx * factor1; - p1fy = p1fsy + vcy * factor1; - p1fz = p1fsz + vcz * factor1; - p2fx = p2fsx + vcx * factor2; - p2fy = p2fsy + vcy * factor2; - p2fz = p2fsz + vcz * factor2; - } - else // If vcms = 0, don't do Lorentz-transform. - { - p1fx = p1fsx; - p1fy = p1fsy; - p1fz = p1fsz; - p2fx = p2fsx; - p2fy = p2fsy; - p2fz = p2fsz; - } + // Compute s' + const auto cbrt_n1 = std::cbrt(n1); + const auto cbrt_n2 = std::cbrt(n2); + const auto coeff = static_cast( + std::pow(4.0*MathConst::pi/3.0,1.0/3.0)); + T_PR const vrel = mass_g*p1sm/(m1*g1s*m2*g2s*gc); + T_PR const sp = coeff * n1*n2/n12 * dt * vrel * (m1+m2) / + amrex::max( m1*cbrt_n1*cbrt_n1, + m2*cbrt_n2*cbrt_n2); - // Rejection method - r = amrex::Random(engine); - if ( w2 > r*amrex::max(w1, w2) ) - { - u1x = p1fx / m1; - u1y = p1fy / m1; - u1z = p1fz / m1; -#ifndef AMREX_USE_DPCPP - AMREX_ASSERT(!std::isnan(u1x+u1y+u1z+u2x+u2y+u2z)); - AMREX_ASSERT(!std::isinf(u1x+u1y+u1z+u2x+u2y+u2z)); -#endif + // Determine s + s = amrex::min(s,sp); } - r = amrex::Random(engine); - if ( w1 > r*amrex::max(w1, w2) ) - { - u2x = p2fx / m2; - u2y = p2fy / m2; - u2z = p2fz / m2; + + // Only modify momenta if is s is non-zero + if (s > std::numeric_limits::min()) { + + // Get random numbers + T_PR r = amrex::Random(engine); + + // Compute scattering angle + T_PR cosXs; + T_PR sinXs; + if ( s <= T_PR(0.1) ) + { + while ( true ) + { + cosXs = T_PR(1.0) + s * std::log(r); + // Avoid the bug when r is too small such that cosXs < -1 + if ( cosXs >= T_PR(-1.0) ) { break; } + r = amrex::Random(engine); + } + } + else if ( s > T_PR(0.1) && s <= T_PR(3.0) ) + { + T_PR const Ainv = static_cast( + 0.0056958 + 0.9560202*s - 0.508139*s*s + + 0.47913906*s*s*s - 0.12788975*s*s*s*s + 0.02389567*s*s*s*s*s); + cosXs = Ainv * std::log( std::exp(T_PR(-1.0)/Ainv) + + T_PR(2.0) * r * std::sinh(T_PR(1.0)/Ainv) ); + } + else if ( s > T_PR(3.0) && s <= T_PR(6.0) ) + { + T_PR const A = T_PR(3.0) * std::exp(-s); + cosXs = T_PR(1.0)/A * std::log( std::exp(-A) + + T_PR(2.0) * r * std::sinh(A) ); + } + else + { + cosXs = T_PR(2.0) * r - T_PR(1.0); + } + sinXs = std::sqrt(T_PR(1.0) - cosXs*cosXs); + + // Get random azimuthal angle + T_PR const phis = amrex::Random(engine) * T_PR(2.0) * MathConst::pi; + T_PR const cosphis = std::cos(phis); + T_PR const sinphis = std::sin(phis); + + // Compute post-collision momenta pfs in COM + T_PR p1fsx; + T_PR p1fsy; + T_PR p1fsz; + // p1sp is the p1s perpendicular + T_PR p1sp = std::sqrt( p1sx*p1sx + p1sy*p1sy ); + // Make sure p1sp is not almost zero + if ( p1sp > std::numeric_limits::min() ) + { + p1fsx = ( p1sx*p1sz/p1sp ) * sinXs*cosphis + + ( p1sy*p1sm/p1sp ) * sinXs*sinphis + + ( p1sx ) * cosXs; + p1fsy = ( p1sy*p1sz/p1sp ) * sinXs*cosphis + + (-p1sx*p1sm/p1sp ) * sinXs*sinphis + + ( p1sy ) * cosXs; + p1fsz = (-p1sp ) * sinXs*cosphis + + ( T_PR(0.0) ) * sinXs*sinphis + + ( p1sz ) * cosXs; + // Note a negative sign is different from + // Eq. (12) in Perez's paper, + // but they are the same due to the random nature of phis. + } + else + { + // If the previous p1sp is almost zero + // x->y y->z z->x + // This set is equivalent to the one in Nanbu's paper + p1sp = std::sqrt( p1sy*p1sy + p1sz*p1sz ); + p1fsy = ( p1sy*p1sx/p1sp ) * sinXs*cosphis + + ( p1sz*p1sm/p1sp ) * sinXs*sinphis + + ( p1sy ) * cosXs; + p1fsz = ( p1sz*p1sx/p1sp ) * sinXs*cosphis + + (-p1sy*p1sm/p1sp ) * sinXs*sinphis + + ( p1sz ) * cosXs; + p1fsx = (-p1sp ) * sinXs*cosphis + + ( T_PR(0.0) ) * sinXs*sinphis + + ( p1sx ) * cosXs; + } + + T_PR const p2fsx = -p1fsx; + T_PR const p2fsy = -p1fsy; + T_PR const p2fsz = -p1fsz; + + // Transform from COM to lab frame + T_PR p1fx; T_PR p2fx; + T_PR p1fy; T_PR p2fy; + T_PR p1fz; T_PR p2fz; + if ( vcms > std::numeric_limits::min() ) + { + T_PR const vcDp1fs = vcx*p1fsx + vcy*p1fsy + vcz*p1fsz; + T_PR const vcDp2fs = vcx*p2fsx + vcy*p2fsy + vcz*p2fsz; + /* factor = (gc-1.0)/vcms; Rewrite to avoid subtraction losing precision when gc is close to 1 */ + T_PR const factor = gc*gc*inv_c2/(gc+T_PR(1.0)); + T_PR const factor1 = factor*vcDp1fs + m1*g1s*gc; + T_PR const factor2 = factor*vcDp2fs + m2*g2s*gc; + p1fx = p1fsx + vcx * factor1; + p1fy = p1fsy + vcy * factor1; + p1fz = p1fsz + vcz * factor1; + p2fx = p2fsx + vcx * factor2; + p2fy = p2fsy + vcy * factor2; + p2fz = p2fsz + vcz * factor2; + } + else // If vcms = 0, don't do Lorentz-transform. + { + p1fx = p1fsx; + p1fy = p1fsy; + p1fz = p1fsz; + p2fx = p2fsx; + p2fy = p2fsy; + p2fz = p2fsz; + } + + // Rejection method + r = amrex::Random(engine); + if ( w2 > r*amrex::max(w1, w2) ) + { + u1x = p1fx / m1; + u1y = p1fy / m1; + u1z = p1fz / m1; + } + r = amrex::Random(engine); + if ( w1 > r*amrex::max(w1, w2) ) + { + u2x = p2fx / m2; + u2y = p2fy / m2; + u2z = p2fz / m2; + } #ifndef AMREX_USE_DPCPP AMREX_ASSERT(!std::isnan(u1x+u1y+u1z+u2x+u2y+u2z)); AMREX_ASSERT(!std::isinf(u1x+u1y+u1z+u2x+u2y+u2z)); #endif - } + + } // if s > std::numeric_limits::min() } From 40e9c17fbac25e3836e69f376cb6a2a845621729 Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Fri, 22 Sep 2023 01:14:11 +0900 Subject: [PATCH 022/110] Fix check of PML + Silver-Mueller compatibility (#4297) * fix check of PML + Silver Mueller compatibility * complete check * refactoring * fix logic bug --- Source/WarpX.cpp | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 508790ef20f..6857ad85bcf 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -871,26 +871,30 @@ WarpX::ReadParameters () quantum_xi_c2 = static_cast(quantum_xi * PhysConst::c * PhysConst::c); } - for (int idim = 0; idim < AMREX_SPACEDIM; ++idim) { - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( - !( - ( WarpX::field_boundary_lo[idim] == FieldBoundaryType::PML && - WarpX::field_boundary_lo[idim] == FieldBoundaryType::Absorbing_SilverMueller ) || - ( WarpX::field_boundary_hi[idim] == FieldBoundaryType::PML && - WarpX::field_boundary_hi[idim] == FieldBoundaryType::Absorbing_SilverMueller ) - ), - "PML and Silver-Mueller boundary conditions cannot be activated at the same time."); + const auto at_least_one_boundary_is_pml = + (std::any_of(WarpX::field_boundary_lo.begin(), WarpX::field_boundary_lo.end(), + [](const auto& cc){return cc == FieldBoundaryType::PML;}) + || + std::any_of(WarpX::field_boundary_hi.begin(), WarpX::field_boundary_hi.end(), + [](const auto& cc){return cc == FieldBoundaryType::PML;}) + ); + const auto at_least_one_boundary_is_silver_mueller = + (std::any_of(WarpX::field_boundary_lo.begin(), WarpX::field_boundary_lo.end(), + [](const auto& cc){return cc == FieldBoundaryType::Absorbing_SilverMueller;}) + || + std::any_of(WarpX::field_boundary_hi.begin(), WarpX::field_boundary_hi.end(), + [](const auto& cc){return cc == FieldBoundaryType::Absorbing_SilverMueller;}) + ); - if (WarpX::field_boundary_lo[idim] == FieldBoundaryType::Absorbing_SilverMueller || - WarpX::field_boundary_hi[idim] == FieldBoundaryType::Absorbing_SilverMueller) - { - // SilverMueller is implemented for Yee - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( - electromagnetic_solver_id == ElectromagneticSolverAlgo::Yee, - "The Silver-Mueller boundary condition can only be used with the Yee solver."); - } - } + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + !(at_least_one_boundary_is_pml && at_least_one_boundary_is_silver_mueller), + "PML and Silver-Mueller boundary conditions cannot be activated at the same time."); + + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + (!at_least_one_boundary_is_silver_mueller) || + (electromagnetic_solver_id == ElectromagneticSolverAlgo::Yee), + "The Silver-Mueller boundary condition can only be used with the Yee solver."); utils::parser::queryWithParser(pp_warpx, "pml_ncell", pml_ncell); utils::parser::queryWithParser(pp_warpx, "pml_delta", pml_delta); From ef3fa0518975a5049fb2a5cf314d7b7c5b0832e7 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 21 Sep 2023 21:37:29 +0200 Subject: [PATCH 023/110] Lassen (LLNL): Sources & Builds Off-Home (#4311) Moving the source directories and build instructions for WarpX itself from `$HOME` to the per-user NFS directory `/usr/workspace/${USER}/lassen/src`, which has a larger default quota. --- Docs/source/install/hpc/lassen.rst | 18 ++++---- .../lassen-llnl/install_v100_dependencies.sh | 44 ++++++++++--------- Tools/machines/lassen-llnl/install_v100_ml.sh | 15 ++++--- .../lassen_v100_warpx.profile.example | 1 + 4 files changed, 42 insertions(+), 36 deletions(-) diff --git a/Docs/source/install/hpc/lassen.rst b/Docs/source/install/hpc/lassen.rst index 7b5630e0272..5726254652a 100644 --- a/Docs/source/install/hpc/lassen.rst +++ b/Docs/source/install/hpc/lassen.rst @@ -41,14 +41,14 @@ Use the following commands to download the WarpX source code: .. code-block:: bash - git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx + git clone https://github.com/ECP-WarpX/WarpX.git /usr/workspace/${USER}/lassen/src/warpx We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx.profile``. Create it now: .. code-block:: bash - cp $HOME/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example $HOME/lassen_v100_warpx.profile + cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example $HOME/lassen_v100_warpx.profile .. dropdown:: Script Details :color: light @@ -80,7 +80,7 @@ Finally, since lassen does not yet provide software modules for some of our depe .. code-block:: bash - bash $HOME/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh + bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh source /usr/workspace/${USER}/lassen/gpu/venvs/warpx-lassen/bin/activate .. dropdown:: Script Details @@ -99,7 +99,7 @@ Finally, since lassen does not yet provide software modules for some of our depe .. code-block:: bash - runNode bash $HOME/src/warpx/Tools/machines/lassen-llnl/install_v100_ml.sh + runNode bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_ml.sh .. dropdown:: Script Details :color: light @@ -113,7 +113,7 @@ Finally, since lassen does not yet provide software modules for some of our depe .. code-block:: bash - python3 -m pip install -r $HOME/src/warpx/Tools/optimas/requirements.txt + python3 -m pip install -r /usr/workspace/${USER}/lassen/src/warpx/Tools/optimas/requirements.txt .. _building-lassen-compilation: @@ -125,13 +125,13 @@ Use the following :ref:`cmake commands ` to compile the applicat .. code-block:: bash - cd $HOME/src/warpx + cd /usr/workspace/${USER}/lassen/src/warpx rm -rf build_lassen cmake -S . -B build_lassen -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" cmake --build build_lassen -j 8 -The WarpX application executables are now in ``$HOME/src/warpx/build_lassen/bin/``. +The WarpX application executables are now in ``/usr/workspace/${USER}/lassen/src/warpx/build_lassen/bin/``. Additionally, the following commands will install WarpX as a Python module: .. code-block:: bash @@ -155,7 +155,7 @@ If you already installed WarpX in the past and want to update it, start by getti .. code-block:: bash - cd $HOME/src/warpx + cd /usr/workspace/${USER}/lassen/src/warpx # read the output of this command - does it look ok? git status @@ -174,7 +174,7 @@ And, if needed, - log out and into the system, activate the now updated environment profile as usual, - :ref:`execute the dependency install scripts `. -As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_lassen`` and rebuild WarpX. +As a last step, clean the build directory ``rm -rf /usr/workspace/${USER}/lassen/src/warpx/build_lassen`` and rebuild WarpX. .. _running-cpp-lassen: diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index 21650f09ee0..223b41d1967 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -20,9 +20,11 @@ if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in yo # Remove old dependencies ##################################################### # +SRC_DIR="/usr/workspace/${USER}/lassen/src" SW_DIR="/usr/workspace/${USER}/lassen/gpu" rm -rf ${SW_DIR} mkdir -p ${SW_DIR} +mkdir -p ${SRC_DIR} # remove common user mistakes in python, located in .local instead of a venv python3 -m pip uninstall -qq -y pywarpx @@ -37,70 +39,70 @@ python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true build_dir=$(mktemp -d) # c-blosc (I/O compression) -if [ -d $HOME/src/c-blosc ] +if [ -d ${SRC_DIR}/c-blosc ] then - cd $HOME/src/c-blosc + cd ${SRC_DIR}/c-blosc git fetch --prune git checkout v1.21.1 cd - else - git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc + git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git ${SRC_DIR}/c-blosc fi -cmake -S $HOME/src/c-blosc -B ${build_dir}/c-blosc-lassen-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 +cmake -S ${SRC_DIR}/c-blosc -B ${build_dir}/c-blosc-lassen-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 cmake --build ${build_dir}/c-blosc-lassen-build --target install --parallel 10 # HDF5 -if [ -d $HOME/src/hdf5 ] +if [ -d ${SRC_DIR}/hdf5 ] then - cd $HOME/src/hdf5 + cd ${SRC_DIR}/hdf5 git fetch --prune git checkout hdf5-1_14_1-2 cd - else - git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git $HOME/src/hdf5 + git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git ${SRC_DIR}/hdf5 fi -cmake -S $HOME/src/hdf5 -B ${build_dir}/hdf5-lassen-build -DBUILD_TESTING=OFF -DHDF5_ENABLE_PARALLEL=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/hdf5-1.14.1.2 +cmake -S ${SRC_DIR}/hdf5 -B ${build_dir}/hdf5-lassen-build -DBUILD_TESTING=OFF -DHDF5_ENABLE_PARALLEL=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/hdf5-1.14.1.2 cmake --build ${build_dir}/hdf5-lassen-build --target install --parallel 10 # ADIOS2 -if [ -d $HOME/src/adios2 ] +if [ -d ${SRC_DIR}/adios2 ] then - cd $HOME/src/adios2 + cd ${SRC_DIR}/adios2 git fetch --prune git checkout v2.8.3 cd - else - git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2 + git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git ${SRC_DIR}/adios2 fi -cmake -S $HOME/src/adios2 -B ${build_dir}/adios2-lassen-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake -S ${SRC_DIR}/adios2 -B ${build_dir}/adios2-lassen-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 cmake --build ${build_dir}/adios2-lassen-build --target install -j 10 # BLAS++ (for PSATD+RZ) -if [ -d $HOME/src/blaspp ] +if [ -d ${SRC_DIR}/blaspp ] then - cd $HOME/src/blaspp + cd ${SRC_DIR}/blaspp git fetch --prune git checkout master git pull cd - else - git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp + git clone https://github.com/icl-utk-edu/blaspp.git ${SRC_DIR}/blaspp fi -cmake -S $HOME/src/blaspp -B ${build_dir}/blaspp-lassen-build -Duse_openmp=ON -Dgpu_backend=cuda -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake -S ${SRC_DIR}/blaspp -B ${build_dir}/blaspp-lassen-build -Duse_openmp=ON -Dgpu_backend=cuda -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master cmake --build ${build_dir}/blaspp-lassen-build --target install --parallel 10 # LAPACK++ (for PSATD+RZ) -if [ -d $HOME/src/lapackpp ] +if [ -d ${SRC_DIR}/lapackpp ] then - cd $HOME/src/lapackpp + cd ${SRC_DIR}/lapackpp git fetch --prune git checkout master git pull cd - else - git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp + git clone https://github.com/icl-utk-edu/lapackpp.git ${SRC_DIR}/lapackpp fi -CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-lassen-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master -DLAPACK_LIBRARIES=/usr/lib64/liblapack.so +CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${SRC_DIR}/lapackpp -B ${build_dir}/lapackpp-lassen-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master -DLAPACK_LIBRARIES=/usr/lib64/liblapack.so cmake --build ${build_dir}/lapackpp-lassen-build --target install --parallel 10 @@ -124,7 +126,7 @@ echo "matplotlib==3.2.2" > ${build_dir}/constraints.txt python3 -m pip install --upgrade -c ${build_dir}/constraints.txt yt # install or update WarpX dependencies such as picmistandard -python3 -m pip install --upgrade -r $HOME/src/warpx/requirements.txt +python3 -m pip install --upgrade -r ${SRC_DIR}/warpx/requirements.txt # for ML dependencies, see install_v100_ml.sh diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh index 2ff90adb521..6e00be035d6 100755 --- a/Tools/machines/lassen-llnl/install_v100_ml.sh +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -20,6 +20,9 @@ if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in yo # Remove old dependencies ##################################################### # +SRC_DIR="/usr/workspace/${USER}/lassen/src" +mkdir -p ${SRC_DIR} + # remove common user mistakes in python, located in .local instead of a venv python3 -m pip uninstall -qqq -y torch 2>/dev/null || true @@ -29,21 +32,21 @@ python3 -m pip uninstall -qqq -y torch 2>/dev/null || true # for basic python dependencies, see install_v100_dependencies.sh # optional: for libEnsemble - WIP: issues with nlopt -# python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt +# python3 -m pip install -r ${SRC_DIR}/warpx/Tools/LibEnsemble/requirements.txt # optional: for pytorch -if [ -d ${HOME}/src/pytorch ] +if [ -d ${SRC_DIR}/pytorch ] then - cd ${HOME}/src/pytorch + cd ${SRC_DIR}/pytorch git fetch git checkout . git checkout v2.0.1 git submodule update --init --recursive cd - else - git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git ${HOME}/src/pytorch + git clone -b v2.0.1 --recurse-submodules https://github.com/pytorch/pytorch.git ${SRC_DIR}/pytorch fi -cd ${HOME}/src/pytorch +cd ${SRC_DIR}/pytorch rm -rf build # see https://github.com/pytorch/pytorch/issues/97497#issuecomment-1499069641 @@ -63,4 +66,4 @@ cd - # optional: optimas dependencies (based on libEnsemble & ax->botorch->gpytorch->pytorch) # commented because scikit-learn et al. compile > 2 hrs # please run manually on a login node if needed -#python3 -m pip install -r $HOME/src/warpx/Tools/optimas/requirements.txt +#python3 -m pip install -r ${SRC_DIR}/warpx/Tools/optimas/requirements.txt diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example index dd938e85c11..652af2a2822 100644 --- a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example @@ -10,6 +10,7 @@ module load cuda/12.0.0 module load boost/1.70.0 # optional: for openPMD support +SRC_DIR="/usr/workspace/${USER}/lassen/src" SW_DIR="/usr/workspace/${USER}/lassen/gpu" export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${SW_DIR}/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH From 9956295e22978fb10ce9bcfae334f5ecc815a4a5 Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Thu, 21 Sep 2023 18:08:44 -0700 Subject: [PATCH 024/110] Implement Galilean PML (#733) * Implement Galilean PML * Activate the PML * Add automated test * Fix compilation error * Fix more compilation errors * Fix more compilation error * Fix more compilation bugs * Fix more compilation bugs * Clean up code, debug CI * Fix CI * Use both collocated and staggered k vectors * Reset CI benchmark * Update test * Add Doxygen, clean up methods signature * Improve abort messages * Implement correct analytical equations * Remove class PsatdAlgorithmGalileanPml * Add Doxygen strings for class `PsatdAlgorithmPml` * Update clang tidy --------- Co-authored-by: Edoardo Zoni --- Examples/Tests/pml/analysis_pml_psatd.py | 29 +- .../benchmarks_json/pml_x_galilean.json | 12 + Regression/WarpX-tests.ini | 16 + Source/BoundaryConditions/PML.cpp | 8 +- .../SpectralAlgorithms/PsatdAlgorithmPml.H | 66 +++- .../SpectralAlgorithms/PsatdAlgorithmPml.cpp | 362 ++++++++++-------- .../SpectralSolver/SpectralSolver.cpp | 4 +- 7 files changed, 312 insertions(+), 185 deletions(-) create mode 100644 Regression/Checksum/benchmarks_json/pml_x_galilean.json diff --git a/Examples/Tests/pml/analysis_pml_psatd.py b/Examples/Tests/pml/analysis_pml_psatd.py index ca3d7cc3012..50d0b2ac1c1 100755 --- a/Examples/Tests/pml/analysis_pml_psatd.py +++ b/Examples/Tests/pml/analysis_pml_psatd.py @@ -8,6 +8,7 @@ # License: BSD-3-Clause-LBNL import os +import re import sys import numpy as np @@ -19,13 +20,18 @@ filename = sys.argv[1] -############################ -### INITIAL LASER ENERGY ### -############################ -energy_start = 7.282940107273505e-08 # electromagnetic energy at iteration 50 +galilean = True if re.search("galilean", filename) else False + +# Initial laser energy (at iteration 50) +if galilean: + filename_init = 'pml_x_galilean_plt000050' + energy_start = 4.439376199524034e-08 +else: + filename_init = 'pml_x_psatd_plt000050' + energy_start = 7.282940107273505e-08 # Check consistency of field energy diagnostics with initial energy above -ds = yt.load('pml_x_psatd_plt000050') +ds = yt.load(filename_init) all_data_level_0 = ds.covering_grid(level=0, left_edge=ds.domain_left_edge, dims=ds.domain_dimensions) Bx = all_data_level_0['boxlib', 'Bx'].v.squeeze() By = all_data_level_0['boxlib', 'By'].v.squeeze() @@ -43,10 +49,8 @@ print("relative error = " + str(error)) assert (error < tolerance) -########################## -### FINAL LASER ENERGY ### -########################## -ds = yt.load( filename ) +# Final laser energy +ds = yt.load(filename) all_data_level_0 = ds.covering_grid(level=0,left_edge=ds.domain_left_edge, dims=ds.domain_dimensions) Bx = all_data_level_0['boxlib', 'Bx'].v.squeeze() By = all_data_level_0['boxlib', 'By'].v.squeeze() @@ -58,8 +62,8 @@ energyB = np.sum(0.5 / scc.mu_0 * (Bx**2 + By**2 + Bz**2)) energy_end = energyE + energyB -reflectivity = energy_end / energy_start -reflectivity_max = 1e-06 +reflectivity = energy_end / energy_start_diags +reflectivity_max = 1e-6 print("reflectivity = " + str(reflectivity)) print("reflectivity_max = " + str(reflectivity_max)) @@ -70,7 +74,8 @@ sys.path.insert(0, '../../../../warpx/Examples/') from analysis_default_restart import check_restart -check_restart(filename) +if not galilean: + check_restart(filename) test_name = os.path.split(os.getcwd())[1] checksumAPI.evaluate_checksum(test_name, filename) diff --git a/Regression/Checksum/benchmarks_json/pml_x_galilean.json b/Regression/Checksum/benchmarks_json/pml_x_galilean.json new file mode 100644 index 00000000000..b32eb7a6518 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/pml_x_galilean.json @@ -0,0 +1,12 @@ +{ + "lev=0": { + "Bx": 9.111505955244123e-09, + "By": 1.743610723263872e-08, + "Bz": 8.692076437162089e-09, + "Ex": 4.8377358061392215, + "Ey": 3.8096703194246215, + "Ez": 4.747325170136437, + "divE": 480170.26408400893, + "rho": 1.7297829145620754e-06 + } +} \ No newline at end of file diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 56139d3da17..a3a04f609b2 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -2754,6 +2754,22 @@ compileTest = 0 doVis = 0 analysisRoutine = Examples/Tests/pml/analysis_pml_ckc.py +[pml_x_galilean] +buildDir = . +inputFile = Examples/Tests/pml/inputs_2d +runtime_params = algo.maxwell_solver=psatd psatd.update_with_rho=1 warpx.do_dynamic_scheduling=0 diag1.fields_to_plot=Ex Ey Ez Bx By Bz rho divE warpx.cfl=0.7071067811865475 warpx.do_pml_dive_cleaning=1 warpx.do_pml_divb_cleaning=1 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium psatd.v_galilean=0. 0. 0.99 warpx.grid_type=collocated +dim = 2 +addToCompileString = USE_PSATD=TRUE +cmakeSetupOpts = -DWarpX_DIMS=2 -DWarpX_PSATD=ON +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +analysisRoutine = Examples/Tests/pml/analysis_pml_psatd.py + [pml_x_psatd] buildDir = . inputFile = Examples/Tests/pml/inputs_2d diff --git a/Source/BoundaryConditions/PML.cpp b/Source/BoundaryConditions/PML.cpp index d34a2f73222..5fc3e2dd3dc 100644 --- a/Source/BoundaryConditions/PML.cpp +++ b/Source/BoundaryConditions/PML.cpp @@ -738,11 +738,11 @@ PML::PML (const int lev, const BoxArray& grid_ba, const DistributionMapping& gri const RealVect dx{AMREX_D_DECL(geom->CellSize(0), geom->CellSize(1), geom->CellSize(2))}; // Get the cell-centered box, with guard cells BoxArray realspace_ba = ba; // Copy box - amrex::Vector const v_galilean_zero = {0., 0., 0.}; + amrex::Vector const v_galilean = WarpX::GetInstance().m_v_galilean; amrex::Vector const v_comoving_zero = {0., 0., 0.}; realspace_ba.enclosedCells().grow(nge); // cell-centered + guard cells spectral_solver_fp = std::make_unique(lev, realspace_ba, dm, - nox_fft, noy_fft, noz_fft, grid_type, v_galilean_zero, + nox_fft, noy_fft, noz_fft, grid_type, v_galilean, v_comoving_zero, dx, dt, in_pml, periodic_single_box, update_with_rho, fft_do_time_averaging, psatd_solution_type, J_in_time, rho_in_time, m_dive_cleaning, m_divb_cleaning); #endif @@ -846,11 +846,11 @@ PML::PML (const int lev, const BoxArray& grid_ba, const DistributionMapping& gri const RealVect cdx{AMREX_D_DECL(cgeom->CellSize(0), cgeom->CellSize(1), cgeom->CellSize(2))}; // Get the cell-centered box, with guard cells BoxArray realspace_cba = cba; // Copy box - amrex::Vector const v_galilean_zero = {0., 0., 0.}; + amrex::Vector const v_galilean = WarpX::GetInstance().m_v_galilean; amrex::Vector const v_comoving_zero = {0., 0., 0.}; realspace_cba.enclosedCells().grow(nge); // cell-centered + guard cells spectral_solver_cp = std::make_unique(lev, realspace_cba, cdm, - nox_fft, noy_fft, noz_fft, grid_type, v_galilean_zero, + nox_fft, noy_fft, noz_fft, grid_type, v_galilean, v_comoving_zero, cdx, dt, in_pml, periodic_single_box, update_with_rho, fft_do_time_averaging, psatd_solution_type, J_in_time, rho_in_time, m_dive_cleaning, m_divb_cleaning); #endif diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H index 7f50d1cd8ea..cef6331888e 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H @@ -21,27 +21,59 @@ #if WARPX_USE_PSATD -/* \brief Class that updates the field in spectral space +/* + * \brief Class that updates the field in spectral space * and stores the coefficients of the corresponding update equation. */ class PsatdAlgorithmPml : public SpectralBaseAlgorithm { public: - PsatdAlgorithmPml(const SpectralKSpace& spectral_kspace, - const amrex::DistributionMapping& dm, - const SpectralFieldIndex& spectral_index, - int norder_x, int norder_y, - int norder_z, short grid_type, - amrex::Real dt, - bool dive_cleaning, - bool divb_cleaning); - void InitializeSpectralCoefficients( + /** + * \brief Constructor of the class PsatdAlgorithmPml + * + * \param[in] spectral_kspace Spectral space + * \param[in] dm Distribution mapping + * \param[in] spectral_index Object containing indices to access data in spectral space + * \param[in] norder_x Order of the spectral solver along x + * \param[in] norder_y Order of the spectral solver along y + * \param[in] norder_z Order of the spectral solver along z + * \param[in] grid_type Type of grid (collocated or not) + * \param[in] v_galilean Galilean velocity + * \param[in] dt Time step of the simulation + * \param[in] dive_cleaning Whether to use divergence correction for E (F term) + * \param[in] divb_cleaning Whether to use divergence correction for B (G term) + */ + PsatdAlgorithmPml( const SpectralKSpace& spectral_kspace, const amrex::DistributionMapping& dm, - amrex::Real dt); + const SpectralFieldIndex& spectral_index, + int norder_x, + int norder_y, + int norder_z, + short grid_type, + const amrex::Vector& v_galilean, + amrex::Real dt, + bool dive_cleaning, + bool divb_cleaning); + + /** + * \brief Initializes the coefficients used in \c pushSpectralFields + * to update the E and B fields + * + * \param[in] spectral_kspace Spectral space + * \param[in] dm Distribution mapping + */ + void InitializeSpectralCoefficients( + const SpectralKSpace& spectral_kspace, + const amrex::DistributionMapping& dm); - // Redefine functions from base class + /** + * \brief Updates the E and B fields in spectral space, + * according to the relevant PSATD equations + * + * \param[in,out] f All fields in spectral space + */ virtual void pushSpectralFields(SpectralFieldData& f) const override final; /** @@ -67,10 +99,20 @@ class PsatdAlgorithmPml : public SpectralBaseAlgorithm virtual void VayDeposition (SpectralFieldData& field_data) override final; private: + SpectralRealCoefficients C_coef, S_ck_coef, inv_k2_coef; + SpectralComplexCoefficients T2_coef; + // Centered modified finite-order k vectors + KVectorComponent modified_kx_vec_centered; +#if defined(WARPX_DIM_3D) + KVectorComponent modified_ky_vec_centered; +#endif + KVectorComponent modified_kz_vec_centered; + amrex::Vector m_v_galilean; amrex::Real m_dt; bool m_dive_cleaning; bool m_divb_cleaning; + bool m_is_galilean; }; #endif // WARPX_USE_PSATD diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp index 1a4039915e0..be91d920cba 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp @@ -9,6 +9,7 @@ #include "FieldSolver/SpectralSolver/SpectralFieldData.H" #include "FieldSolver/SpectralSolver/SpectralKSpace.H" #include "Utils/TextMsg.H" +#include "Utils/WarpXAlgorithmSelection.H" #include "Utils/WarpXConst.H" #include "Utils/WarpX_Complex.H" @@ -29,59 +30,85 @@ using namespace amrex; -/* \brief Initialize coefficients for the update equation */ -PsatdAlgorithmPml::PsatdAlgorithmPml(const SpectralKSpace& spectral_kspace, - const DistributionMapping& dm, - const SpectralFieldIndex& spectral_index, - const int norder_x, const int norder_y, - const int norder_z, const short grid_type, - const Real dt, - const bool dive_cleaning, const bool divb_cleaning) - // Initialize members of base class - : SpectralBaseAlgorithm(spectral_kspace, dm, spectral_index, norder_x, norder_y, norder_z, grid_type), - m_dt(dt), - m_dive_cleaning(dive_cleaning), - m_divb_cleaning(divb_cleaning) +PsatdAlgorithmPml::PsatdAlgorithmPml( + const SpectralKSpace& spectral_kspace, + const DistributionMapping& dm, + const SpectralFieldIndex& spectral_index, + int norder_x, + int norder_y, + int norder_z, + short grid_type, + const amrex::Vector& v_galilean, + Real dt, + bool dive_cleaning, + bool divb_cleaning) : + SpectralBaseAlgorithm(spectral_kspace, dm, spectral_index, norder_x, norder_y, norder_z, grid_type), + // Initialize the centered finite-order modified k vectors: + // these are computed always with the assumption of centered grids + // (argument grid_type=GridType::Collocated), for both collocated and staggered grids + modified_kx_vec_centered(spectral_kspace.getModifiedKComponent(dm, 0, norder_x, GridType::Collocated)), +#if defined(WARPX_DIM_3D) + modified_ky_vec_centered(spectral_kspace.getModifiedKComponent(dm, 1, norder_y, GridType::Collocated)), + modified_kz_vec_centered(spectral_kspace.getModifiedKComponent(dm, 2, norder_z, GridType::Collocated)), +#else + modified_kz_vec_centered(spectral_kspace.getModifiedKComponent(dm, 1, norder_z, GridType::Collocated)), +#endif + m_v_galilean(v_galilean), + m_dt(dt), + m_dive_cleaning(dive_cleaning), + m_divb_cleaning(divb_cleaning) { const BoxArray& ba = spectral_kspace.spectralspace_ba; - // Allocate the arrays of coefficients - C_coef = SpectralRealCoefficients(ba, dm, 1, 0); - S_ck_coef = SpectralRealCoefficients(ba, dm, 1, 0); + m_is_galilean = (v_galilean[0] != 0.) || (v_galilean[1] != 0.) || (v_galilean[2] != 0.); + + // Allocate arrays of coefficients + C_coef = SpectralRealCoefficients(ba, dm, 1, 0); + S_ck_coef = SpectralRealCoefficients(ba, dm, 1, 0); inv_k2_coef = SpectralRealCoefficients(ba, dm, 1, 0); - InitializeSpectralCoefficients(spectral_kspace, dm, dt); -} + // Allocate this coefficient only with Galilean PSATD + if (m_is_galilean) + { + T2_coef = SpectralComplexCoefficients(ba, dm, 1, 0); + } -/* Advance the E and B field in spectral space (stored in `f`) - * over one time step */ -void -PsatdAlgorithmPml::pushSpectralFields(SpectralFieldData& f) const { + InitializeSpectralCoefficients(spectral_kspace, dm); +} +void PsatdAlgorithmPml::pushSpectralFields(SpectralFieldData& f) const +{ const bool dive_cleaning = m_dive_cleaning; const bool divb_cleaning = m_divb_cleaning; + const bool is_galilean = m_is_galilean; const SpectralFieldIndex& Idx = m_spectral_index; // Loop over boxes - for (MFIter mfi(f.fields); mfi.isValid(); ++mfi){ - - const Box& bx = f.fields[mfi].box(); + for (amrex::MFIter mfi(f.fields); mfi.isValid(); ++mfi) + { + const amrex::Box& bx = f.fields[mfi].box(); // Extract arrays for the fields to be updated - const Array4 fields = f.fields[mfi].array(); + const amrex::Array4 fields = f.fields[mfi].array(); // Extract arrays for the coefficients - const Array4 C_arr = C_coef[mfi].array(); - const Array4 S_ck_arr = S_ck_coef[mfi].array(); - const Array4 inv_k2_arr = inv_k2_coef[mfi].array(); + const amrex::Array4 C_arr = C_coef[mfi].array(); + const amrex::Array4 S_ck_arr = S_ck_coef[mfi].array(); + const amrex::Array4 inv_k2_arr = inv_k2_coef[mfi].array(); + + amrex::Array4 T2_arr; + if (is_galilean) + { + T2_arr = T2_coef[mfi].array(); + } // Extract pointers for the k vectors - const Real* modified_kx_arr = modified_kx_vec[mfi].dataPtr(); + const amrex::Real* modified_kx_arr = modified_kx_vec[mfi].dataPtr(); #if defined(WARPX_DIM_3D) - const Real* modified_ky_arr = modified_ky_vec[mfi].dataPtr(); + const amrex::Real* modified_ky_arr = modified_ky_vec[mfi].dataPtr(); #endif - const Real* modified_kz_arr = modified_kz_vec[mfi].dataPtr(); + const amrex::Real* modified_kz_arr = modified_kz_vec[mfi].dataPtr(); const amrex::Real dt = m_dt; @@ -154,28 +181,29 @@ PsatdAlgorithmPml::pushSpectralFields(SpectralFieldData& f) const { } // k vector values, and coefficients - const Real kx = modified_kx_arr[i]; + const amrex::Real kx = modified_kx_arr[i]; #if defined(WARPX_DIM_3D) - const Real ky = modified_ky_arr[j]; - const Real kz = modified_kz_arr[k]; + const amrex::Real ky = modified_ky_arr[j]; + const amrex::Real kz = modified_kz_arr[k]; #else - constexpr Real ky = 0._rt; - const Real kz = modified_kz_arr[j]; + constexpr amrex::Real ky = 0._rt; + const amrex::Real kz = modified_kz_arr[j]; #endif - constexpr Real c2 = PhysConst::c * PhysConst::c; + constexpr amrex::Real c2 = PhysConst::c*PhysConst::c; const Complex I = Complex{0._rt, 1._rt}; - const Real kx2 = kx*kx; - const Real ky2 = ky*ky; - const Real kz2 = kz*kz; - const Real k_norm = std::sqrt(kx2 + ky2 + kz2); - - if (k_norm != 0._rt) { + const amrex::Real kx2 = kx*kx; + const amrex::Real ky2 = ky*ky; + const amrex::Real kz2 = kz*kz; + const amrex::Real knorm = std::sqrt(kx2 + ky2 + kz2); + if (knorm != 0._rt) + { const amrex::Real C = C_arr(i,j,k); const amrex::Real S_ck = S_ck_arr(i,j,k); const amrex::Real inv_k2 = inv_k2_arr(i,j,k); + const Complex T2 = (is_galilean) ? T2_arr(i,j,k) : 1.0_rt; const amrex::Real C1 = (kx2 * C + ky2 + kz2) * inv_k2; const amrex::Real C2 = (kx2 + ky2 * C + kz2) * inv_k2; @@ -218,42 +246,42 @@ PsatdAlgorithmPml::pushSpectralFields(SpectralFieldData& f) const { const Complex C22_c2 = C22 / c2; // Update E - fields(i,j,k,Idx.Exy) = C2 * Exy + C5 * Exz + C9 * Ey - + C10 * Bx + C11 * By + C19 * Bz; + fields(i,j,k,Idx.Exy) = T2 * (C2 * Exy + C5 * Exz + C9 * Ey + + C10 * Bx + C11 * By + C19 * Bz); - fields(i,j,k,Idx.Exz) = C6 * Exy + C3 * Exz + C8 * Ez - - C10 * Bx - C22 * By - C12 * Bz; + fields(i,j,k,Idx.Exz) = T2 * (C6 * Exy + C3 * Exz + C8 * Ez + - C10 * Bx - C22 * By - C12 * Bz); - fields(i,j,k,Idx.Eyz) = C3 * Eyz + C6 * Eyx + C7 * Ez - + C21 * Bx + C10 * By + C13 * Bz; + fields(i,j,k,Idx.Eyz) = T2 * (C3 * Eyz + C6 * Eyx + C7 * Ez + + C21 * Bx + C10 * By + C13 * Bz); - fields(i,j,k,Idx.Eyx) = C9 * Ex + C4 * Eyz + C1 * Eyx - - C14 * Bx - C10 * By - C18 * Bz; + fields(i,j,k,Idx.Eyx) = T2 * (C9 * Ex + C4 * Eyz + C1 * Eyx + - C14 * Bx - C10 * By - C18 * Bz); - fields(i,j,k,Idx.Ezx) = C8 * Ex + C1 * Ezx + C4 * Ezy - + C15 * Bx + C17 * By + C10 * Bz; + fields(i,j,k,Idx.Ezx) = T2 * (C8 * Ex + C1 * Ezx + C4 * Ezy + + C15 * Bx + C17 * By + C10 * Bz); - fields(i,j,k,Idx.Ezy) = C7 * Ey + C5 * Ezx + C2 * Ezy - - C20 * Bx - C16 * By - C10 * Bz; + fields(i,j,k,Idx.Ezy) = T2 * (C7 * Ey + C5 * Ezx + C2 * Ezy + - C20 * Bx - C16 * By - C10 * Bz); // Update B - fields(i,j,k,Idx.Bxy) = C2 * Bxy + C5 * Bxz + C9 * By - - C10_c2 * Ex - C11_c2 * Ey - C19_c2 * Ez; + fields(i,j,k,Idx.Bxy) = T2 * (C2 * Bxy + C5 * Bxz + C9 * By + - C10_c2 * Ex - C11_c2 * Ey - C19_c2 * Ez); - fields(i,j,k,Idx.Bxz) = C6 * Bxy + C3 * Bxz + C8 * Bz - + C10_c2 * Ex + C22_c2 * Ey + C12_c2 * Ez; + fields(i,j,k,Idx.Bxz) = T2 * (C6 * Bxy + C3 * Bxz + C8 * Bz + + C10_c2 * Ex + C22_c2 * Ey + C12_c2 * Ez); - fields(i,j,k,Idx.Byz) = C3 * Byz + C6 * Byx + C7 * Bz - - C21_c2 * Ex - C10_c2 * Ey - C13_c2 * Ez; + fields(i,j,k,Idx.Byz) = T2 * (C3 * Byz + C6 * Byx + C7 * Bz + - C21_c2 * Ex - C10_c2 * Ey - C13_c2 * Ez); - fields(i,j,k,Idx.Byx) = C9 * Bx + C4 * Byz + C1 * Byx - + C14_c2 * Ex + C10_c2 * Ey + C18_c2 * Ez; + fields(i,j,k,Idx.Byx) = T2 * (C9 * Bx + C4 * Byz + C1 * Byx + + C14_c2 * Ex + C10_c2 * Ey + C18_c2 * Ez); - fields(i,j,k,Idx.Bzx) = C8 * Bx + C1 * Bzx + C4 * Bzy - - C15_c2 * Ex - C17_c2 * Ey - C10_c2 * Ez; + fields(i,j,k,Idx.Bzx) = T2 * (C8 * Bx + C1 * Bzx + C4 * Bzy + - C15_c2 * Ex - C17_c2 * Ey - C10_c2 * Ez); - fields(i,j,k,Idx.Bzy) = C7 * By + C5 * Bzx + C2 * Bzy - + C20_c2 * Ex + C16_c2 * Ey + C10_c2 * Ez; + fields(i,j,k,Idx.Bzy) = T2 * (C7 * By + C5 * Bzx + C2 * Bzy + + C20_c2 * Ex + C16_c2 * Ey + C10_c2 * Ez); } else if (dive_cleaning && divb_cleaning) { @@ -266,80 +294,80 @@ PsatdAlgorithmPml::pushSpectralFields(SpectralFieldData& f) const { const Complex C25_c2 = C25 / c2; // Update E - fields(i,j,k,Idx.Exx) = C1 * Exx + C4 * Exy + C4 * Exz - - C9 * Ey - C8 * Ez + C23 * F; + fields(i,j,k,Idx.Exx) = T2 * (C1 * Exx + C4 * Exy + C4 * Exz + - C9 * Ey - C8 * Ez + C23 * F); - fields(i,j,k,Idx.Exy) = C5 * Exx + C2 * Exy + C5 * Exz - + C9 * Ey + C24 * Bz - C7 * G; + fields(i,j,k,Idx.Exy) = T2 * (C5 * Exx + C2 * Exy + C5 * Exz + + C9 * Ey + C24 * Bz - C7 * G); - fields(i,j,k,Idx.Exz) = C6 * Exx + C6 * Exy + C3 * Exz - + C8 * Ez - C25 * By + C7 * G; + fields(i,j,k,Idx.Exz) = T2 * (C6 * Exx + C6 * Exy + C3 * Exz + + C8 * Ez - C25 * By + C7 * G); - fields(i,j,k,Idx.Eyx) = C9 * Ex + C1 * Eyx + C4 * Eyy - + C4 * Eyz - C23 * Bz + C8 * G; + fields(i,j,k,Idx.Eyx) = T2 * (C9 * Ex + C1 * Eyx + C4 * Eyy + + C4 * Eyz - C23 * Bz + C8 * G); - fields(i,j,k,Idx.Eyy) = - C9 * Ex + C5 * Eyx + C2 * Eyy - + C5 * Eyz - C7 * Ez + C24 * F; + fields(i,j,k,Idx.Eyy) = T2 * (- C9 * Ex + C5 * Eyx + C2 * Eyy + + C5 * Eyz - C7 * Ez + C24 * F); - fields(i,j,k,Idx.Eyz) = C6 * Eyx + C6 * Eyy + C3 * Eyz - + C7 * Ez + C25 * Bx - C8 * G; + fields(i,j,k,Idx.Eyz) = T2 * (C6 * Eyx + C6 * Eyy + C3 * Eyz + + C7 * Ez + C25 * Bx - C8 * G); - fields(i,j,k,Idx.Ezx) = C8 * Ex + C1 * Ezx + C4 * Ezy - + C4 * Ezz + C23 * By - C9 * G; + fields(i,j,k,Idx.Ezx) = T2 * (C8 * Ex + C1 * Ezx + C4 * Ezy + + C4 * Ezz + C23 * By - C9 * G); - fields(i,j,k,Idx.Ezy) = C7 * Ey + C5 * Ezx + C2 * Ezy - + C5 * Ezz - C24 * Bx + C9 * G; + fields(i,j,k,Idx.Ezy) = T2 * (C7 * Ey + C5 * Ezx + C2 * Ezy + + C5 * Ezz - C24 * Bx + C9 * G); - fields(i,j,k,Idx.Ezz) = - C8 * Ex - C7 * Ey + C6 * Ezx - + C6 * Ezy + C3 * Ezz + C25 * F; + fields(i,j,k,Idx.Ezz) = T2 * (- C8 * Ex - C7 * Ey + C6 * Ezx + + C6 * Ezy + C3 * Ezz + C25 * F); // Update B - fields(i,j,k,Idx.Bxx) = C1 * Bxx + C4 * Bxy + C4 * Bxz - - C9 * By - C8 * Bz + C23_c2 * G; + fields(i,j,k,Idx.Bxx) = T2 * (C1 * Bxx + C4 * Bxy + C4 * Bxz + - C9 * By - C8 * Bz + C23_c2 * G); - fields(i,j,k,Idx.Bxy) = - C24_c2 * Ez + C5 * Bxx + C2 * Bxy - + C5 * Bxz + C9 * By + C7 * F; + fields(i,j,k,Idx.Bxy) = T2 * (- C24_c2 * Ez + C5 * Bxx + C2 * Bxy + + C5 * Bxz + C9 * By + C7 * F); - fields(i,j,k,Idx.Bxz) = C25_c2 * Ey + C6 * Bxx + C6 * Bxy - + C3 * Bxz + C8 * Bz - C7 * F; + fields(i,j,k,Idx.Bxz) = T2 * (C25_c2 * Ey + C6 * Bxx + C6 * Bxy + + C3 * Bxz + C8 * Bz - C7 * F); - fields(i,j,k,Idx.Byx) = C23_c2 * Ez + C9 * Bx + C1 * Byx - + C4 * Byy + C4 * Byz - C8 * F; + fields(i,j,k,Idx.Byx) = T2 * (C23_c2 * Ez + C9 * Bx + C1 * Byx + + C4 * Byy + C4 * Byz - C8 * F); - fields(i,j,k,Idx.Byy) = - C9 * Bx + C5 * Byx + C2 * Byy - + C5 * Byz - C7 * Bz + C24_c2 * G; + fields(i,j,k,Idx.Byy) = T2 * (- C9 * Bx + C5 * Byx + C2 * Byy + + C5 * Byz - C7 * Bz + C24_c2 * G); - fields(i,j,k,Idx.Byz) = - C25_c2 * Ex + C6 * Byx + C6 * Byy - + C3 * Byz + C7 * Bz + C8 * F; + fields(i,j,k,Idx.Byz) = T2 * (- C25_c2 * Ex + C6 * Byx + C6 * Byy + + C3 * Byz + C7 * Bz + C8 * F); - fields(i,j,k,Idx.Bzx) = - C23_c2 * Ey + C8 * Bx + C1 * Bzx - + C4 * Bzy + C4 * Bzz + C9 * F; + fields(i,j,k,Idx.Bzx) = T2 * (- C23_c2 * Ey + C8 * Bx + C1 * Bzx + + C4 * Bzy + C4 * Bzz + C9 * F); - fields(i,j,k,Idx.Bzy) = C24_c2 * Ex + C7 * By + C5 * Bzx - + C2 * Bzy + C5 * Bzz - C9 * F; + fields(i,j,k,Idx.Bzy) = T2 * (C24_c2 * Ex + C7 * By + C5 * Bzx + + C2 * Bzy + C5 * Bzz - C9 * F); - fields(i,j,k,Idx.Bzz) = - C8 * Bx - C7 * By + C6 * Bzx - + C6 * Bzy + C3 * Bzz + C25_c2 * G; + fields(i,j,k,Idx.Bzz) = T2 * (- C8 * Bx - C7 * By + C6 * Bzx + + C6 * Bzy + C3 * Bzz + C25_c2 * G); // Update F - fields(i,j,k,Idx.Fx) = C23_c2 * Ex + C8 * By - C9 * Bz - + C1 * Fx + C4 * Fy + C4 * Fz; + fields(i,j,k,Idx.Fx) = T2 * (C23_c2 * Ex + C8 * By - C9 * Bz + + C1 * Fx + C4 * Fy + C4 * Fz); - fields(i,j,k,Idx.Fy) = C24_c2 * Ey - C7 * Bx + C9 * Bz - + C5 * Fx + C2 * Fy + C5 * Fz; + fields(i,j,k,Idx.Fy) = T2 * (C24_c2 * Ey - C7 * Bx + C9 * Bz + + C5 * Fx + C2 * Fy + C5 * Fz); - fields(i,j,k,Idx.Fz) = C25_c2 * Ez + C7 * Bx - C8 * By - + C6 * Fx + C6 * Fy + C3 * Fz; + fields(i,j,k,Idx.Fz) = T2 * (C25_c2 * Ez + C7 * Bx - C8 * By + + C6 * Fx + C6 * Fy + C3 * Fz); // Update G - fields(i,j,k,Idx.Gx) = - C8 * Ey + C9 * Ez + C23 * Bx - + C1 * Gx + C4 * Gy + C4 * Gz; + fields(i,j,k,Idx.Gx) = T2 * (- C8 * Ey + C9 * Ez + C23 * Bx + + C1 * Gx + C4 * Gy + C4 * Gz); - fields(i,j,k,Idx.Gy) = C7 * Ex - C9 * Ez + C24 * By - + C5 * Gx + C2 * Gy + C5 * Gz; + fields(i,j,k,Idx.Gy) = T2 * (C7 * Ex - C9 * Ez + C24 * By + + C5 * Gx + C2 * Gy + C5 * Gz); - fields(i,j,k,Idx.Gz) = - C7 * Ex + C8 * Ey + C25 * Bz - + C6 * Gx + C6 * Gy + C3 * Gz; + fields(i,j,k,Idx.Gz) = T2 * (- C7 * Ex + C8 * Ey + C25 * Bz + + C6 * Gx + C6 * Gy + C3 * Gz); } } }); @@ -348,71 +376,95 @@ PsatdAlgorithmPml::pushSpectralFields(SpectralFieldData& f) const { void PsatdAlgorithmPml::InitializeSpectralCoefficients ( const SpectralKSpace& spectral_kspace, - const amrex::DistributionMapping& dm, - const amrex::Real dt) + const amrex::DistributionMapping& dm) { - const BoxArray& ba = spectral_kspace.spectralspace_ba; + const amrex::Real dt = m_dt; + const bool is_galilean = m_is_galilean; - // Fill them with the right values: - // Loop over boxes and allocate the corresponding coefficients - // for each box owned by the local MPI proc - for (MFIter mfi(ba, dm); mfi.isValid(); ++mfi) { + const amrex::BoxArray& ba = spectral_kspace.spectralspace_ba; - const Box& bx = ba[mfi]; + // Loop over boxes and allocate the corresponding coefficients + // for each box owned by the local MPI process + for (amrex::MFIter mfi(ba, dm); mfi.isValid(); ++mfi) + { + const amrex::Box& bx = ba[mfi]; // Extract pointers for the k vectors - const Real* modified_kx = modified_kx_vec[mfi].dataPtr(); + const amrex::Real* kx = modified_kx_vec[mfi].dataPtr(); + const amrex::Real* kx_c = modified_kx_vec_centered[mfi].dataPtr(); #if defined(WARPX_DIM_3D) - const Real* modified_ky = modified_ky_vec[mfi].dataPtr(); + const amrex::Real* ky = modified_ky_vec[mfi].dataPtr(); + const amrex::Real* ky_c = modified_ky_vec_centered[mfi].dataPtr(); #endif - const Real* modified_kz = modified_kz_vec[mfi].dataPtr(); + const amrex::Real* kz = modified_kz_vec[mfi].dataPtr(); + const amrex::Real* kz_c = modified_kz_vec_centered[mfi].dataPtr(); // Extract arrays for the coefficients - const Array4 C = C_coef[mfi].array(); - const Array4 S_ck = S_ck_coef[mfi].array(); - const Array4 inv_k2 = inv_k2_coef[mfi].array(); + const amrex::Array4 C = C_coef[mfi].array(); + const amrex::Array4 S_ck = S_ck_coef[mfi].array(); + const amrex::Array4 inv_k2 = inv_k2_coef[mfi].array(); - // Loop over indices within one box - ParallelFor(bx, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + amrex::Array4 T2; + if (is_galilean) { - const Real kx = modified_kx[i]; + T2 = T2_coef[mfi].array(); + } + + // Extract Galilean velocity + amrex::Real vg_x = m_v_galilean[0]; #if defined(WARPX_DIM_3D) - const Real ky = modified_ky[j]; - const Real kz = modified_kz[k]; -#else - constexpr Real ky = 0._rt; - const Real kz = modified_kz[j]; + amrex::Real vg_y = m_v_galilean[1]; #endif + amrex::Real vg_z = m_v_galilean[2]; + // Loop over indices within one box + ParallelFor(bx, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { // Calculate norm of vector - const Real k_norm = std::sqrt(kx*kx + ky*ky + kz*kz); - const Real k2 = k_norm * k_norm; + const amrex::Real knorm = std::sqrt( + amrex::Math::powi<2>(kx[i]) + +#if (AMREX_SPACEDIM==3) + amrex::Math::powi<2>(ky[j]) + amrex::Math::powi<2>(kz[k])); +#else + amrex::Math::powi<2>(kz[j])); +#endif + // Calculate the dot product of the k vector with the Galilean velocity. + // This has to be computed always with the centered (collocated) finite-order + // modified k vectors, to work correctly for both collocated and staggered grids. + // w_c = 0 always with standard PSATD (zero Galilean velocity). + const amrex::Real w_c = kx_c[i]*vg_x + +#if (AMREX_SPACEDIM==3) + ky_c[j]*vg_y + kz_c[k]*vg_z; +#else + kz_c[j]*vg_z; +#endif + constexpr amrex::Real c = PhysConst::c; + constexpr Complex I{0._rt,1._rt}; - // Calculate coefficients - constexpr Real c = PhysConst::c; + // Coefficients for knorm = 0 do not need to be set + if (knorm != 0._rt) + { + C(i,j,k) = std::cos(c*knorm*dt); + S_ck(i,j,k) = std::sin(c*knorm*dt) / (c*knorm); + inv_k2(i,j,k) = 1._rt / (knorm*knorm); - // Coefficients for k_norm = 0 do not need to be set - if (k_norm != 0._rt) { - C(i,j,k) = std::cos(c * k_norm * dt); - S_ck(i,j,k) = std::sin(c * k_norm * dt) / (c * k_norm); - inv_k2(i,j,k) = 1._rt / k2; + if (is_galilean) + { + T2(i,j,k) = amrex::exp(I*w_c*dt); + } } }); } } -void -PsatdAlgorithmPml::CurrentCorrection (SpectralFieldData& /*field_data*/) +void PsatdAlgorithmPml::CurrentCorrection (SpectralFieldData& /*field_data*/) { - WARPX_ABORT_WITH_MESSAGE( - "Current correction not implemented for PML PSATD"); + WARPX_ABORT_WITH_MESSAGE("Current correction not implemented for PML PSATD"); } -void -PsatdAlgorithmPml::VayDeposition (SpectralFieldData& /*field_data*/) +void PsatdAlgorithmPml::VayDeposition (SpectralFieldData& /*field_data*/) { - WARPX_ABORT_WITH_MESSAGE( - "Vay deposition not implemented for PML PSATD"); + WARPX_ABORT_WITH_MESSAGE("Vay deposition not implemented for PML PSATD"); } #endif // WARPX_USE_PSATD diff --git a/Source/FieldSolver/SpectralSolver/SpectralSolver.cpp b/Source/FieldSolver/SpectralSolver/SpectralSolver.cpp index 362af06a354..80fb2c39545 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralSolver.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralSolver.cpp @@ -52,11 +52,11 @@ SpectralSolver::SpectralSolver( // - Select the algorithm depending on the input parameters // Initialize the corresponding coefficients over k space - if (pml) // PSATD equations in the PML region + if (pml) // PSATD or Galilean PSATD equations in the PML region { algorithm = std::make_unique( k_space, dm, m_spectral_index, norder_x, norder_y, norder_z, grid_type, - dt, dive_cleaning, divb_cleaning); + v_galilean, dt, dive_cleaning, divb_cleaning); } else // PSATD equations in the regular domain { From db87b8685f89a011f81b323f12d2391554aaa68d Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Sat, 23 Sep 2023 00:27:50 +0900 Subject: [PATCH 025/110] Clang tidy CI test: add almost all misc-* checks (#4268) --- .clang-tidy | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index 5a3deadf304..42f4fc36065 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -17,15 +17,9 @@ Checks: '-*, google-build-explicit-make-pair, google-build-namespaces, google-global-names-in-headers, - misc-const-correctness, - misc-definitions-in-headers, - misc-misleading-bidirectional, - misc-misleading-identifier, - misc-misplaced-const, - misc-uniqueptr-reset-release, - misc-unused-alias-decls, - misc-unused-parameters, - misc-unused-using-decls, + misc-*, + -misc-no-recursion, + -misc-non-private-member-variables-in-classes, modernize-avoid-bind, modernize-concat-nested-namespaces, modernize-deprecated-headers, From a471915fdd1df3154b316650dfeff54f4f0a1525 Mon Sep 17 00:00:00 2001 From: David Grote Date: Sat, 23 Sep 2023 10:43:51 -0700 Subject: [PATCH 026/110] Use cupy when available with the particle container wrapper (#4203) * Use cupy when available * cupy helper & copy-to-host helper * cupy helper & copy-to-host helper * Fix get_species_charge_sum (revert) * Import Updates * Pass copy_to_host through the particle gather routines * Add documentation for the Python particle gather routines * Fix fetch of particle theta, passing copy_to_host, and use numpy when copy to host * Avoid cupy for copy_to_host - explicit copy in AMReX possible - this way, we can support AoS data copies * Add cupy struct reference --------- Co-authored-by: Axel Huebl --- Python/pywarpx/LoadThirdParty.py | 35 +++ Python/pywarpx/__init__.py | 1 + Python/pywarpx/particle_containers.py | 363 ++++++++++++++++++++------ 3 files changed, 323 insertions(+), 76 deletions(-) create mode 100644 Python/pywarpx/LoadThirdParty.py diff --git a/Python/pywarpx/LoadThirdParty.py b/Python/pywarpx/LoadThirdParty.py new file mode 100644 index 00000000000..ea62d558eeb --- /dev/null +++ b/Python/pywarpx/LoadThirdParty.py @@ -0,0 +1,35 @@ +# This file is part of WarpX. +# +# Authors: Axel Huebl +# License: BSD-3-Clause-LBNL + +from ._libwarpx import libwarpx + + +def load_cupy(): + """ + This is a helper for + https://docs.cupy.dev/en/stable/user_guide/basic.html + """ + amr = libwarpx.amr + status = None + + if amr.Config.have_gpu: + try: + import cupy as cp + xp = cp + # Note: found and will use cupy + except ImportError: + status = "Warning: GPU found but cupy not available! Trying managed memory in numpy..." + import numpy as np + xp = np + if amr.Config.gpu_backend == "SYCL": + status = "Warning: SYCL GPU backend not yet implemented for Python" + import numpy as np + xp = np + + else: + import numpy as np + xp = np + # Note: found and will use numpy + return xp, status diff --git a/Python/pywarpx/__init__.py b/Python/pywarpx/__init__.py index 8b57f60fb79..f89926399ca 100644 --- a/Python/pywarpx/__init__.py +++ b/Python/pywarpx/__init__.py @@ -34,6 +34,7 @@ from .HybridPICModel import hybridpicmodel from .Interpolation import interpolation from .Lasers import lasers +from .LoadThirdParty import load_cupy from .PSATD import psatd from .Particles import newspecies, particles from .WarpX import warpx diff --git a/Python/pywarpx/particle_containers.py b/Python/pywarpx/particle_containers.py index aed05121668..c74549bcdb7 100644 --- a/Python/pywarpx/particle_containers.py +++ b/Python/pywarpx/particle_containers.py @@ -2,12 +2,13 @@ # # This file is part of WarpX. # -# Authors: David Grote, Roelof Groenewald +# Authors: David Grote, Roelof Groenewald, Axel Huebl # # License: BSD-3-Clause-LBNL import numpy as np +from .LoadThirdParty import load_cupy from ._libwarpx import libwarpx @@ -183,40 +184,63 @@ def add_real_comp(self, pid_name, comm=True): ''' self.particle_container.add_real_comp(pid_name, comm) - def get_particle_structs(self, level): + def get_particle_structs(self, level, copy_to_host=False): ''' - This returns a list of numpy arrays containing the particle struct data + This returns a list of numpy or cupy arrays containing the particle struct data on each tile for this process. The particle data is represented as a structured - numpy array and contains the particle 'x', 'y', 'z', and 'idcpu'. + array and contains the particle 'x', 'y', 'z', and 'idcpu'. - The data for the numpy arrays are not copied, but share the underlying - memory buffer with WarpX. The numpy arrays are fully writeable. + Unless copy_to_host is specified, the data for the structs are + not copied, but share the underlying memory buffer with WarpX. The + arrays are fully writeable. + + Note that cupy does not support structs: + https://github.com/cupy/cupy/issues/2031 + and will return arrays of binary blobs for the AoS (DP: "|V24"). If copied + to host via copy_to_host, we correct for the right numpy AoS type. Parameters ---------- level : int - The refinement level to reference + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. Returns ------- - List of numpy arrays + List of arrays The requested particle struct data ''' particle_data = [] for pti in libwarpx.libwarpx_so.WarpXParIter(self.particle_container, level): - aos_arr = np.array(pti.aos(), copy=False) - particle_data.append(aos_arr) + if copy_to_host: + particle_data.append(pti.aos().to_numpy(copy=True)) + else: + if libwarpx.amr.Config.have_gpu: + libwarpx.amr.Print( + "get_particle_structs: cupy does not yet support structs. " + "https://github.com/cupy/cupy/issues/2031" + "Did you mean copy_to_host=True?" + ) + xp, cupy_status = load_cupy() + if cupy_status is not None: + libwarpx.amr.Print(cupy_status) + aos_arr = xp.array(pti.aos(), copy=False) # void blobs for cupy + particle_data.append(aos_arr) return particle_data - def get_particle_arrays(self, comp_name, level): + def get_particle_arrays(self, comp_name, level, copy_to_host=False): ''' - This returns a list of numpy arrays containing the particle array data + This returns a list of numpy or cupy arrays containing the particle array data on each tile for this process. - The data for the numpy arrays are not copied, but share the underlying - memory buffer with WarpX. The numpy arrays are fully writeable. + Unless copy_to_host is specified, the data for the arrays are not + copied, but share the underlying memory buffer with WarpX. The + arrays are fully writeable. Parameters ---------- @@ -224,13 +248,17 @@ def get_particle_arrays(self, comp_name, level): comp_name : str The component of the array data that will be returned - level : int - The refinement level to reference + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. Returns ------- - List of numpy arrays + List of arrays The requested particle array data ''' comp_idx = self.particle_container.get_comp_index(comp_name) @@ -238,101 +266,226 @@ def get_particle_arrays(self, comp_name, level): data_array = [] for pti in libwarpx.libwarpx_so.WarpXParIter(self.particle_container, level): soa = pti.soa() - data_array.append(np.array(soa.GetRealData(comp_idx), copy=False)) + idx = soa.GetRealData(comp_idx) + if copy_to_host: + data_array.append(idx.to_numpy(copy=True)) + else: + xp, cupy_status = load_cupy() + if cupy_status is not None: + libwarpx.amr.Print(cupy_status) + + data_array.append(xp.array(idx, copy=False)) + return data_array - def get_particle_id(self, level=0): + def get_particle_id(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle 'id' + Return a list of numpy or cupy arrays containing the particle 'id' numbers on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle ids ''' - structs = self.get_particle_structs(level) + structs = self.get_particle_structs(level, copy_to_host) return [libwarpx.amr.unpack_ids(struct['cpuid']) for struct in structs] - def get_particle_cpu(self, level=0): + def get_particle_cpu(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle 'cpu' + Return a list of numpy or cupy arrays containing the particle 'cpu' numbers on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle cpus ''' - structs = self.get_particle_structs(level) + structs = self.get_particle_structs(level, copy_to_host) return [libwarpx.amr.unpack_cpus(struct['cpuid']) for struct in structs] - def get_particle_x(self, level=0): + def get_particle_x(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle 'x' + Return a list of numpy or cupy arrays containing the particle 'x' positions on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle x position ''' - structs = self.get_particle_structs(level) + if copy_to_host: + # Use the numpy version of cosine + xp = np + else: + xp, cupy_status = load_cupy() + + structs = self.get_particle_structs(level, copy_to_host) if libwarpx.geometry_dim == '3d' or libwarpx.geometry_dim == '2d': return [struct['x'] for struct in structs] elif libwarpx.geometry_dim == 'rz': - return [struct['x']*np.cos(theta) for struct, theta in zip(structs, self.get_particle_theta())] + theta = self.get_particle_theta(level, copy_to_host) + return [struct['x']*xp.cos(theta) for struct, theta in zip(structs, theta)] elif libwarpx.geometry_dim == '1d': raise Exception('get_particle_x: There is no x coordinate with 1D Cartesian') xp = property(get_particle_x) - def get_particle_y(self, level=0): + def get_particle_y(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle 'y' + Return a list of numpy or cupy arrays containing the particle 'y' positions on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle y position ''' - structs = self.get_particle_structs(level) + if copy_to_host: + # Use the numpy version of sine + xp = np + else: + xp, cupy_status = load_cupy() + + structs = self.get_particle_structs(level, copy_to_host) if libwarpx.geometry_dim == '3d': return [struct['y'] for struct in structs] elif libwarpx.geometry_dim == 'rz': - return [struct['x']*np.sin(theta) for struct, theta in zip(structs, self.get_particle_theta())] + theta = self.get_particle_theta(level, copy_to_host) + return [struct['x']*xp.sin(theta) for struct, theta in zip(structs, theta)] elif libwarpx.geometry_dim == '1d' or libwarpx.geometry_dim == '2d': raise Exception('get_particle_y: There is no y coordinate with 1D or 2D Cartesian') yp = property(get_particle_y) - def get_particle_r(self, level=0): + def get_particle_r(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle 'r' + Return a list of numpy or cupy arrays containing the particle 'r' positions on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle r position ''' - structs = self.get_particle_structs(level) + xp, cupy_status = load_cupy() + + structs = self.get_particle_structs(level, copy_to_host) if libwarpx.geometry_dim == 'rz': return [struct['x'] for struct in structs] elif libwarpx.geometry_dim == '3d': - return [np.sqrt(struct['x']**2 + struct['y']**2) for struct in structs] + return [xp.sqrt(struct['x']**2 + struct['y']**2) for struct in structs] elif libwarpx.geometry_dim == '2d' or libwarpx.geometry_dim == '1d': raise Exception('get_particle_r: There is no r coordinate with 1D or 2D Cartesian') rp = property(get_particle_r) - def get_particle_theta(self, level=0): + def get_particle_theta(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle + Return a list of numpy or cupy arrays containing the particle theta on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle theta position ''' + xp, cupy_status = load_cupy() + if libwarpx.geometry_dim == 'rz': - return self.get_particle_arrays('theta', level) + return self.get_particle_arrays('theta', level, copy_to_host) elif libwarpx.geometry_dim == '3d': - structs = self.get_particle_structs(level) - return [np.arctan2(struct['y'], struct['x']) for struct in structs] + structs = self.get_particle_structs(level, copy_to_host) + return [xp.arctan2(struct['y'], struct['x']) for struct in structs] elif libwarpx.geometry_dim == '2d' or libwarpx.geometry_dim == '1d': raise Exception('get_particle_theta: There is no theta coordinate with 1D or 2D Cartesian') thetap = property(get_particle_theta) - def get_particle_z(self, level=0): + def get_particle_z(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle 'z' + Return a list of numpy or cupy arrays containing the particle 'z' positions on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle z position ''' - structs = self.get_particle_structs(level) + structs = self.get_particle_structs(level, copy_to_host) if libwarpx.geometry_dim == '3d': return [struct['z'] for struct in structs] elif libwarpx.geometry_dim == 'rz' or libwarpx.geometry_dim == '2d': @@ -341,45 +494,101 @@ def get_particle_z(self, level=0): return [struct['x'] for struct in structs] zp = property(get_particle_z) - def get_particle_weight(self, level=0): + def get_particle_weight(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle + Return a list of numpy or cupy arrays containing the particle weight on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle weight ''' - return self.get_particle_arrays('w', level) + return self.get_particle_arrays('w', level, copy_to_host=copy_to_host) wp = property(get_particle_weight) - def get_particle_ux(self, level=0): + def get_particle_ux(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle + Return a list of numpy or cupy arrays containing the particle x momentum on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle x momentum ''' - return self.get_particle_arrays('ux', level) + return self.get_particle_arrays('ux', level, copy_to_host=copy_to_host) uxp = property(get_particle_ux) - def get_particle_uy(self, level=0): + def get_particle_uy(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle + Return a list of numpy or cupy arrays containing the particle y momentum on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle y momentum ''' - return self.get_particle_arrays('uy', level) + return self.get_particle_arrays('uy', level, copy_to_host=copy_to_host) uyp = property(get_particle_uy) - def get_particle_uz(self, level=0): + def get_particle_uz(self, level=0, copy_to_host=False): ''' - - Return a list of numpy arrays containing the particle + Return a list of numpy or cupy arrays containing the particle z momentum on each tile. + Parameters + ---------- + + level : int + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- + + List of arrays + The requested particle z momentum ''' - return self.get_particle_arrays('uz', level) + return self.get_particle_arrays('uz', level, copy_to_host=copy_to_host) uzp = property(get_particle_uz) def get_species_charge_sum(self, local=False): @@ -457,13 +666,13 @@ def get_particle_boundary_buffer_size(self, species_name, boundary, local=False) def get_particle_boundary_buffer_structs(self, species_name, boundary, level): ''' - This returns a list of numpy arrays containing the particle struct data + This returns a list of numpy or cupy arrays containing the particle struct data for a species that has been scraped by a specific simulation boundary. The - particle data is represented as a structured numpy array and contains the + particle data is represented as a structured array and contains the particle 'x', 'y', 'z', and 'idcpu'. - The data for the numpy arrays are not copied, but share the underlying - memory buffer with WarpX. The numpy arrays are fully writeable. + The data for the arrays are not copied, but share the underlying + memory buffer with WarpX. The arrays are fully writeable. Parameters ---------- @@ -500,11 +709,11 @@ def get_particle_boundary_buffer_structs(self, species_name, boundary, level): def get_particle_boundary_buffer(self, species_name, boundary, comp_name, level): ''' - This returns a list of numpy arrays containing the particle array data + This returns a list of numpy or cupy arrays containing the particle array data for a species that has been scraped by a specific simulation boundary. - The data for the numpy arrays are not copied, but share the underlying - memory buffer with WarpX. The numpy arrays are fully writeable. + The data for the arrays are not copied, but share the underlying + memory buffer with WarpX. The arrays are fully writeable. Parameters ---------- @@ -524,6 +733,8 @@ def get_particle_boundary_buffer(self, species_name, boundary, comp_name, level) level : int Which AMR level to retrieve scraped particle data from. ''' + xp, cupy_status = load_cupy() + part_container = self.particle_buffer.get_particle_container( species_name, self._get_boundary_number(boundary) ) @@ -533,14 +744,14 @@ def get_particle_boundary_buffer(self, species_name, boundary, comp_name, level) comp_idx = part_container.num_int_comps() - 1 for ii, pti in enumerate(libwarpx.libwarpx_so.BoundaryBufferParIter(part_container, level)): soa = pti.soa() - data_array.append(np.array(soa.GetIntData(comp_idx), copy=False)) + data_array.append(xp.array(soa.GetIntData(comp_idx), copy=False)) else: mypc = libwarpx.warpx.multi_particle_container() sim_part_container_wrapper = mypc.get_particle_container_from_name(species_name) comp_idx = sim_part_container_wrapper.get_comp_index(comp_name) for ii, pti in enumerate(libwarpx.libwarpx_so.BoundaryBufferParIter(part_container, level)): soa = pti.soa() - data_array.append(np.array(soa.GetRealData(comp_idx), copy=False)) + data_array.append(xp.array(soa.GetRealData(comp_idx), copy=False)) return data_array From 8dcd35cc8bcf4dc262bca6f033f32417c1fe6b6f Mon Sep 17 00:00:00 2001 From: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Date: Mon, 25 Sep 2023 09:29:30 -0700 Subject: [PATCH 027/110] fix distribution map initialization for particle buffers in BTD (#4318) --- Source/Diagnostics/BTDiagnostics.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp index b79132e83db..34bd720be96 100644 --- a/Source/Diagnostics/BTDiagnostics.cpp +++ b/Source/Diagnostics/BTDiagnostics.cpp @@ -1450,8 +1450,9 @@ BTDiagnostics::PrepareParticleDataForOutput() { // Check if the zslice is in domain const bool ZSliceInDomain = GetZSliceInDomainFlag (i_buffer, lev); - if (ZSliceInDomain) { - if ( m_totalParticles_in_buffer[i_buffer][i] == 0) { + const bool kindexInSnapshotBox = GetKIndexInSnapshotBoxFlag (i_buffer, lev); + if (kindexInSnapshotBox) { + if ( buffer_empty(i_buffer) ) { if (!m_do_back_transformed_fields || m_varnames_fields.empty()) { if ( m_buffer_flush_counter[i_buffer] == 0) { DefineSnapshotGeometry(i_buffer, lev); From 9b5d93333422c4bf8024416c802554f23044dd64 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 25 Sep 2023 18:37:42 +0200 Subject: [PATCH 028/110] Doc: Perlmutter (NERSC) E4S 23.05 Boost & CCache (#4302) Update the Boost and CCache modules to use the latest E4S (23.05) GCC/11.2.0 stack on Perlmutter. --- .../perlmutter-nersc/perlmutter_cpu_warpx.profile.example | 4 ++-- .../perlmutter-nersc/perlmutter_gpu_warpx.profile.example | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example b/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example index ab5a0c963b8..a722c9a9104 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example +++ b/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example @@ -11,7 +11,7 @@ module load cmake/3.22.0 module load cray-fftw/3.3.10.3 # optional: for QED support with detailed tables -export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.80.0-ute7nbx4wmfrw53q7hyh6wyezub5ljry +export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3 # optional: for openPMD and PSATD+RZ support module load cray-hdf5-parallel/1.12.2.1 @@ -26,7 +26,7 @@ export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master/li export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/lapackpp-master/lib64:$LD_LIBRARY_PATH # optional: CCache -export PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/ccache-4.5.1-ybl7xefvggn6hov4dsdxxnztji74tolj/bin:$PATH +export PATH=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8-eqk2d3bipbpkgwxq7ujlp6mckwal4dwz/bin:$PATH # optional: for Python bindings or libEnsemble module load cray-python/3.9.13.1 diff --git a/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example b/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example index 09d6a61b458..629a2073a9e 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example +++ b/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example @@ -9,7 +9,7 @@ if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in yo module load cmake/3.22.0 # optional: for QED support with detailed tables -export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-22.11/83104/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.80.0-ute7nbx4wmfrw53q7hyh6wyezub5ljry +export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3 # optional: for openPMD and PSATD+RZ support module load cray-hdf5-parallel/1.12.2.1 @@ -24,7 +24,7 @@ export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH # optional: CCache -export PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/ccache-4.5.1-ybl7xefvggn6hov4dsdxxnztji74tolj/bin:$PATH +export PATH=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8-eqk2d3bipbpkgwxq7ujlp6mckwal4dwz/bin:$PATH # optional: for Python bindings or libEnsemble module load cray-python/3.9.13.1 From 751b6d5b4b83102cc1d1543e2b28d410a9ef60d4 Mon Sep 17 00:00:00 2001 From: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Mon, 25 Sep 2023 09:42:29 -0700 Subject: [PATCH 029/110] Add `get_charge_density` to python `WarpXParticleContainer` (#4300) * added `get_charge_density` to pybind `WarpXParticleContainer` and exposed useful fields.py functionality directly for a mf * reduce code duplication in `_WarpXMultiFABWrapper`; changed arguments to strings from f-strings * add `mf_name` argument option to `_MultiFABWrapper` instead of creating a child class * add back deleted docstring --- Python/pywarpx/fields.py | 114 ++++++++++-------- .../Particles/WarpXParticleContainer.cpp | 7 ++ 2 files changed, 69 insertions(+), 52 deletions(-) diff --git a/Python/pywarpx/fields.py b/Python/pywarpx/fields.py index ab66a464628..fd647287a53 100644 --- a/Python/pywarpx/fields.py +++ b/Python/pywarpx/fields.py @@ -65,9 +65,14 @@ class _MultiFABWrapper(object): Parameters ---------- + mf: MultiFab + The Multifab that is wrapped. + mf_name: string The name of the MultiFab to be accessed, the tag specified when the - MultiFab is allocated + MultiFab is allocated. The Multifab will be accessed anew from WarpX + everytime it is called if this argument is given instead of directly + providing the Multifab. level: int The refinement level @@ -77,7 +82,8 @@ class _MultiFABWrapper(object): Note that when True, the first n-ghost negative indices will refer to the lower ghost cells. """ - def __init__(self, mf_name, level, include_ghosts=False): + def __init__(self, mf=None, mf_name=None, level=0, include_ghosts=False): + self._mf = mf self.mf_name = mf_name self.level = level self.include_ghosts = include_ghosts @@ -99,10 +105,13 @@ def __iter__(self): @property def mf(self): - # Always fetch this anew in case the C++ MultiFab is recreated - warpx = libwarpx.libwarpx_so.get_instance() - # All MultiFab names have the level suffix - return warpx.multifab(f'{self.mf_name}[level={self.level}]') + if self._mf is not None: + return self._mf + else: + # Always fetch this anew in case the C++ MultiFab is recreated + warpx = libwarpx.libwarpx_so.get_instance() + # All MultiFab names have the level suffix + return warpx.multifab(f'{self.mf_name}[level={self.level}]') @property def shape(self): @@ -544,143 +553,144 @@ def max_index(self, *args): def norm0(self, *args): return self.mf.norm0(*args) + def ExWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_aux[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_aux[x]', level=level, include_ghosts=include_ghosts) def EyWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_aux[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_aux[y]', level=level, include_ghosts=include_ghosts) def EzWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_aux[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_aux[z]', level=level, include_ghosts=include_ghosts) def BxWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_aux[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_aux[x]', level=level, include_ghosts=include_ghosts) def ByWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_aux[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_aux[y]', level=level, include_ghosts=include_ghosts) def BzWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_aux[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_aux[z]', level=level, include_ghosts=include_ghosts) def JxWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp[x]', level=level, include_ghosts=include_ghosts) def JyWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp[y]', level=level, include_ghosts=include_ghosts) def JzWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp[z]', level=level, include_ghosts=include_ghosts) def ExFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_fp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_fp[x]', level=level, include_ghosts=include_ghosts) def EyFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_fp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_fp[y]', level=level, include_ghosts=include_ghosts) def EzFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_fp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_fp[z]', level=level, include_ghosts=include_ghosts) def BxFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_fp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_fp[x]', level=level, include_ghosts=include_ghosts) def ByFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_fp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_fp[y]', level=level, include_ghosts=include_ghosts) def BzFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_fp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_fp[z]', level=level, include_ghosts=include_ghosts) def JxFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp[x]', level=level, include_ghosts=include_ghosts) def JyFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp[y]', level=level, include_ghosts=include_ghosts) def JzFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp[z]', level=level, include_ghosts=include_ghosts) def RhoFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'rho_fp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='rho_fp', level=level, include_ghosts=include_ghosts) def PhiFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'phi_fp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='phi_fp', level=level, include_ghosts=include_ghosts) def FFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'F_fp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='F_fp', level=level, include_ghosts=include_ghosts) def GFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'G_fp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='G_fp', level=level, include_ghosts=include_ghosts) def AxFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'vector_potential_fp_nodal[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='vector_potential_fp_nodal[x]', level=level, include_ghosts=include_ghosts) def AyFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'vector_potential_fp_nodal[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='vector_potential_fp_nodal[y]', level=level, include_ghosts=include_ghosts) def AzFPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'vector_potential_fp_nodal[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='vector_potential_fp_nodal[z]', level=level, include_ghosts=include_ghosts) def ExCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_cp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_cp[x]', level=level, include_ghosts=include_ghosts) def EyCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_cp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_cp[y]', level=level, include_ghosts=include_ghosts) def EzCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Efield_cp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Efield_cp[z]', level=level, include_ghosts=include_ghosts) def BxCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_cp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_cp[x]', level=level, include_ghosts=include_ghosts) def ByCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_cp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_cp[y]', level=level, include_ghosts=include_ghosts) def BzCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'Bfield_cp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='Bfield_cp[z]', level=level, include_ghosts=include_ghosts) def JxCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_cp[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_cp[x]', level=level, include_ghosts=include_ghosts) def JyCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_cp[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_cp[y]', level=level, include_ghosts=include_ghosts) def JzCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_cp[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_cp[z]', level=level, include_ghosts=include_ghosts) def RhoCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'rho_cp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='rho_cp', level=level, include_ghosts=include_ghosts) def FCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'F_cp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='F_cp', level=level, include_ghosts=include_ghosts) def GCPWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'G_cp', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='G_cp', level=level, include_ghosts=include_ghosts) def EdgeLengthsxWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'm_edge_lengths[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='m_edge_lengths[x]', level=level, include_ghosts=include_ghosts) def EdgeLengthsyWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'm_edge_lengths[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='m_edge_lengths[y]', level=level, include_ghosts=include_ghosts) def EdgeLengthszWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'm_edge_lengths[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='m_edge_lengths[z]', level=level, include_ghosts=include_ghosts) def FaceAreasxWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'm_face_areas[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='m_face_areas[x]', level=level, include_ghosts=include_ghosts) def FaceAreasyWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'm_face_areas[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='m_face_areas[y]', level=level, include_ghosts=include_ghosts) def FaceAreaszWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'm_face_areas[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='m_face_areas[z]', level=level, include_ghosts=include_ghosts) def JxFPAmpereWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp_ampere[x]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp_ampere[x]', level=level, include_ghosts=include_ghosts) def JyFPAmpereWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp_ampere[y]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp_ampere[y]', level=level, include_ghosts=include_ghosts) def JzFPAmpereWrapper(level=0, include_ghosts=False): - return _MultiFABWrapper(mf_name=f'current_fp_ampere[z]', level=level, include_ghosts=include_ghosts) + return _MultiFABWrapper(mf_name='current_fp_ampere[z]', level=level, include_ghosts=include_ghosts) def ExFPPMLWrapper(level=0, include_ghosts=False): return _MultiFABWrapper(mf_name='pml_E_fp[x]', level=level, include_ghosts=include_ghosts) diff --git a/Source/Python/Particles/WarpXParticleContainer.cpp b/Source/Python/Particles/WarpXParticleContainer.cpp index b5d3b16269c..386225c8968 100644 --- a/Source/Python/Particles/WarpXParticleContainer.cpp +++ b/Source/Python/Particles/WarpXParticleContainer.cpp @@ -115,5 +115,12 @@ void init_WarpXParticleContainer (py::module& m) }, py::arg("rho"), py::arg("lev") ) + .def("get_charge_density", + [](WarpXParticleContainer& pc, int lev, bool local) + { + return pc.GetChargeDensity(lev, local); + }, + py::arg("lev"), py::arg("local") + ) ; } From 02dbfb5f4a24836d6da1ffeb1ce0887e0edd2170 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 25 Sep 2023 20:53:19 +0200 Subject: [PATCH 030/110] AMReX: Weekly Update (#4322) --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- run_test.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 0f3adb8593a..03a3044848f 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 && cd - + cd ../amrex && git checkout --detach 2e99628138df3b5b0ecf50b0c1201d5547f821a0 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index c20dc5ed93c..c6caa6b915c 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 +branch = 2e99628138df3b5b0ecf50b0c1201d5547f821a0 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index a3a04f609b2..da9d8398869 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 +branch = 2e99628138df3b5b0ecf50b0c1201d5547f821a0 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index fb80973c96b..3e0f044f020 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -257,7 +257,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4" +set(WarpX_amrex_branch "2e99628138df3b5b0ecf50b0c1201d5547f821a0" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/run_test.sh b/run_test.sh index e24095ee8cf..81237fb6971 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 && cd - +cd amrex && git checkout --detach 2e99628138df3b5b0ecf50b0c1201d5547f821a0 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From a5c1170fde7f01697a4ebdf8b8a2c6776d851abe Mon Sep 17 00:00:00 2001 From: aveksler1 <124003120+aveksler1@users.noreply.github.com> Date: Mon, 25 Sep 2023 16:43:09 -0700 Subject: [PATCH 031/110] Particle fields diagnostic support for PICMI (#4262) * added particle field interface in picmi FieldDiagnostic * added documentation to doc string. * undo line deletion * catch unexpected keywords * simplified logic * fixed bug and removed unnecessary error check * passing in particle field diagnostic information using a class defined in picmi rather than a dictionary * Added ParticleFieldDiagnostic class to be used in FieldDiagnostics * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * error checking on repeat name required if not using a dictionary * fixed docstring, changed var name to be more explicit, removed error check by using default list, bug fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- Python/pywarpx/picmi.py | 62 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index ee4422daca6..80efe3ea804 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -8,6 +8,7 @@ """Classes following the PICMI standard """ +from dataclasses import dataclass import os import re @@ -1941,6 +1942,35 @@ def set_write_dir(self): self.diagnostic.file_prefix = os.path.join(write_dir, file_prefix) +@dataclass(frozen=True) +class ParticleFieldDiagnostic: + """ + Class holding particle field diagnostic information to be processed in FieldDiagnostic below. + + Parameters + ---------- + name: str + Name of particle field diagnostic. If a component of a vector field, for the openPMD viewer + to treat it as a vector, the coordinate (i.e x, y, z) should be the last character. + + func: parser str + Parser function to be calculated for each particle per cell. Should be of the form + f(x,y,z,ux,uy,uz) + + do_average: (0 or 1) optional, default 1 + Whether the diagnostic is averaged by the sum of particle weights in each cell + + filter: parser str, optional + Parser function returning a boolean for whether to include a particle in the diagnostic. + If not specified, all particles will be included. The function arguments are the same + as the `func` above. + """ + name: str + func: str + do_average: int = 1 + filter: str = None + + class FieldDiagnostic(picmistandard.PICMI_FieldDiagnostic, WarpXDiagnosticBase): """ See `Input Parameters `_ for more information. @@ -1970,6 +2000,15 @@ class FieldDiagnostic(picmistandard.PICMI_FieldDiagnostic, WarpXDiagnosticBase): warpx_dump_rz_modes: bool, optional Flag whether to dump the data for all RZ modes + + warpx_particle_fields_to_plot: list of ParticleFieldDiagnostics + List of ParticleFieldDiagnostic classes to install in the simulation. Error + checking is handled in the class itself. + + warpx_particle_fields_species: list of strings, optional + Species for which to calculate particle_fields_to_plot functions. Fields will + be calculated separately for each specified species. If not passed, default is + all of the available particle species. """ def init(self, kw): @@ -1983,6 +2022,8 @@ def init(self, kw): self.file_prefix = kw.pop('warpx_file_prefix', None) self.file_min_digits = kw.pop('warpx_file_min_digits', None) self.dump_rz_modes = kw.pop('warpx_dump_rz_modes', None) + self.particle_fields_to_plot = kw.pop('warpx_particle_fields_to_plot', []) + self.particle_fields_species = kw.pop('warpx_particle_fields_species', None) def initialize_inputs(self): @@ -2060,6 +2101,27 @@ def initialize_inputs(self): fields_to_plot.sort() self.diagnostic.fields_to_plot = fields_to_plot + particle_fields_to_plot_names = list() + for pfd in self.particle_fields_to_plot: + if pfd.name in particle_fields_to_plot_names: + raise Exception('A particle fields name can not be repeated.') + particle_fields_to_plot_names.append(pfd.name) + self.diagnostic.__setattr__( + f'particle_fields.{pfd.name}(x,y,z,ux,uy,uz)', pfd.func + ) + self.diagnostic.__setattr__( + f'particle_fields.{pfd.name}.do_average', pfd.do_average + ) + self.diagnostic.__setattr__( + f'particle_fields.{pfd.name}.filter(x,y,z,ux,uy,uz)', pfd.filter + ) + + # --- Convert to a sorted list so that the order + # --- is the same on all processors. + particle_fields_to_plot_names.sort() + self.diagnostic.particle_fields_to_plot = particle_fields_to_plot_names + self.diagnostic.particle_fields_species = self.particle_fields_species + self.diagnostic.plot_raw_fields = self.plot_raw_fields self.diagnostic.plot_raw_fields_guards = self.plot_raw_fields_guards self.diagnostic.plot_finepatch = self.plot_finepatch From 8330e27430ba038172f4b3e2fe8a1ab42ef5ea9a Mon Sep 17 00:00:00 2001 From: Marco Garten Date: Tue, 26 Sep 2023 02:05:06 +0200 Subject: [PATCH 032/110] Add particle sorting parameters to pywarpx (#4323) Added parameters: - warpx.sort_intervals - warpx.sort_particles_for_deposition - warpx.sort_idx_type - warpx.sort_bin_size --- Python/pywarpx/picmi.py | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index 80efe3ea804..9e88da7c66e 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -1708,6 +1708,28 @@ class Simulation(picmistandard.PICMI_Simulation): The domain will be chopped into the exact number of pieces in each dimension as specified by this parameter. https://warpx.readthedocs.io/en/latest/usage/parameters.html#distribution-across-mpi-ranks-and-parallelization https://warpx.readthedocs.io/en/latest/usage/domain_decomposition.html#simple-method + + warpx_sort_intervals: string, optional (defaults: -1 on CPU; 4 on GPU) + Using the Intervals parser syntax, this string defines the timesteps at which particles are sorted. If <=0, do not sort particles. + It is turned on on GPUs for performance reasons (to improve memory locality). + + warpx_sort_particles_for_deposition: bool, optional (default: true for the CUDA backend, otherwise false) + This option controls the type of sorting used if particle sorting is turned on, i.e. if sort_intervals is not <=0. + If `true`, particles will be sorted by cell to optimize deposition with many particles per cell, in the order `x` -> `y` -> `z` -> `ppc`. + If `false`, particles will be sorted by bin, using the sort_bin_size parameter below, in the order `ppc` -> `x` -> `y` -> `z`. + `true` is recommended for best performance on NVIDIA GPUs, especially if there are many particles per cell. + + warpx_sort_idx_type: list of int, optional (default: 0 0 0) + This controls the type of grid used to sort the particles when sort_particles_for_deposition is true. + Possible values are: + idx_type = {0, 0, 0}: Sort particles to a cell centered grid, + idx_type = {1, 1, 1}: Sort particles to a node centered grid, + idx_type = {2, 2, 2}: Compromise between a cell and node centered grid. + In 2D (XZ and RZ), only the first two elements are read. In 1D, only the first element is read. + + warpx_sort_bin_size: list of int, optional (default 1 1 1) + If `sort_intervals` is activated and `sort_particles_for_deposition` is false, particles are sorted in bins of `sort_bin_size` cells. + In 2D, only the first two elements are read. """ # Set the C++ WarpX interface (see _libwarpx.LibWarpX) as an extension to @@ -1746,6 +1768,10 @@ def init(self, kw): self.amrex_use_gpu_aware_mpi = kw.pop('warpx_amrex_use_gpu_aware_mpi', None) self.zmax_plasma_to_compute_max_step = kw.pop('warpx_zmax_plasma_to_compute_max_step', None) self.compute_max_step_from_btd = kw.pop('warpx_compute_max_step_from_btd', None) + self.sort_intervals = kw.pop('warpx_sort_intervals', None) + self.sort_particles_for_deposition = kw.pop('warpx_sort_particles_for_deposition', None) + self.sort_idx_type = kw.pop('warpx_sort_idx_type', None) + self.sort_bin_size = kw.pop('warpx_sort_bin_size', None) self.collisions = kw.pop('warpx_collisions', None) self.embedded_boundary = kw.pop('warpx_embedded_boundary', None) @@ -1774,6 +1800,11 @@ def initialize_inputs(self): pywarpx.warpx.zmax_plasma_to_compute_max_step = self.zmax_plasma_to_compute_max_step pywarpx.warpx.compute_max_step_from_btd = self.compute_max_step_from_btd + pywarpx.warpx.sort_intervals = self.sort_intervals + pywarpx.warpx.sort_particles_for_deposition = self.sort_particles_for_deposition + pywarpx.warpx.sort_idx_type = self.sort_idx_type + pywarpx.warpx.sort_bin_size = self.sort_bin_size + pywarpx.algo.current_deposition = self.current_deposition_algo pywarpx.algo.charge_deposition = self.charge_deposition_algo pywarpx.algo.field_gathering = self.field_gathering_algo From f52205bd770978de7a39ab11e5cffd31f36ccce4 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 28 Sep 2023 01:04:18 +0200 Subject: [PATCH 033/110] Glossary: CEX, SEE (#4325) * Glossary: CEX, SEE Add new terms to our glossary for charge-exchange collisions and secondary electron emission. * plural collisions Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> --------- Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> --- Docs/source/glossary.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Docs/source/glossary.rst b/Docs/source/glossary.rst index cb376af481f..131b3a63ae7 100644 --- a/Docs/source/glossary.rst +++ b/Docs/source/glossary.rst @@ -18,6 +18,7 @@ Abbreviations * **BC:** boundary condition (of a simulation) * **BCK:** `Benkler-Chavannes-Kuster `__ method, a stabilization technique for small cells in the electromagnetic solver * **BTD:** backtransformed diagnosics, a method to collect data for analysis from a *boosted frame* simulation +* **CEX:** charge-exchange collisions * **CFL:** the Courant-Friedrichs-Lewy condition, a numerical parameter for the numerical convergence of PDE solvers * **CI:** continuous integration, automated tests that we perform before a proposed code-change is accepted; see PR * **CPU:** `central processing unit `__, we usual mean a socket or generally the host-side of a computer (compared to the accelerator, e.g. GPU) @@ -57,6 +58,7 @@ Abbreviations * **RPA:** radiation-pressure acceleration (of protons/ions), e.g. hole-boring (HB) or light-sail (LS) acceleration * **RZ:** for the coordinate system ``r-z`` in cylindrical geometry; we use "RZ" when we refer to quasi-cylindrical geometry, decomposed in azimuthal modes (see details `here `__) * **SENSEI:** `Scalable in situ analysis and visualization `__, light weight framework for in situ data analysis offering access to multiple visualization and analysis backends +* **SEE:** secondary electron emission * **TNSA:** target-normal sheet acceleration (of protons/ions) Terms From 9bf35a1416f5425da3c98107359f193f62c82c8d Mon Sep 17 00:00:00 2001 From: Grant Johnson <69021085+johnson452@users.noreply.github.com> Date: Thu, 28 Sep 2023 18:44:00 -0700 Subject: [PATCH 034/110] Add fluids in WarpX (#3991) * Added: new fluid-langmuir test * Implemented MultiFluidContainer * Implement WarpXFluidContainter * Fix compilation errors * Call AllocData in WarpX.cpp * Fix allocation of fluid MultiFabs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix example * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix compilation error for Evolve * Cleanup include statements * Fix compilation in 2D * Add fluid initialization * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fill guard cells * Make the code work for any dimension * Fix compilation error * Add call to fluids' evolve/deposition * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added fluid contribution to the current deposition * Fix calls to `FillBoundary` * Fixed calculation of J to properly handle nodal versus CC values * Used different boxes for jx jy jz * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed intial index * Minor Indentation Fixes * Split `DepositCurrent` into 2 separate MFIter loops * Additional comments and cleanup * Prevent double counting of J at the boundaries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prepare gather and push * Filled out GatherAndPush function for fluids; currently not working * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed bug with fluid source * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Corrected speed of light factor in the momentum * Added analysis script for 3d langmuir fluid test * Fixed typo * Added docstrings, cleanedup files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added in MUSCL-Handcock Fluid Advection Evolution, currently nan's due to issue in the x-plane update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleaup WarpXFluidContainer.h * Currected flux jacobian to SI units and added fillboundary commands * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bug Fix: incorrect velocity being used * Added Fluid Rho Deposition * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add separate .H file for helper functions * Extended the langmuir fluid test case to examine J and rho aswell * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allocate multi-component arrays * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Regression/WarpX-tests.ini Co-authored-by: Remi Lehe * Update Source/Fluids/MultiFluidContainer.cpp Co-authored-by: Remi Lehe * Update Source/Fluids/MultiFluidContainer.H Co-authored-by: Remi Lehe * Update Source/Fluids/MultiFluidContainer.H Co-authored-by: Remi Lehe * Update Source/Fluids/MultiFluidContainer_fwd.H Co-authored-by: Remi Lehe * Update Source/Fluids/WarpXFluidContainer.cpp Co-authored-by: Remi Lehe * Update Source/Fluids/WarpXFluidContainer_fwd.H Co-authored-by: Remi Lehe * Simplify calculation in MUSCL-Handcock, Undo removing DTtype inlcude * Changed Q_midpoint staggering to be correctly placed * Updated the checksum test with a new json file * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated functions which changed due to pull from main * Updated the fluid test to have the proper J time centering * Rewrote MusclHandcock to avoid MPI comms and temp variable creation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add instrument fluid code with profiling annotations * Reverted to faster Musclhandcock with comms * Eliminate unnessessary communications in MUSCL Handcock scheme * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Some variable name cleanup, optimizing A(muscl)-array calc * Removed unused variables, hopefully fixed Azure pipeline for the Langmuir_fluid_multi * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bugfixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bugfixes * Bugfixes for 3D langmuir fluid test on azure * Added 2D (XZ) fluids * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added 1D (z) to fluids * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added RZ to fluids * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Uses correct velocities in the fluxes * Fixed bug for selecting the coordinates with V_calc() in fluids * Formatting * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed json after bugfix * Fixed 1D json from azure pipeline data * Bugfix: Correctly limits domains for RZ fluids(), still a bug in RZ implementation for Jz(r=0, t=0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bugfix: fixed small error in fluid-RZ SA/Volume at the outermost cell * Attempted patch for RZ * Minor Cleanup * Bugfix: Fixed volume element selection * Added: RZ centrifugal force term * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Ad-hoc fix to RZ, and print diangostics * Moved ad-hoc fix onto the temp variables, corrected mangitude * Bug fix: RZ volume applied at the wrong place, removed ad-hoc fix * Removed last of the ad-hoc code * Fixed for new plasmainjector * Minor Cleanup * FIX: Corrected CI crash for Langmuir_fluid_Xd tests by passing geom to WarpXFluidContainer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added RZ checksum, removed commented code * Bugfix for RZ, at the origin, also changed rho output to be called simular to current density * Updated RZ checksum * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bugfix/improvement: RZ test now correctly outputs and tests rho, J in addition to the electric field * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Appleclang compiler config, shows warnings when compiling with appleclang * Updated fluid-1d test to also check for rho, Jz * Updated fluid-2d test to also check for rho, Jz * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated Checksum with Azure Pipeline data * clang-tidy cleanup * Cleanup * Cleanup Flux-Jacobian Calculation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add moving window code for fluids * Fixed issues introduced by changing InitData() signerature * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Simplified Communications in WarpXFluidContainer * Minor cleanup of unused variables * Minor bug fis in Moving window, Added: non-periodic copy-BC to fluids * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor cosmetic changes * Fix compilation issue with single-precision particles * Template Higuera-Cary pusher on the type of variables * Use simple function to convert to nodal box * Simplify injection of plasma * Positivity and monotonicity preserving limiter added * Changed the default average routine to minmod, which is more diffusive but still second order TVD, this significantly reduced numerical oscillations for a0 ~ 15 1D fluid WFA sims * Added alternative options to take averages for MUSCL-Handcock * Added a working fluid-envelope model * Added boosted frame capabilities for the fluid * Minor Cleanup * Minor Cleanup on 1D positivity limiter * Major cleanup for 2D, RZ, and 3D MUSCL positivity limiters * Fixes to the boosted frame simulation * Lorentz Transform the envelope field * Minor fix for envelope lorentz transform * Protection against N = 0 * Minor bug fixes for zero density check * Bugfix: Improper check of N = 0, updated * Bugfix: Fixed Lorentz transforms in the fluids-InitData() * Bugfix: Small values of N were causing gamma to have unphysical values; added a check * Mergewq * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improvment: calc of gamma, added additional slope-limiting utilities * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change MUSCL scheme to a primative varibale model * Fixed RZ for primative variable sources * Updated tests to match primative vars * Small Bugfix * Added a 1D WFA physical_app + test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor fixes to pass CI * Add include statement for fluid * Use temporary variable gamma_boost in device functions * Add missing .H file for Python compilations * Moved WarpX::beta_boost out of device functions * CI Fixes * More CI fixes for device-host issues * Cleanup of some var names * Minor Cleanup * Removed unessesary autos * Fluid Theory Docs * Checks that there is only the m = 0 azimuthal mode, removed TODOs and declared RZ bound default behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed indentation * Fixed Warning about azimuthal modes * Added documentation for fluid input parameters * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed per Remi's partial review, adds checks for do_not_push/gather * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed bug in matrix(J), updated all fluid tests accordingly * RZ iff for centrigual forces * Changed variable names * Updated the 1d_laser_acc case to include the full laser model in the theory. * Changed names of c_{x,y,z} to dt/d{x,y,z} * Code cleanup, mostly of comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Large change: Rewrote MUSCLadvection() to simplify readability * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Minor change to positivity limiter function implentation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup flux and velocity functions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup for inputs to ave by definind diff functions, fixed RZ boundary problem at r-max * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup of the flux functions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed some unused util functions * Simplified linear slope calc function * Reworded docs to not inlcude contact info * New CI test for boosted WFA with fluids * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added current and charge deposition for Hybrid-Ohm and electrostatic solvers * Fixed docs, removed extra .get() on rho/J * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Declare myfl as private * Add flag do_fluid_species * New function to extract species properties * Update injection_style use * Remove PlasmaInjector in WarpXFluidContainer * Remove number of particle per cell from inputs scripts * Add missing files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix GPU error * Update call to extract particle charge and mass * Update tests * Clean up code * Avoid life-time issues with parsers * Avoid additional life-time issues * Added string that we don't support mesh refinement * Fixed Clangtidy issue, added warning that fluids only work with one mesh refinement level * Minor cleanup * Fixed build issue --------- Co-authored-by: Remi Lehe Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Edoardo Zoni --- CMakeLists.txt | 2 + Docs/source/index.rst | 1 + Docs/source/theory/Fluid_Loop.png | Bin 0 -> 300102 bytes Docs/source/theory/cold_fluid_model.rst | 135 ++ Docs/source/usage/parameters.rst | 45 + .../laser_acceleration/analysis_1d_fluids.py | 174 +++ .../analysis_1d_fluids_boosted.py | 173 +++ .../laser_acceleration/inputs_1d_fluids | 72 + .../inputs_1d_fluids_boosted | 79 + Examples/Tests/langmuir_fluids/analysis_1d.py | 137 ++ Examples/Tests/langmuir_fluids/analysis_2d.py | 159 ++ Examples/Tests/langmuir_fluids/analysis_3d.py | 179 +++ Examples/Tests/langmuir_fluids/analysis_rz.py | 174 +++ Examples/Tests/langmuir_fluids/inputs_1d | 72 + Examples/Tests/langmuir_fluids/inputs_2d | 69 + Examples/Tests/langmuir_fluids/inputs_3d | 74 + Examples/Tests/langmuir_fluids/inputs_rz | 72 + .../benchmarks_json/Langmuir_fluid_1D.json | 7 + .../benchmarks_json/Langmuir_fluid_2D.json | 9 + .../benchmarks_json/Langmuir_fluid_RZ.json | 10 + .../benchmarks_json/Langmuir_fluid_multi.json | 15 + .../LaserAcceleration_1d_fluid.json | 15 + .../LaserAcceleration_1d_fluid_boosted.json | 14 + Regression/WarpX-tests.ini | 108 ++ .../ComputeDiagFunctors/RhoFunctor.cpp | 6 + Source/Evolve/WarpXEvolve.cpp | 8 + Source/FieldSolver/ElectrostaticSolver.cpp | 6 + .../FieldSolver/WarpXPushFieldsHybridPIC.cpp | 9 + Source/Fluids/CMakeLists.txt | 8 + Source/Fluids/Make.package | 4 + Source/Fluids/MultiFluidContainer.H | 78 + Source/Fluids/MultiFluidContainer.cpp | 73 + Source/Fluids/MultiFluidContainer_fwd.H | 12 + Source/Fluids/MusclHancockUtils.H | 484 ++++++ Source/Fluids/WarpXFluidContainer.H | 186 +++ Source/Fluids/WarpXFluidContainer.cpp | 1353 +++++++++++++++++ Source/Fluids/WarpXFluidContainer_fwd.H | 12 + Source/Initialization/PlasmaInjector.H | 7 - Source/Initialization/PlasmaInjector.cpp | 280 +--- Source/Make.WarpX | 1 + Source/Particles/MultiParticleContainer.H | 5 - .../Pusher/UpdateMomentumHigueraCary.H | 52 +- Source/Python/WarpX.cpp | 2 + Source/Utils/CMakeLists.txt | 1 + Source/Utils/Make.package | 1 + Source/Utils/SpeciesUtils.H | 39 + Source/Utils/SpeciesUtils.cpp | 240 +++ Source/Utils/WarpXMovingWindow.cpp | 38 + Source/WarpX.H | 9 + Source/WarpX.cpp | 34 + 50 files changed, 4461 insertions(+), 282 deletions(-) create mode 100644 Docs/source/theory/Fluid_Loop.png create mode 100644 Docs/source/theory/cold_fluid_model.rst create mode 100755 Examples/Physics_applications/laser_acceleration/analysis_1d_fluids.py create mode 100755 Examples/Physics_applications/laser_acceleration/analysis_1d_fluids_boosted.py create mode 100644 Examples/Physics_applications/laser_acceleration/inputs_1d_fluids create mode 100644 Examples/Physics_applications/laser_acceleration/inputs_1d_fluids_boosted create mode 100755 Examples/Tests/langmuir_fluids/analysis_1d.py create mode 100755 Examples/Tests/langmuir_fluids/analysis_2d.py create mode 100755 Examples/Tests/langmuir_fluids/analysis_3d.py create mode 100755 Examples/Tests/langmuir_fluids/analysis_rz.py create mode 100644 Examples/Tests/langmuir_fluids/inputs_1d create mode 100644 Examples/Tests/langmuir_fluids/inputs_2d create mode 100644 Examples/Tests/langmuir_fluids/inputs_3d create mode 100644 Examples/Tests/langmuir_fluids/inputs_rz create mode 100644 Regression/Checksum/benchmarks_json/Langmuir_fluid_1D.json create mode 100644 Regression/Checksum/benchmarks_json/Langmuir_fluid_2D.json create mode 100644 Regression/Checksum/benchmarks_json/Langmuir_fluid_RZ.json create mode 100644 Regression/Checksum/benchmarks_json/Langmuir_fluid_multi.json create mode 100644 Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid.json create mode 100644 Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid_boosted.json create mode 100644 Source/Fluids/CMakeLists.txt create mode 100644 Source/Fluids/Make.package create mode 100644 Source/Fluids/MultiFluidContainer.H create mode 100644 Source/Fluids/MultiFluidContainer.cpp create mode 100644 Source/Fluids/MultiFluidContainer_fwd.H create mode 100644 Source/Fluids/MusclHancockUtils.H create mode 100644 Source/Fluids/WarpXFluidContainer.H create mode 100644 Source/Fluids/WarpXFluidContainer.cpp create mode 100644 Source/Fluids/WarpXFluidContainer_fwd.H create mode 100644 Source/Utils/SpeciesUtils.H create mode 100644 Source/Utils/SpeciesUtils.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 2501c9dcaff..f9b1a8bbc2b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -372,6 +372,7 @@ if(WarpX_PYTHON) endif() add_subdirectory(Source/ablastr) + if(WarpX_LIB) add_subdirectory(Source/AcceleratorLattice) add_subdirectory(Source/BoundaryConditions) @@ -380,6 +381,7 @@ if(WarpX_LIB) add_subdirectory(Source/Evolve) add_subdirectory(Source/FieldSolver) add_subdirectory(Source/Filter) + add_subdirectory(Source/Fluids) add_subdirectory(Source/Initialization) add_subdirectory(Source/Laser) add_subdirectory(Source/Parallelization) diff --git a/Docs/source/index.rst b/Docs/source/index.rst index 2209b360e26..7fc2a1c5833 100644 --- a/Docs/source/index.rst +++ b/Docs/source/index.rst @@ -116,6 +116,7 @@ Theory theory/input_output theory/collisions theory/kinetic_fluid_hybrid_model + theory/cold_fluid_model Development ----------- diff --git a/Docs/source/theory/Fluid_Loop.png b/Docs/source/theory/Fluid_Loop.png new file mode 100644 index 0000000000000000000000000000000000000000..8d47cd606c5d8ee945d79fea180e6f9ec54a3f48 GIT binary patch literal 300102 zcmb4L1z416*M^}%L?uKJ5Jegkqyz~?x^w6hX+cuj5d=XArMsmWa*!B898pj@hCx!0 z?(YAcLD}8!``q3C+G}Bid7tMzC+>5fbDkksRau6Vn1&b&3yV}v_P#n67U3-{EPNh9 zeBdX=o>E4@H*9BhnR{5p?U!ePe*{?Q$URh2!eR$r6Jp_EU%(BRqu&{!xv2cIBM-TXpekqUsa{3)74f~H3(?Gx8O?WE}=htg|9`tiO z+@)>6HzG$_U1uyT-bVEQ*sKrqhOn?Cu;lLF)qH}zFih}{qHnlmHT?xeZCY-w2{ac; zi{RcWrVf=`fZsRPr!Y4QzD~n0t@MhiG!|L=jEnUnH`V>Sch8kR%|)7wjQTYjJj_W- zhP&_kcC24^|50&!`j&Ihd~&%Z$^#eooCFs3$(J&c4RDtsrC%I*g9$ z3xB-fKVQjG&cr21;)4jUJjFWs@)+VPFU*)?pxHXWPN(>2hqHoCIk9kKHH4hfPo3%X zZGSv)@`~>@R^+9GnFqHte~FWZ3tEp!3*GGvL|ELa;8(D)yVtyc85dqRO0^> z$m$9<^DVy4&ktCC?I3}NE37p}Jud0e8K&e}B6KB*$8t#9zJ!RBfi+Cq+g#aOBNYb1 z#r~;+|6EWa?JQp07u6t^Q0{BL&ZtF^yYV3Ko@C{yrB_F=egN@4iXOUCV|KHtNvwsn z?2Z39!e6`p)XJ}aN^p_fj8Bj0IO`+%OXoH!X%x`-+z+{^*(*LjJ{aC3I4~M=;6O12 zU8GD+EdPp2aQEDQ3H9sYKm8(C2GZ7Rt-qP}dg4szG!bHy{ZY=lh^xOpR|se;lppjb z)9K>j=bDI1>8Cdt-8SX17D*ClXmtS_Xg)SYoDHeGY?txT3UT};80OimBlYL!ejN^T zR*C1hfVvp1&ibDoJ%YSsRNQ_FClP7F6E}HoVQAVc1U~&iYD1UAG+&sA0@_v1+KYqv9_o`VT#& zsR6d^%(aYp%=pWeUl~iD2399c(1h3qn%%GmvwIie;1viUaf8U%T}{INIK{6APEO00 z5BMw1<`-V>Ut6VB<1J`WjejQfsdONxw_B5}Zq}h|WadB6Uu8q+hWt6q{$w=?EXY-~ z!f>gi&YwEyx2u1VuUG?uH;;+tPRoUp2ziW%QNTI-AI&N6$8Zt5QuLbV| z!L@pr1zxFhB0dB%^tCro+FyUM_5XT&{Q=@4r2RjoZGV$hLZ~GNgko){4MGh;>0ujid=@&Pl_y2fyJ@@%AC4**CHZmrd zR3-Pw1mX7ocpBdO-hrZp4Q8Rn}VSWz5Yzk~0;2G~;og@p5?8AEQc!Q9lF3G(+9q-Kx*|FRO~4Y7;5tqiNOs@>PDKp8Cu(%&b6zv@X<9pcf4 z>Iz}tpc2U&XX&ffD-HU$QOKGnVOi;xH1Pmo#=@_tk!;L?vLX8|mj3m$X;~n{UQ}?1 z2F1(d-kaL4ZvPtiu3wzVr&7^Q%GX)Q3Ot`*9`>)rt#0(TSye<_3PC8^#AUC6Ika?YU)y72tvvR^RM6SM_Gj=tKVuIPBlTX_Qo2tR7s3XCBb)cK*cNzXldN zkXP=MT!=0U{};_cbIPIL^P3zjKUtTxh+s_ANJtVQ8@6AvZ)qBswddFeZ3o#N8(_4n z`l|66yEkOQG(`7!bV4oq@Og$(z?I+SfnbpC@mQQr{QaN&BsX6IOeY!_benExrM{!cBds*cG=f7(-pzL`49jdUhPzp3aVQc5d_{UIEB zC}~k!apc&0|F5*-zhD|t2!gFOfkSi|erV-?qu9)I?!T*g7Ep;*_EWg@^j$>ZHf%ss zV4j0rZ}58lDR2Faa1t`-zo>uNy8>08YX8xTmWYx6`2&)#(FV~}H=(3kxQ^i5t{G%^uQwz3UD)%*KjrdL`gvB! z8cAW1~N1n6+Ko92>*69;zUwZ$)Jj{>IrcJ7A7~*W_uK^{8rl znWC3^JV8OHDzN*;xcJP-K$rh76A8*GP`d z00xyvX*@vHO7gV%*uh$pD`f=(esm!!VLatA=aEmGn)fz#2BNIzL=j;dsw#suu!Ko>KBpnk0 zl(cq~=Cnw;NVjG^)u2Y-R!RZ^%<6jGF~d_tfoE}qDr@M_>9~gGQnJ+V6n^5;Y3;ba z*+)lN)OWHJ_LK5>o0@B{S04IjhWRwdqaCF;vypplV<5zO`@Q3y!`$Pi8<&2a>%Zb3 z^9zXHMIl|fGvaXyLn*mV^Uv{}Q!+>Friv)o+nCYr?Esq3Z;qtsSqdKCXwxhaxwogs%E;S_XWq6|={s1lF2wqqZ z)Mi|LH5?Ehy1OS7UFya9+kVr@PC&w*VZ_#-xbq|~W@|UEb)zlK^>t>E zZmT}GBrOG692zAPCDIvkz%lJ29+9`%J-hUeOH!5$h}LCL-?^`_e;LZl!A3>AUeL&g zsBl|i?Zw(f11k#vtR?dW8iF(f5RUZ>3 zxY80pv@>HcGv$pGx{<2=?T38}OfAy?fDJkjvHX=hvz;ce%Ra%>vZ`Hk6b4FjUo>8! zoALtW5~(|`UIQ1)Itcv}wX&zsQQef`2xNF7Rb|QM*C9Y3G>6F;Qn;E*tjU>PPIM;wKL)~@+h%-`=E3*m3ay@rj>ghzghY|UZKX`Q+e-j z{3d3H-+!Uta;&roU~s%{uYc*prw$1XGK-mqJDRPx^A&^kE?p4)d!MqR!q&HEJTT#~ zMo%32Xixe|o7YStDVD&BHPE0QZfGiFm&V^72yK2f|D2o3wiG!jK2m_n2BIulV*7Me7iy8=wtLUjSQyJVfAlhw^=7atoVr4h#Kt< zQxg4N!5CQ6G%zuuTa`l0NEXf<`BE-lA~;zmN5YT)2yV9F%P``1Z zjQx5bgTZzC)MU-qe3+BfTok&0O}1tk&~)*>8ZXx0Yr5tFQUOjXULudGFD;h7Z;5Hi zhfuzgrhKz>JYM1R(mrbKAC6NYZQ`MoP)zNsLpkiLyt_q1eBwIaiCwxorP-yANeP$z zG7suQ6&gSNy}4}eHZrr;Wl{U8QVt(U?26i?mv5Xz>jFq&jxdrk^O!1NdlY>1-eQ@E z;P(KGwxpFFgoA%=1t)(7D7=hwtX-*vfSBk^0wr1_#aX3~l?a!`vku(8es0eFV+`@X z#A{_M$)O<=j{_pBEQv179;4(0N+Eesg*|u1BsZt8TPb0GU;XrEH?GZXx*%30^Jcc} zU2pRWV^CC;9nw&cKGQ%t?xPPk9ag7%<@Ub>_~T~@`X)xC$JY@nIKVnOQ#U%lDZZjd zpVbr6e&LB4&z{5mCw{NiYWd+5SbuLcd#ZQpTmcOCrK%baH9p5tS7p}A{fO*)~0 zm}#IKKb)sEO?&nCOeD<{NIyANn7T2y=0Q+>YYn4Ba5 zCvwpYl3_2Np4r)NEgw+uo??|mv(={(&ct5|OvBX|=Bba$TWNEJ+SCF$0XsIX2Td*?n+o1Q;r1Tl-T;kIjDn--?oGxA7etzS6Z zgQW27_%XQjrN3$JTKnV4NI)tj=1$(n>B;V}K(Ug~fgO&2cp5v$&3m@QV%eWbB{{rw z!q5z>!VuWTUN8#}<6(OgI)=kAIpXBY!{X8&VBxjTCjixmdlp=Wx@%g%dq&l}9diyV zN60taz+)w&lGkVLR>pbnoSp@OqWGj_r#c{A;v{LGa>Wx6X&0pU@YA*lhQXxW#uc`T zJ68MF$n#Aksw(zcDq1T*^t=Zh_qP|12YlLkvIx!Fab-#d${k>vPNqNzshDQqACwBH z)9T&NatQPe$z?VW^2MTO+VkGG^e|^QCQ8Z(C!wSM9$f{SaFMxGcH+u}B%Jxl0~E2u zEYdJ&=TYfvKDYA{X?OABK6IwB#F(DOQYcHKEi27(67N-98fic`q|WS@Flkjn8WOFw zq=je?Thr%Qek>!Vd*ba(L}%194)?pn*fgxmDFwk5eGnl3bf$rdQa+S)HdbF9U#!lv zJ6#{BV{}YVFOH*$#7G8qjERc6(@2fPU-!Lq0)BHIn5t_A#QS3qyZ6OYQ3FrC#~Lzz z#Tp~&VctJa;z*PP{h~pKH=>qObhHl7JOc=#ngdGsbhFoXIypISU%=FK@U~QZT}1G! zdw)b^(YYN}X9_zr$^Orzb)FfNpEQBKo)Zx;CpRdoc!_pq)t8`_JdN1f!UAd@1suM= z92pD54e^NDeKsK+cCP_E5|0rC7}p=1A8~Z`N-Ljl!o_F4Ar$}Vj}{zjF4FhCs6?BVPdx1g41D$_t8SxtX`*!M~gu!hG(Na9-0kekJW>s1@jZ;0r7e zKRYnTrV-_F!ITQXcw%dLI5Gr>uIQ}B$lX&lvP!3Q&NCZno71wkv%ghqz4}zbp6E;b zCld}{CIV(2rrHQJw?8QfRB1s1dFh?(q;+l?0_~J=i8ML%ggc8R#S~q6D=|9;eB8He ziDTjDFuJjtjWG$jlBc-Vsr%em%V1D#R&&a5<`WAbk&TW7`K3V5#kz8hTBKL*G*@Cu zvR?R^)9}H@W#>}0QpX{uVfT?10!qp+fX;AYPtsc^31a}U@}!MPBD}0qp?*EviS7#Y+RsQn!6<+(xjY&=fs${ zyp6<_ZO`B=UnGN9tf&ZE5Qf$KME^OJM&Bx>l+K%-46k{stB$1Q!U5zsi@r`kz!J=j zDWNLl$0+*rP-^SkQ^QLbGXtmtQ75`n7mRam@(m37T7*GUsV>~Kb@TXP{#D9a!*wS)Fb_5 z#Mb|f*MgTnigZo$DcWPBRGvm(YPD2o!#GMG`8(Ah<-|`W@a_sTW?t%(>pN*?AmbTX z7Ll-iHILZaXy1Ua6KDCZe;Hv>{1ZO{Zh_cY1XhfCL~3){Y7%?|@_-gF&yeh!xsC3g zK9z{a70|Agylg*IWd?P_q%QPNt8o{!$ZrVIjOuyRVjK^`PMB41^nQdfsczvBrRQhH zU$GCU2LPJGcH4&ikN)sSW+r#yDS$9aa{;mqYw~q24CZPYHyWk7ct9Q{rFrYOCZf*> zY?6&<$F*lpX=vMRoPXfKG0YPNWS63ICGLdeC+Pxn4q&dbKP+f{^+v%hzUD#8SA?4A zq^vbHJ%-WFbt)m%#iC`ab?hn8iBqFHk-X~04!nJ~ENM;I1#>D&3diDB)h1s@);|S# zK6k{O_}IQn`Mzc|SfEfYlTnlFC$8N|2wJy3XHF3-d+AHKys^1*eAHE@lLnvac}R+I z$YCPCUu##r_Fij9B*p zLmkkK=8>Y~ecS|ewp4P?zr+eG+1Uw#^9v8W`KA$lJQPS};)qe$CbMI;@Igvu&pwC4 zHOzvl4c)8bz=m`YFDp;!|L>wd*&K-;kNZBd29`NOHPe+ zx%RLT2)@6h^C-2k1CTuf^gtJ{^=Csvq7v86HIRXh;5hkOIKYBfBPHfIm|I=$B>bt~ zO?!l^Zp5EHnoiLFm3jtHZf_{ScjkU*cB!Ft1;K^bN5LN7^Netb6$m5{@ji)hRe#WO zzRG~`UL|DB?IgIOFN{%=N5AB@R_brjOKMC$XFB0U_yYz7%GZ zl3925G~fRTnJ}dqEBXdw@9YC~dqSdz&@S)wqaW0Z#!-TZQyx;uyizT^AtRHctvv!c zp*{COM~)6Ag?W<>DTt6icZ-1~m=3~{c4}REp59aiY#9VtsWn|b5u@lSDif_hCz2j9 zOt|sma(CTraJ;<3Ipyl~%YT0=5A$}kNX>+R_Dq$p3DPDJ(lyoQ6$9+c{4pS*Jz7G6b{sh7=}*iz zB@mo%@${ojo%EPzPtMT`X{?(2?h)q8omZENNfoD7MH}Jgwv6a;^tqhZchT6QbnRv@ zkGi&n<5&cy#8L%=26C&&FN=!F{q(#4&=mTgs@rI%*w^@kg${|g=Vzd=&$_(Y5}r~z z)vpAc&>I?dG}1NL7ggndT6O%C;g>XX=rC~gBff-QwUZBIDI#q{=EHpgXQSpockh?rNsaCCegQDC|tX>7Ox-Ksm?b(aS1?f)SWde{VDxu>}Q z1_R&$R(6muniQpnvLxGV-ygd@ z9s-PBGF2W67+wCVyNWlfK!E@)MmL$X{E0!>eaJ3(lT= z4b)-ipI#&RmeKT8_L7N6Hp81YGC%3*f2hTk1u%4th=mI^+R#t1_p4@QTUZ%1MyOAH z4;b(i$-6|FI})%#BYzu{XlCK+oR3a^=$I6-%JR9K34i}u=4ZVa%I#QW{!}!oD)IMA zq?r$IM1^6JBj3x$t}sZL#)$u^3anfpvHXp(6az|^X42&w_^)H;m_Q06satF17>PCZ z)m_PLpj+WQ-9EycXg9$r3CxiB!WzTHk7hzI-|qHW ztQ@Fg(|9{+1~r1$RgHMI>6<|C^d>?$if8Ogn(#-6ISM<4{4JS&s!eqOZN2#(nQ653 z0?Boxuc1&p`2|seL#OWe^@>TsErUqj&b#te4PxO+pII;y@3ea$@q-orDg>4Olb*Lq z98W%!w+pw4OeZ-L$8czlB|6lv!bUu8mlvx>xix$DJN%Bm+t;mr#A{OnJYr76WNTq? zgwFlj_^|i-*O5pSv00zZK5#ri*-XYu&v|G=z57Hk%a0|tcnSnU8G^ z#dw@~7@N@Rbi-P=YPmD>#fh*_Dr}zjXo2|V!XDj*4>>dp&kt2Wt2)H?p8y(nS0lev zIO=xnxicQ*oZ3f2&~i$a6ahamAi$`zlF20UQuq6>t*muowtDuzVC6D4P|T3ibicpb z8ppyF>4ITUt%RJVA1;i@F!)R!{t#?nd(x<``0(B8M_PEp@!@j2Ii2s$cnUo5y|%?n zZ(d|MXFMR99|*ShURh*uauL2gGBVSZxsGoF4MW;O;dSfnA|12>c5&{jh>KoBk816W z;#ZUhiVdCZpZ6e7sx?6C>|Bku%CAAPsX3;?LN3NcANGU%vgDc*gEW+QycPr z;IP{eSJgfL3VFOV4X10~Lx>kpp|GrOk_b7Ub_Xx)tCp^j@6ML$^hg2iVXx+LD~IEPxVhzyn3Li!6V^&AoH} zSBF1M4>ahTn5Vrggt>4iQBHW~@tF;-Dfl0C&O^$$w-E6O>Vx{QJ14}S{fY?(^D8}4x z@5p2MXn(nbqv68^g(rvm+Z_t#Ii8?VpY3FHc(e#RcHSlMpmnHw%qeOMBG)dHS3}>) zL%V+~S7+%rg0!DLNlf@9^8ij~@u!s9L-Oq^Bw3vTrndJ!#Q5#6@OTK(cc_$d`4I^C zHKT)*WQ4}TgFxN#hf^`p#|P^j{>*j$WMar#Kv2Cce^PpR{0rVx*GvkYBj@EZOgluk zzZ}>fTN!RC+_-6nEcrGsp25Z7v(eSXG<_t`Q8YR1T@D{#_cI-`uOb!LJ?(hrqg#&o zYtD`s*@=+;F;a9F6d$5FEc#MJMrm^PyMzoMM5i{zzTT=KVw9~uOEy;&yIp^(<1<`% zQW24LF?}eYX_)f(?aB48+-HsURl_mvn))?|tsKxV3gyXW#FNJz46ks`qihO7datwXg)8zbeR%u0h8j!BWFQ#{cF*@+%fV%@yzGn zXi78bON3MD8o!vn=tkthq-FecU3=HrB+{E+_1mqf&MmMt^R2A02sBBE(inSw(P-sk zIK!yZ3*Z%U2#6K%B2Yj0-eNJllB!|(ndMT$w>s(Md5V&z{=~=;q;4234EUu{f>6wecJk8haR`ki8{B^ z`0mZJEca@tY;spmzR{sfyu7|SVC)qxwr9n8L$;*UZFywa3jk5Lir6lbY?u%FT_ZHM zh36HjaOazWv>YFIkqK?eltV=+N!vr4!@ZEOo~^~{>1eP~u*FxA3T}vuWZYa^#V%^&5hQqC)YN3Z;s5@HSmc9*^gUbzogde{% z>N(HVVYF z$$Alf&-pOXFJ*_%^YMkmIw2BF)w4v}oEVh8B;D#!(9Sm0yt=*?xkgo>TqVQ~+GHCe z+Ya2jwsD95atm4Cx)un8AWCw4Pvp_iw$N~VRE_U@w4OShJCALie|KVZ;O*zDV&s)4|(P45oBWmPD_sX`rg%R#mjX$Lv@mVL>&? zQM2t$6uG6B9uZ@aNWD?Vg!_CySLG3VLE#&{D(VKk`g(N&>Q-vOM+W(?4UhFMUD2#@ zNsN96?p@>0>N9bQb06`rd@HBux6{m^zIuGL-LS7`%Cvds?O`F_tlsPEm5WZpnd)S+ z(#`-UxGNs^i>CneUYe@JqO|8I=PP{*dFu_6`3pzdkDr;Ls$~@zpu%9bEiFLnHfXKu zRq!r89%28Uq=%S9-4d`4Kt@GWg9rbV&{mofo+u$t|ZH4IRv(6h`GF4dw z76KzrzCSa32Qc`xxgwzWs243PlRGO{0Gg@9#~V5WsC#NE)>sVcGoNT=I?mwyh3Dw2 z;OIib_K4RJkdI&MS2i)<_HW20z^KTB^1*Gjc4<>Ogbd z0rlmh&U8s1IB{m~-u^=_tKXbErs}H?Kai{J#87cr zk}FkQ_EGTaB8?4CemU$uD%$^WgSyM7Ad>3naM|zTuu+B2PA% zK4Q&~gtOmkYiX9#*!_liWQJ1g$iZ)4e{_ilw&!oeF-b;$M`%^2I5pF0G}8>{YPtu| z=1603{v%o>GKo3UActvjJK`4d$GVwk>!d$czSvuJm(usWvv`>z@+b*4sQmh=L>}%F zRwMS$t*~Z~flot*a2Te3zxkpnV(pFz@XG;k6GV!* z5U5y#r6Kz$Tj=6YD{VGm9}s3J{s-HsEBQA_xZkBU0KT@5E~h5gxoGVx)LxA+4y_2we|L3ryL>l6}JnR@ss&Lr2Z46iw z|Dy$29W%$<-Bu*d`(J+xpz8V|iiVoO(CQ2jf&`s{6dE-)W$ zCwp%iL>Cilxj5LonWvG1FQqOHSZL$}!D3F5Q;2i0Cv7Izrb7#H3*w^johspbbJ&uX zaPZH3DxiJT!VE@oTz*rp^$2Q37^{SWfwcmRMb2OD*H)BecYxx4zF?RA9H1tXm8Ox3 zoWqnblubdMaDUKxhxn)#!hIKcpVX`Jh3Mof{91uGA9bvFB|F=J@+(p27yyE^ArFo! zf=1j;L)l)?NNq3DRLb3jy-~20tX!h3^dude zV;A80YE*qbAd6D@Xs1OSf3Uh^z}Qdp@l}7NZ;zkjoE5s_mJ;b3#_kdBv>98+e8Gg* z|55c{Eb>knZ4qQrg$RvnC;!c2pS~dVqK^@7NBEVdNyC~|IlxD2NjP8{Onf<5WQh6o zI_HsmuaTp-%{jw~%LR`HgoHtx*8uv>yoci*A86C|Y(B&xcx#OBz+T^r^q$Zt45*6w zdPk5UIOA&s4Bn0kx?V!o&}W3xLohC5BUWU{+3`O5X98#MtTf)!a^7v&>w}uLR03`C}iEi)L)vNd?_&V!r3Hh~n4Y z2HclRlYh1HT+n4FSsT&VaHS{@pwLNn&HP^OlQ{PJI&zMtYmcM&omotNDQ^&i?~}qJ zAax69bF(!SFJ@^m`)=2cUZ?C(aNg~k+l&@`dez+xWt|h}ym1+I1aQIeZ@$7&b~&;|a&?V^y5o^p^-}++v%JIY_$}Uf55vkd=R|xzM@C%-d%m znH^}!e!Hp@A5|w++VL&xTYv1{GEqqL)mhAk2M9mlEva|k`2M*%ifd!kgR(*8S6Avj z$P<||@Fs>~`$wX~(tGD*;XyOu-SHL%zb`6r1!g(yWa@apI8&4zf41O*Klye_895d>t7Oh=_ z9MvD`M+)De?cr_XByS_R@aChfhh9-*?d;pcx?%S!DXd6;ZQhj)_5hu?+8k8&V@Jjld$(sBsm?{*+Y zFL2N*p<4s)Ep)5PR1`HS8H>pJN>`w6{7}Ysndfa# zVW#oWQ)7;`*xt*I*&9t9Sum;v=#xi+RV4VUFL(3yM%|arFm3HsKHY-rQqUT!L0G8# zXy)44_Os~D$7#R3^C`%O|81S_)N7x8ZL#zU5Udq)i1&4hzJ33$#i)86c$E>s6B0wk z9SQTvZ>6Md*Lq&DmBI$&X%!5>8eM^Q^nCItSn>cq7}EiKKqU2%#_-Z(7u-2X;GRIj z%us*@bw0fgBIew2i}=xN_bdhOY9_Z-<-M39I9T*R9%ujKGP*g23ModVrIvlOA@Lkb z1#{O#ZrOxfGI4k>uhtnekMO{VIizFn!#A)0aO6-OdfxE6ElUs3a1?N=!;vn@qN^lSG=WBabYH2$#H4K;319JYFC%wG&u^k-N7v?RJk zOr@X9X1OI~jr z&@D^T=ltW^V%mAUBNhLU!|-oO6=2%Ojr=#?ii><{o}Lf@*bLSx)_Z+a;tO* zNB9aXdDN_VS(EXb^ppKTGuljVNx~J?Ix&@N(?xsJi>9uO-ejZ>#@dHVT{yTlXs#W) zTeT%0I@cw`KsxH~<0T<(kE*C^Hhzqt^sv~_7#}K|8c%cB+S~i#U{u|3vr5x3n{tJ)v|e;ZH=uv^ zd}oSB-oU_OMnK_Ynh3Ox&RN}uoTt%T@SFURl|1?ar+6*Sb7lt?W0O#w-H;=sS{ip# z;yHunE11`Pp3pCBKx-y^zS z2Crp`9FhcNl8p}XKlyzZE>-7K11QJLs1;5zWEH7kV1B%QS$kkj|$ue_% zwW;5AF7lEdq<>5c;LNEkaqHTL>#%1v=e~tUip4^GFV13vD!CdT;2#N%>OmNMcbbd= zH(6hTHwTiwtuA7N`M1di@)RgSHbD2q2^h1W9QSO-ESV#$?L7UA_8nCdmFP{ z78$g#`!&%oKcbtK}++Nm-G$%`!F*Zl+8K&)zX0orp;-fp&1^^m9T;|55L~;1N3Y&q)3)> zDmQ`LPg!V-=eq?U_+A;%V0@TAMAaZ~Y#fi@8+h-UOxQ-Oo=sky#STsN*fWv~;i`ah z!;K`>z5to+c9;0PtyspQ)ONw+@Q^LYA@fBvQ_2IF5^#%U--+Sr*2!e_1)*K`p~qG0 zrX?5q0~X|X`2&y_h1D$QRD*8XD~^3Ri${g;U=u$=Zm^%Ur1P)c)DEhQqr%Uzh`?Nm z>lDBi^ZbOfKS)q$4C1qBWj~>ih@Q)01uq|Dt>&xWerOw zB1x$qOpSdg0*qteRgEA4zdN@Le0QgUFL{>4#HGaKPECu7)(2ZS0JT7Ah2wj4rw-`w z1w}dyp-VzeF1B%?vGMBqMH*+-@e!}(&`sS@MU^`;ca!;`R~jl)6Cc&&4HW5D>1@LG zz5&C7QUU&^k3$a=?agMpaBJH$?m{5#9bq z&JjCBV_jgJjn20UNhMvut#p-weJ9Zu@9wObsn(A)uAR-0WQP4Of|4UvsblaoA9 zR1{IURFFfZP%D$IedXlF351u}h0}b9@8mN_5^ox}3eeEXCbP3*aP__KavkF zex2A~(OcEZg`o4SDn7OsUIhHnRW@WpGnmN&ecKV3R0=?5nyY|8Mte^PNvX8`vwM8# z08de}LIPtf(W3XAZ+yC>E_wi+L|dSxzL4&E6Z@)cxyk^}D0*thEFI7`1QRu(ziA@D zH&~s@_xe3MGR|-u=pU^-{F)FA&QgfwPSJk&NkIX%yV0Wp&q?uQkC~!zepf;#LSuLJ zzDtIzsx-&#C+@?p;F6)WYM>qJu?O>N^R*SISd%cTgzf0~%bEA))276HQn_OkVKrdf zty7M|Uqo8!4|&d{2i0A=^)MiDqaq`)YQV^AD*O6k1p?YV09HS4dkq6BKu@=ZlKoFW zL)0v&b6(xdx({ep&GS<{S9I>RWvSBw8}jSb%^cc%%M4j!qCB}rzzP8%L;k5OIfkJ~ z5xrx+bw^WEfuj9tr}A?ucM5NPkI`RKrd)CC{cS#b-e!p@SGA&ER-KZbn) z50w8{R`jI}-zS)kYDI;YLbw}n(q*K1q{DPqFu8V~?LhGeFrw>K7q58?OdPH&KP?Fw zM&Kser4LYj$S-_15t8DQI6(PkPxQ%(i9`MF4`;2mscg|)HvkHH>Dj{o)_}C%RTPoY zZqz1tpbvut9Jhw! zE5FC;aDXrqW+K~L)kx?nZeR9H=>hf6zm3A0{9v*QJ8nU zMa*YWJ~9-EA4<~F3we=LWT8ZIqJZR?lXU^xnae^qS>Mb$BTGz69oqE~=3%#)w@LRZ zq?uJx7DU&-+$vpw-I*K#I^4Ii8BDzwV${j&p9Bg zfB0t4uzkMZ%7b8%EJFfYsJ+9>FJy89_0hS#@T;>1z0Lel&>0U{tE@WGhQn*$W=OKz zgMC^we^K-;=3@=mxOyyOp6|M9sPZs3ik?c;Jhj!r_mXPL+Q^@*P)~D)cs_3cX{AB5 zAKb_k?y7YCZcGqAR3iFB-*oBGT)*bjmqQXKo6-+w6346~?eW=)v);Wk>$sh|#jEcX zTw3J12b)NAo}k^kToBiwTzoF;MG>oMaPP62l!y%ZkNaMxW$>LHeS}Sbp3t(u zl354*@C;p%m7hn2^CR&VV-DZ6os5aYkPZ$Qb8oYU?bmk+MPb9@3)8lat&NML-zOW2 z-)dgLTn}jEI_ILJSO?SRQf=|%-EFoWlS*73&JlXadIo=Ma36dDIudbkqnOn*_L^S!&Oe%)De zBS5)Ey8!d9`S?v;mbPo%+@qMF?}v!@yF-lx2K0dU8Px6*^tBI^OUx}Qi)X2{g+9el zrP!n(1Zhi{!PsU?m*LutYTZ+E(O>-zi1yhQ28>uoQoK;d+V3Rw57>Qts%6wuzGZLq-_P&uCbU>9p!|e-?b+>LuiYOc?%SHbSbAHl7h%g0py4XJ3>VfcO6;YRJ z+Yon0dOJ|?K2$%OZVC@?k9?zmKXnVMVFi`x++uzsY*=QM8__|j@2t2kBH2aXU3J|> zd#ZyXa&KilAaHSf(>g>T{%W4GdVI|Ls&by%u**j%s>I`qX7wQaitRm^&Hyz2R zXpVFT29Z!z8!Ai>*BE`Nt_tQJ;6U}f~SnNLaKWl;lR;O>LMak^Sl=QBM~$zf|DX zz_Z~V_03>c08)paxn|-mq5D^`jlC$Vt`9yy=ctc?&q~#7(8>&BHK~20;KOTSLRTLa z1YfNmrhT7|+79sqZc_xpMBNsUaaF^TfoCpV;6ml2GlgEC8asde)5EYkSJ%ACePjDK zF;^4inN3Eee}0x+l?dC5uqyAn1ex&KCmZ-vpYNi_1zfxKgE*6-Ymu!r;CAytewb_E z$PoKEOAg=MCod-+g~aqHnD{nmRKT_zlyIEqkONEI%=hm%z<6XWMpf^qJoZ2ono?5M zsdf4G+hqr+6X*|4ct|A%lqjDM9r6&-S>-|)Q>a0a8I?~)?9b?Sz>~{!D8I6{;btvr zax3I+uE$uMiISswB6Rz9Y)#;iEyuHy51PwgSpnwuPmKKzT@@5ySN)5=L{J%U+yd$W zfswqNlmzX*eIRp_gU=uV8;W{WO6hgtw0On{*aj)l1T?@^hp>2O@U>sDZ7V>_)XmWb;#}XxM^`ZvzZ! zjD+d3$K>l2!*VKM8oOy3JsJ40J!y6@guBZ`90%RYR4Cp8u$Lw`$d;8%O~m};t6=7- zEwC=)sH$!e=)mW~w&SQyQoa@XI;5V1j1qb@4w=9eP+hGcaO{b`c-*G_cu_;x(Bz({ zHutpIq2TciYCQG|Mz2+Y-mmKX_V*-I2ei#fD2W5vQTGGXpFgshF%F51&2I{D`=N!R z41tL`Iw1P-^^K0$5Gh6WSJFprL~j`*HZQD|F>_@-gUvlXj4pESm%r53x9(BJSe(D} z%KHn}))5uW{ zmMID9I0l-VcZ#k?#G!kOgE2S41MiV}v#2N^u{1FoJs`bk;0W37R=9x(F?Q|KG76^` zHJcp|0NT}D0_ni5FRB&T=!h@BYY;H({p5Azk$5YSfjIOws}InO$4%r)-Ab!*p58FN zjh;@%l#P>ulO3zgs)i=aTSAQdV;({umVo)*JyTkH0!(+zr7CFdadIg<(_VT(IX~La zu$}-T7Et?<6{#H3x7eBPA3H@!X}?&WIW{y0M7DaoXo}|8y*`IU6o8)HMlfEpK?h{e!iqpL5@~v&-I| zDc$#C_VBHo#f^qXESidPdUt2TIZ*k8_M~0~ka>#5P10<_+Pv2?6!uP%Z1+S{-Dw+? zggsnP1sXt@V6)4~C(W_5HxFeIn=}HvK_B(G#&mS%7}(q# z0oJ##)X@_fFM!ssxqW*q(a?UAqUWh=hQYAT3CTNT+l+DAFa3l$1z= zD5aE0cO%{1jdXW+OE;V~;QN02oc-m>1)q;Ty0pM; zh=01=YpVemZ&GSA#4;?5H4w^+-@E?f&DZR=Q`kvAGoo%uCY9w7s4fWuWzweAHupFCo%-N}vSMOM8+@Nt)vU;nVTBhiCB~mj~_vTnR z+T6ihtUUS15d=myfu&i$&I#Gzh7w5$o|v-d-cox{?vLtwEsRphaw0%9VSZ+$DL@3# zI-S=AlsF_=I&sH#J22pOP`5C-5l;68h$`jYTa;tTZ-OB(0C9BSwG6X#l#AsW=ALtZ z@an?r69*_5a%;~wKD=Hy28*~+12`}Rubd*(sK&_PqS4S$R9nsLa_eTWuYaQ+=Z|Qu z&OZW4?1SzxpaU!#IN+GfMwR*!)t$DytlnE}a{?1rXI-Z-V>ZJlC{^2rHqJ=ibywS# z1%hnU6rGz0Gp-lgDwirE$^7+cKWM7L%G0Sshq6rGe4Wk_bU+LOJE;b+|eRZs^ zedJFa^GwsOY3G4zu7GLN@5uQSiL;yNYS8REx&b0XlefuuE_h-ixg_chsQ_QqAb+S% zz6kIcx!><4iZ1p6G_*3lw@nm%xZy83ldQ(o`adn7K}R_U^M;l8Ldo@YF?{%!eLyo> zNyiG=N;Vl(+<_Hw@=*)WLCvUB_)b*z7CSzH?y-Z(-_G1g9+a6QiQ&KXM$+vUh zlkc)7;9fnUzLwYv^h2eRRqT4r{;TW5a5vujv+7Bl3G1B@v(LpW$&Q?ss=Y6lr(iZ| zxXUS$;%8~{G)^4t^V&rcS|47{&&a!dQEEnQ+Y>sH4t3|Rb*ErFbxer2cbm4T+TKSZ z-&$86sjWqy#z5s-Bi2Us#gts3TyOX^Q*fC4b8@ul`tmcPlVip@;8E}>xdn@0?;U;M zN^HIPQT*I<77%k)Y5);teDuMm67DvCwiGY z&vshVwmy9rZCdP<5kw{m*_bs`7@6#A4|pP+CswXnfRC}-1`drRr&!*aS|4*G;2CWCy(B#;S zaIZvB-id8aR{xNZQMhthosu*+YQOD;pX@JX7=)!QRz$Xj=LEfjuEnSg*X`ot(Cpfe zv&rT+$^^1oZPwu+RuSFetiW=Ba!ewQ1!*PE8(>UPi(ffo&ZCJxHS9WSaVw?tmi18*^Wo2}7(hcb(@<0-zpN++6$ zD?8kl20u1c{iGy^D^W)3p-u;>4Q`Vbq%odOyE^NfD87>O$mE9&b_YR^MFiMz-Inh=7Vc;e2wGLz8uey;tjyWdSaitWL2k{ zfo{ho=XrnOvs`uKMkyX`ZN~a9u+jV&#|W0ttkf5^`A7}%1|*7ua1~u&m1$m=^HWE1 ziN@glkW8z*e4?QNR{asScL^`v`LmAr#CpyV(3eeFSNO^u7gd7k1Enn9=2$^8Ah4}o z0B~D!OO$*^!55wa-6J2G`Re`SIS9?#VLcwO_!QsxI{iY2Q=!z$AAUH{Bcyg7lDVkm zS;FFZ6Ps+=8@7Tr^CDPp>^y;kSEBfoLnMRQha+LH_ZNv)_TgF@ffs=OzI zJ$b2euZw*jw!k_O`nqII4(Bdk(Fi3U=O0P8jQf3^K-%{eO3%sBy4!zHFHN5lB8ULX&f)$1=V$>S9F*4`o( z@WR3Y)Mu6OHW`F^6G6yrau!OIlX4 zul>nlPY*fwnlY~c|D8^0x;DoWwSs+2$8)H5Tpy_hhP<>=u($x~*-QLW_9px=pK3c> zEi}EwfAgA9IqHhQWxs>6mS;?cCf(rpN+DUaC14a`RQXVY3?N*mAsU}qYA~Lk5KmDI zZhRRCtj3!3M0b0N>32#P!k2B|QPhjH8oOQM6u3_>%_Xwnq7Ywi1sMhb3xUCpryd zNofKv2D)3|1H4emf}=5#>}kpidU*L<1x@CeOa&>2NNg z9^lIEgEm_=--Y%AKy(Z(+@MjJ=^&BWj72=i8nfGLc%REGYw$kJd_KS_-mI8A@ zH*6VVVg@5ttTj;UzX{s&9=m1GAkwH%R%3{fYHH8h1Q*91M*Sp{Q;w3+EnkT;y#xbG zigmPKCV-A6jH>l*G-$l^qoAiuuB18-`F6}WELJpwKCPKP;&Qo|0VllynlZ2u@N|tq z{0}zlPppI5IC#y!S0sv~L+JmG0d~!wT z==tPrMZCsk!?+W8Y*QbaiZATxrj^i6K{jJ198hw~$HG=pGf7q9KDRQKPzTuXRfsGX z^-1@sG)gbEKJB9~lUC+p+E#l2JvPjptZ}lx0Y#qKdK3@BiIwMsI!t=w>L&a#KXyAG2r6EN7e9 zs%+O6b}Kh?UBf1# z&>^SQg76I(-nU;7SQK_pME2Iy#Il=UW=z;XPHNy9 z7O0fmX$1Nrsgvy8h@qJIZa2NJU#SK-_2dKI%$cK@z|Ib4#*K7#Y7 zfK~rzQM0AHv4a*tuJII?d`?C%_QU0@f6_Tq9fWCojR%M&G{}WlNc7T7+od&_Gz}`t zI}Rh$rB*og6fl0IAWKL@i5Ce*b)JoIaamr()SMKZaPCIV-ke{B6`iQ{p1So+y4Lhg zY-B@Uv1xqK!O#Xo=yOMLG>!YRg>@+&=;>dt1ZQ@5kNNR9gu7wtIl=s9ov3S3F- zG}};-6=t~UWkR`9XD!KIJL3Rh4QcB@HJGeP@ypz$w+Ns{PUkK z+FLa)&b}*I%^ysY`mJt~G~(yV<(Bt3ejQe4Xu|~+ENoIo(MpU~Zk~FMXmuLbE5j@B zc#*yZ22|wA1V_5;^cES^o*5G9B@rFM2DUE2DU*GXV zs#mH`(blN{#&@IDq%g@rEFRlMrTBeeC{p!hZ`hu4;pq>%Zn@IWVh-xZ$TC9Dlvsq^ zJIFTni;bCZz>`($HmpB&Snn_nrGi=?fmaiMj9;wgWzZXp&@`Q-!* zUKL#Ksl=EL+>6nHJoEz7Qq*o3g{>dmcB6XSlC;_CCWzRsSJYwD#YLwd1pfDz-aaKG z=!$So(&l#qZs8sGX{Mnc$hLflbW0<%MJX~rY(ECKxWO@=V0eFKivQC_PBjY3GxVNUGh}4EgU3Z zrJhjSpMG+TIe9kiYL{;mU;k*`I$@18Nk}9R4F%)sN;krovfnIi5fv{y+|7yc7>h^N z7FEFovrfg}4IYIOZG!*9``6A^;i-1YD=mcTJIYa zZr93jIm;KnF6E~Gaee(LLM-;4Tke#B%QTYRwDpk-^@yu^12T62dQs2xq!~oPC7RleZmC$ka=mca=9fu zT^v4f9QW~`NOh+I%&xfy@XY^h(cSrv5z)GWTviSdz||yVqvbzj{?IIJ<%~4a$+e0-mFqIN>yll?G=F5EnyYhQE>P1;@mQeHS27_*Y54+h$ zms}0(-<9ULtjdwZ3{3-TkFlT8|F>@g*PtH@4u`6ktT(ioej|HmzF>~E*qf)JgNcRl zs$!^r)~ZP>O(GS!{esud|Iwet8A1Lcm9YTdUzDyW`0f5375EJPo1L8=#2D|?Z= zLW2prqJn2mQ9A$^s2bJw@W0;WJ&z?2iGm1TN(w=%&CdbDnle_(D?+M!)6nK!k&iGG zHeZT>V=eD^p>iZsfE3v4MO3OUEig`6E1t&qJL=bctFwE<3L4Mr&{}Z3N?tjKD42XFQJ&2!(Fr}mvt{5<`_|yh4vSF0J z1jxA!GJ0)NG$+-o`TVp;xxZhUP^Va4gs<3#C3HP5J-i5eC!%>yVWS$7AdM~d2yOD@ zxzA|M>$#x>K@#X{QvO~I_~b3*ZsvXx^Z#pOVZo%1D+_}8rPiv;U{r2qoX|#Htm%Ru-b0J`8<8F@cN;4*>&hsEqd#Tx2*%m_g)9%|d zR{Ka|ozAYHu3mGBR?+DH74YtM5fv0{5FgRsRb&3JX3-@2hT^1a?r9s2r>NMIiv8$G zZGKAFD75Ml0xqVmQ5i(q??UDmE`KW#H=JlTx-4vW=pt+|7szh$W^9|L&RG;R-j6@8 z8mtqsbM44SGA^B)3WWV$?seMcSNe!5ea9;)S-5vJn&r{vj+ZLJ<_ZyOm@Ou81j)3p z3F?0hh{UAO!XYT(eT5co1(Br`Ey8GTUh?Jry4C6lRE*&+!~>CCbRHpFZ3Uj^^{R3Y z5FCB9D9)4D%#-Yd%?s6H!p9zcIyLk&&++Cl9M~O7Ps-8r>W;^g{}8cOl(hSe4MXPV zo^HN@Yag2y=60wG<9u!GR3LVZLb2fIyzJ>(GR$i(odKDaysf+^!*gE3{d)3VK0O%2 zyIPfB?d&mMe6h0$#FMIoqNYVsG5mi$ok8d@0_%y)1JwSvTb92c6NFVdn)C71Uf8Ez zIFb*jZL9<%{$>Bm36rR%Xr^WFUDr4+W)a2*gNz{^O$eL912C7>8c-M72nlXNBQ_UjG(x*|+z{->8$$Yx?X=sVlRL4HWqLd4HdA3XQ4Sv!@8Eu%BcUq(p-dwoW1G z^xwA)F0^Yz^9QJIQ!ybub(PA zBChqL1g4cqn3|UP*$0(;nda1(3fU5YsW~S+_;IBBO5jlLr4h9mpSUK<9933Q1Ke)9 z!#;jWO^EO_IiafTD)we)ZT4LSJ(;H-v#E(#y#6NnRd6iC|9y?n+Brcuw629kzKzHI z=*Iey{Y-m%rRfjl;WSnyqr-i2JNPFHC{{aL8j?nvk*Qrb>=)yc>QYs-v&L6`H4uXO ztklo8U2H(yUSU$bXMK$_ezI>-;M^~bHsM`m@vzLHNWU%O<_^U_=LjRh)Fj)t#T)7F zAj#*$vYAil^IrZDIsg(|>@5Nu4%q!*_iHmi$?P$k%D9ORc8z2)`GL!*`{iu&Wu5)g zfmzBI{{+<_KIq`Zm*|i|H|Y2Eo|yE~sN_cLRZUX2uqE2C`wv^++G(dUWA*-EllO;{ z-P@^!oVjk}5jYHwd43B9M&V$|_pE5q^e_uM`K5)V=kz)sUy*oniyt!e+kqD$)}`C~ zZcIqa7NDrJj!es>qm6X)yr)}#o?ky8@A7X~mXLsutz9=$4aV0iKuyNvkI&9_v0WDL z0EXB|?;j&$!<-4uEqr^)7RzPFl%rO{1W{Rksh8#cQY*>)rBa}KXaiLJthCCx(wiTE zH^lqLJcsgw>2l8o0l(Jo!KPj&yW0WO#8exlp#QQ!9?jc*AQm1FNxlE{!!@A0!<40&aM~FL%W|2Usq z&+j2=HErnmw>Y{92H^)>ap}U(u#eQL7Ri(uxO=IB}1h{Y2HeMM!W z+a!v0@buzPf@yPOtHgTwcbb4FTz|p(Ik6k2O3C|B2Oz}vheJ5?g<~JeKwoj2y;A!y z36G2;zkk4DIrGYm6S z_jtVKm%B3`s{8KHXKs!d7j2YSbZtgYR*q+9y4@c_(Vs@srj0^z5a)6bwiWAVOUhw4 zwL80fqsYE!`@p$y?P*daWAu09WBBgA{Ma?w$L7?dm62t_wzU~OR}U0W)c*XiY|iK^ zjth{vsVtJV4KP;ecaGT>w#%KcH&9r&>gbWlEZ5IeDMg=FRlI3Tse^yd;CH?J z(ZiTacyy7epH>7FBo*Z4tjq?KN(&niaOTH@lepygDl`a$giC5&DUU9)8t9Qk-h?`< z*~bd8jqv)sCCOIrBltrp#((3#AL8ixcnGe@K9BqA(6g_nA zbpmG2l`T}tklx~K-uu~tSt;P+owXKQ&d!vz19;kSr3>+-4rcjxD08JimSqJcDPGVX zn7~DQ6FE{jnzeeN;Aam{7Mtk7p%R$|@Eog{Rt#_f+W>Pd=e?FSJfJAmxGaEUGnQ`x zLvvl*>eNvE(cvt|G}UTP*H) z00$^n`452d1s5g2*B7pZz^WNt^a0`Z#q5nJvrx0M^FJ}^KNbFG1dW#uZ5M~7SZVLU z{LQ)i7Kj=zrk!}whtuv0Bf$P<@o4up{Ggxh+q^#2fDQ|T)@}*&CQaH`Jji47thAeL zQg{c9e0@^V>}B|wIE16vEq5|WZxAZYeryvDB#Q;MxBBC2ZZ{+EA3N%|5!%}hAHA1j z^CkrvfBo$DYrR0qK|4@tJ}K`)cFgU1dD?eH_C}tGK^RcttHrycSl{4<+r(fX2TPB} z8HA`R8_f!g5^~x4h{GZG@^<Z7^-T3;f1$Hx4|32>8Y z7msIobRXdhSwbBt<@F3*om6LCI@FJQ6r)qcmxdW5+(r36?Dt)SrsW!!cMh}s3ICQ* z9=2$GU2LvNjSCZg5AA-SXq_~E3JV0|7NHO8P3RDo$rXn z)SB~}j?f(de%sF>sp`Hh#BdF3U)6jak2An?k@@-v3*+0*DdjAsZiw{00x1743LG>w zfb7|@;|KW)FupjkcABt_gDh+MyHEkm%Aa9o- zoGR(bRZO{8=-rBYbDpSpJo}B*l@wgLAlCOnl8^pu9LMZ0@oQ0M{?mB@;XGZH8yrBs z@ho{9`w%ApC|jId#bUPIfH=8-HDmf zM+;AQtQPm&mdwr<>kQ^e;PEzksO}V>}q%G(ZTNMbLJ_MR*!Szpi8EKjnncjc;sN^oHS(o`X zt8#1#_^qf$b^~#qrpxHBH>0XzF|Ady9sZoqCN({ zPu$AkWWHS%kH+WuT}&D$rdeT)<$qVH;(7judBGx}kBfo?VJPqF-S4jeyB!Ab;Vv=L zA4%KDa5$egzD)ssJcd9*Meh_OP{jO6B1Bth^eQUBxwH!AU^`nr9 z;>dQ>CIWkyUfl%Qb+Zzf8bm`<)1q~NqfE5-KQsi%E$v;CSDe9&+l`C=qrF3{Dl5Rm z5{1Fj7<>hip=PNxj^ojWfh_0p&+;_#j1FXcHoBfxlz;X^ix4Pf7gVJTp@>4P4K-;m zX&X5HuEPv0hhbwhlG5#oqC$hIl*qv!=+Psmd7G&ZJdw`Utl+R{m89QO0Jd#_(HW%B zUJHKM@f$4Dw(4@yH+rTYSoMReM$Q0r>;gFR;#Aaz_Uc5f%LPZ(0mWWB{E>;fSm71- zF5RF=Q7wH?g^=td%d{um#{*vA*{rLm|xfW%1*B{v35 zJ(bw6E{-V<&K3bzp;SEr{4%_vCmaI>{<{Lx5$VUs1>dX)s z-TGFuRg0v7YQR6`FpxG=6&omjZ3atBM>(ptMcM1YW--R$WNR{^IVu^gRLYQVAOLO_ zi4&!vwBI6eNAk2Sy-=U+=&n9JoTrJP5CS5+Zr+P6%+U!ztTUkkm+*sVltn{EKv&%cO#W-S;O{MtTg{BU=6)&R*=UKXC{ zQ-W3DXyv(AH5`gDg{Qu7V9E!{O7>>+%}iCYM)4ZQ8OAtP(-CRM+IQ!jCrpmjqPoQ2 z0ywAJJip8*@X7fySoZ*7g9XV5r20+CuxG^*JpM^3ZkA5W0-({T5{y0*MODf&~gbv ze(uIEi`+VpLUK!of+U@^aEVw*J&B&a3xTZFux#@I`AUpgZEWN0jLmEejYq995`|Rh z80hWTLH-llVmnMHlKKFgMMOI{3vjif2NUrRsMgAIJIt!(;(4c&4fMEd*K};V3}isL zAu{^YZX3v5;lp_b;ZU+Ao^l3}u_3MiOCQ9Yp^bmH=JNbP^1G1L;S$Eh;y0(=?U zbr|R;S501f67kO9mk+k@6MrK$gYK7 z>}d9N+vmReGp%hCwy#a7d{au@gD0R|sn`OynB2L-vSx)3d~K2k_1}-?n#X-w@a0fv zV#i}x2yMLSV(w+h{3SkFYTm7BKOHJLEN@pmLNabg2nuT>ZC^3{PuMg}ixVZL-N2w_ zKI$!v09UT)YX%mxvA(xbp16PWH)y~DK|j(`mevL3eSUgG#R6Md2G56&|4t$K!w@gu zt>=M>|KIK}$PVWG_y(jU!qHubRyv*VvKxON2>*(3`COefsvHNilwU5|H@j!$H!#IstT)X|^6lndTfxjL({h;vC#&EZ~WerwDcY~!FAT%de01du| z1rL*uC&9kLKtXO4;u1)&hcQ|UaM!?aPa~2c2J5IC17+sV574tgnd4K1q*;`IvTQ6D z`H8?&u49Q13K37YpDh6AeXo?Qf}W$4EQ*qxxxEbtPHDh6GEF9$&11F~om4a;lvcUb zE{D$vs7L&ZO!yoLz8aSF#l!RL3$Nx~#Q^m}e|=Q>BkGw?gyT#=HvL3B8Sid40Ng+t zgf?d_!d5tFbNqnn zXJ@Lyify%k5%pcjp74m-UpL-U?YFZt&Uu+`( zIbavGzsHAJ{hL)aK+zWp+p&gDHqi=vj@}vQ@0&1b@&AvgYOh(J#q) zYgMi9?h_Q9K2l1+)Z_XosFB2SBFNZo{{f7?rq}F+nQ=b(7FtYKmV{?DmpY$W+koOQ8Z1Fhb6D3`xI@sm z@%3}o0Z2h(fW!TB82gEg9el)&^_%cSSauoQ}(-`Bd!H%Y7@6iogF&KcN2RYW5}) z2nRWGJ;6hY0xXqRJz>B7Eiz2)1eAn3adS#?U@D;fBH^~?XuE}w)mZ$1!!=M z9D!)$Pc#_Baprn;E)uB!VZzS5?&jJlSuF9BFI>nNlinI5N@dt|mHkO3QN-{f*v~AW z8Ts&#X$u&R0u@S}FVRuH?fUBjpmvOa0=La{6r1_`;_7%~Vj;J<%<;g8?<^(_AzgrjiC$RemWj#s!on+9i`Y8c=^Pe3CMgzC$ zNZuW`y-VKxTWioQNL@14S#5j}}W=g5B%ASQ07RPmVGjB47m zVp2%=bz<~CxsK(w52&5qc_qouZhf@vEVHa#=X&*H|Cbi>Gv2ALc;JcyiMQE6>zdIS z_tI-1x2Y?Vx!e8z1C7D{?}Sj4?j&M1K1M;Imw1Ar)Tc*amUN#BPZ*kzRr-zB2J&4U zzG2z)KHxyUB3iRTeX41zprL*qI|f>pZb0-|`~Av#%BoGZZLLMID2)JQFi}q}MHZZe z;z8MOhQTUzt~CyTS=ycdu4$natx5v@p-O}AG?+qy73#w zFJGC(iOmkQw$~|+8sMK1lp-oufR4N3Gj4GZ^MfxITAN;Fh6)H=0rkTkEeWcO)k`rp zJ$p>6{@td8hvJ@)g6qR~M}6;H<%JVjTF}nbq=6#7)?6AS-a#WI(GN=EuXu%BwkJxy z!4-3Kw&WfeBk`6;*sGm zoZHFrXJRnO5un5L{BpM{I84lc$iZO*g&OAWvVt*J_+o4|9+J`!Z`n((6>oxJqCd z-~cDJCTg{?N5u6wwiE0G3;frzF0PxI_aCvoAIcAq349I7A|ruCOjeAc{u+?ybndWx zGN0I7soXyGQgP|lXRuysA2aVrf6>l=VMFvXDYSIl5WhDYgB-=87FY`KNJ>Zj5@2RK z?Ep!18GmXJaFdeTG>pCAUIJ=TVEFjjNp3Nd!@j;UX4O{Vw@$CWhv}Y2BPG9~SEII& zA=noD#PQIkk;`iMXY_RqPSad39O~V!iKs_mpVAr0`4(7Z+Lz$LIsXN7fQc!u_{_wF zBN2Vl&{nX&TwP}VrV;671V~XWXyUQ zF}iOj23g8^AIDia^gzdNHGa@CSC-3F3%Z?pWLze)%KLe3oNEdgt{%vaik^n_r=UYf zZj=2I8Ly1?m1wpPjF-WiUnBki30BqbaTpSoc49^wUp59HTr?(@Cd05Q3HK8yl36>A zd0p##M%A0Z50}I2dilv{c0oo|yBAbrb4JVe6gpqmMYyev3- z_->pB0nar23RouEjSuyK(6AW{6Bk-&e5gK}KmrCBO~vMuBQe0Bv40B-8qKUGe*9eXz|yYUB@)eOF~}0qwA1R!*t)IO9JQ3i(qo?@0PH8iad045Nft z6oc;>ue_a*S6 z`{#h;)Q^8ej6{{$VdG6Jx<;V@CXeNvXl znctuNliah>`&J7_57f-hUx@*@A5oZ>#gkOeK)e5!4SZ)DDVVD0N4dj3DUXxUthAL- zF9Y9PlI3#NMsE#9gXu$UppE#nv%D-tMQg}qQ0CByg3y5_q%lT%hDbOHd& z^MjRco6ta9S>R4v?r5zqwgmM0-zpY)mrwo63nywb&EmwcUuqF3v&CCnj+B5x3HR9w zXmOWyDOA}6k7&iBgvo+h-`;?MHF7qe{fJDTFya_u zr8k!<6zBMMrX^L1r0inwR_zSTD>b$^G?rC@2H0d=z}dlqIp^LV`!XB7Y;ST_Aq=fl zia-BZafz_d-{sOXvvg2`LHpWxNZxQp{t^e~{_X9wPiai*`MuK6_g zfy&B(4eQwap!v6!<*;ca!!`;5N8c;J?zfT|0S)Xq+aN_bA^JPiT)e@H5Pt^b}@lI0mY=&L(JpR(22l@JaG1$bDF`c9L#)68&3Fdozo%!MI99} z!~0Ow(MA|!;}chneVNt#&#bW(t*gEa0*=90!mmCy8zaVk=b9D zL&b6zh0#tgUK|NO2FIBd1&>9XCEYep6&_@#W3!=U{dqthtw8=e7{ z6q}I(4XS^D-TKd%DfIM?7~25}Lm=D>1hY4&lZrDp)%i+5F8|tjr7N=k9ttN$Li0y3 zj*7y!9D0*>*#b(X0GU5|KlS7)!D_63dPrJqF+H&bkQ0yYY^|D!YETRTQ`-k{x>)u= z^J?tn4T60H&>xc;9$&^V8A#6A^V0%pYgTT3Qcmdvjjc}b({Nz8(y9oCnl>8m{JC<$ zJcTs3yp&ojzj?g;-v+cCYR)w_JItV?v=ntO64$y0<`DHZ7oj1D)t8#;ImLOK#=gEg zULIr!Ypu1T=$N4OF~-Ol`{rn#|K4ApJy5SEWg?E~MbPO;710aG>%S%H_-ZHtrf#nm z_{pz;n86IXD+b``6sTJL$x4D757r8_T%l7Jnx7sz8IL>5&c|#Fm>G^mAQepX2ROoXv3M6F%ti)VS`mBX9P7Z zb)`wn+Mvu^9*TMjU;uAh0k8v0%wsBe9vVURni^qd?zOa!bU}%rA6NkhDI5E=ZHTf} zxXkJ@`3v>ZLninMFmZmy11AtD2z(+x>YV8#r8sGZtb=Vd%MSYNsw_Xla|BXH7V)ID zJu*jW2@_#Y>NMHd4X2@wLW9oi?w2b0^8-k({=mUM^?L|S2Hv5>i|AsJMM`brTPxQfVsM0oW>eu z&>&|kq&=%4_oO=oP3aK96B5W4l@n>^113O59>7x!57+uBBD=LF$9TAr8MJ__d>TOZ z6o#`g1n}4IcUMfUYbh(kV>bcZb8-LuhwnY?;$VnsRK4Mif`jnB4!nnB1%yIYXaK%$(Kx#4D z7|n|}V;#1Zpe=%Yfek-@CN6%6A{oVLvl7 zM56|~D9UYSKpdM{DmIT=+H}VGD^o~NLHx9mw3+OSR;DEVZ=)pc)L+Z7nqt+7l?-&90!100Ug=98=wkQm5|x`OtMz zDyn4=Zy6kIe}k8YXm)SfE7XN(CA9?^=X<_a35ZHv1hu6Nt35F0uP|fu>@mGO-7A>< zAy+3&O#YBf`xTID`cfo|<=JkIbQ{REG5~m?2yD7?ny5tKI=%lC>JZIrL0iud;w2MD z!hS%a!(*=jS*IBA9U#skuK(3l-gbIe50-r%J3#W-Yn)C# zj$@*Lm-ifW#U=0=35?TVOj^%uUOscXv`VLTgls4contj)b^rEx{KS+{+k*I!>y{!} z3@y$1jkZhwf{yOrfLTqhgTz$K(|hny5E=P4c03BNZ~Tg+>1jD$nA#bZO@iTz17x{U zjb?omBZJ4lT{uA4=I|O|^?SwK{;rlYoA`&|Z8>4%#U>#|pQcJ`P58{l7>dB)4v>S+ zBLD@c?JSlZAijQKRz#?YkZ*g>7Wj}A&(EWA+u*rnQvi#UOY&?HVCKiUfxyA8$U z=vqAe0&x0_Cqa7i8*u}XJ=p^ti?;Q~MCAcGi&H&l6Bf?TZyr37+;^S+=9+e|BC%1B|6rgn^(BiQo<8UvNHAppFAFKsZSga<;yD-c zl;8!EPi4EhY(hvVr=g0%B|ievWS)cu-wVk1BpP5z)7csoOt~`v=#T+T)`D39{?*yw zZmBYZV8o&%a5FH0SRJPHJ4H%4YE~y5Q&2N?6q~8|>qoydMQX=!P!|pr(p>@b&*lX! zxJW{>wAhXHT+GyY#_CK2sa4Q}J8S}B^h7aifJDoikH_guS*1NsQmMP1`S zU6da`pek{gl{LnEjyNxHf+#k(!o)sMHGuNf$7R8R) z0pq^s7;9)_aj>*1lfETJ7eCpTNkJLyBK#TudEt1hqLcY#slT|p_bvyy@$=U?YOJ$t zGvmP+t&5a#-mK!)QvjCQs=eVe$cP6(%n5*Ao7$^Bm%D z34ca>jzKj~qh0cTxE<7I>H}#9f^MzgvUPd+4FNv1+5Oh9F;|2Jk7O7O>by7rp6#F3&=6+sbZNxRwvA7%!23FV;N=!ZpGI?csy zotuXGeqiL0>r#90NUj-vjh49^utpy_Uh+TlNog+&=*ZG6F(GaM#*tXTq!bJ1LF{`E zCMQ8bNNO!Cst>$ht?X(nSg3&ANt%cGx&NTasJpkVc;RLxP{yYutM{^zg$M{UseD-| z0vSqV$tS>=8#1ufAw+5R^(9b8SDESH+Eeqq4_<75X4-Q^EvEg&Lfg|Q2EqY-^fM4N z`NrL`ui_|s#2y4SI=jU#E`bx@&HzfxrFa^M6UO3ks-YXX<9~ctPK+Va6-npU{)?mD z^VqsY2zw8<0)EWg28RS`w7w@70Jxx!+ysw&dALV>;MY<~5A;I80olH5umW{yF%=V~iw>b))ImbP^-THsGZdQIMZfmiQ1wB<*)1RCFcq z{x@7@MK}8D0B>K~;xkbgy)H24oLk;6mT(V-BJra05C7&@XGpf9@sD^mB;~MEzHiG)^KfQmMK*I80HnsXcXM&H)Q!@PnU) z)q<0m%3{A89)FN-=E>?R_*g2e`Y%0{}u{Iv@}%Xk3f zL0LaP+7x^6ghiYr`w|cvMQ57X!623e4EGtVxA<*^CTZ_IH;Y!24Z!pXumbOv5@m0& zv#AALrNS`lfa#UlRCz(L9?B1s%DK+pRtWvOxw(nfBS$|-iMs%VxfwMHiZ2T?8!;Af zG2mN_03ZII-zR>pl3um2nNGcwF7w=_zeIj#(yU0PxEJIDTy=a`^VM&fb*6aIMSvUg zgT&;64?()_Qbsri{!?&VWXz>ZVx^z` z{0nXI6WhU*t`=-_#yJKK{RMVZ5Vip3PrvJfPO8BM`apgw43#pASWYnJ=>~IlY<2;j z%-fk%k=290*@J)lIp_(-)hrF6d_s&5>sn%~JNc?WvFRUu6h9u~SX>({a|$|(%pI`L z0RytGH5LKu`=7S}3l%R6KDz}AKxnHaE+a?hya2N^<&>i977d5)Kpc`eO3WuQ+!g>| zh;=4m3sOrS_p^xHZeQpx9^6-F@O(kVh@v?NdsimZ%OrrJE)C$KRsmor842*6w%e}W z?^``!M|VLj0Tjv0ePgX!aaTZP!|3=O{x8qUoqwDHx-Ts^L#WXw#)LI%KrZbLk^cUb z?KVDv@B9G7BTpyd7I(t(|1R?Ye*%C7Rsgo=Ybr520+|cP_=7B(@Cm^gkQCFv7}OBN zXR8ok;eKrZ_$i2Q5(U3c(wSx7SstVdu`7GZ=cT-HV*)jL_%|YsHX3VV$ITLdYh%h z3fY)JUh2O7NOu_?LNx| zUw7Yz27{IO_eXK&`;+*wrNtvAL~z}y{QRiN(ku3!GOt}VA6@-S^IM6WQZ4o{W>wgB zD_BETAe1Xj`D$C7l~K!;R_Sj{+V7JQU^H*2NgN;l?V2LWGRbsW5(Gw@4yM_10N?(6 zc(>cNu*Vq0ZtoiOYM?2LfSg*P7qpQl`v^Z@0i_!YuxV#upZeel^}oLV4_$8=mF2#D z4ND402?7EV1_;uPNGV+cV$huuk}4&oNC`-X2qKNrf;0+Bw{%LUh~9L)>$+j@bI$)6 z@0T;iJ~-g|)rz_1nkx*pi17&j19WOsLCI_m>iYtSAS12)w;+mk(3NWkY9+$7pYj+z zxdpH^rtmg@oQ9J%8bJuVP*ROglVRIp>MyBkwP}!gI*56%(g&AxVfma_z#-#q(gJ^X zV5&h-hK9DH$+V0>_(k<+lu}gcL|2-koMZgMMnBj_E=W_Wka-%Q#JW{woH%YYpK`uX zZXCb7(<_19A_Dg!&d|T;aMkb-xjK%@lR3G0ynxJcL*@K6WREJ%mQWX>Y#+6U<4=S- zV{JnR97A|5Hoe=zMKr4)P>BYjJ6Kc*I+*nc5YB+V|Gx86*l!TprGlR|dh_H@`=rOn zmO`)ktiN|n(hc2`Fv}tzLJ{NkC^e0ez=db2RDt3wB3h`c-A6&+89dnh&!zkQ_&e+| zFjq>|7ecfcx>|Y1MQ09+yXqrA^xuxAo~R zQXy}x%d7a{xgo9u?{R-AqcwI1q^y?AFxj-%K&Pw;7N%mb9T=1g zR84BtZZ5sGUS52_GO^6S#iZt~Pr85dhQCQeGHa}@GKV~SMhU#OVnRQA_cL!Ul%w+f zXhOpAyHEMCNzVt2Gbbn(Jipr-zrrbI{rd}F>f-*dhic!QQYC`uFQbiYlc3LB3^BYA zB>~+N7mFgZ94Y;{10nl+tHnK}e0S5Uji)j0eVDjNHC;C_qr!PG6U@>fkDr6Ng#z_2 zNrb>c2qb)b2j@k-9O?g$r2#ztMQOI0E-D_O$ofoODUi?jbiX905tS3HGJ#uW_ULz( zKdT@1l6k?ZX78@vd6r=XlyLWi^j|GhN{745UT{q7q-3JfNPd7gyPHoIy_^qnSDW}T zJc*b84l1E`K6W~s)FZ=%upy|hx{K5@bM9~KPPA1~U&8`0lnf8}7Id=r!lM6EeUZEd z7*IHYz;T18_bW;1H>G7Fy*vJuwimYlgwM26QXmq*9`lXPwmBd%_>3Hrj@eid{L4wP!)Ro2CSo z*am{gm$sd2MGC=MXr%50w1DkE2S$MKhMa0to*@*YLw?Ni5VN$Mbc9(^U5d4j4MZxD+YGx4qRc)mo%zEFlXZ4FHT2dcnp(q5y zf8wyZ=PsN}UjGOq7rOIh7|1_f8di1owxm^;xES{(jPNns`y}K5M*iMNGD+b${-giT zreKpz#sx}BpF@A7&p9ZRNZBN^usX{`i(Vvs?GQpPCRg%t!L*xkC+u9-g}W?gp0!`= zmXMY>lh$@N^=i;A4)0Te;@cr%!KLJV>@e7Kl1ZBwIc?y;m(CZg?j$Wbbc_G%6#jQB zrT~8@ed%|cjmnc_jF<=5k~67{^(u$rTi*Bel-}6JNnBy+_-lhxG}(LkB7Ya7e1@XW zX6hqcijr-CNt3h%`cyv|p@8qE>AL&Wl|K>ck0dTcCH1BAPjFvK_x^MA|7Y**p&|B5 zJN6m$cSXJRw&hF9=I{KQi3<5!{X&4ZE>hZ0aA89;NVg@aXQvbcD~{<9OU8?cg{uGK zO+ul7r9#R_HhHKJkB0?VX!{b0joQiWG?FM3t36mU{D&y}pPPsN^OCZ71@Td5KOjvf zk|}NEnw;2&zu%=a5@6fX@7?{TYl!_UYJHr7dv2`3fG?p`*~tlA@Yvy z|L6Xs`AK#${O;{OZF{`Pp(`o0b*r8kHNk@t>+43GoS1IX^;D^)-Y>pik8`A0EcTMD z)o3`9l%yVo6y$bA_gdFYXm_L)Dap~zC)K-Fq3j&EfZe?%!}nTUDuF1f?Tt#c;e9_ z=T|38-T!A#?BL~njyFVGB_9;&wtm9(ubHf3+NW4=V4Jtx1ucsWNAD@@#p0WiwXuO- z8t}34Oa*29N2=Z8sjt}etRzF6A~#3d%{ExV9>}bb^_|;|kMW?%$8}wpbN={TVfjhL z%Pic>!ZKm>!uL=%-C}Qu8(qlDLl4-OyIE?3Ygk02QF7{6j_|i}sF#{H%3mGt9St5b z8w6=bfW%0EbF9Efs^lf$;R(d_#y^>vBbUKZMRMZd7Ws%dVk%!K z8LHQHZP>2cv|3@%Eqz}(jbUbQ5n318gYuFQxFKnkmUQ=iwLD59bt^cJT%~Mg`l$e$ zlrf`UhYlZm$vGEXKIibgOzFsjGOZB_In)((v;8!dhzV>PYa4gF`K=G<=@54%dighu zH;R`DUa7@ggFLNYy)#j$vn?TAp(7#^(y(~wj{SV?0JtIMe`cCFE0fZT!7cq@m_d$QGqVV9TMcO;s@FwQk`oDK3NDF!ST%}0=3NHnFU?X65(woc zwOJomXPFbo$69WNy-T$Spl25`;McojIqlM+gGKASKM=UoDU(z5>F$2Ar7C=VEK5W( z7(;^*ryCL$^;w9Co?-PtEsBLqj>uV_cSWNLtZ-3GHTf924c zB7t~2aLneeZNfiG&`|BWqWlvXBw`M8<9-ro5d7<7jt|lwFRD|mlK%vV>vHZ}_T>IS z>8`=RFgJBsO~_DD33Gq(@v_cBANwoW2037vtNH8hx-Vj-3I%8Ig@3s|TI$HBU-GFp zZp!7|DQ@lEYZb8lQJ6Cu)_r|>c7rSUv;3lz`4ZvHixP!QQVR?>yO z+}P_-YHUZ;+gf6u94*=c7EfTa0aG}L*{AQN0R2keJQ94<>Qlkcv;WVRe?AJkCPmlH zery}BW73*^?`1D8oxk_#Zo@r$>`_;{>CEs|L;qjvW0k*98|AK?Z2rn*T{cFYfp24* zZwr=HS9^q#_r1S(C;ArcqbX7AAfx9CuLDE*M-5nQEg$1vIl>zZjq95KBwltFZByhg z`ye!4w+;TAcdHVIG?{zFI6eyMIIG95ecPEH3g zs%qe&6pE|cG`;J4qHwJ{84>*!-?t1OF((;;NQE=q2h#PX?Voj87HJNy1`l-VFe@9YsXrp<;2%a*B3=6I)~%%5DfTAR2ye1Xdh_0j9#$9u`DnM*vs*xZUO)o_V7tli53nJ>Q(95{BzDBTTQTt1|ipkLCTgQJB@5r9_OwiN2ic&7F}Q~#6+j!i<7 z08s0iHVC>TkOfUK8;)GS-=eCTn$uV*&SfIMK*Oyt#=S8TN1T-qq2smHUAAq@+dS70 zdml48{gY8@H2cSgbI)64w$2{h{juq(KB1{jWcgYD$Nl@G<^`5g)p)G1xbWT}N*%T% zF_pW;_nzYLTE4?c#gbN~9AkVKdFC56s@Oa?jOzf~rt!wm+kr&}e01MF%Vq4E{Xa?r z%3FgbB#`6d#Ex|q*bMPXJP14homtLu;)-oZ!90WCxb$n)h+p_W0f-~wjCIvARncbd zrFRC)R%B9xkwigXKJ(_Si@zDDExXL;Dz)A!+b(u)%9aybXvZ|(jj6mL%RIq+s_xrV ztp*?u? ztxp@c;Vt|vO;2YX`3sK!xXw9k)|sE}b+ZNO2xH}B%(0enlf z2|)u2HQXn=cyAcGevnVqN~DCef5Z^TA0Y_HueBVU2s~q3g`p8vVbSbW8Ni(QdgfyG zU!xM7qoWIpYhNSHFn6o^=JR3~oV1e%j^A=xeBJi?v>wNjI-+ z_)R@fEWtf^%cI%y;cR=x52A&Yty2=eP-nZ|n{Sr#XUa*B+O5BIIS@)Q9P9T7%ATdZ-g`q@Aoj?%bMXyf%xxs=JBQ^1NKHJe z#?(7Nz}y)MNiEmSk+l z_7$Y=8KrAK2HmKykRDN0Jw4_AeZujdjgc zZ}4qd>AZ77MJ6@+#)j2F1-cywC{xTeFjzPdvbpa=HbVOGbY!h-<(aF=+uJ<`9_ufx zl?%z5OUQHUucCMP-_Q3^CeMmnB(^*z9<+%1G*e=hmKAaI*_*wVyt>%By!GOt@jz`u z&ST0|)?sP>kcGK<{M2=?F(Qp(l$ozDwGC}?`dSj@8*E?w%NLf|bz7Dj(`w8K*CV+r zPrVS-uPy(+V$t$qbz)xg`pCS9Lnx_O#~t}Fp{HY0jnQ`kd=PyL@m(iA%D`MdI z?PK&gyu6a~MnLVms&nLliG&5gW@dUQy_Wh=rX5vbEcr(3w65j!M!fl(>vURNC0~A~ zmKyr^J5)PHp26IG(>MPy*<)$Gju?Hhbnu{0^gA*=ZVqBNJ`fCOqDEukKJV{7H)~B_<+LOf77m5-;IZlm3G;)&TdpDK8%a;LdeQ zY}{q>QV>k^HQ<&4&UlEqtE8h8){O&(`~3yZj?`{GY?96_i9ZJ(wSUv&b`CiK8m739negf-5ZfQ)@FGpVq6ps-f@$-g zUti1t-oy(&luF1-85&4bmivF2?E8kssf{ z#J%$mQJ*l}ykW;klA(4^DvOr;jHiz1SP^&17ZFOYcdrI`qeDK&DC9oVQbAb|G;_d$ zVhsI9^N6ZpL9VwaLG_;-W^)J7njvy0P_{IKnIO0tg;OySxdL{hgCR5mEl9OO7V9*S z+Y`7Rba1KPHnuYP3PPsj&nII!Zn@vipxcr!OjBeWo8UUIyYydIixe~s_hWQy*jhZ$ zG5%Em6%;I}i#WzSo^=GMHZ@Lq z565Iv3_s=2Xbynx(0x;DhvyUJ>()5#IgsS%bZSX-8%iDL+M;-(2dG{^mNy25o_1fx zuq&U90g}EZzF5vrWQ4%jcTZ;UuJ;KE%q<5C-dTYvj4nedZS7^|65=m}f{yX)>3e(8 z$}ypBprZZ{B8Wt;;nbKqx0?8R#Xp`v@=@OOJSo{t)pxcZP*&1Q{0?Nfqlw+$Y9AZ? zuoSDk?ss%n-h149>}nh?8OKXE6k#d*o}Dy6$$HG4Fdh6@Xh z96WS5WX*XLyr*#0p=x!$;VmVmzK=?Wzko0^2;rvO95Hu?EXgp(>d7NBtHO%yrZz*2 zD5F?l>K~NpC3hZ&3|C0ci#{#TD!e3)TM8YX=0JYIXBsQZeE`IZ_0+|qjkv&qDXOQF zACi^1>mea&gNf7AAM_oQly5Lae`cZZarUZ3ZIksE-c`C~_8*7rBg$OIL+)B7Oh|-) zPp{7wq!*RimY$>b_e~77+g~234Ac(rA>`SPG^P?yrfKswjj8kZo}5lu{nqS4ziCUVoCL#Mu{;h9eD^;Ni7YvkTnTd}=U zpT_;)PWg}jf3>Y3x@oovD@?`n?PIA&1T*c}5@*f_K4WOsr#O`z^=kriNr?D&=G3@& z%ANh?*!34+cGur`?vFi=;p8VJXePq&%v-9pcxHTt_wDjd@;_Pts9bb~6n^<T=S&z2{c}JuPiU!9Y$?DEDT#xrjBhzS+i+yZb@28zIB#UXHpZ;N!9VEZU8X|G%Lp zA9h5CueELwY3GonW=?4&Q!Z^gGDM`fdKBu8`bs#e-;~Zqqx#ki_q1qo9J#l<_9j0 z<4`prQeYe^xF}Ao2P0lz(QBN&4;fc$U25|MA|xUy2C0-z!APsdVvG=lvrP?KxzLZ9 z0Wr10E5EP*;|Z<+%6G2kA*4C<>6SD37XA6!eY^$_UCood)_P;46GXu z_VqAZCU3v0Bh22cOkc{I8X1{V?RN>hn`M~IKR%`L05OFk>Q8#PUtqQ@{QPlT}X?kEzS{0p;8AmWT`Yz^qz$2lsWSG-B;F{~SfpwiO`r%yoyfqZ|N z`Bf|OjFfccC+n+*>uf^4)a>583q@k88`nQ{njAae3)uUi#)B}5`9k_Wm)t8FvuB4p z77u((i&!jY5rk#fo(9-M7A?`ePruqjSepEl(xyB0g}O|U6r%>(Jr(ikxc%25cuVdg z?nykR1Zh?~h4~Y&%9{0Y`1aQ$vyL88PQ2%g_67mGNoj4?snJqVDy8z5v+eU2$hr-r z%P2XN{utGnZe#@xW0-yY+!=9;rVCiEE=F&XdyTkC)F*C!DAm-8plhJ##C)KC?P*6Lw2;2tl%9q{G?_{U@CbotFhk~v zgM)s*ePPK6$$Tz1ziOHQ(f6u-ILHURF7TM+XjJ~^F~3J1GmAdWabAgmB#mRStszg<^Q_;Y=yt~Cx6JTfggWni^1fHI{i$XOhgTy%Rnyf(Z|Acy zko+f1Not9*Ul*!4Dyj`I3XfQKqrR;Oz?OTAB$XvOHK|LjMb9)53w8qj@$->isY{0i8AOi%*-4!|DbVo{XxTa zkC88%u>exQuMF86g~{E}=u_G#qw8ST1LPFAq+#3&_57BY|9NbuFtrfyJk^N~iWUQCD={wb%In*?DJK z>0Ly#@>>U~gh0PY=sjJ%Qf(9+VNa)C40&GcOvmgS*LT>r+n#K9bRSO`U%VU zIUcf#sFeiv{&aw<-~;zT9<#UJ;u5y`7|VaDj*SWl=E$S9+jAw*)ryEe_GYPH4x{twj4f`V@fAjaz08vFPnw5^*QRDNi#IzKjl`VOLd$=CC$ zbv%{p={EOGz`;9W{9X9AvaW~>-fwvuEA6V&9U$ygK+#mRR>!}UDe-rAt~OTp%6IzI zo3t}}X>CraZv4-2FR_#Mogx!kGNc71M;c)=+4~CbG-SrSev;(rSxytZRrkQ36;-^8 zMV1kG&17~&o8P9cedCUx-WTqka}n4Wp3jz?ekh^Jb0vb0-Op1%K5Y7p0qiQh4(rzw z09C~ZCCgqqG)A_VcpGvls2Fsgj|LTqoN%N$-5fE6%mE)nA9 zIKT*_3Dv>~5Jph57%&6>1=LAd?qkjdyBN0}*AV|Ccpp;CdA8z(p?w-TcE93#PZ@@G z>FJBZ9?~rfuc^LWqp_U4T49wpbk3RRYg7N71oqVhF`NM(QrXyNu?}DE3JIjn6?`ibG#Cvkl~@ zHxhnz8~<1vyRF}G&eo0aWofIwYtShrEtTZN-F+0f=dVlXKRPLu0=3Z$=(L>+y!tkP zjmLJF|2=J^+y#f{T8J_TP@vi&DOGB*jCn3ae7%8?PNbxPvT_@t<)H=xfIvL754kx{ zG0@}!pl8Zc@fSH4fALqiCni7uj#>yBx9)!dUCD{aJhBp-p&n2R_|g`~Q(py^%^aZS zw=UWxmPR0>L#I)A$%_Dm9c5&aGm)=jSN`n>G?~)~PhYqsrz~uRrojC3D$&sdvVa~{ zNg?)uz=v+0Tw<&TdOfM9eM~&J8~YoM@;}hOG8>F)FZ{*nM9oGs9sWyIYylN)p8qz3!b{I-{aL(i~SSl2JQtAf6) z^OYw+{aw^0heF#UK5CYzO08Z#Y+gymKkuk7P_wzeyq$DIQ~#@6P*NQY3?*JMCKnqj05*f0)X98ozYmvsf-0vVuSLfNohREx zl8o_d51`lU5{dY>q4QmkVtV390=*8s>>+W-y5}Guij8_=I=|ND#sM;$YfOj%0GUmo zo|L>xyNI@4@)6!OylJfY2Mt`-eg>{9r1DoWJc;sNF8oaG1sQqTw08qCHv*SvS0;ky zZq>ZLJz!cmy>oFdfqC{ZMYS$!Al$__Xwf>nqPa45AtdFi!Y;(_fxSA+{+=?YgZ#zj zE*BCVl2;o7F-TjyH7U;K@=xu(ZOZkkO^K%tMx)-!l9h{&~ zHRfC|JHB9d+44&g20P9gVcif5fQ#D!34hIf!qeKWr0@`;t%Dc+e1hyw9Qgn}&ePZ$jEoU_7ew8qE0bd&VvZESU(w~UxU1S`hEPL$K} zP95|wTa0f%H5VgiF+ce6SM96}<>gDpB4@GjQ}M20Ub^N`v$bm0zqlw|ZZT)jsj=dG z&&6fqky}~p>XRJQL9N1{#}N#X`-wB$i)>cTpG=9w3p*R>%pVP#>PW7|x=o-IpKa;n zd$P!#!oY61GYL>&ag24>>6T^%KZ4sZ(2>t`*O{$}v9Jc_h9Pp88Tw*-$-~ac??Xje zEG54Hq@Vm*X>$3M(D5%=MHl3Ims z(%+Re{PO0YU^NGI^3oX2?z63gbp0DK!n;tFNdz(j6XiM$%x#o}kAna4iJ|IYIYOz;&QdS)o(_AhsZlzPaYq0w>VYtk6=WfL9srTc8z9H z#ZkfBGS#0?vVR<#Unr`dU&D3$<3{JkM5J$jCa1g5y6zS3x(!QtwQ2RUd4*7e0h8HJ zBQE-iNm>01>WIM368S3=j?)&asLN&BB#G-&fq#Y;h zkP5x|>D{V*j{wR9DZ>4p606=1)D*x$NdM{JdFpFQF=gS>_hKi%Uqiz;KunL3G;751PT81l{3}5m-g^9-)ZieAm@F!?S zN_4lcTpQo~cU`lwgYxdKn@Wdq+4V)m#+)x031_1DyG)ed5$k6(_9atreQf?Fsb({+ z!rxaY5usVPoJuM@Y7ym@+_x1C+W@~awJsyk?je6>gy4t`-}~zivO%}Q98!^D$Gp&+ zwL53=oIIJhc-~zevC2&P!&k@7AmisN8%A%7%qWe^(-RPU1HY&I+^aP_?_1BWVS@iChh3Dh< z1Sc1afMb%f4P;8zZjrF1Pamy$9r3|@%EBT0dZLfBe4MrWf1)-sc`)(PuG;q~T#ZM% z)AWkuV+-P;8QzVqkP6lYn}$PaXzq_$i*bP-`L%Ub?6|dx`ixP$ zv#M3y+OOtqJ*e)CIJw^`P1uMzb7p#Mv;X$`#~7MgFr5$6oN)%RPxLiE`|C&B_yg?_T@MteU_ z#c}FYJh!DUe;lu0PeDP^zZKZg)LUex^xhycHg#RZO7D^t2nF2#RAB((0ds&(2q6sw ze6wyWgF*vcRUxa?)55&xHrKW!0blQDVfp!bx(5zgSq^(zsvndXui<82~>L& z_q}`f%)dUzC;kd?f%b&cu9z!`$q)J}0f)sDkymj-lp|(aV{dGxAK!eREadP$J$b&4 z+|*gt-wAIvFNJ9gxc52GyPUg>e)4J^ur$=JMAG!G;+JoQc04@rNQeg~Zm-m5R! z5^lS&`qmyDti5Z@RboVQ?x1<&?cRb7$>|IB-WHx!sX&sNgH~@r==Xk9mBms!H%-%T zrpupV1nitVS?F@%^}{9Njbv7FY@7NG0c9dIN;K~8=H@BO9e*fe1tvu|9nEDaWW6kW zJ}+lIRDTQ4Q{Sn{o{5taJHDHbeTYn*@~WFkKPB1B&QIHw))&2&!8oWLb#6`)+r7X% zd)4iSddMkEam>QPjN!X+UrL#ia$YT3auxLz`+x;ARR>jXac)E-^}QUC{lZ9D{`;yf z9GWI%Dl*XGl}FT^uep2HB`Ne6`ZY&h-FsY!RPCW2-Pkg!l7W&e>8mGqOL2M!^m8DeJSw|FJ+uqL!4IMV{gPUFx9F?{^Nd; zJ>~JyoAodjcikepVQ?FHAWiAsxP03O%BwN0e!Wlhu~9V(TlCAGh^@E6v*ecg@V&vt z_5LT(-iwIA*g5+dAP2p*b#O>V&v3)Ps?T6pDPwY7n5Gl9HzcTjDil40alx}S?6J;} z6p7f(n9q-4`XQ727r~m(I#UBc{7?U3 zaddc~w-OCn>#Wj@YLZ@Zw-2keu>2Ij6i8CyESVK@Rh--{avveH`Fu|i;l9k!YsXf+MWt@7 zCTmNhI{#lAj)w!4qiK;PX#Z|Ul-OirPi-R|`6Dxx_7FCigG@kaeM|iVr2BEGrDtp# zbonr!td3QdJxT;XiUwcX-j=*1fBClA-7KY-*i1<(m!eUiUin$@2uY{fG_YU}5$Fp_h7X$}hHUyWSfh-Lc+GI&a8gmr3PCD3(nXSe8Dw`nZ`v_ z%(mZRW#z9u9z&*mylShN%<5&Lggo0)u3YB}NC+YP564o`fY7ykuY=qn2r%10n`G2K z{Ek+azy~EM8yH~w*7N?|yKzfP4?B0C%KYKwTK_YXzx%AkhFLZ?LtYt1Zxro--60Ep z`U;6qyd|$-(?}biBZLGB*65?{^YhQ_!)79DA={&eL--MoN}`JpLjeD92Y=z9YgIp+ zjO_}^0M(Tf-_9Zq)7ykWLeJxL9*Ls>m57zv3tY`B8DKIlTbo4AUTlL1kF4@ZRm|+= z5b+hFNWt-^CL?~%TJ)Dupnc8NCkPW<-p1c}k~hEjVq<{VuJ+)uoJ2*fPWk(H=PZFQ zd^`0fZV|t4-wYVRC)Nq~<8gVLc+Vpgh5P*`)sQo1fS0h^pqiuUU*=}9g^>t1RoZ-U z&U{#FbldQis2whMvT%a7`$WLH`!*^8CA+LNdOj)L2OXh#OGW{79ne$xMH79*{Gvw^ z5hNQa!ZIz7BHA)l#JpM1p%#lP#({P9I7xNHxLgv$lfMsD{&j?kk`^?dK_Ksv0CU-w zg{VygZu*!5!q;=4Gf@<6Ra^`H^&L*F z*6Y?AVLI-ce-B`=22T=JOStg~#6^%H6osGq;;-bD&(Y{m)1E9y8|aU>YOyXbAlI2u zYm$gH?zL`Qfv5IYYkhQ_|M?7i#8mwiiHoi!Gc?Wl3eOaaIMpX54D$IIW$xFliDzx8 zR)_s#;ZFIy!C$}{hJNai-F$=En(IK0u{DdgyTWj}Tte@yWNbX+?0rVsFNE+ec_9N& zF-buL1Cj2<_oE+EKoc4f-pPQVzjSGvkZ6%Jj2jA!jUj|?r>ZkRT@q>3Men{oFCI zi!S{JBt~R+5j1kNybe|lABZhztm@blzUVf)KH9U?m%D6k*fo%b!s*v{x1C6;Z)-#3 zT_Q(s=f{G;_U>kqZuS?piiW_Ey*xhtG$=`dzHzIF=r&n(dvti4V^`hv>iN@XtCL#CJ){}0 z8UDNCU6r`FQDW@zMnnbItvjZ=F*qw#Xcmr3ATI3KN_AkPG=6uAW%3kn&TQg8au5Dd zS_3M|eD0ez6G>GW^G87A{&>2iLfU%{?C@3V3F5f)phW0~($#(I!Qx9Ov4;szC!$i*U7qr9Kot!b9O zyCC8>qc!;>n=J0x=5u=Yo9NXw^~#U)2~|Iy_kZ>btK5=gJ&vKvsVu|yr0lVe-_jk* zxU23}SB9)ZTZ+?v-EY>%Ah{!R;*Zj>QAc)eM}HOFCVuQ)#XmOHP5A6UkU{9`vJAXb zyhtl@1^^Z>F5K#7aEDkK#5}H-{_V>CGe#dUOq2fkEWcRoi(^}STrDvBvfqp1#_gv zB3W;btr%DEDO0{Au3p6qLbpQjkAb{*t?*d&F|M!s!BinCSueXCCc3{L%a)RxdX#7a zoZ+~KA3sAzXMk=^JE*T&vsqvrujLlFWi>pP&_Lu6No6M|dYahcZc^P9z4gb$NFbG8>| z+ws0emhh4X&x|^+}vI1d$XgU)xR$)@G@9!)G zbR%sFMmRx6bByTqTH!UACVMHAcDig?$mz|_(nlmQoY^TR%0Jm09E|FTMbA#Q#e+L* ziT?y1l8l*{yWV`)7p3}an2BToTQW9`H8R(=gu?FPaNpWMR$YCRr=PPQHDGJVNC=C+ zy+b4s799lw@+KyUwxxxbAU=4>!S(E_A{LEv4A#LCRg*n<-bB7mQNCC2Ax7Gyxoshf zLnJN?VoJ!km*P@FX$aovu1-;Yp0Sk1I6ebcn0kAWr0D`Wf?~Awd1==9u=%5y+h^(j zK08wgDd(e|UmI2%oOqXj50Ws=t__rU&WP6l21w;W_)XM1T`4vJ*dD_U1a>61=w97n z>GBSQ5SQbHSXJT@|KaZM*r*yw5fI6f3g?`a=T}N90F^XBIqfuRYOM zSc@Fmv929Biw0J~{YpV-n3b2;Q9$*WtHk@ZSFPK|!$$5=`sX4v>1w0~B9~t1i6;8T zSATJuKX(mLKYrheot6;6P-V>kOSa7!(_Gd0^w7)~lcZxBTMW!PZ>@7NAGW7DJ+lkXinlW;iax~sunQz}d3xV^SGYftfBWk^$RV2)Cs;vyDima_7@?1Qy+zLTL zEnyfhRu==6#cV5}YWCS7!?qB8TZD{sdC)jk?=k)gTQ4p=E8ZI`1%BP>IJ_!uzd+z$ zJ~#0+gGU#ld1P;+I>${0}ym=!qm1f!85|4nOd@jMd z4M;07>pWv3@LF@_Jo-tKHcb0JXd$AZ=DVmxJ3Z}zO_u>Z3q%y{FVb=b8D)8?QfAkx zyQY^c;#fZSr8^OB-bAA*jC#O)=!rIwKwOoN2mU_GX&TNQ=P)ed@IPLPZ&%e-0V_l{ z?c233b=-qe;*HGdL$LeGv~a!^U{CX3puSk~ z=TpCSD*MfkI@0*Q5z_ECX}wO5fOx7gt5xT7I?~su#97{;9MAjhNl_QYc4JoteLQt4-iEkB6fJ0Tnaa$qf{RM+OmU|F z=JHP*FPSn3zGF^EwP2iT<52$JbNO=#aNx-eQvNZF#XDzjW`i1So(kL%6Sg(bw6{%s zg~Imry+`Ip4b*%=K8RgMd=P@Z6L6hIS{%#Q0ojRbX;FNC6Gwj$WOXwU(W9yyzH1~- zMD92<4J4z0?Y>va-cE-Ap-nsHYre{$NdB5h+1gHlXAkl5o3p`x>e+Ht=b)G5_zn$` zeby2Y8GK*tlD5y@Jn&z32LBBQR{vh&gPy%h$1xD%NfPTIkp3y$FS5LSP%&jmuStr; zIg=D^U5Gd|Xr=biO~b^F5AI2=R0uN$Cbp<+zXSy^as(7rD_cV(BK4sFQL%bgpWmwI zO7z^j5#07RQUmi_vCf}&%;)p7wC@Auv8%B)?Hw-f-Fx@qAtJ1< z?PXW0FpPyG@wJFI;|8RnBI#&nf650FuO!Muf5Az&nqkTF^>IDHN0Obg)$q|{6&XX| zw$H`r*zo{u!}?mKS@kM}8aALA^*Z`l>-w(4-t%Xr439$@){fXhIcd8=&1zsvjiJK8 z;TuJsOF6p@3>QH7;@U3}=D2OG@@|n!)VEhD5D;bMEh3P3b1T^jfYj#jVmhWQt15#! zn_$|CvfO*9Hx5+6tN?ScC#4m^Ts4CPmcrmUH zouCoTLZ@Zf%%<2{e<<&9xa-IYHAzmFMOx$%Fpgl-Os!G0rI0-1jPY59br|jpVE%A79Tu3L{LRbZ$fNLW7srOrHHaE{MY#)*V?)E#2fy#Fat$S zs{&^i{|H9aZU4$kzOPDr>i3vWO9&1$`0y84AR`3;11Y6Ln4R}YBR_J>(Sl|@NE23e z8{!gaEgodlRKz|h zs`JZjgnfN|%&sw^&_Ub}EYQAggVHnRoo{V9I9`ihmXeWqL175lfraMFQBW;v2kEGf z-u_5EMKA==+RlV`KVdF!5#-*l+pUG$KAb_kj!Qt^YIRN67-pRAjTF-3mwJH!xQkM{ z6=+1HslvMh>xSl^VtTP<<0LGmJ(c#=LbekZDSbawTW+ROh$Z26e5uvm62 zV^r_L#_~YtQ+8lIKQm+iMO}VxB%|j+r2q7i5f6y3jhT|i!v(}ybMr1kw6IfNbxoHi zI^Wmg6x_$8K0H|SLZ(}Nd#st1Gz+$<1d|4?fJN>NXixsQybXv^S#)O6**O|w+VJbQH&3s8{L(RV9tuBg)L9cNc|_# z(b4D!R=*H&mP8B{ejS{M?d9h$_|q^C9Ibt$FJ@7vCR24&#?{K8mi$mgfD0 z9qdk!xwP;Z2qR`{k!l;DPHbP&JFBZ*iqhHXC_IC6^0W{En>PhL=wwuq7zfpx6|3}Y zX!`uCfggKfgntI3jgMa{!L^hvq3SSKrjHi)ak=PP&>L}ytpN0oC|aW5fs|3@v=~r-1C)956pxh}t!x4_|gX{Xo}2=FhMGJJ_MO2rzL_?@H%@ zM2lV8hYQU`d=MI0WtA=p{()?R`WA>(+l5}AJ3o;pfM*Eon0|qE5U=d})JHAmY>vA+ z59*LPY$SV^+5|?SI9c}#(f<{$>5O=?1b%A>LZQgu-r@eWpED)jD0Sj+iJp2yt$YGWCvNjEc4fB_1+!}N9 zDq21ihfd^Jy{mMzT>yZDWE`Nr!a7O+rYbME|%(~AEcp~1|zCy0G2 zvl<6h&tB|<9xZD5<3&NVeiLVu-X7dtEOppo^aEcLLFf>tdwM}ne-Z#Vmy%_e1AUvJ z;0Cd#F^+Y9_2VqQo>M+hjN!RE;!Lv`L1w3k z1JY?j?}_T}I(Sj;?}zCKaEcK!U+jJ3mJ*DZjU58c+(AOxT_Kp0Bj^4JT_=Hr9|ywA z)Zd)kxM;WNjQzdkwu#5%AJ=iperIPM*Q|ZA@D>i1ER(-19Nb=W|JS6Xq+nK<$~)#{ zXDe_jTIn#1jsrmBv$u`I#}4B0gfMBq?bK?&I%MrSJZ2YhzT)zY!sY0P0o;`;+3^n; z{Wv}U005Q%1sMMH!>LzL61PJN@W8X7=PIhn19kW}_FwVe#Wdl&6IJ5yq5C;u98Hlg zKa5cxtea(e45JK@8biF>M$unOi=iSjblt;g;1#eLuD7rp^EpyTG1=Yj{)2FYb6^}q zE#=WC^~L_ly+ZQ)KaK$wsko-2AE-rLWqOiE!KfqxWVi>jtJ@xw6usu)(E)r>qF~LA zln7xQGXCfmGMf~QrRo5`!o|k3>j3`M;1Ze8?iP?$nwQVve8uKl0V#cCmV$EO{O|Tg z$cUW-0~^u?{KN+bmU$!7_?|5MA@c1E1G4t4+df82ed-`fDdwHwg_ztyHdYD^`qU%rVT5_7bta`NUcxyWkEltr!a3U| z1W@xaM7HB?X7rN4O6eOvi+OyCk2+<7(Ay>OFnkpSS(8Y`C`9!ae568U&zoqu-C%fem&u|KZjoDIbF#$Rk!3nif&%Ldl%6DKjk?#s&^I! z%~lUR%>CH*o%WFp(prE}0iAPDK|us;%U69A@TBie7Sz9Sh&$TyI*R>c4;tV) zpS&&byWKhgV;!E8>1SLc63v;6zKv^3Y<{KuNtj+(yLcg-x`RfL||a^o(0Z_ zGHQ?t9k&r+CO0v@ja=EcR$N`EXpJRjlF+Pdu01w3(y}M3>{NU)K7VY1n6CUrrsuf+ zo!fbDYAaPN`>(HYY%KT5&Cm-k~Tw(}hV&;9YdB(g; zixzx;LK_KL(MyC3S#jvh`dl{+(hqA~yy}<(_}amE-Flj;jAH)<9)4jHuu4Kmc06SH z{*^L7oyhPvOQ0TDv(<4>&1ZWeGqC#0nK$#lxRmR1URl?yI$_m>061k7Wr;Skq%J-; zdF!yQi>TP*rMNV0=4I?*l9;I>@e z$a|FH%HiPPNlT}Is0%vM0gO7CElY*%{s-<|)(ybYzXxO1BBVxHKGt1KWX^Z_QckY! z>0tmSi@)H~e(s;m;eOylxW2v=f@gH6(4ZcvrBCy7eV+8KVZDN&>X=|iKD93-D()Z)&>*4EmXQ0zVXO8C?g|p7e~vT#=7|9 zScAY1-d4XrY0@HT2vnO;CT7&zv@e7wW;+$p5{Vio`tFOo`S8HHzM{W{5!G`gqB={z zVnt59~!e=qYd9s5$vWBa8!62ar~$>y3glt$f~@put9gk#wk!*iphOY4!r_sj#u`h&zxxW{?t zMeL;)mQv)$&o=0$VIbijhX0SP?~bSP{r~3}nVFH5J+k-8I70SFW@gAH6++fYR<`Ud zWJETR)xweND49hWq0H>xb)Q3TeSY76Jl+qz-S>50*K0nn=j-*7!*b>FY}gsT7J_!U ztpLR<*k-Q~)#cv!yC77*g%Cn&>E5_B8hSQLCH-j#lal#EFYQsh1DdO_N?V752XOxa zt<=*YhHz*Vk|V=dFsY!n*uChzjJ1S@ftEUMg*JIMI zx*-0DUx3Upb{6ytC%6kH?xq5m_)9lf0Kx$enidGbf%UJTskJhj!`H5?Dx91a)XsEx zlB@%z4{;H2O)oj4>VQr42K9H>Es~Bcbetk)%h9}l7f(oItk1Fr>UIWI9O=UlF;tGVE<6ZGDCd==f9of zIbPhtfySy0dgp{m&&yBlu*zq#gDdkblQ6^8$rgs2l%RnJ0^)2z(_!jYAQav{TIo?yirc?^Tej!{}$wm z*|Iu+&+0nZ#(h{vXxj zL#!6&JNrqSnl4`WFVu1YfUe4_JOFe}eUDgB!k36>A1u^})_#|hx^Y9V^c5&TjB@u| zll>@1-r4SqJ$TO8xx5ZFWC>zpnNkCKQ{B@#`NiV=9zbJu^>hJ|=L)uURVsvIeFh_T zQB&sv4mSJx_5tYfc0P(5_|*>+81ppfxSSU%_+wXXLY-=OW_z2h(vbg^kCuztTYp1B z7r;okvU&-u)5v$~c%2dL5Ai9Wwy`euUkWv-p{Y%#X;U3QrtF@#Y+YBOcmS#BtXeLR z$?w1+2tsotpoC#5rj*1$tUnLx{0lDcW1QQbGlSb1CLS0xV-^b2KZAG}m~|T$e?x!) z0W&ym=8nlzIcD_P2YyIa|E5OeMT>Zu(IR34v|qGk$zAtb zI_MBsfMPxL;j>U1H+xs5fjA~j#-}>Tz9;ntNFJP^<{;wZ?us>l6qpX;NtySXR;=zcpNJg!9{d)>BT->D*(Xa7d|M^0z4uVy^d@?9eu}<;bJZOb~ufcR?KO( z!MS%4l){iKnL$Y4T`>LaJcE%;i4Lfs{g&aUjDeuXD+AZMBYnAVpNmHaYzMFqRb z1ITb~#(?ExQ9I(c zO_#*Fc8DYg7Aw3C25i{P{sN4{K9`owk>~!%6dqK`*AEaS-gII+RQWN|BGWp7a7ZYN3&E z%Z%4S`esM_`Hsv7+gGsVOiKXslx*ZVzXsHvtc&zIkC~$ReTaatMQq!jZM$_=pyoueMd%ix49Y1CbEjG8c(N!W>o{Ln4j)xv-kbHnspWrxut4C9Q@D1QSt!)4y(@gpG&v+YIz`C z9=|E=yiaS}s7;L+*^96o+>4(~L8nd1T)zP6I0Hg!=dr%O^#CkxA6((LUly=#H4KAk zqYc)5!dW}aQUGm7g4PRijj7A@1mnQ4Ll=JEK?))Ua==iuoBLj`-GWQ+a9_(l^LFUK zXW4`yfQ4t9*}6`73~50C)Bue*u(w90FG1EhRHVbGBKW2|iN|#r?dFd0j8u@%b3MHm zS%qR%*>dNSq{PIHkLKq`7wlUgRqh|(4#(Le#1Mo@PhZbp{+8rOAK+g0)UAS+Lgbq* zPYUK)-->~~2qRw&?gtMTnzss3@IDBIcky>XI6Zb{5Zqdbc+BgXrhN(JKZwipa3nCd zV@aB&>LIoB3e%5K_+o{bg<}J#)pnpx{YpxzswMzCRmW>BI;SME>Sn;rj>oX`D?mK@ zcKw2YABf}VLH^VP_yF%LNBlu=4c7^03v}L&vdOwGAPT4aa;d``C9lF{{ju6vygMSu zXjy4#^=`|@`^6{3YAMR%49~pRM9FXT?RIo&Df9e0u;|OCanl;}q%}IF40ojliz?&` zv{x0kV%K(m9-nqCHkc^q#77t>Qwnb6$6m;)YP%J^S7a+HZ!oqoAZuEx55~=TM}(^yApzW7fMLF_6N=N8Y3*R7E#z{HBYVrk#O*9 zV|?-A51&bF-RUZEB^<02HDfHxKbK$a5^lLQfA9u`EJa>!j%|j+s3=Fta2mv;qR64G zTVPnW9f;oX#lC}3Yh-}`Bh5JvuQTQ&Oj2Y$alCpjmccXyRLtaw(Il!2Mh}@S)w%tA|>}P z-@(ceN2f=&3hh*H<8m-?x34Z~lHwzXSE6M!=+vVpB{mLck~Uf?WR6~pPNK6FfWHLe zd8i>~JwTad+M`F0X01n;p>gKj01RE`TIWd>yzziF_X%jG{vBf4PPq-wU7d!yVS(YM z_vott&FVWepD2-&3M4oe0o7m=cwo2vT*`lJBaN2m~iG81hWOc{=BF+ys2EXaVgo^%WwH?-3eIr7UR&90xbiTxOvKJk7jjJQv{M$hyV8&{bc0Lz^O7vYhHBbirY(MR z??x&p;jLeWD6~@;c&0YuConjO(Zh2ANHUNMF6nMzSt7R;VX&ea-mf zTnd=t?vN_VxaZ|}$f`cNHR={}WQ0X4>YOfHu=VVyjn?Jr#cy>9I=YIWv}!K|7(L?m z8h`MwhJll(M6`qJ#rgAdfP4laVKZF!c+LYdo=KH;NVyXTbVQmomAR38sR(usJ+dBu z(*|&R$BTD;O(8`P%x8LH69gu_zR2fooCf-(&Mypt)w_vT{}ZerxDd!PbTwD0Y)}{J zUG?qm3+_5di$I#eSn=q%af3A&677pbar6kR`-5<@JQdoW(ly7)^uBGCHW(f0O{h>n z5ZBeHoF{g2$9HhxV_5Kiy_#RCqjBJV{HH8b{W{G@+igdlnF=>=7UuUnn+E8Fr+#76 zTP0p95s>M&<}*TB2@an-p`;Wj1cG&L0H(aO-BEuQKy9NKUUI-hX1Gesa`0dNCP$@5 z=p3mF@6W*ipK!1(nnr~;t2-^aaKB#XC@{Y1%*oY19;q=!S8m{qiVfFN%roYNX?Dbt zU4~4-qY9hn53K7tBm4HJKh}FKM5Ja${?t#|ILzI~2Q*RYmOeBg^O2bqNY?FM!R-C+ za+5?(JfP7tx3=z1y!ahZUF#he*T|tLq&zD6HNM~GMV=DxyiJZtfb7XG!2c&y`AJub0RoH1cq;f?It;k{s^>+I)z z0|)kq(7?s0b-dXF%i#e!0K7} zwE4oi6Y`^+|B=QbhidlhP$=p~27!%ehvO2naM_Y7zBVpT$tpeKHQ^ON_ng$>DlZY4~b;g3)P^$w8@ zYsvlV)uVSPMMbHgR>WliQNX!{wC82O;pLW{eejV+1ti^fN8A5lS zUuolDYl)pdP+H|dD>pfraitaFnkZ(hZB!ncdwqWQKMojz(5hN&qR^;$ z3T#&>bqIF0-*WL5frBEHBkLx^XuFtdgt`)OFQ^5>JUW7{HE4&Y$@+7Po`{IDVT;cD zzul=)cSYYfb6rP(Y2wly&k+O)3*h=n+zk`2$r8Y!!Cnm?0TrX~JD7x5m$cAvh0d|3 z?tVIGM+qu7AKhm!G@KZXG7BIE7%?Be4)X zhq!A)dts9*$G|@aAW)v^q$xLR9CWA*GHL>`><2I{s<||pUnD*ie2yJ7O{Oz=!JDpu z9k2E|m^J&GqrO5~0}c|Gftv&!v5dE#*|18$+%z;nT?oG#dEy|k-5-#j)0@A<<^84Yp|Gv7ydF|AJwBf<$(c`+i4qX8qoDf3}( zhi(su4;!xby|TB9NV=8I_f2c(5u_x#*I{n2h>+sEKeji^>8HdK|9{~m{}2*N%crr|ORKM4SK&?p6XQe|_v+}zw{pJ_m` zCFL!>ZBM{r9$A8js-QOT(?0NQb@fNNlDzu1Q8dsW@ z5?)%N7((bnMRXB}hX3q3e;(~YLeBNK@jAhS`Bx_UZMd=u_tlX1)hw0=LLuftZUIhN z*!DTQ$|bek9_#6`)>MMcak+9~yQ|Uu!`93kPoH!X{Hh$SLLvLQwXhEv5)vR>ieh!Q z0<}d?0TXVd=gh9)E;QW>bYpmSsSm_elR(+zpw5n-H-A&3Dy9SWmOp^sIk)b7&+x8l z1V6O^@qe-Yn}GDa6&F!S7OKd1k!2MSy>EA6A9Mgo1*uIsD8m|@*ChGv0r+X4-GLNm zD{_<`1(wef5h5>wo%upe0cg1;117Q$5=`l`B4(g21u#=Ey4j${*K;}0r~&ZsI>qRR zFWz-QX=pKFGY^)G<=TWC)`a5X^CX#mGzJM*BS>^*Qrw^T$t6UO zd_{C!1+Hio4%Gy~)$q+c29%-s@-bEqpz)^!a8(&FNEBSY^~zfwUAhq^mwMrn;gbn+ zJuR*10eYRLJ^<$0^~xF3*OV1rfF^tfqGO1G86YElq0%A9Yejz)x7&86HV?udxMVRaXlOQ=+ujQkv7gEPF)y~K zjxx2T~)@pW)f;|ucZd#zwZ zfF=s#%(mPd+6DTLx*WBq2JE>KoRFaE`FN_Ko)2j z7!gDP>gZevZGoNjuxO+W zUVcopOT!hNmwL%wVMWqVpNRSZ%8(%K2^0w$w2`^g@dbner~!{)OdYDJjNueCvkFv= zU#rMI(7D~z2a2<9f@VFeHK3mC%hv5L^MJD<_T~-fN$}0k6-OCjNzAWNgS4Q*nU3Ht zD|F#l)n;AX9~0oGJ& zXBOx?nFf-z&0p7380SIl#%HTwV!!R%&&=wak_P7ZYTLlf?o_}bP6K0kdmwNWVdK6O zTm6;VxYi*aYyxnUt~lcJgB-N`3H0!5`}W;5$G!r7L{fG{t*t2}qI5`-sh|2O2)a^* zJHd$coVA@H1q1%h&?+W#2LoB5vDFk<9|vxzR=Iqh+d?Ukt_3Ws^!<__MW11x|8Gef zs*h~R-4Euj6uGK$*Gk+VM1?Y-EbB(qcl`u_$MlLW=uyzh2W%PY^QSaB zF)_&B?PY?rm^LnYr^^V3Rb#1fH%dh$9pH*JL0_pJTV(10`m%tHlftUpIx0EQPNARlz;}`Q=Wvyn-3lF7`79x_^KeeH_g!BetEDiT3 z7VnQac5?RXhTWYk?3|nuHM8O_f0-GlR>>3;1aXn+&K%d{0TR_`nrsq>#<1b369F%V z1pJZHTb%P{dqn4?FDZIrUU|ce#wO6<(q9kGbsg%sw^eR4jd}+Tr7>Tp+?BOGOdCms zMbB~%)IQU~+(d{3d4dmGZhGOqJaam5M4cq+M*&1PO?O9gGj+FMZISJV2}L-n`Wh{` zVo;)kctZ6n(9UbCGw)V#2=we{f+M}Q!CI)fm{nu%VK_?>k$T@#BrimF`MPRQ=9kqEB}6UH8)tK zq0|BqN;ZgaT9fZi>Qwg*PbV#EyrW2{Lune9HYT_P+Q^ednzzgL=O=uo8E@o5^T7W; zkxLkJJ7_AMz^T=}5IHLD2v+*baXE1q#%5nBlA>QnX1;&t|9|s3Kdij^Ct~J1CT@{V|4x;4;2` zsFO15|H?V)1>I&4GT(+F=@0!gqKwdtTXvG7e@R;JsC0Ju_*s-@U;`o^#FpXVPFD>- zOf{r>HBZ9=o8#C0hT*<5o0d;}Or1nYRAO+@-fD-~ocm%XD|Gt`Ua;?bR}}U9HBw1a z?e&?UYp%?j`Rupo`ePu|&g2Vod(WwKD{+8i{!58LuL{;m z@62^Gn~uW%&r#&i-<<=(rZ@0}5540v*w%wTDUmwZqPN&LD>Noqav4z4HiONr+4h>D z89PRES_Qv3BJ;8d!CRtu!karCcL)E|(L8KBgD|Hn&^|F;0}mEHXB!AD8WVdQ$57+< z?S!MgmZs+({yt)cYvP{l&CtI*Vb@(aAH$ymhcZ%i3R!;;!MMT4F>+M@%BF!Z0J zo}q5o{_URi4ntFuZC@INBH&9x2#cWaSyZi%E-s8`jy^VV|4w^j{bS#(&~khDrBa|_ zfPw%5^+SL6hEk#_Og=JrZV*f?#^$9~(bC^KE7T{j^GXud^nO~U`Jf-m43N*B%;>{LD!=<6`%mM8!_BFd7HE6)G>OqkR1@;}D>8&#+ zu{wEkUQ8P?KxB_i1^glQpDfACw{R5|ayM{5T7R&eDuQAOb2YaM)4WC`Oc9({7byue zSd+tCM&&#NoYQMot?t1yUEm?`Y@7{}hW+2-ee?9_`UXgNnS?an{7~?jhzMyvX~?}Wq2nz9n=^YYXbQ1j@}Y_p zZWRrZKr;gL^Jk5$s_5NFkoj0=pKZ_Yv0GC9yKa2%-rni0h+6t+gxe`Q8vZco%W6CM z{(G<;-uKR9!`OMC?P#sbLz>y#21?7DfrOFE`c|Y z%z^#}=qpVqe`RI8`WD+{A|qpddVh^E53)vW3`1X^jmifbBn6eJo?R9Gfr3s*4G;#x zl@x4KG}b{y>KR6C4aZM~*JC@;Gw@yb!Sw_>LC`w@7}Sl}rbs9C!wo5gzv_YzKG#3x zIRT@27;Hk*y^%g$OV(8bw$}&uhdX+4Z1`_&_so!^Y)LO&SNKC#)mP>X)o6}oW*iId z#3Fqiq89PPMt%}mU=rsTHvUu~`@1O^Fm|Tt&6m$dS3B6sWd?T2p@lIDW!!W9 za6B$XXeeeLjtqXsLbhD4Ej+c;$=GE&***sYBTKQ(Ceq>)|H%VW+&#>SsoQh+tnF@g z?Y(@3NycRgG_iZD{Ks3=IkRxN*F-IdBA;P(j2JFDvoX>h3kHG26pok@|9$wZl-$#3Y@dt0Ugw=Zv!QP#z=a1CC@zyTC1w0ug zllDz}8^3n*PTHYr;nsIXe~$wb`;iJ0lWNM*d#XNDq$(MhOKxY!m2@v*9p$09*b(+~ zwru;K_s_fQ?K1tuMUtqyo$pUZnRasoZD-6%POrOYJ{-I9t1vuQ6vujG)oJC&(;Vvz z*|#k*dZ*s=ggdwG&F%;B@7tH$_4^`jGY7#32!UvA++yZLA)~raVBhANz2f$7hOb{z z8*AonGzhO3MagnkQP6!sMJ>6ocA$++doba;=e?%i7^(!Hr3Z!ntQi8%==h-E&#^`0 z6avFG)z@BDY+kA~lAQzp1djo#(M85Pfx5V3io-#q4R&fLbUvJSHga5#pIuf&3jwJG zlU;`t?RtdG88vkrx2G39;WhyEe-Kn>SUFWRv=RHNsfS2jB>$I}5Q?^qGF6xom&h7KwQi^8$C z(JCwvtj6%uTcp66wJ>=h+BXxCzIG8U8Uhs=42nIh(D)0A?z?Yk_R( z%Pnc`J6khUkadN$GuGJ3Ng&PcHue|lpapu3m6XJ+KcBr-du|?EsNjjRDJaT&3|X-U z-vy6%+9Wr4V>@%!f_XPuEyJ0eB)M5y>3%x2TW2wo)7|6#P`~KwwlcPUofCZ>TV0DO zKQ~nG&9om{(@_BkcW$JH)3m>q{+X^9AW8PaSxj0jx%(ace9F3ZdK!O5k)Vsw)ob>CqLxCB}&c0$;t8&1hxA4-mix>I@w`dbY0~GEum|{6w@Q0@iGohU>vW-Ke%hHL|Ap(I+eM9_ z(p;7mJx~jEa9X;WqA1uO&a?&#Od^vL@EM9@ZBZFB=iS+kTXuwM%g&oPU1p)yER4Q9 zIyw@BJD3Bz@QR36=r<=BD41_geb^ov@bzlMe})|RH>%>2rP!Nni(KA-r1%bM2YS23 zGKvbVv+@Wky`At`tbJPcVXgSlPSr!MyeO#8Mos4-jOLRVB_AOK49_Dp92*=_t*mLT{=$Z-;^ zviv|`^gkSyr(y~aCS4{R=GnlIlsnOH1164^2`-;|Nco5Jjd`^L z?P0eQE~%j^4Gv*zR+9r(SEGfbSrr;Ni2yNmdLKLup-M>FMY_-s_Ucp=5EK*{^u9CH z?E2LclE*u)U(_@kHVHh6XAh|z@5RijDApGNx13lK^y5uj0IiNe2dLoa{`29-F!WFd zLhs3!o}YTUNS7sI2Tna%w*!){j$3j?s1Q_l4K{_OS>ZNG$uhq1)g?nCLxoMAv(Ytt z3z&YyopbIPSqgk|<=jSOeRoq9nx4I<4iQ0d=)xbsUCKNKNeQ9g?zHg74%xyMNVdilrutHVJ%=sLB=ho<0V2d5a^sq8ujp@??>!sw z%`GnX$x8X)__HZD&vz2XF&Fro^Niqf~W_KliSK`N&C3TNPd(dsp)m6hw5sSu-n;&mai1{%3M2aFhYWlyBQYDd ztNv1l-ICd-JZ`TaWYX%E@W2aZ14pnL&=+lUruB;^&hzNV7yX=Z3ZskyUqx)@QO{N8 z2mK1-+Ym&k&Os=H3V!e+b|sIEI{Y(!Pn^gFexGYdQ?;||#03bs#GyG(QGk>}cu!(Q zd+(h^0gVfS&e}`tpIriD1=Wix>E~2df zQv(cJcy=UZ^0Y8B1%fz!wlAF(Ei9z$h^d=`eX7~)K1RrBwljs|gYca3y2tx5vo{k! z8M>|eh+0Ac7=Id(9O=34ec|3wB$M7BgPfKSkn$qPE6#WhaNQ{^;nze5| zSN}7wAdrxUUf7wIkN#Rx1I#SP88xJ|;u{WTToYzjUsXBnHHidH4WK^o1wwj`B1ye! zxmTk(SoH;<-9%5nnqpSM zpHe=@hF2E>+%Lz@YC>QlJ%=k(w1*u~)t6zP5#rzhiLl*+@~I!Y2hXRrvr88JY($G+ ziG8@hBJWsg{_IhyZr>~CE1^%M$#793qU!{`Z|qX{f6f@PWPp*??s+1qw-KGQ?~)Tn z{i38Te(u#(3~MLKXjZQVYfo7eD`9}nk@MR%0hUYK4FNe}>f1-7&EG#)egd3@IEB7f z0tTtC)&xriEATzh@@Q)s1P^tI9ArEtpLj~jxh&jRH-J$`y_57J!u+P_w%Vj<$W^|i zdI{AF+&kNpTlp73II5BxK{UsLJc=6^4Y{F}QeUKTA%E|!Q;+eEUP%RmT1v)z=||-R z4-;8%vJuT(JI(n5CBelm~0t+0baJ$XvD3&z58AAksmVDMh==W19!5LK#4XSZD z%ZQ|4{=a9>$2YEaj%3yX)1n$^uAmNvnZ7M!mrUq;B5X+46K~dws;mTl<{VQT^RrA3bOrRW z#*$G2`02qd0>$GMv%mtoByrVO_;-RSJBAg; z-FSRx)}KSNd-rmn#V0-Gq!MwE;f(3oUkZ294aOCk|FoZ+6nFRW-LgcHlK9L_?(#F5 z$iib-GL9YWOEtEd#PJEBkHHahhwwkR8jrFcqpOB(iL>4myZ*37LDNEx-KTQaF_3B< z_&FA)%Csm27J?qKgljv2udE#dLuLUCpZqh!kFlO@<@415O^;G-UgXhIrH1-j%_MB$ z;dzX?eQxDG|B0xFUOVzMF(~t>TMBF49vQQGxu_?5uNMXTo>-S|4W=ad>&YoZ_9xcO z4Ys5t7{9v?B+}v6Q)}|I{tTrwJ13{EFUnOu_R4Fr{ zTivY~A&V1ZT3JW*%@BFNKr%CF@Q@5iJ1nRH+3hvXezmiO3Phc1ZcI{1qSt9HxppG; zz0}>7_Aodat8m2}x<9%Q&tRQ*>LmN6X4H3=@KZ(o-NR!mXWYJSk^k~e8$NCUs-et@ zIj775uUFnq7P3OWo&}6SkFqPC<HH25zsT9|r5DfqRDTW6Tc@%)4sHg9 z>hXuc4^eq0Vi?0^_v_u*WS!yABGa-|;)1 z!ll4=QHDS}^;=v>6}?lBE?b$?6!sDZKAqKZ6JIvWwfQBNvINp1!zs{X=;W?nKS&-JN(tMw7{lb4;OczwKc=Jj|Y9u98Q~Nn%6L zEv8_hantU-wdtRy+!6pX0P20n_Z=Jk%dVGjYN@Z$2_%}|x_0q@*|SY?@8pYa*;8Y8 zn^@hK+96~oi5q7k4yV4E^PVj*siT!<)mpYpXru?1mNIEo--e;mD}7~blqRa(z7bze zHUq))5{g{rIDji|Z26Y?kDRiB1UZfuUBeA_!&W2rrAI-|%A~}Hvz^EQAf$S0i=um0 z64>hIx!IYy0kMvx!DTgZhJ&j@#7wD)Ba>2}km(cfy${PS#9|};HI(gEn@PW7b>LX< zwb&%lysCEdAUN(lYV}j!49{(bprnfJj2g5ggGyf6xm?=eN~82MIUIWj8f&Zxff_X= zUWyy;bmP}Gil2WGD!XMX3IsE%Fj&msk)Z$snqjE@m0orRCtMkKTiKZe5Lwi z5_%LmYn<+r_EW;_jsS?Jr_;vy#5n_x?+5e0*|n?k*@LlXF(5}nPUns@c)q$8yu!JqbyK=; zc2B9TZ(}d|BpfHB`h{oRKaJaxa=L%-=>-Gp#haT$8ECP_ja-ncJW$;7S6_2WFu9^Z zstS5rj#gN7`pPbOX02>L43J;gJ|`lC2SZ&1g@wmL z1dAjNnu_(`4*dPE#1OcK)O(;jJW`Ltns$*JDeR2AK73#|o^a%j7Fy)^W-V@}BR8K< z3GDjkPsyYBOdzWRP8~+`Xg1n9_5a@DAov5gqK|*=Y%H7oTAb0gIhW6iS${T)PfU0P z5Af$Vi$-VMk-p}3zOGB#VF5Yw+k_{VypL2txWnRGbDsOO6FH8NA)`Lctb1`zK+S?r zYACGjcYA0{p}++ULfFM0fHt+TDQVl?IfKAN^_UEV`DxHUdaa zwMnBpSjaSvgSXAN;xYmzNA1n};FbE=>FVDO6G#%uFo~Abdqv9=&t6dvPzC6N5jx8R zdk>E%0Gd>Ai~bF|u;rB0pp$RxK~e|KJu{9S9%LH70@+kx+D4&>m>KU*1qkfD!BOOm zMpy{=PQ0zi(`;MZEE}z}m?{t^50BpbQ{YBJ<6a8Aq=cZW?~fC{U3jBiz+uYbmtLP! z!S-Kaf(|JeZn?9471^fNyK&>r*TX6Gz+L2@H*EIbhPDQYOH^?-4n~*Y6=z!b$txaf zS}p<61*%aXOsVSKC00+QlXDIF4E5_wGxp6QsR|G1)puv5q>iI_w8AlIVJ4@kQp8Lg z%Q@2^jL+6XMP2Ul9Y*9Pn?28h=n(~5Y{Gsz5WXcnjDkVRes<0Jlzj>0ef1S~lN2eW z;dMTS7WN%Sq>j1;Z<@XxY{UghIy1d)PLy=^keZ0JG`VF^qa+(BFM{al){o|O`aeFa zqIoEW;NML)W`zr4r2yC)C7+2+LE1xs;kE~&{`FXN!)_(E#BSb3Yxfg6H7vN&^fO`7 z>#t)H?jlau#S1YbKW=h=1=JG$&NV zQ_Qol4z`DQPpVt|_@kc+9sV8O%VT&LxfZMzDx8ojQ2k^t_3QTzp*p8g6-s~mFC+#=R-eP~e z##5O`zfzI#QhkNb1nd0Bj>(hSn8vU_kbD!HzDvo zG&3R&s85*_d!_b|!Y)FWPi`G(P8rx5Ul{Et_n_Gpwxy!n47Fa*!9uP>PEbxkJDzLO z5E!r7t(n{Svo+e5*sQ;KIo;L#GRvrA1U)>3$lG8okd2}p7$bLB746bkiASwnBmNnD z=>8R+2#iJe+RHbePCgS-q}+e&QNaHuhfUk9;PEt6cGT9NG`nvq&F=kg@?biy4)~at=fxgox8$)Z7Zm5V27zjs2d!%a4*Y@oaZZYl9t|oj zJk(}t4BDW&yo%q#7B~5LN4{_JWy#2WN9`*!T8iu1(yLxYUYK!Hk8m~;@5B4lkx?Y;p zJ9yL1%ID?XFAo?N`PhTcjf}=b6Yx@1_Fqd;FOa&vKxxh!G;_gRZZPB_{3^Dv8MgCi zQ0T}L{maZKX<{eq`bE<_OwLYLq6i=yvk1h(+El_P zzeK3QKKKrcH~lP&)Y7^U>%62#h^W=r<@$FQ%i5Sz8$jz5w{1g7M3nR-f zTAPUa3k_TEw^q3ND^X_%zrK+3@!hW0x}n>ER};6)VA?V7mH2SGuGi{VbTc^t-!Fe}bu99uktn z&YkNgw7E@ul=j^Ldq_)Cw+jmSxY0{-phLQ^0Ldla~GChDtUa-L=rur0T#@qkuiYL=KmG&z} zl|eX!W~%&DU6q%q$zfuwxkFl?=81mm1~;lwu`okje7TCi4$TceG4 znN~D)Y^rplVF$HWk<+0cJp2!OZx+an;gJ;RW71u{k<5SV9>2h%jj;QT*5{8&i+wO# zjFa7(@Lw)9h>dVBi%qFXLwYcr3SJ|&^>D!K^m~d;z<&tr;o&u zF|Jg+WxmMAH*ywegWaO9`;Pv)D&BAld@RxqTV(j)uFe?%JnUjuW2Xrb>u$>4tVK5i zJ@(H|#^5Aq1J)d*kKG8 z_qkr!>2)Poo(qK<_so5mihtnQFpHO( zLrkG(Je(rxWg@51Tof5#F1sjVhE7{c&|DDmPZ+V+P88_)V^ROpB7|t|1_UgeW~Jci zR{%-*xHsM}e6qT~ZWeqP0Tkva_?4cHauA__c$7C|3Bx$;8jlqP4A922#d}nVoXTU9K@fB#aK07``O z)YDG?!O5$|IQwb3NbiY{KNiW{u{{mNgU2Q+1vzr4zgExtzy7k0Esi}d50dt}Va74k zh11DH@rIi)SVe|+MO63xJ+iPM39YX&F4d@$qX)6bu~qr+xVd@TyNJ#Oz=}~*n*JUd zS3P`GS&D(&e3i|O$7?6iC<{j0K3K-Llu_+H$XUg()T84Tc-yQ-G$Q4Xjx?j|mxQ_9 zb?JH|Rn6~9**Hx-tJf{dmnKASg~;&_bOztdJDl#ns{KUpFbuv);NfrAls=EoD*VE% z>-1rPn=mF2W7EWypvO&|NF7%g+*&j=Ra2uRlJNFL`nnf06Wq=Ys;}{grO)TR!g^Cj z!skOl{+$0W}-hi^?lE*niyh zdaNo@UT%cEduXqoOPs)m| z&oa}Id;a{BYu{k`NiTpNQl8oYfq8FYorQ1L#`cSBW12}XU4BEaidPK0NHfuYN&@~1 zF$Oxxn?|l*s!dJ%7I~z@5oYb7OHyxE6D-fhgfZ^-Bx3DRV69`q56;;p_QW z>Z#VU%1tEH3qnu6RlStm9rd)dzOR+l^Q7@1F*48LRuZlZFQv)vk}lyacI%xpyTux# z`QO^qqDW@_(P+zy2T)W{XZg z)=oKX34r$)w5YXdipb3N^Q4c%GYeOHmi?&|L1?;Y^ZSTz;t2w;MSSpPlGaH+`CK5pwMiy$m<`CX0ik(dM)Xj%s;=) z?Z(-FSu`zh{M4Q`Y_a-cF=x(Yx=v1gNx8K^f#{1zH2J09S>3FP#p}JMZg2I5VK@DSD7R`q~8ln!-{IrZ*wYTgEa%yKqmj7njSg7QNhjvMJ|dFr`(5Fjll-;`>SA1I`;vNBwH}u&)dqcf_ZsI+C%SeZVH(sBN?jvmGZe%>zLwZ)?-W9 zoo4V0qEDYg|1U^l_BAjtPH#r>JxqZE4+{Hvh99n*lSfth)@AAJY|DCUe8#0+iW z&-f`sy>HxSxP2p7wdl5P2zIi6KiRn85m|iWkn{rvTZ7hUlX6bEr>}?~-z?OOS!Ngb zVM_WV$q*R7!Ck$-;S$RD>7E70(VeHsaC%)q@DLnfY*E39hjJ2=JSrFX84s4Ee+&&f zYuf+1Y{uxhAUE!FK240J@=j}4bC_slUJL7s8#6Uy?vYo!Sbf&6SicSZNINcQdPsMP z<%lan#_30*DDZ#<2xjzk^@)2a7J{O~pZNKW|Gr%g=6hS6WT~>&;I*P~)u?9DRqvldEKygq5Aez|pD7TN1dxqaI)rZVRZNHKY}6jj7ZV$G zZpCpfo++;Pz09QODr;GL%&R!w>@K$cU-HH`U*GuWhLA~D2j3uI^9!Sgm7`r2oB=V0%L**8N|6guFS{wwoYKrh;NXn~&eJJkcmhHP_Oy{Lx zARcqY;7a=t8SQs@7UBEVxZ?BGvmeL5%ph%i2gRzD4k^A^%6(|T=nZE984D$Q3@S7F z3$r>w)WR{}R)=Gr^6_5lRssj-os}A15GhURpa1HQ6BH|s`L8&fOkz5A zdGL4~Fh(z(ibmj06{S%ujOV+DrlQafL;vP$Z%xxR1b#Tf2a;$BZ{K#&@wu2nAti1A zo~}<)|IgMTe93_`$j;W=tt{7A&suq7;O@ZQ0#e2I*OYCe8Pyk+ z$>;!Un+s}8iZ*hq8>iZ^fMgOnnGWC9g?~>5+;xBf4V8hMfu--KiNdM~5e~3Oi=TK- ze`YON-cLq&#S}f(2_JtNe94UWeix;`g71K~OJ=?>u5fs5=hwk8N!ks-(m%Cr#gNTi z+-@wi|3}X-U_N-DH>~_dZT`As(lt_dLT}nw!Bu8rmlx_Pu74VOFIV;Zn+c9SmJnTH zF+Y5IjYCQ1qOq#KD@B*CWglnDzhWi~H3m})^Z%b$5OacUvizvMbk3d}#!hN1Wf@HS zoN{NHJpP^Z9mU#>vpFrha!A z7{jS~=>Wm6MLg;wZDxUJB$?+~wvPWSv0zKi^JKWim}EDjEN(Ko+p&12b~~S zIp`zfJLW?j@6#hb5dKyvUnKcw-{865fymAXEx-MfzKX|COpKUgjPUy-Va`usvgMX2 zUeSM*_K^MAZ04uzHN8X%r=9a4EZF&+zx>Z0zz<>LHxt?z?Ej5;iJ9YG#C;-_)moUH zxM=Faq^pitd82$K`}vH>tw z=*DbrJ8IR$qACjiTQ!0f%I!LG;OXf8t|$7DsNVkIe6v?ecri7(XG^*MroLk%wg{-< zxHkIXktYSsoyxA{?eM7k%C>**i9XG{|CGOTe}w$^ z)M(*Ae{j7M7FhmyZCqqfbeC1WkYul8@ufVm+?gq-Uj|`VCU}XKR_}hWFq*Ng9Wp8# z`q361X4r;gf~d|S=8k(EgCdh+oh)lDG*R-e@g(JH!x+*`oXVvANBt3{RH{*}(@2__ zD;;z$P54jK-J9{cO5@<~;qmcBkx@D9)iXItzEM3RaTTgHG8NZ!%vQAGH>(-UwD$0A zi=WyD*Z-RcBvDw|{{N|o?njuT0GD{zuN7(GU~`NSo=!IGQr&jbi)+X_meqZ2+uxCC zbn{4I*=cJNmbU0+f%s6*E;Flzt)~a$Rory5HANmxiRfuCCxS4X3{92$X zc#Y<1hC^!PBiepJ^H&YrbkgUT7^^pB=T5*+E0egL z>9tjm7Vm!_{(BRI`X+sdD*ti+XTSKrZ+H!haAda@Mh--68Jk^a*zrs%d_`pM9Hw3J zOnV@0m-3fkgx>~9e42eYZ=k?kM%lFTeELidlI|&M*J|nYS1T0klTTTw%n|15w>1>r$WBbJ)Ca->!Ais<%SdNM0b z7^|)n;-iN0WSeSHj--a|TxQSmuUSVEcKMj)i?Mccl{UA2IPg)8Q^wx$`+8nSw26i6 zQ-AU&uYCcIr9lQ}uM>8ceIX5s)NfF3mSb(h!{=xH<`O3}8LnlGS zNV%S&!W374sL>#j)|Dq^b!J-#l?}iABzJsVw7SA0Ry;#V?DXf8r+$YO)+qkMnrNkh zinGJ|YOirUXL)1P%Cta==RY3^G9_w<(gXf}tpE9S0^z*!J0%7$p=!*hDXP95)>Y@4 z0&d;OP{DS(lG~WmwR@yvyKlY+2bks4FACO^nq;n+W@Mr+oeDbo)lCP!$l zl-MPNDF+VUDQi~32+mGok9!4U9LuNfjsM5q0kD+=ARC)oQG<;CEF@P6<~t)a8BR^@ z=@Gihv+$1Hy3iQAo3+v-x$8?&q~D$Bkspi>?D_{9>FW~aX9wAzYwSF>!46xKc<`_G zf%pls*M&Kli~n;EUhD)s=OK%&prkKoJ#cFe~+mNu%hz#a}1^bMfjO?H8F`Dyp`X7d69Zkj|z5<;+J8hpLbf! zuzPw}ci$c7P#pfKOg3^NEB7SfrKp(kG}q@XGcqum*BrC|YC175SUQ&cjt9d(b_ULr z4g#2ci&z<txJcr;#FKOC3Pp|Tx!03%G(@&A!^m0?wGTla{hfYL}xh;*0q1|*~# zX`~yaI~4?^Bn9b~?%1TDlyr9p(jh6aH{aTxd#|4J{ZZjTczM@abB;OYSaWFT$>2!l zn{w?{w}&W*Vh3V7uLef5a~$ z-*9kXqvdRn|4B2IpI(s};-aqNF-mGnDDvjN{rmqeYXEu&R`Z`X0)K_a9-7P!EZ)Dk+d25k#IpU$K(7D!y_a=TbQgq|C34t z)%@G9!vV^I27gkgV99#qm-m|oS3w}wq*^G%$|H@X-ZZA=dHil_kb&VY(cgsMe;+O| z1Rrdtm2@Y;e@0=dnCJi{-XB%z=g&nqobDoNGIH#^eNk=a73M@ocI%Tt#{S=T;(u=) zo+d<4gWt)#`R}UyC)_on&nNbGkVil!?&OjsP!CvP(Cx^iTsB+dvFoG}{OiZ6;3Lp% z*Vq%%*@79G2T0RNnvpyS*0zg|Ki;#$%6Sy7(p zzn+E;$7+g8J1J8?{7GgrQW@i;OJxQ}vUjzOGL~=X-H-p7DX4~%kt%KW>Kq?;c+}R# zy8WguG$J9;+J}=&VKRF3h|lKX9SR@ZV)Z^(OHgh^kJmSLfc7|-uGl>CSZQ`z7he7u z))%J*uS?p&$fW#d%(+BPDb9NePzZYn&RcDHsMRm~%3tpeqKN)5B>QX@-`<{Bc~>F_ zg};W*=_le)sp#KIwFxo~Sxr&X%|p>w7m2S{)M`V~`Frxxe-{^QhE0M(D~@3H0O~>R zKk5-kEb=N|Tm#5&*l|1+D_DB;6&(cNl8ws4TbLF_Bfvyn|09)i@=ENK$X00lsmsYwofzI2O z6|q05QT?x6=07kjP@0fIheL8u9ILk-3d>^y5thPy;{WRkTG8)rO?Aa!|mnW%zLasPhY00|9YOb*e%xD`JuLX+I#~jqtddx)XMBit*a0-bhW3=NB%pO4)uVclbh_d%-q9b= zSn4EF`!=z7k9D4zm`cP;ytw(6H&|OGYJzBNu*i!jv z{bH$ChmYspSs7ffEQg!CxaHa#b{2j!_6lBEBjJsG2M5u<~`|V{0bUnC3NdTV>ID zJeS{-KaYoF2`HFV@38LJfGJECeWH9H<-m+$X`5gREzKM&ETl&ChJ25TWBN^cts?cN zCfI3NqO*DShioz9ge-A*xX!;3RSg8*fwgB1$F&Z`BI;27TP*&oTa`(Kl5R^_*^VkR z3Y9#D!*>~?N>FhQ_mgUCQw2vQEDwWv&YBk3ZKt{oX}NZL)}Gsw4!@9g3hv&<9}Ykz zyXzmZZA-!`K@|jPGHzA7X^mSsD9xF;d8fRVJdOKFmiI3CzmlmJVb4zYvvLC#U1~~+ z=el7y7$m=s053y~!?b;@8!+mzLc>FPek+YFJ41<>lg+#dJ+}6fj74u)?VH`}B3b8~ zwLmxiqbX1riuGbq7R!UzGdMy7b9Ai@pQMTjhBUPYsUh<`JM3bK9CAXrTZcuY!%1(@ zS(2yn9^Sg+IWGWzSvF&;| zNtbB0;m48cwnLQ)Uf+MH_q0BKi-u1t{=-cqVGwykbWfr?T6L0*bTKNAFwuO6+d2yH z$IPx|4QXm@#;_{<`+5gF2bP)jp6Vo+}*a2PK7C@BPIm3BU0-IO>1^v*A*W?Ct!$S z9d1V=Wm(4`kFn+niBc2FXQ?GKXBoJR?D;w-G}(FMudUVZR~dF>;z2C%@0a*6Mvp@h zI(?YGQZvC5yC(Ly)4TT%jwzE5h09?2#|hhomUa0+)X261(&7S_M7P0o7PE)AnK!on z{g|x}t9!P1Gfy2K?~4i>#IkQj8SrEVKA#-J&}1;WP5Q`o0!^#X>!VcWW2E=+eLCAW zq~pfsI&%DZPr4VQZlHeJmz#HDr&^g;OAjB+RPOic!(Q7M0o*U#9QR8o@V^}WSD+W} z8}hF)ZW^c&J<4#&Izx@k-^k)Gq^E)J?J0;&-pv^a*8xuM^7A9xT9(Y4MUw12G4-U| zJTIQ7(VUq#8oo%)S|6BjI{&Kc&}Q(Y?!vLI)9!BgZ9tT2*4>^l0unQe$A-4hLLXi= zd?Y%&H(cRCZQE-DtY^ts#uYHQ>&*|it5CO}wp`-MlI2du0`t*o7ELiR>iP6p&>$`; zrQ6=a-#Py8Q+tL&gTf71uq$OAk>6XC0YYT}xt+sJ5TI*Nu-)dFZ(SkY+^#hv>sD@i z_z7Qk65lI!4?~A(%r!ogH7|9($dia8Rv=PT*=!-hLNwp>K#KJ1VGaYDlY%EQ*t4dk zTM0+w(yZw5;#3PuttOiGx3KtVBzpt{+YRvLLj*FA%h4pTI1NGGf;ZSD7I`IU9Y%OI1+sM6+z3-n2e zztwgGs_jT5ESc2F_tU5u)STxQ^G0b**9@>9-%rf?fyraFVl9$y+Vz_7=!x9}5qil{ zNC+(Nsl$k*E*JUP=Q2^faJ!p%;<=eAyOgkCKs^rbhB`eF055bUT6ZYAUzOOEN<}Ns z1etdYziatm%_&kKwKB9Pdjr-;v6CJZOzZRpHGYOi_}}u_0TX}D>FCV1w! zq`ww^FZ~;q_D!tVBAY!3_Npjk&ZicY`DCh_ut_+B%b+Xx$f-o}02Z0te3e`4Y!jBA zrS7%egPBfLYiF^|mdfv^p=V=wR*K@)QwEqL+N&&bRYVQ`^WEf2@1mV7A7myb<3Q|V z$`_GZ1P)qU0Uc-4URlu+V3Ld! zE#~>%I?CihdON5>I>p2|0af&pXD7)MM|a%%%mMA^FV79c60#;aw=sUQq;ku*j%a4h zCbpn%)PS!dh~1vwAh&#d4Yb_BmH>DC5(?$=9cSACL~ST zm=8g{l(1Wv@KX~$x6d1-4t3EzLG%WBWy8^vmc+k9VV*9pN)`+%kf*o=NO#}7`;-O9YG)MO?FnD|u7Fb^L7 zleC0nCL*VB@$o$p9Ax~jK>QbsXhLRb;Qz3?!s0}#2$3Rvixa*#uGHLLwjFf5GLW|jyuBoNlU157S-3Fj75;LWU85- z*dXH%P2MV9S?qSHdZ1i3-Zp^t>i<6sGtz(n@3=3qu9DOXjz2J$E8{TdM;&j2vq&X# zg2Vk-*Sx3uQ^C@&?(VjPp>oX^({j1oojiPRxM4_TQmXaONZ<{A68^I9RD*{ymg*G1bGLct>P<^HdwoT_JLfCDX$=d2Dn6*R!0XJ$ZbJ14rGAR{o5U zFZ(*OS(81ZV=vbGgO_pTrDI-Q9#NWa@u>f8Jl0*0d$T(#E~IT{HdC2D(2lT&Wr^PP z=NuDPm$7$rypyg)asHD9$^3y#KBjPAV)kGpl71XV(IWje zY+B0hR%%GiC38S^H-A-ubA`PY>M&n&3!ChcTZg^)FLlNEgw#G#!_1#BGBkPr z$etvMRj&907#!&&tRbkb@JjX%3~FV3OB*6txu0TXJgK|Y#4mNZ^SRRd1nHAS7GFKq z6fUSavNn3g4V(+YN>Wl&rb;smedcI+%OC3n*CY!8Kw|tdJSUzU<)k-pLLj{p2s%~X zK;pi<)PZ!8ely6!ns`c||1clLX^-w+ASo{37ak(r7xZh3lySJ69LwK25Bi|Kc2rH# z;5V21fq4qw3?l6HfV(zpWJkNZ`FesC-*(LxEVRV>dilXz&ljD$b|;s4qZxsbhW<>f;KY-I0RzGk`h|7FmixP@SKY4nJ<&BX0%+f4U6MYH z_+6us>OpRk$>ELIVPx z^FbZxu|2e{eEeYQO{;^$XAkVP9V7Nr8RB~nbg*t-6pc7<=fQo%WWpzIq|3#YhW>)1 zYMeP5xA>rLTwOA2-Q@VMhD5VWTB#zm_R)?}hyp%ybvGTDbEAi;^9 z(`;4ov#v2#N`^VjelHHyuq@KOS8Yn&cHbYX8Qk$ml8YE@@QV}eM0+JTo$G#Ek4p09 z#aqeMY0v2G*R^Tt$`PkHGv;-&^{qu0c{7)FqBn3phc~0vtygn;K2?U(t z#=WUDLpH)A;nANGJV(pkUv1s9mG2bl70+x!28eh1bNX80^9w_c;xdOZ(*G4lfi?um zuF^8a7Ijj#Y4nx^q{BEV7N0^`F((F-mj6*;qBrd0Is0XVd=y3gvWbpX2Iw3)aMk%F z5NpzViU9xiITNrgtzEA7dI`hPqfSs<)+z1HPX#q z0sFn1%uKh~Y&OG7EJPxMblxe^t(60#njbN1m)SkLE%@t#Dpj^Plh_C(#4(h`gq0@M zke%zr&~RBi7PvFFRgiqh-yx0FsC%2V?%cANIh^dH^QcE)g<_3LMFi>6U5M?aqgr2dS1(nF*y+1XiUqANSus4lEf@WFa;g2$l&A#Q_g6X z0_-oOaryRa7R_2Xs+4g1vflve=^Pz^A10<@fBM(~htXbLXlN>KT!u}B?qd~JpN9hI z@dl6^!+Pty%l?X}hbCeK_+2Vz8v%EIQ{yz~)8`sjtpL!%Gg0-TZC^M97B#jX3s3}p zBu1FRt$`u>@7{A;y0K>X!y9XAYGi>)hS_p^AeY@V$IOdPZ1Zjk_xM;^87f=QR^h(5i2|v_D_33WrFc2m&W;;^#(H zLW_SAr5eQnC_F#ayU!lN9GEZ3G*bPw*>=DvitNvoykHRG50i@_e>iQDf!ljw;6El% zJBLKZQnAjo_ga!rfdu-#B~%t^ z`zlLNtn{Tnqv1Kwd z;bGZW?0vf@sIi<`N_Xew&4Mw(Z-)haQFl0VSVnDhcO!wn`+LKi>98ZdlDoE0=-sO) zNjQayfv0|KT)S4kC-yQA^P_{o^1`|!Vl#lb3)@#L$WYCue8Ay%5+y6mlE@@ ztpXnw-1TD9X*)00hXhn_s4v3dFlg?zCcj1`(bpHclBdaoG%5o~rfd_qq>Fg(&y8)E zO$AG1==p*0sMV`&xNkhoj#|a*7A$o>zqU)>2%qveFa>nic_7dUDH*LxR{C5<<6yjub8}xl4V2_pyoB})S>TnA&Z+Vvk zQ-IoHc7%m5Vk*4B+(eDq{&~GG8}JOIebQctMUBUW2uw{x!m& z_||ua1aSrV-ml!TDBLB$?@gq8aFa`{yg2MHyO{KKxpEUJ+BS^nta*z)@yq~nJc+ZUWOlDA3WCGVG%7r@-FJi<*Zvt*4~U4}n;zk%VHfRj&!-<=?; zr33Th6IL5O5_p`9$gG3;*9pK*o+Hm!#QQ8B;XC$j3&{*bm^EN=oVZsvrv#kATjGzS zA4_bEKBSHbZtp;Qz!ELy(5CL5D|uJo30v#W)in!Ek-FO2USLzTG?5QD;li#fZ{QMA z<+3%(mbxaFKlE;mQ}~xOX*@K*J!3aN5X&FqTw2CtTEjAJv*o3ScGHLTjP10xd!grA z+8n{K)I(HQ=$g{yz22nGNV)=*TdoFKt>;z(p+vew8(<{3Zr1~ByVO`kquP56Q_s9*j*VIu)wn* zA;1IhskgWH7f6WNh;3P8z7K52NZH{LC2$(-C%|W05)M}U^x5{II-@mGlNCr zaNt_c2=oa6krQt-pTb*zd?Axm2BW6!XvyqzCBx%>0k&;J2)kkAJBvy&RMxCPhZpHR znZ6P;B;4N>CKrm;?^RkeUD_q4XkV~WI7w`TpEqYEILa;5wdbcGk~%NO31zXk&6I7# zhx3aM8w>1Ln(EHh@-Xzx^GLpOgZwSi)iRzv5?ZeDp)q;o-vBWi11z_2Ulv$u)z##W z_#Vdt{N})+DHgbikOcZo?jXI{(s3m&%L&I3mk&N7MH!|>w!d&`hB zZPS1N-lL`+mncW0(H{Pyx>#AJQ@CNPPb0!xOf)TO!p4_P5}WQ<%FU#@)si`^Eo0W8 ziHTt}{J^iIM^LP{<@$3};N6e58mRsm(mP)Uz;aT;6(%DdwewI)<>XQ!hcj?kvBzTH z;IMH#S~zJd=FNO==0PujP9Q~9wj`9_Tfzh_+Wx12AM-HG9fw81<4R4aRX2f0C;si0Xg*t!pQ6DqoUTOYvnp|63a3CeBc?Ki! zH}DH5`FnD=E5k5b+oYkdz<4z!=|kxILBWyu4cC$mv^Q@jw-^^iv~%Z3X?AYJ(o9U^ zRCqNLU*<*)9(s*?ViuR|O)Qe9+GKBkzhC6WGaCxgLFQ{#G3&LZ$ymZKarh85<=ko^ zm_reSRvih^3QSf&r!Wn~(o@7V|NG|a-^CSV0aJKNu!wnj%7GWpTKRhETr7yh=YEM+ zTYKY1Iy11PyT^Mnw*;@7ey2PO+*{W_VV;J98hfLf=ehSeTI<3t{(y>3ef&MopR_dg|oQTU9& zkjQm!yN?W5UV<_H6}Cc+CX#z%=kv~0RwLP6RUSW9%Uq$}2LAeq09|NFM^mN=OO><& zfT3^bcd_1e1dOV~C~U{9rfblMjpjLpof!M~ZQh!ipKebafxOU*$=*x_%#z}P6B=E} z$-zW%k{xufZgh5*iEmx}e1GP^#|ME=fm;r)vrd)S(^byK^g()^wnmT=B91_w=pnT& zp+-1aAWToHz?K!#-5ccExw&e)i8~&uCv#DYLqTWJZ&0!83v%hb6f*0Hae>Mw`Y*T^ zwY7lU<6u(0Vk4X2;eU1IJ7wVB8-#*+t+RUP>?d%!!5=DZ83K+N6J1xMSoPB;m;%6W z*sML^7NhxEH7Br<_dM)CTR)vLQf^k-s}=Mt8XriYa+hSY@dCU-xhWU z^P>=j^*cyERjlIalERof{*T0h^wEd@V{Ig3rrv$FMMBdORA{$uPVjoiSYn>95mzfc ztoHmvkD6JAVrQeJ$5PmlA=t`|#sx#$Y?8w8-t#IQZ)+BL>fcR6t)dechEMNxhKJ4O zfBo%&{3J)qA<@SnqZ@$NlI!-tZ%_ZNQ7g(&!#({b?Z5|ou+Zdsyy$-e>uMC72t1!^ zgHMdLhtzB!{DBNkP|i+PbIV^H+SG~i z@bkN0SXBhO_tGVKn0|TtRIkkwVadJ{jwE1FCQhBMBob3sVrIl8BKo5XKk%J=8!%Q^ zemz^FULD`M`i56y6}IP?64j1#G-IA*&~(=mWUQ74H$S@#ms))#yXi7^RysqyNeM<= zmDre-MK5yQffKSP>F~Fi9ZnsKCK+tILx+ZykP=NZ?{b189lR3#+i#l@fH@uYp}D<( zli$TL@8gHhr25<#MfKE}jBf`PBAtoM+_cP0+OwIdO<-=4Z!VlPdOtk5o^gJB{I#$= zz)Nf&e{Z?)qd%8zVc@o>cT1bq<^v>N_QS=DPyTa4@X4X!Hi{OTP1?|Wt=NEt=ilLj zBU;CfB)_gI|L2%P43RRFop0vk+k~##`nb7*je<`<5;m-cGk?e-e@h{x<#!nTAU=1& zt)OiIBaM;5YctE*eJGV*8Mq_HH8{_?#?&rIqrAvNn2;qbBv{g7OfM|+FSx{ga2b92 zZqMUJ$(lCp5xsXKWet~t$#+I7GMy)cw` zuVI-0-wRg{R2~)C2}ZKwh(kGNdvFR4%tA)Ov-a~mX2L}3on|J-7v9g=O&)hY zvR(Q~Z=&ecE_dVgQ9ph}o9?$SbFcS;!BMhEu@wwH%#Pww`PNzP!WaB5uH6EC?> zf{QEsh2oV^dv!&hqUqOafUY4>HQsv=_SIWM+Oz#&it4AY&gcK9ip`3_X+|{SJ)b`kBj{X}r#x zs(r|~5HZPhzxpe1G!o`~(!98^C;a7sF^(`q#x({cA;U@iF%g++Mb78qSVRi3dnS1a zz&rOc$H$V%e5O~>?hZQC*l{dq`Z!4;)xPfiHT=9S38boLF7dY;fmzGvqoCd>&MOy2 zy$2<27N$eubr4j43V=C_{JG5+LcX%|#B|)Lshc@LSy!0c2g~!rLwIke_=A1%I4Fc= zFwH8gajuJsik`ag;390k*PD+AyN5dTVg6tvfC>1ndSv>&`5G@#p8t#S$*bI@(bqFwfiklc=F+0 zCji?46{{0v%>PGwC6azQ#c6<7B-JRy`q7D{pqeIzzCY}Akhz`%UaM+c%SM2bZDvEHbq4?4eV+6L|fm|Nb z6{vqsPs&c>`JO?QL{2r7*;hkSvfUGfLi46#ktMoOx%GW3;5)BJ$}U4eu1NmA3i8%5 z!p=-t#^>-v!g|WDZ3+M~vJx6|y@SznJDr9XS^OS{%PJ&dS2II+S8?5ql&7n{a_`rt z5fz!FeamiDf1j%Bq#zRvUWaP4EZ@^@NDM!ZYl*fC$8E7XP<+qz`D(JhbzQi7@ak&m zCcSdW>-myr8LEsY@Uz$5)K7IY2mf$N&$(rg_052!>|8D@gASaw`huLHSF`40<_(KG zXR}MdWl6F-nv9< z_h}g3pS@yIeX36y_<4Id$&>GxY9{zE)fL2cOV&4)q$ z+?B1LiS&Aa9X(#(J&Wec?c$D-E{auP)!i^nG0!I$thgRSJ_=9Nb9g#Gu_wl8ASADP zHeRH@4xHMFT5x72UL__HjYOn#yrL7%nIlA$<9=HF$0-m=RQ(B^`*07iB$Fm^iT{tU($dXi%*LkuxX-Mzw zE|L8*Z<_rKZrBVq`%QVpZlx}5hhC)h0Y*%PFR+!gydpMg{O5LM2b1^H>ZoVVX?FNR z5{by}Rgn;tb_a6I?_t?&Z7OLWO_ZHM#J=xia$Cnz{tOf4MC!BAm#+TrnMP z+nZP>W2O(|43KLhayvA_s~&i+#ddvGgfeU?^C`+SK|ceIQ20CvcN-_jadIhU@?Y)r{Ftbz00|RS{gf2wD zBH>N%7p^CM+F4qv_9BRR+4qKMe1EGa7(yV4MUU>k&${0eugVoDc5^W_ShcrmBEa@# z`*43ys66`dD!L^!=)D_%<-!Gpg6d#~u&0v=LCA9Bq$Am9<1s~k?0q@?C%#9$^yVYk zQenP)H`f6EQwWw#oA1`U{4wHe#h5HbhZ-s(t(TTEGTeKF5-WSc?uV&v772Lq?z zd<5FY;p=m<=>9)DA4t$5Pq&&?#ggbVj_rV&T@(-V(bGqmkE{Or3v;+H-lZvZ*nzC2 zU7EBvakK?gYb>DL^L|n=hpy=jOwY|vs_^y2<@nU(ns6U6{g}bqzK`G(`2lm!rAGg) z=~*y|nmRv4hxP6RP35?1nbWtj|IPwtTO?4ZEj8Wj`D+-E18ZrEY;oADjoDYQ6vQ1C$_`d-WrPTy)wG zSUfuSy@m*SjLl~&&dS!>dUj_WJzuvz8cj%LTnc&u09Lg^XzqghXF}~)XZCHdvm=nT zULSFrC-0@$mc8GK`EaJ$dF2hf8wX?6?ZydH;h&JOx$?Rxy;*9!cc&{&)C+Ru0R_-H z?ZVOcNc*t8;pI0(i$N#C>DM|`#)|s=h%kD!U6UEkkTdZtK%2w9c0#M8q-;;lr_M!l z@DAAY2?0}&6h{@yMwMQNzNw7RXqhZ8h9@>~`_^6yoj87a4AZ+%4cqzHzdB(bcsX~bg23x6|z4BO%3O`X4&UhD)TE$)|-@_w;q#wsa0_Lnk(Q>iF0#&j?Mg<$C2 zyWbR-jb1x!;c@ydK?l}T49a4qW%$*>y@f5u$eIXuP^IeW%Qym5 zasV!k!_QTbsCPfrMz-Q85Ov|^?Js-6_e_%PiRsxJ*YNxa!%oMPgcPsUjbGi}{OSQa zo%+nn)%k_Y@xy+6*K6OTAp~&Fk%?CFHgpQzp*e}6mNe^UtyiFYcQ$gGQt{V2z-kNc z3B2Ba3_BSEtOPqtO>)_B{j|h}-vB2sFE3rbeviwFKoU*y3d~-;XZmV9qa?%sqTi?r zm1|mn4<^nbh4BU>JeRA@_CP0s_~WB(wbKXnKQO&$%rgVH?COVz=y&&DoX4d&y$nGS>corDR zR~fdmo$%t8aQD^CH0_r}J2`MhZ%q_y=i*9bPa;xo?HFOQ$t|QgyK2Qr2VwBaP1L24 z$G|>#@RiA9F33?7d`d-tZ$YVU%?nW!jxH|lmvB)=3mwDWmj!&_oqQ%4B?kAnd2XqnyCj(Xt- z?peIyd;9(daHfzCUOAnWo5W*x~|O`Bt+&&bz0- zzDxUei}LS2We9G?L67sCdXw!^-szIg9vGolQK6~`bi8h}QX5XW)?@hs>{QYuIvukc zO)7$jCJV2P#^iVCV69&{biYYwXuu7^+3)vrC3L;0z-50JlEUwpZkGCp=7n5ezM{2p z=VXC=ndq|^`FQGcM1AG?Es|eCO2*R=(^v5oQ)JBZFO5nVxLMO5?g=0$U|@VIkdd_i z3h;s(U*CTthdu1uX}?@cVKN!c6j$4gC9%^80iMo$T~-~U?~(l8(KjqIzxGKK231>c zHk6Wo{)Hropo`U1ovr!V?#vNLCbTs>xp7MGI)c$ckC;zq@NCc#G5Z+lGMX|KZ-q4M z>GG0@BBJ4Hy>E2vTk-+_kh)g*Z*Gl$QYKVg=OzA2< z%B9}Mk4apC?RY;{#^kDp9zmS^5}y9=4~8dHhdGASQZLz%KI2zp^(+C%rxNHjmGp!} zSH#7H5+IcTVCEgkL)ZbJ@8^(;d-)8+a(j2}pjMP1ePXBM&$GGBAd*&)CV{y($ab-! zzL$^tnQHEpMGO-mR{~F8p`ByPCPRF_J&Alq&g+ehYKO5G8j?(3cGzd4Z9M$v2soIv zN@V#Me^}*slLM&4=()wGU!GGA>ROW=%C(eTUf5ATdxT=AaUkEBRX=dwO}M}C@bOVt z!h`h-hLe`n?NHx36!my(`PQq2gOGges`CZFRgtr>RT_kWqFAyA46L6eeiP=aGs7_+ zd+IV7c56k4UZOh|S8AC6rgs|dr+i}>A1shB*)Cl@?2_a_5doY*#$zDL6cd(`u(6|U z&JO8&3Hwe(7L<5E3Eqj>B?)e%{i7)UE6@e#;Bw=r+p>}PUO}1d8`l%7zNgqWxe@XX zu{q5i?sgEYBW0&K|A(McQ>ur&MO4IeL$xnEvH9O9S}rT|b=`=gQLGL1*|sE`&sp7K z%Nfa$=?4HLP5bH{z3(@&t&K9antN`3928DH)H>`E#Fr`PlO(QJ$)-`bP}y|bCV<-4 zZH=yU`>eI~UrMb-oHEjHhbW?{lxToJaM}q1L7*C+lw93OA9HyRCj0aN7?`xZ5uC>#SYo43pz`Rxek;3LrcdzHAt{Dch4-CEccsdwi8eve&)RFMob!_oMDmB}z$2Sp$~Fai-C(M$n>h%ZRqU51z|`AJ+9|>+TFe;&Vy-g|1iI&X4>#b=>^(cB%`tOwat+I2-TT&y2u~+0lf02zuH!% z>Q{TX?&LzJt;dApX&c#)9GbTBt9l=$nw9$fSp5tdNA;mm5l$V~U`xk`QhG6zJ4@1| zaD^MxEnGZMa65+&3LTJtWt(-&@@)W?)e`oVm|yVem9W;)xb@4!5dBlWY2)DKEW#y3 z=WLqi4&0^v7Mg|>Q1KLv9ypR=Q{R7M+Ht!h;)kEuP18>83t$~ASMRWJ9hz4nb~>(p z)cF9ek`9`W@ji3+`RtyVq{P9_=9JxLt?3bw)+B|{_M5J4DP2eGJ*EpC1di_oz%g|+ zlukbJq4X^Q@GZ-Yx7RSY=j+w|#O-w^k)GsN7kxY%O%BJ{w4$)TOw0Wi6}=&Nds#+lbgOKY z&=#NYx1OC!#T5y5%Ith3bTohF=6{85acEm%$cw~h)XZ$({Nqd6ZuWQ3d}rZ#*rbW$ zTIi2y2K;+frDxDa0nQQ_1)_ZD!VCl3Tzp2 zAG|)JUXZeawx=-Wc`H77&-YNhF>lr#v@iPR8=cj|qn&OEufM``&^UtGFF3W_Z9U0j zmm-C&U2J6BGsvvx;=Y)bf;sx`=w-4a4#`VtE^Y-)itNU}QV`4Fg%xqo#y~3Z?D;yozd-y(UjtwQC1?2d zf^0p`mIKQoO5Exj7I?v!WQ!R$PQm*X&-bdKee&<`Yz0T69s=0=DkW))#^uO3wW{MZ zchR#Un?SpC*Cn#z~O*#xkCzjXl0wv=k*Chi0^BzC|uL%clpAB7}I#8 z5s9RPL)(DwsAF>^C(ZSO+hBv}TXk2yjVCydFims)dI%e)Uj;5>zEWBWC*VOpj$6uq zQK|q4*hUWAdurP9_aBiikPdSt%XA+$)KK>=TKO`&P~ZjSV$6nkG*rK#cLH{k8iDQ7 z{&-uJ4h__tjqOnN%l|)`Cnf^&eJ<9SG0YT~Asf;Z;eDa?{J75OMF&dQn4Yd~7n{}W z2~wF0kYK&4LId3;z#rH%h$a76x`Au!Z^$$n!}p;FeW27OW#Rfpv6K}&W<~(rIDc$(WE^U$J`@&$p_Ah))p58=66JwYfG*EE$KC-eJG-)^h+v<1=3IHEQt1XA( zo8^A((IN8}CvmO5SK*Ogap)TJ0lA^Qo9)Xnkish;CGb6Y_zl*XrweG~qux`V0Uqma zR)T0kneQ^_hz(p~uTKiZ2+4}NM!}P@A*h6PU~~VJg>T^%%3w(w{L%m*60R4^?Oi`q z;o4zCi41$I_d#%=4$L~D#WtN59A7>4XC3u@#*q5<9R_($4tH~z|M3u$4)&?~A@^*; zkZTwmt=jKWNg1cmH}E^0l*PTW2QzDgKt`7RBGT=!yLRcR*Z-QJzxpH^N}#p?GZURr z1NRr8Lg%_T67iWxH@$qLpI#FiPA8ZXXO4mGAzpSdyO(WHhgkt6hzk0fJ0A<3Kkt8v z;*xr8@NnXRue#E~RJoDrej&Tt3=nqs$qB`g zb8VyGzTld~@~na?59HKoDa&eOa}E-f=k1vMK#Xf(8PWQ>O1w)89f}+Q7bW?dl3}s#`WPPIREuh;nT~ozk(S*f?Q26L6Z)%xZLA2MtNrNmVX8 zy5-|Xl*3_u_2)%m;QB5UB;jV@JEi?CyhaV|SQ_3m6QJoNMfQ4w z-8sQt%Xm3N0X60OBId1T&{t)y?yH|iyuDzw+duj$rRY`BBb)C<*;(khxYLLCXcT>e zg;IHX_VLOUf6@}fKmUCt2vJ+SVQ3dCq>*^}%oBy>ul9u9Y~67);>_~}ccY7wF=ch7 zUYC{8AV6=aLjn!jmp}t;J>7f%Wf~Je4u#xH83MkSE(in$(+N8bzAMNOpJXe|*r&vA zcb%7HNZN$IFkuM2n$@>Mhl8F)pR(T({?u~=PYY8RdOpxvJxpGohXcZ&vhM+(T{7qr zuw`8ve*rL${HvTDG@r17f={BGKEr2KxM2K!1TlLeXwHe`k#fuN<4N=dn$H)GUtNVW zm!Y3l0p(iNkm%K`qhJ!g+8}1x_WZ*ak+Vnbx9`2mPpaVJj@+z7NC>=FWyCI3D=xd- z;&BZTuX{+9>~%+m-&u8rz}2qUqoq?P|GKp~r9t5J4DQm#tzPZ)OK)me0(r3MS#}&|Af?wZ^28aehi$v4E{6`Y9xx=GY=xqm|y$X zlmaS#FapBqcy*Ht!w%FPgDG~&=Xwr}pBslVJQ1};0&ep+|KifRcR=Mk?Qhk02|Z{{ ztxB7Jd~<#LIltks0O*|_eUGSDB}?~*d)}1AyJ8mhBqF$ZfqRT!C{FVj=Jvn38Bo=a z+=~)9dv*cN^aog_F)pV5G!WLV=chVs3x3hmno=^j*mLyXSyCv`;}9o!fcfE9M%$;v0S{jCJ`lOFm}*LBH2EA}X0P{N*t5zG&`l<^_mB&SWd$mpkP-D56L-NF$7CPE zzx6d5hMm{97{PKdA70sZplnIsmZda}Cc^o4M{NcWAJ?I-rO^DGiZu0ng)PA^Z6+D^ zK=qksc4xYe06mtac>3TR95qeRoB8}7aGEsG(l^OTOD<{|1PZP_{gY1<#Pm}7%;%Q! zT^279S*|g_JZ8u*|EjQi3^E1I)gJ?^P|WGiN@RP>Z8tE-LatP9 zI6wgL%M1N(D_`>yenEbsugh{XY`gi;1`)?a;Y#X7`@TD5c`G2c#}ug-xEZXGiTX6y zRdk>p9d_Oe{YW9~@d7Z;F9HA7kNBpbYtXb4bx@Xut0kxDb_0i*gBmUI?Cea1d3XO5 zEV56lQIqB zdyv9M>m8T6#`!?vH(iCrHvbfJT9e&y;YV<;)nCml7#9!jiT*UM`dQWuLWS&?$bP@1 z>K2))XQtRB+!k-A8MyhXx-3%dCpDvN^FWiX>=N7!3?wDIPa|md_EnCiOa}8PSB12Y zru)F6oVQCUyj1Afjc7?3bdj(?65Q(G`^*@w20|hY5+^LoK1cpZ{v3KhYx`pI$U`I-T?h5 z@fGd7+#1bW1TvKEy^7Ae*7m0oZ3TgN>5DpxuIaYxfcsO{lde#%Td1~FJ^&4GOp&Fs zs)+ae{_yK33vfPPaPnC#2&Gty$qx~fJRK^$_k$0Y|N4owfXbU$vMD8}@46KMO1 zBv>TeOP4tgCC*3%LyW)@m2b9n1}$ORd8h1ddELZmFCh&s=~&ijuH`+=`%^y_1ex@w ze%xk;|5XKZ$ljpt$BbnL5Lr34Wa}$b2ZY)^Jy<^5rGd7pRXqk=bJ$ z1q8HRJ01eqXnBiqj_6^gdzTE8azWXl%g4z0N+07rBA zD#81+hT}l^ltB`TDz3oYTgV@u8DEeOd3zKxXRg$Cjmu74sXpnKqguXBOE3;$0({G_ zG5LRUv2vKmZHfHg4v%zZ3bGx;8PKsuEN-zONWyuZR8k1LS$Nbem1UZ1;{TELmSI^& zTeq;JAl=>FA>C5a4bqKtNq2)Z(jp+;-Q6Lgba#W&-FnvJe)sjg=Y0S4QsTMqwbmSS z%rVAV@!HRdq|t-oVQy1YJ+E~1lfMc!J#bofbZPthd>Vk)Nu-A=aGN$dDS7`x3Re(&uN&T6`yW!6= z`XvubnDEzOVwaeLFH`+WV})oy;ZZ2kiJ!^2D)C)uSlZUMoa#23nTR$3Xsu1vsan*k z4Lf&6jsAQv&3N&}us?jJTJn_=2M6$pATDX-8GQcDy!)sZqB2kv@i8N~8pbDU=?Ysl z5ni9BNaLKWoXYj1;$%)fgG^f~Js0|0f1m}YTe=5$!| zg3PAc`7eE`&*PeQ?{jl>3aA05F4N1Vc*sfd{Sjz*?djSw zQ@>uC|DGw0d}sqUk$mY`xgWP(bGo)+QCWa37rNg`GVMimch-xjLSw--RvjNGmcRX5mQ5N%{+=grY|6Ciw|lq^1ZRxvHU@p)bhv+?;! zI}LGk2owZ-sHY_zz5l~X_py`L)fH#**L%!2LV!DOg4DUlTWS8~Px1z!k4Xl~L%lKn zePW#&<2UoAxYX~1WCc>9nSt3rt3rFSaa`-D@ea_J)WMubep_;2AX@cwWGU)${9%u~ z&oa7x+k7sr=LgRx9m|5I`J~sTl@D1N8k|t4Jil1{c4?kHM!zM?wN+Eq^^a1nJ`|bP ztGYp7Hb@B`dhDl_RaxNJCBSf9BqLP$F_zQK`?}m8Qnruz-Gfz)HcrW-! z6u*;Y-mTzWZ(9S~ZJ>lFUgW{*9mZUQ5#V0(rk%}+HEF#1fXs^ZEs0z53V_-p6;*GM z#7AksW07YiB?i}OW>N^IOz9>oEq}+?KKGoe_ys)qs+zBRU|=jV>O@_j$YO>)SgUvQELkg;Ia;p=94|GL z1;~_B+sb_rk5pOIO`r_Bx&JzDMf&=7O(*-=t83k*gY>n_Ue#&q>-GK5J}3Mn5|{GO z6y&g>V(3t|@IGerQh3(ZWhBxXwlzh#Qe&74A$m8QL|nvu@F?B;c1*7q*!OA@y=rVR zqDakHLynWmLIcr1CcO|RoqcATY#Z(%KsxX}WNEA3P|@8Mrg$qZhJXO|@7v4LvisgS zE!Q8;RqEM~50BxlsZw~pL>})wLuR3b1>c3<9hPVO{kXf!k;CHJPj^=*9-qc9EJMhH zCpPjC;NQ`q!8fCp-T+KQ{BQzPVGKS8C94E{@q6|3NCCIs*tVxEO6wL(;FDw^CKr{r zh8qYsVkgts_2`P zYCV|;$8Gm5cRw-N)6>NCJ`m=tQA-kmTIofu9vQ%oJ&!OZ;e!bj&O?dn2na#MbT1^r zHKhazb;JZs>vV2%@n7%U6(;Q={(De?=-&{6z&sDbkg>%AG>px{%q?!Jx2G-Js4Tw* z@r)EaF)05%FPY>(?%0fapOai-(>|uy1-)AbA8)r0$!xH3W{$8%b?>0@OFyFbrkfN5 zf=8E=j!v)lwS;1Q%GS7jBN194sPpy8$Ql7|OG6!aW*`NEC;lgq~b%uoNWDswds?7{3Jr9owAgBk=`NyxfT!9+}EjLb?E zLHeJMjt*@I4cN~M#=8U1925W#gUti`Ab{9Pj$QXfbOidFz>=avnnnC?QEk!IFyYiv zgv__=e$NEqvwP^c8zsfs;Z1Je^jSVieIC%;zp@DRRJWHeG z6L-6!=J#62nj$BG1yf11K02K@1h(^4g?NUZL10oLUa8ky1R%8}>&@Rkr++Dn6}c#o znG;hyJ!{{vx*g8NvQw}oF2e$qn4mXk)PI4)>D;%;aXlLgg$+>9SwPavW$_(*{oECZ zAonEo3NavCAxHu1tPA0X>#^F`|Ey>5ITEm7aanW=8eyuaw{ZqeLNG%0j15G*E_Pb9 z>{>2IyCDDZ1#HFVnG9YDF2b|#Dn)f*XRcv*#-I>tT_A1lq)qJ_%zN;pKXzyF|1kf# zDcD#?2X=uW7QK3835SpD^M$@6@2e8J;o#$b4&$GrY@MaM^Z|%J_KG~3zZh}#SPixQ&kq`fayZ@8O{8uPYaNT}@N1N;-A^C3TRoFiJ$pYw zF~5x`Rn+dFNPL8)RY(^DEqMIgg#M*_V68XraWh$NvUdezr6~Z8q!&C2fSc?JNRrgjSPh6%?EZaW zBnG_{v1evS_O5%W#EjTyVz~;}krY2@i&jLob4hI`&fjIz>wgYm1LiNPTlJB?Oa?*Hy=6%Gy&}k z7&uS)j-5n#D{#!|9-q}r6v$#JqDh3mGxQ1rlr8)NU=A8CpH^eXgf12O`ohUUH4ozW z0oSL+cS>q3iB2v0=Z?zlcxl$^hZB&< zzt0paf4$O3H!f`v#YmJ&(SO~hvP?JpZ<)Db%KS9Cb!}4l4%Yqu9KlmcIjZ^6x+I>9 z6P-RDgrfdJ&wtua&b>j~dIdZf6VJEn$}GAKIRJainSBe?#POt{bU5Q`$b)PWCFnWi zrTBW{N{eMbY}H>E8M@eB@-J9}47by^dgU)JntVPHp_;*CfR`496$H(Ly0%TL9>1hU zk!2@9@t;>&hS;;wWBzTT)x|QO0_OKRcv%QZB`*78h2STzFBGIOy_>4D7>(H{F0`g> zRs6zp1rX9Sfc`Qj;mF!tLFkx8-smUL@OWp8pXAQB!#qgXPtZq@gP>sG7uMJ|pr|I% zi@*>!_jWzN$o5S2Er}GuX@Y!kO(jtHrp*<&cIwH)8CBLoK8_edLAuzT+9>ur`oaI6 zy6DiYZ#Mu{M%AHz2c|i~v0r!{NjG1E6Rk|Ao_4m%VA+4<#e5tv(Oea{q1FULiPoKT z-Wm$V=ciTvJ4Z~Q3$2m&uOF6I8YwWe+M=*|p?zE;Xlvd`7~M?dbPqB)baG!sJa%S2 zc84n;JMghEf`@gHB32nBnSw^WlqWe|YZb#CHiUOYv+B)f^vHM&IkV^xJHXg#$X!b3 zJoXIjosl0m(8sK654E`+S^@DcXLg7h3P+F!68Lil$xOI-p^MOdLM#hcf z2IuXe3=tivvOC&k)DbpFc|f2d&nDyGRWTI0DM+Oz6W>t&Ts4ElfW_}KHdZ=ybHhNgjH$53jFP((_8rult_j0gW8>6Ic z^XV~Zoaik(*d~7rQm-eQPNDq@2S308TG7#k?0A8V992!Z_!pfd732rYums5_(OSdR zVbGAc{NTt~D>wjtBU$^yjox16vnr@HBH~_X1ce3ZtV|LfJN(bcNI-e?o`{pn?wjaZ_ISQ$ zQ-R8z-q#0#FsM1C!rm&yN?9@C==buHx~$-MRzhFcOzB-t{-7B7JNx(Sh^jLcd(1m! zmVLY<58UQBsUtLTMdbK&twgXu6WCzOYAnMuj&e=5qt|fpCJN`Yy$g)-pMU)i#0r{0 zzXP1cu?mKbwL!akVad>oqO&CWYH$+HlxZ?G9xT)nnr=Z#DuhjT7qPOuI+(n=+m5%D zMOTAP>hd{7DK;#{6Ey^Ug|RtR%uI8cc?yCu@!8rYZbyi~-A2f2*|$gbyb-t0oEER1 z!Neo`{g9c{&Ita81e-33*%52VvJg{)9Z08O@J-bDFX%g`Qfd8QApi>OvGVz0)|Ct> zfxeW+Kl8*hB8alxF%M*tseLc>pJ;y-_`$0kWFCqJ`66 z|I#N+ek@DZTN~4zuC?%3Oq=oA5;e=I4^dhf6LJco5b-e6`%LX)ZeD{q`vjO~hyxUm zf4tnhtY=BKDX^Kl1$<+LMT%dBmR>3oK+23%H0~t8pbimtp%F--V+ZrqWXGVXu5B1O z;=5)x>{vjhY4g6eS?ch54e=B?7d;L-p22@tXe zA!s5wh4N`jUw22-mMIiWW`b)lAna^~kBkpUKA0x)SizX)7i5p=<;&!;&&Z!@Iv5Aw z>3s*T71I(zZ~6+*nl(PDVk7)~0BB|!pxq>@SW`2ROp}P3&9gWD6~cU?Zr)!5d_4so)KQ0L(S|xG7dDG^ zG`)8f;9Y3cQu@D-MhFt3?O{WSu2_7!oqg8vLiyW?g8`|c|4{gU)_ae{i}j?{^)I#K z>&2IM`(k5US^WM5wa+D47SEUQ`cn-+m=z!~zBU-Qcc@fSA*teFf$g(BlR(inmHXM!TL=!m3`DP(0dk54K zx|(Gg9CL0%1eTHvl)tzl&jf&uGuGa{C5=IS2=YXan`&!At^o_9L;Rti7cF9{hPC6Y zui)VVxIOjko=naQIvs_T9v5nQ+wS+i+!t$|Iab#A=dCJ_gp;J(OhiLl`rOdl$1cm-adKmEE@ z#Q@n4-G~U3|MyD)WeOFF$DA*!_`}V-?#zxr_SPqU(pdQs71s?1k)PcqFSmE)M-Djs zK>FJFR{@%W?kPnIl{$nT!G2mkV`o{Y#-h)zT-P~x2D*zlkQYuZ)q~9$MEBbY$UHxN zA7uOkvGt|nW@D<+K%f^;K2D5_{;j}S><$PRf922e<`oXDp->RXHza#B%!(P{q!l@b zZ6*<|0i*XKFpo_FUP1QXtL+{tYRAM{*7fIztFF_Ep(%H&uxr#C7$9;1MYWdg0uS}LhV~6y4OHAQ1|>bp z7+4X?O}v<%)IskeJwxfb5h*ll ztll2cr0JXbm^xi#G_~y61)7WY6QePM)d(RlUPm&->%DZO#_xxiJM`>Vh8N7YSsz7S z(;}#D86Gaw<~x?jvvnpL`s*M@43#fS};#78MnAYy*Tpf&jt)l$SemkST% z3>XL@-$~3A7I?g6J9G_3VB ze+tw|Wp;X*=DYJPoQt52HQ!4`NLy0ke!Mh9OX_n}+r%*dT$vI8cAr2G#gQC}Ag{hH zK~gw~CwM(%_U%RUMMqucn%}L#mgWN&;69gUJ9NpXe9s5*$^_hr=afUI0x@8aE&6Dq zRq0#KqWlGs-&Q2TdZ^%vtA;jm{1AJQGzH}7gxq!u3G9)_Jmz8x5rBDE%AWE2U0Q5M zMHY(4jwr6%0VZ)p>KP|Mb5h%H)}q|AVYMOl^dzgK_H%wU!9KzZ@kDdK9zZWT zy$xKmmqRM-(rw_&B86imInAA3`2pEhrolQ(Iu$Gma`;NH3I`ZUAcNyI1_+^FnmSV6 z+zs%l!BJ=>tWo1P1KyCHf;i^k+^6gFcZ@7_8Ez;)>f$zd2Ic>K&iy;|&{KL8`lj-m zbT4SlxKillsDtD^X-4p#@-EIXr+k{LmQJ5R|K6PQQc+d; z+kaRGLuiO^jYEDRv?}QrXmWD&9475$Ia=cpt2Mqg*)JoM=O`u}RO&Y7fohks5bMoY z_KJmyz|elFfdUmr3`DWiXrZ+y#1q)`eB7z1AFF#O{jO4XmWPLdkulH6|6XeJC66k^ zhF$kg@h>0*G88_%%PNM$`l5+)`TotI;R zHw>tvZ-}RYH!unSFVU3Si}Oz3gv&;^)-+5va~~gG{4KFFEZ}cgiz0$hViF*tk91T) zd_i`*K4DZC;%2^ZQHQMh_(L1Q<`%Ze01S!*7HB{RvRtTDLpr(tUDpJH)LgrYDpq1k z9!vmCrAh6Pu$wsJCgS?R3I+S#iR|xYyr(8J6rNt2HZ-Efp{z#ayP@ZcuI9_akCns& z>Agr&abT~@ga%qCx&m!@WWyE_DFHi1)hn%gGfnpIvRdo}eCFTKIV)w{X;;eMRcG6A zf4ZdWM@li)Y>;JVh?4z@K_xv=qMBa<8va`Ibn3T87^&o{pZQ&AQSjM&>f6ZXUnW(t z3vnO{p+ThMTtGN7Ze6j1hNUj*%p!WX0n@j`*zwx9u35kJgS#piR44M}{^luhj?vuU z7w0w?g90%RzANx7MLg2yGNyzu+PQzhk-0ptVgcib4#Y`ulj%@Hj3&X6+kgzR9$E2i zNOERW1CVR@3(;5W%nhLvofsp}eKCa8i`~)Tx?c2`LvIn_@}ri?UrECvOm2r}IZgGE z7umkMnJ7f)>$pP{m+0A!Tuy+pcK%ul4j4}3AJFlyPU=F2Qmvozg;rqj4=KgT^&pLe z-cF$*;&dFKNNz@P>1|iZ}Hy$K%W^0fN#;^ruNk;%b9-Qi1Dz*3tA}-{<0`*N*t_ipzN;B z;5u8Vj(wBVLXVETi8#PQBH(()ZSFLsS)$Urcx`t$Q@YL1L(((Re@p^nyFW zsG1eWH+tIhl8A2OH@~MXiG9@ z2tyLbD{=3gMPDs)?1n=Whr}PFL-9ZFU1C>b0a@-n9olUAqb4Nm>8lp>2~OWh)Ql{{ zfXv&N>%nNeWeQAo>msE(fSUUa!-N25N0=2Zk8(Cr@^NEs? zt^2p3*Dg-}@;yj0_)?_s*)&4=Y&2R{Gbvrmv3(lS+`9H5(hbDGX1ZvX`T=;!7ty76 zYhzD03ApeqgYCKMulQ7F7lD7upz~uu)%XZHLz}OC=6)gYC_*FXrlwm1k)$N3Gv{SA z?DPuuF15g-jz_!lzXZkGc3c~KrpwQ#Qs}KSFjMfi*B%4O zKh~u`raGa!v9|N|?7L;lV4>BValIzHNjU`0zzNJ~aqv!ZI07t&BEr6QXM2Qz^A1^i zbxh>RZ<6?KGv6eqHH4B99wp8%E7-l#p`0|*DR*Q#u}$J$6dTH4Jb;V6Z4_BZk% zcB|s*ZO1)1#1((u;A6U){K7HgAd-&@V6dH%;SAMQ$BkI8O6jl@XHdpt<8|8Os%2Qz)ay zSQm{8ZXvr(&Uhq|rl}ZC{SSnjck&;M!s!Y}cJdH$KhhR-cmi!^nwms7o{{gTG2t|n zRFn_n!nYF}y&));Q{@-iv&G6}oMEOadLf!iXL32;0sE;8=vU3zrE|cPRR6M>tB^fj zYMj1`GX)bb;8;cDkElHcxQxKMk*BJAg-bO$Q>N~8~-7B03>Kb163@5s>c;X$DkA4mKS3RbOCqOu?iDH^VXjj zg4eXU7t~4UGBi-Wlyna__UdNL zDI|{#H)5r_yEWi!jJ-hHRYrgU6d$FW9Ils^BGFd6yK63m2V946ZABZ}7x3b@MTu1Z z%9C^tmvbr2D;13rl^|5nC#N#)T2ryIJLOJbz;M@wRd`Vj9gm?(3q9*w%6GGk;|5Kf zG^I^(jH{rOrwn==A`Pg3!Oh@fQ&oc<|Jq-~PWa_;t|dnJ+&Hu4u$iTp!_Gpz?5_xx(q5($10A zBL|bi@Lyw3BQdRFLUTZ0E79p{N}508s-G*ll}Qa>YN0WnD$zXtTl`@F8#VaeUxd~` zYv0Gp)`F$CIQ+4&%eWU>8PO@%AJ||fKx}{{NSnIMl)o|tZkwB^%XR-tq5Y#RCH`IP zw$1RP#lj3v2x&9Z9q?CXm3vq8sHR`MA34X#Gwcp{=Ni3(kr^N@ONMAbSL!}sfr@I_ zF?4Cups{K}tdWa#M|+XK=|v|PkgSC|lLC4Cs1-W;Ku`56bs}dLFXX>nU^9C*hz-G5;^`!IwdYeD!&Vgi4X$S~ zmtd4(`;c>pnqSn{Cm!s9oV4-K7dotdB?U0SxbJve2;>&gycM){U z<2g4Ow5iL@j>>N+*c5gP^JD3;NTr5iiQXb?asVj&8+6cqWGR5c@r;*jy#XF6>W+X9 zNGB32Nu{a(9(C#wA0%nhJ<@mvi-!6qU|#Y%v0^cGcfdnNyIO$i&j8IsB8U&Nj-R$) zZnCz=4nzND3BC21;%jwwg8{-0Mi9t|ng$U%GC1JLlmjGVbMJ>=6c5bPVAgd1x07hV9o$;P*w~> zOr;zzgzc5(p-3==o?~80iWbsmY6@~#&nd(RLqEgp!5c|4ndm8hH|MRqdy8?q?{k9Z z`?+cSG+X8n(4h#_FXAN+hrN=pCq$l_>-2gluHtS(iT~A$AFzg=n{eBR-H3Cm9_y>9 z&&Hheo}gt;yh#sSxMDXCbPi?A5^&2y#G*x~4XdX)zffDY&Pa|ME{C}&{{SLm9@~8W z0!#RjhptUFIGUG+7|TGl!crER(#E{g^INYiqt?4j4? zo^{griGbpx>XHK|dmx1bHVmP9mVeuSR=(0mOcJ1DsdrWTUAx~RvP`u&n|TAi%IaVu z_vTTn`!KdOmB_K<6RmQ#9Z^}Ym7*j9_Dkr+$5a*K1B+ZVC&5O*tjln3%VVtsMIr>* zVFgClgALEc6KQRu3<}R+LNBbeD+5&^n@ha#4gM;K4{C6)^zfI6<%hqGa%#q>hBU=7 z%O%iMQ66iE0e=w=A3@3EwW;O*j4FBq6m}s!O`GzbJW|>6iv?_MNKmmav@%UKpOqNq zlS_s$1mi7~5hUqU(vrG{u>P(tyx2%PSiGR5fxuGu{$ygbZ5k;IO@)Umh(`W*pbMvf_j;WOUnH zOH)yk?5ex}v`sRFrt-zU5;Z(C$Bb1N!zs}iBl08u}lSa zha$97N~r7(I8fJa(d=hLmlluS=DGEoV%Ea12D3F8QzrUDvr~Ek-?=>%yc^C7Q>eb* z1oPi83Qi~UBPOqNli>9Rk|9*jYFW=SfJ-590>4XoxcGF+G#DPhL~ENf8G(hXA8|tY zsiq5bOvha%vu1;EU4fSBat|&HUyceLC^l$&m=u}n2>Jt71$?^y@Y}7CW1Mrb=6#gQ zA))h)Gm=3y4uP*eohzZ)c`wjW+s4fMg|-qC%vLR6H?rPvzpt^%;g0={cU5$%rWKD( z4Jfr*>Ok!yBfsmEIbMYZI4_CgmDc*yVO2=94}}?g2VS7Oe}&yUI8xnpz@7PsYYV51 zY=hpD0qM45A#MsF!I(q6#I%Zcb5;+`KN)DT>?$N(Ytt z_dkQ{toIYGsc3!2fwXVAyhhgvY~iB>GHv1buzZ9`=u<@eb#M&RRm^zEC)=VqQnP0} zAyt_q^nMbw7kV7IU~$-H`RojRLYu8BY}11-ZlHqaXoZ2Fx=Hd{d|4kzbB8z%W8#%X zpOb+LQGgI=#Pbs?(ZA_IUxc&Avy#v*oJCSTO~paT8a-%fSvKoXF#*J67QIx6J1K+H zBI-07W}FN`7hCl=J!wka39^qt)P%ya3kQ4LuK`njBE8NKY(x;jII9i;FgQ@ z7~83WNN+ho!wyf5RsYB1n_;ffGK-JupXk|&2^HkZlakU8hIZa#bIgibO^~nG5Fro* z$|`7j-y9Jw&V!PqwgcCHENDT{w;tQ{&CB}|mokbUx6KWFxMAxeF2hL>$^9QGSQ zivomDE!{Z1HXT6Ne)R%hphBlUysGUtBi$G0X!cin&C*|$?*KNmO&qG&(9MIpzpv&@U^Er~MAvhuayIyag0@2~qH{O76il^0q07Ydcy(<3 zIvS|kGi3eq1#a}_SL-~dHRL5R0N7#xHBVH8rz2*m%QjKthEB-{fv6iO3rt#ms@atT2K-qY4C z;+Wo`jVqRrJ8GkGJ0}E{vinxL=;4gC4zON9ibkE@jT~II6K-hC>>K8FzR}B`FyQjB zXz%fiwabXGGiO%iHu315OO|XZe~R}!4l4&-Z<@fXg3u$`1|ltF5t`*@P$fJ8JKsLJ zf?b56?*&jKc2Zyl5L~yv%0G>S z*3t42(}7AoizRZrP@y)(J5-GRKcM|)YO1%h=_9%fNs!N-)U`<|6YC)j+{?E=y)O%( z04y&E74zeHPp`5?gr`DMrU;{^)`bmJhmey|3NM{r(*gvj!k&hjY*`uyPL!T|;`;$V zy50PqjbA#pKM0S4k)Ut66uS6E)s&|G7Npo2Tk_H@n|oCnw5Ngy7~x75Rk}jOQ{?lt zZ;oo1IDUX}()Ck;t^C3B{-eZaJJFD`;AKBe-3F^ScLP+B4H&nta7kNH2vqg~)B6Ub zhrnz;r+hUIz5}^W(@Jk!Z_?2tC}Q=Dom>}8B3d;W7L`*SorxSHW3X<+*jMY%N-(hd zhDRLRPD)>~1b^5d0At?YN^Z6G$xVZpqjBipOd#}cKi+V)wNb!x<0Lc|N5)wM4;idc z-udp2`yMfgf;+50?gQ|Tg>GC?k1QaRt?1zTY3F- zpe`ZSGhsUvLofq~+X>E0SciDjIHlFH>ymxN;g9|MsX*}bbxio3QTmZmQuQO4(3Z`) zKJk_7a(ig$TRyZVM_ES0YhsI&8zEvRI5es>;;Dvd-7isDdpKay|NQA4WQ5-|_H%D= zwanUiKLHuMt{t)<1Fs-C#r}dUsiQ8Lf8t7G+A|;kAX1k6Y%C=$3d+`~#N7-%bT!f$ zp7wB>boyF1%th%Yy(JgbJpNk1nhsjFP%aIjzfo~!&qR{<-;+@nS)nm(G$fc1_I z@PC*Yr0|&M-+;U^Z;+=0#GLvX%nho$R~1TqLr&U{(kJ@+*_XBO_hO>$yzUYeEGC7q z;{I%*WLxzze5o7B7nx;hb7#I(9E6;`qETFJ(+j@LT7)_A6p0sqT^;^|g z7|-q?BPHNJ>z|(v5qz!XTaMZI)kfr&0~f=@Uo1tda*Nj?{#Lh#sUOspN-f2CqX55y zr|%F6AY`(X+6?p(xul#MwfLJ%8j+^3xqFC$N^t92Tx&f{pKD_}e*5_2vq_zoK%wEC z!RAM;KgZoJBX|TW&)k*(6$X{(q$s z(8-DwINm$S>`b}?(plHm7E?e*(Wmyu5&9au%*_}0|f=98ISclZb}LBEi;|N zkIRrxzOQ@*ZuY&OZlh`8TQ7}4Q;7(V64&DiCAyQZ`z&~$S@}!nFw><8s2{GLTHLSv z_3-msC5*^d;7q^Zy!8STEADrG+6*tsxAi|Mh^DDs+l}D(PoYvIUrcNawGDrdEI|{DhuRvxPm}1D{s+3 zpu~{^ORsLfVZGohohtFmkqg}#W2nun1V(KWGAS^Rmw%!%AwDnP+tGiR#Oe6r;LcU2r` z;e1EAcooE6h#N%c+f)rP5SU@QL7 zI(tmp;MDpT7&X0=K!{g7`lw-nCsq&Rt>mhsMWRC4Fnw2G`B#OTV(OVNKDWCI1ia9= zK>jUu>3fRJ9cM?Apx<#w5jZbuwBNxQFo~mH=#d9hn7X3cpjbO&km^gHl9SCE;r`{ZNN9e=TD%Cp3POP4obO_ z5vg@1x(o)CVS{+1@(j{;BXgS?7ue#-=FsU@AWUvK1|iRzKZum|&5h4ria%ZC=%$W^*haKkbqQv<6n zXQQ+@TW$8{m18mnwP%FGtG9ttA|UsS5u#@pI?{fE8OI0Mj?UyflB%dK(`Dg-AXoBN zaXaz!AKrCeGqq*AKTns+i=lOY3=a+5c=DvP(&;%a7Pd!n2}pJDfcbN zqyQo-9}k@ydnivT`T#V5TJ=70&VS3F`|BlVTVI=Ckl`AY1$zyFD8o(D@j1E~vwWQuR(3wEMForBDiFd4+PG z50%}&-K34C&-D{6{y7 zPh(%}2+EYpfU*h9AAe^`Hr&uU9>dM_dPnQx5Xa~MmM1_zXvOY==Q(gqfd3VlMUdPoo z?#(1d#4XKoEl+B)(AV5y6`{S-3k6)XXv%=k43_uQB;>X!C~MdT02rN?$Z|ZpGAo40 zx5FbC3TdIA>Lspp7Z`&VR83W@s}@C+aN@J^#jbzc+K9gNo7xoKp+1KTo2gjs7%BI$!fShm~`0rJ)nN03e^uv!|XsBC}6|yLx2x<8bMIc zeU<)**df=GA7ZVx#FiXlq@#Kbjt2c9$kD)kLxhBCxgz1e#+Simy+!@nf#W?p*?UgT zi_!9~@)aPSGjJZnaRn0wdGLtl=JfG&fS zfZ5Qm%HS^;iLw<7(3x=+v%W1Df+U?UwU~}-MaOng?%v7|7|{#wj-_+-gY=3oLIV2) zb9EN--KIj`afACKiVd6NKM7n#)G2CEPpUEUK{+_agoht;dfb_o<{do zb`NkI`qRxi;dvB|kHG$QlW0RdGCQ%L3+iVn9bT7}fLi zthmmNjTAxbr!8CmvHKSL;x=37#?mC$Vza)iL>oo*mx4RO*-pt|Gy=2Xx7_DHdwOg| z+at_Kr(AijB0N6sx}}Bqb%eYRfsYmjhWYc*4h!#hm3#6xka*DOCEs;dRj)?d`LQi2 zrlAiDIPKiPARbK@=||w~>mB`W!}gy4DIBf|6pV|Xdil~Q(RBMPJZBRZ?Knb+1PkHp zYaJi^x1`>jPzpcV@u4pP->K<&K{uUI$4mS)6MMI>`xAw@L>=08=%zF-EE8j3AV*u( zHW~MOwp`D69G-lr17bd1W#xU@acc&H6)jARf0?7tgI}}2H)QIFeNrKNO4mPc;3X2@ zOm7BBaO54UbjF821g%UR-&6Eyqgg&tv~X>k^WtMm1b=euETFj7JC0iCNFz=>`dipm z&DJ`{;SJ`v=k?8MJ79!)suPndIRO*cTud4eiccW?emk z`4j&)9b-kW6D)FAJo6fjJvODLaAd_eIG4}W9yok6RPf{6oXw~-T z86|jd+UKMmE6&+PeX|%tI9t7gOxl}qn(I!8;hbkd<*eGG`JF(OPWcBI9d&-qe2L0!{lV#fgOw z8}S=ozJVQ>fIpv3N`&=ddTVA;@sm#21wYnXMt$C6;M*Xd-KNN4e8m2vMgD#n$V}`+ z>E#==qG@F?DQccSd3eF5xxFjW(~~mm;r5g)gZNbJ+yUj3`Wg1L>FflD^~d=b?ECh6tAHl;-| z7@fHG!r@H-7VXIE?~xJ@i0c#p*AT9h_4s*mh}}gz}_KLBoWYt(NKW5Vz*o4rXlQz65);3+V%Ft#8q)k(5fX zLl;{Q@}h@qR z5YP8Lr?C}C?f!l7q{sbOyFvDX0IW0$oqPbMbFBIj3$vHYTQS}EcY8tC3>6Cfuv%?a zdI^K2?Iiw>LbBU>IPKnbEW!6^agIJ&9~S>3$(49#7jZ8Bmjnd4E!MVkMWej&J=|w2 zcYmBsSyO5UHyCJDnm+GDsshA6R%@KQpUG=)B0HK?QR{ntQZpGg7{Cs*d|HJ!w{n!q zux3+{A15Gp2+U`~TZ|DXEX?jOmc3gX-=LSQ*#agTDs@8`Wq0vWh&oRxl-jxehqV1Q<=y&0 zgDM|UBC}qD?xS{ukCyQM$dY(1gLD42J-@xb{QXXjXAi`;rL<=-crDX;f)6TV-f|8e zT=OcWw6wbrbBGso_od^XMT-umDsV`sv@;tvpr^CiJO)pl2e*AYzL8IVW$39)G^2Fe3b_Gc8r$%| z+@!HVP;tUvgKY>hyEy(Fg7@YwHCFR4%uCb;KIX7uMgNzc`(!gbx2;Q!wtAqxRF}wj zcdUo-B73xsiLC8zuf~3-`fR)DfK9ZceOB~BS}E&CIEte(3GH-@ zBq$<|{Cf;gwM5=?2Z;^ZI20xjXY);lPN zkl#?(`;3`m2a0|)Nm?{9660U%SmjVqtGCEKe>0RXIo%E9BVA$(^+AK>nGHfkAA()& zpe(sW0w1%=tJ=QzncJ%PPq(s5#IPFzv3!_16mdbIjK;=7nsMXlIxY8wx+9f9l!H@ep++YxiX z6!*S&qQ6@BjrWr_m`-NZ9%;l+);Pm7tDVcxR;g-#nLfAs7!hQo8{?7W!_!oEJBq0j zPu2qFZ&xwAx6~bEuYD1bF>iV{7_f}ER*hUkF1GY(heDR%Tf{;ize#3e{q{rbXTG9P znDQELKUCUl-C-SW`1;#2TK6r_Q1B8wzZT$^Vb(7u2Qy^nt%s1wJhXDY=c0e34=U8x zn?1qj=M2RvV!c-HE-teFaeipL#JD$<5`^8)?f3$|FG334f7L&|shX zWgaE7IwX*pwB<9ti!rlm`+OdQ`;{=P3@-{29X}RZMis+G_X0pZ9`_TTNsK*k9zF?#t4yG9i5^i_4 z6R6&d2=8<9lV(Oi@}_{D^X8&~;CVpU>pMg7f}sa6emK@Li9IeB-Y8q_cs4tIf7krE zWq;L;_Co)|KH8GKKAZDc~G~8n715g27_=(xhFjcE93DDa(|HF7wFu>DB)w;tUn*!UWbL z-_8$$%g4Myd%UXculL7<6|&FcnU2*#9#VRhR;BKZ+zOCB&-N9NitF8l2^>rpkB{}C z_&DW;p;ge=zS60D1lh7puiIf0-<7H~j1j zn+*BNXU3XhHsiPw>ZRhmlpbqqZu1(y=x}l4iu)6bkbSziaLiJO5}=#_m%~D7-R-D@WP0x&TidDGxVf*toXP=0WWR^}H&FJJZE?N=u75H{_0SA9*uUR9ooXqwky_eyobEU$)==}~B zoooRM?ZQmTgE2Y}!$SExw#q-?syBv)F>=E*)ECtl4$Z%cMp{5f`KUVD)R!1UViC9B zGqnbAy#`5)XXYDtWqCCv!U4nEsqBJPdf`XucbW0m_D=B4w(jX-u5>g5%q3?XX z?`QqabIyAIa;b~hSIo>c^NEbMj@>Vd2^WtV0X$V0EE+t_d2D&}NPHu#GLL_nn!wxQRK%=ShtCrubRxA8!(I{Bl)vNAm`c;}f%PN)R zzC`QA$I?FTQf5VyHcek@=}*^N%uc(xuSqCYBreLg1$WjK1m4Y&j~gjQA~ z9L|Z?k5>fL)F8i&hzdQrMkSpuQbeCkURcX!$%aZGW6|DNyqa5EUg1IFzFW-&1>r#` zxqyUR5FI4d8wmA#-vw_MZJS$r*ttNX?d>CdaJkV*38!0JWDp*VIvW`etWrh`6r#RD z|1fU78X;4+u0FPidub8$WBXivjI3m{$>DFPNaK49Ttk{73D4JGo^Fn~q#Z4t>B2af zarH*1A9Sdg0tobEYPLVLT?lG;bt2R_Ox9m>yCc^)!65aUq&LV9{rPD^KkBkc`>TP5 zQ0u6UdE!@`G{Q%!aX3&(0h*3NFKJ}T5^eR`T{9M$Wu&J$B^)p%O|WDwj(5>Cb4={U zXXT{4m$FlI3OY=#?qh|d?%mNUe$YewA6zg>yXP zi#Y!8v&8SGYnVq1Z=yrnYVv8`3pd&gcX%e?%GlU_da}P-eF;ITwq)O`5R_pNfPt6atdB=O6pTX3@Ze>Maw*Cst4Rq!}5k!B79oe(j5JgX(AmdZmGK)!c zqHu#08M0*w>S-hVt;u((Tdc>AkdNr>r<_lR6Sbesu#Du5rDxE#@Q-Yw5`_ifKYe!<%PQ9VzcBPN-lA6wI-x!SYCE3)}teKkD% zuvB>d##oY84h@9(hOZvjrT_AR-jK1@Q_#l4`lyDn{mR-YDpnUrI#$P(8s>vS&dtJ3 zB^nCIehEaN8k;IVB)QHlyI;huR!pEV=DErD-vkpFaWfU1)CyA0@yO+x!Tf^Xy4`FO zK_MhCGuKByUHrK?Gi3W@dFuNZ%;k(IlZc-8n{;8Bv;}-KZV2!c>Ftc}2(8t7vtr1^ z7v*NMi@>BeUSqswIv@EDPu?a^=bw z<=?-K<{OrMQ-78+L<8wpXJ_4TY?y~7xFh|D>gwhHMfQmfj}Jec5;hoQ&UF*P zdPk7WQBv_MY-)!xSQWO-b;BJ39IJrYcF%$!%g$F zX6=wDJDFY-y5AaO@D^N=>Q|ay?S6cOZ_v-%SIOOrwGk<-XlGYgbB)?oDsqI-sLZ}* zMHeshx08d6rSCiO8DB~qG-$SMzmL7U)wTQnM-F=Px_Zs4GP0Lz%jp>^R={B8igY(^lrIvBMSk5%sA(JIJoK z3OZD=AB!rWj*txb$1#sEOy=|ldfXQOD%lHz2R6Ti4kPXETAV6sUveXM?{<6|qfd}A zpsG$yZPZoJMH2l3m<9KJ%e~;pQ}UVF-aTh{G3qxfo2s#JVhPBL{iMCE%s-hb@F7$# zWw^vKA1U=XCW^>V+^qWdaNL}_k_*M!e*slKhatZ8@cQI7X2kwa{u)V!)&9+Cl= z^g|?BsKrjpU)qGf!8ya0=h;XV`M;UMoDDLCc{HQx0nF800hoQQ7J%8^fv>BxsH?_> zD~dG7G**%N_MQ7hg&*5<7Nb@8qbsK1K+nz98`YM8C`)elq<#wo5utRO+V)f0`0;5V z0`2AWZESL zwt3b5Vm2)B@~GVNH(-CW4)Rd7i5}Ij*MN@SDC=83FRibdDx9}P?LU+CwHY=*v;LiK zeoC;!B`4)MIe`CM`}lxf8iy9gl?Fxbv$jKoNk(qp&0PvjI3yc5RDnSH46_?N(6^+h z0yGeLKY5~P z`AWR4-M5SU4FR`sa_8T|j7~&z(f;@IU)Xm7@9-5j^j)3y($L7C;_60)IMZbW)TCh@Boya8HQ^lO?&}jt(s& zLTSVS%<9^_xew*sXGwGwthf_Tfw++9(IB*6eiOQ;m6cOBYTNCxxur|vpPeJR#3XS2 z0yNOcx;j-Ts2`ULuh*caja?vXl!qQ|yOTupS7zVhya8z@$I8p2vdyL=?3F6pHO!}G z06mk%;n2xa28j$&TH}m;sn+V6$FSlXT@nv(qnnF%xU96v=IY-tbOUvvR-s2z!gY#9kLND!+i%AavqvP7+{cW&bnYEuD)8>SglD|d zPeY8k#u(W5Wx`^t6669mOoze_@pc0X6}_?LPK(zNTISD}+T-2yC;J`$y*b3F2!tR? ztQVk{#bGVfg)@BK;I*+T)Q3u(8~Spe^Lwyccnmnokt#PM+L+zbsWydynPx2Q?9vg4 z-t^1T?4;5-WuF)9*k5w~jY$6Jsx5te?+1Kw(oa=YB7WuD29yV90LF1Q46QXc!r@xf zQOr$N*@w3inT%z|3IIC9W1HeW+60+VOM27E0>r$Jn05QuHr1%kS?5|?0B9WU?7(<+ ztVo4GXUCLHT6NBeUBtUKIC11qd1mW}RHjRmk?I(KJ4g92>gnMXxGKezBpKU~kR}y7 zqhm6B@RiHsp-7XyTl37}?c(9#2q3FwEtL2k)}|!xVeHVBt|_;B>Nwyr^F((4x->UD zCY%4(chJpa%&T_zv1dX|`Rx=xMJ8qxcJHfdii)=kC2YOXC>0{#91Ok|&i#dF=cat* z19Q_aCgu;lgJ#U9X3CHF%*ocMwh5N*g^&C@L7UtTp)SM@ID%=Gmv|g|-g|7tKINJD zkjK$Z;SfvDw{qvdpQsq9#9zFka(t)Tua|>KOFywlsblQbE~(m3Xp7j9O5S7YJB6CI z;fVo9+QY#W8dOSu*!K7wMJ7knWsvI07`e~FaUBB`dqiT5;tZ`*#Rm8HtCj)tRR;X0 z5M2imiH&G(5tfcYdKnoLI^gf;)!wlEjC&g>;Svv5VU+R&A-!we0TIw8yQ1~N3?aM7 zFDjUmM5_OxsN&V^ggAwBo`rCJ^#1QE1^peEZvwJ5E{k z6SGe*Ulbb~_)yxDp@8(%9w}lJa`*}J95@#JZ_O15Wfu^xTk+(O7)@kptWI1@Th!o1 zfA%{u_!OQ^GuwV)C#4eSX~onD+zP(*Yq~uBX~Mp-B5Ub44E`VQbb*C`p7461(>`Le)yw8)(mhhY^ zNTd3K$p$O^T|?P0u63&7j_Q7`)yjAG=3Xy>sHs8*eh{|Q)!mWo+H?Efrv`|Z>p zicAcs3A34)t1JPN`@jW_RVEKs!gxHL@=ID%@V9qmo5CA!Nx7Acy{JO>fNbUl5T#5e zL5WX!FKueIn^5Sx5V`3$EfQZ_`=7d||4ADWr|ck*c5`>e4mDi=u>p$DXtQmD*N`&C z>v|rsYqh2>D}el|Okr!Zs&h{Kuv@5=E)ao(U4!~PmkFvu^zO>ze~vo-uxkfp zVF~XDw^V43Gz;svKNhgcQca{)+0{q&Tu1@IJs~Sm?cQz&mE^y{D0_xDAm8abZQ-~0 z`ssCFkzt&&gz{NLBB|B?IJDWAM$~iRl@$RdnV9K#fJ2VB_a`%xQ9yL6z|8I5I@A*g zCGexPKL){AQ<$>*m~AZv zG{Uk-7;xqV37vv&gq9F#Q1E8P$UnxT>pi%VwJPB{{{3jR!(2*7ni(JB%RSDB{de(0 zO@=2pNAVwQXe(M)&HlSnf|RyVNHV|`#OH_-{#7&-j(HAUwOKp>!>Dy=lZ-x6%`F$z z2>_UDdTZb$)nr8)$n6@K7~H4i-%1MTE2wPeB>5*nNEQ+XSx}g*U0Tp(=%TAJE9tuHB5Lp^*)#hx2c?cfo`@P8uMHftJ>m2%siUU54o6Wy7Ss#-Ai5YkLex# z3UB@bfse#kVTC+{+8e|DCzggMG{8^zn+Af8Io)F9KVfM;WY8;e*CV>cc}3x~GdIo6 z#7>RV+{r&$Hx94y7IM_~TZ2SuCZ~SSu61Gf+wEnMh07HN|97cLQTrUi?9@gU13*N- z&L`xl_%;_Wj~+@=xc8&*65H1O(sn(onocIDn)0JOKZd`(N(WEa_SCrFr9P}ojmU2D zjF0`rT94ChtT-NkT2~pI(O*t`Uznt$?O>oQ{|h`&0| z@i~d?bZlP(jP0 zoZ!)aDpTnY&%+Jk)#=5RBo~4m%*rhQW4_9}JXUYlzi~$M#h}O7I@QHu>YwQ1qm6pO zWq@@=BGbe4G?GW*>|LKM&bExhM<-Iy5CN+8krjD1%{vkK~kCH#RA33@G0t}|T*&No@s6Qi02d}-t z-rWIAT)k07`Tft+b|A!1?>Q~Pe`mzqf^)AdKTH#h#{-ERW=)@o9o?<*-5~1b%y5L| z=V*Ab{V(b_>QqLySrqXB+*S&`!XwDP+<_lG6Mhxj z(s9?$RGiDrhaCXrtRDp(gQ5cA@M<n^ zHb44k8qv=C%2VG3FD@@ZL&)#Ie(kx9dU`JRz~K`S(4;6{qRnfeznn4G@3Neibkwr4 z?b3cy5!C4Dz}Gvgry+<{M7o$UXCmdL#{ZI9)Ky*iVH4Kk`!j9{*RQF(vWp$d3qBTd zZ_*CcrI8&Fjo`7GF1kb2@JRJkGDy#a`>#kU=Kri;B5olKI;nuw`JiROVqpyX-ksX} zhpgjM#~=kV(j7z4;A_VGKhU`49#?-C(y^4LjSU!9rUsXH1v!9DRi2?vAz8957$wi} zgz5F&xv)5mNw$Nw32#1t_lywx{q;`#x5n|DjsB(dG3gDeAdjl{1oo4Kk8uxIi!Iom1M|YdM_P zH_jpkjXM5zEPQkp-8|CHwE_U1h02wZ}hCwvx@T$vKZ!-Fyxldo6{ z(}@1mN$t0gP)t2VB!WC)rg%hnmsHDf%5AEe8Ab-5e4Ov{3}{89a9>>R8a~&vGMIul z>~y8;JMFM)iZIDO)?Czd^V;%lfnNvG_v9%4MjWo&DPlB(uD8{CQQQ8_?{r*%Mxr(6+ z9xsz_Pp@Ii`Oj`)wepxp%Zs7A>eTf)J-gHcZ$}FX|!bc2@Yy8An=f zFM!FKWFuaA%}S_I%u2YAaWyj5FE`ClZLhcat=or^cWxYmgktz(K66rD5S}ovG5G*8 ze>J0aC@YHSC=j%*q`!G?`|Z**6j>f3UszFP&ud&R4}dpZz(8p8a|FmhSKYzv0$bc? zZ&}mi+H+9^9U8x2Vl|TzBc!eG1nUtHs#B<{EB#HZ733qv`DTPOn3Q8%S!ldk!H{Mh z-V$T`;pb$`7iXHW$2 z-3gcshvKna7oGsB9xMHz(pEcKgQw3`{Z z!0SwxJdHQ$f`M)Wt{5vC#6B1fI6)5#oS^wtmsn-|G9)K)uNo8_D&NJ?_i&v*Q*bQ? zZ1B_(Q^GKm){bS|AgzA;4nmEsk$F#)w-EMdFjVdXQg#)8HTpZAiu)+30jw1kefiea z$mk4pbcy2-0m`c-?N9%LNz%#!@g0|z*E=`Qej3^^zxkHD0iBQmEKF+Ar!>pYD#$59 zC=k%|f6Cl{gC>_M)}==4hvM0M!!E1hr`D=$^jJaNuS39f@fl#v8S3g^FKNaDC*ruBc4wdOpEV6mPp%d269vrqF56vHUdydHt#PYmo7{^+xC* ziSzYSL|gFlpkbgeWvDoM>wX1a({n(?OaLfgm3#DdXDoSPyROWh@_o+nxoMW2I;v=W zqOHm~`(><6xYUiRNeo^f8S&=xpd3G=-q+dnAbZK1XP)E^%=y{zb;D%8Df`=+7(%`> zSHaL9=?p)9@Y2T&DM1>7zdJ8n4xT<=xBVY2fRe`7vFwyDTqpOX{r7rHU-yYqhOUkb zflRE+t?yF5#*v@&t)>fyr?1_!*KWxg{m2p7_{^D7ZA;d%Ozmu=U!M9WjH?+hD_tjp zp;u>6>+U9GjnIdyfFOP?liEcl-m^vxwV7@c_F+Gkd*X%7^}XI0Q-RRicyDN>^DFii zjpITey?pNKWx6AEkU=QMclg=91{c%8TH;#WAJfd=rjDQU^bT8Ab~PNc-vXx0rnEn@ z&3jre8RHda)k5yC6S0LW?50m8-7X{!Zhd@wuvUu2tOY6sl33e=F(Bo|{39j~!Y+G6 z=fdfUtFHNI;KRo)yE2a?c9V2QLnlmB*zNo@8UMRYCa;L4ct?%8!|}ebl45nSV>gzB zP3iz6HGTO$v{}^N57*3%Z&7lT6*by{$KSQErbt-=vkbl@Lx#K0t1qoYO1HV>%c9LP zjFSu>r<4LNk}?SR_}h`wZAz9-9QGrfyjjbYJ#HPX*u_u`w`4O0|A8`HoK~&ezr5e} zColUkm-Y8GdV6G~Ad(7cEuNa-863Q?%@7Dq9C8D{*04~{#c zWno7WbCX=t5@q|-i`#)?XCh8RLC<}G+EDB~{;?*VM9+MqIq*S})E8?0>*UyX-+maf z0pv~U(;BgLh6^?Ke$UzM2tOXQgd@b4Nv}`Xg;p7L>lK&^UOQP>WLhvzN4r1Z;2PE(~sB0_ao=?AiG{O?yyEr7xUd}f*yS3pQYM{aX#caN)Mfc%E8zYS zS%qG*is;QQ9HCwYR+-hd3p0&g9&LSro1ac#LxcJfL-JcT19!^cC7b6sle6Ix)wW=8 zMDn?le~UYE4#$o`R!#v;emF+TZ8YXlMj3lD%a4GLcbu98Jg>|xD?H0jTz5VGu^uph zKA?L*;*Psy4KB)fCt)s8EQ(y%(It3f61^WEK@(K(H8O&vyQiGbU&ntkQ$i>3@pQHXvV$5bWz7@ue2JVCD0q4qU!^JWO~o>9&F8?g zP&GrWTUk;L&Nse?rt0wbdycc=omSkCl2$qk8PR0kkW`wMM;sneq|_mR8bW+p2~V)o znrz}Enfql_(-H9P+H4%wWK@qh@rw{nYcH=mA5C1w0+x?!anH|Yp!yhGuYx|o*5<{A z1m%w$ceaukWS4gFwu%ZOrY-p z+4SH_Q1~8sp4QiMc^?2o-{dvm6k@&H@_L?TYmGe5OTEWhbn98&h2=J;!3kN$u~Oqa zt%ExD6bI%sI!$f!s(W#!Hk;p`t4vbT8Hyn(`WYHUvk*=?5~g`4O?}vz=xu!aeT?$h2_@MkAh(yJ`HSAy7$8O zWO?UnK3i2G5%0gYMH=my_cEooo1S;`*IDH=a);PlS4TadLu0fPcWOmjfHIRZNYeOK zIsPN3vy0_3EIU6ky6apANji2Vb@cDXo(Fsz&IQj+es`_5kunCSihIgdcUX8RJio3C zvceAk!F&2ez|T1<_F(!e{zcJ6E#nC#Yscap!!ly78(6=8MSY#CRQ}_66fjrRc)Bm> zp=vVUm->lS*qN(k(mF%{D-!J!=Q!)YfA`Om=T>bv>}kI6V^6hg;j#DwxBIP#kDAEhIlZZSo<@wXi0crG*7t&qb8+NN^NsZ}zH#1h1kK`xDhXh( z^n{^h-g>F}ZDb8FnHTjI0rykDZB#k}3E4aJyYj$C7;(P85(SDPicG2)s^>vRsTla2 zhCsXkl8$#R;YE8qM}VG1*y|K%*KFtd>7pGoFxmw>Z?{@32AsvQKJfWx<_0w$nkcBo+H933(bPl$0A7RIt-f%;MUB5`+z#g{0%Qj*dZ zcauZqRP4c$#{N;tw_1MN3HfZ!=B<9*b~l`jBbqFuCkV+B@_|V*W!Fe@j{*VIeMI!Y zg&N?+PzseEfS1h0=yY%>^G4e{W=(L(d-qesP`E}piyUw-N6z)Ya8 z4kBWpbAF$r0sTaSl%EWfpn85eU`rk34VdXeTz_8e_M7wv;i6H{7QH=taTTk19J+X* z!J!d_d!g~90gQf?2eXW`j2q~2XmII#FbL@`jlB9TBoTM@vll^v4YUAVp5heN?B$79 z=0LF~26}i7`|wT}xi9!i9RXH-YoRSP?#4P}1h0s>pykBI8YTT>fX9#KNcn!vzWyqn zeYW0>45U`WZNK^e_Qo(%L}ZR8id7m6l}M6>$X;(nY6jM-VVFLIZq=&^j9WFN$k`4< zos#D5vQP6P-Xmw&$06v0LZBbK#?-Dn`%g6d;=HUV+ z_z_Qu%b%zPe-5axw=UT?cS|y+W!3jVzZ2kkqO@ChSobrV^bV~JnY19XVdQ!!(EG{s zQujf+IVv6`8^{_& z_*Zc*yJ=MzmXs%ZebrCW*nL1@tyH859bvq_3*HJ}aj7Lco1Y+XB~V=X0FokH_n6NO zO}F~<&Vff;BPYogF5dY&)r;RANN+QZhhrb52JI6TaRPkgH<^BJ=Ogj^_ol?39$ARL za+KsA`ZKS20l`Tkg~K7EO&~=ztd&W*?dgxqh&txSbi z+3>Gq)<+Fn6Oxll?GqMJDwF&7J>(2(qwlEIuPl(vJJFX+6S_9(uoFc8FhytJEq?P7 zl$glzkjo9>@5q?X);g0pVAqO=HC|zhxR6D9*7$L@zN5lJK`(2~WXPZqz3V$YWIFdZ z(_}JZ?BHbN&_f^CrHK0ZkG<`mNWT~^FpdPyJk^6Mzk@=+q_)9t7tjcQegZ5dTqrXk z`6*&*^_cP=oNhAHwM*cZOopTrsP6iKhdoFcA*>#(c3K_~llDGn5a03$M z5x8++b4ixKr5zc1uF$Lq4GNyT#^>VTwPLQQtBlD7vO&*DJ5Nm7UHtDaNSjiNn$->N z-!iq1+L>j6CHzUe7^Fve$Gj`aaQLoO@UIo(2FSbyc-1N|UfVpDJ)*oL+ytDySLG@o zx(Jy^{usR&T6;W4)7QnW|1{IDz0ck*=|@faqZs+$BvS)A-j4_S&rLQ-{1MW)4^I;H zxX#aA$AOPB{lYl%r3|kMMMOWZazrGauA{5pgjNCEAn)kvsc|*fZQ9;{iKo1Qd)Kb| zQhlW<&Wn3ZQ_@k_X+Rva<}t26BY4WZ3KlU>ZzJ(ED{M$hq~go_8OVA6Tb*6_Yi2Ay ziMJ#Bg-Jh=s=4OJ`X}KA`U6m0_-OVGDzc}D3YL8o#zNyxO+Q0lyPal7V>OAAnd$H^ zpc7Ef#BrUr0 zspTkicsll-+B_2-KJF4bfoD}^B^`c4*Ozmu8qskis!}lVd{qbj(`>KRuPMkAb8%o% znq)1A{zr+HkUPu+g#Dwlu!w~oYW?uX9I3fL& z;Jj20*}D>^M%eEMBjEqhs7Uz;;_|`(arei>WH@X^%A4vvtTo>0+ zi414bH^Z9na+>JKvR~HAz*AvGzA&6A84rbsZf$_tR>*ouI_mX#2AL^B_VDp_1Jng9 zpUEVOtuQGVwU5KWd_g3$E8lUsIw|e@TwCCUuRh_#6gew(0(Z{x!;ULzvKZEY-?SLZ zy<2v0xD>pEsEMNY67+F%I@>V&@O`-@W+AXU9N=Rn(KX{Bt=P?y@VcZpV*Kv}8okb1 zU_xi?piY~+v{g-gD z^GI$dOlrjir?N%5tbQ$ydUzVA{{%VK|5<7htmwD;EL56r`Q)$`yEnlYNFQ(+#-KoJ zN2shuDXTZ5qyb_(EJfe7##&sn%cJFAwOI@t{q}y!mbfx;YVPo zIh#$cNRoGpRTvR-v0yc_TX2lndOVYv{$ZK7@z5O$Q=wo4{q)T)B^|RU`EjlK8M5-f z(YE$9DSH{zhfjcHu>r)~n@YMqDP7i=uiqcRMDy1;4wCAvdXwmW20vyK=x*A{zhDF` zqpl?eUIARB5W}?L)*nMHBQ)y8v(MLhggG|6)(6f-vg}YDH^xix}Svj8j;irF+?<|uU;U&1LFXX@DN5MM{W?Y-2+;{AR&o;&JS*+ zxLf#8X~lFRiq?XSP?%+E`j;bILHq!k;a{t6co=^+bMNu{=Y>GFAfl{q!>OAXO$QW z9r66f8!EEzDwUhp6|CULRAQn|r30ncq#6RmU0;eBz&~L{L^6d_9Kh)6Dw9|cY9AFt zsSmOkhsF;rZmzwk_IWQzt;HbjZoINmosL_zeio?`bX>%gGwfm&7FvA;dXrWqT=h>c z12~ybwhXP6?lF8m=UcNX`B0ST_{cX9G>Uz1KE`MXX&gJQTR}lrWOm9i1Kt*7VXdP;TasH-HMO@Ye zo6OkS_9TDL8dgL%^P;xFJ7Yo$zubJ(bVmty^M5roFesD& zk94bd43jTx^2aBiSxg>$>dbo^`O#TAjt>#?GU+9DtAh+DFriQX z`zB1rL?zhNB??QpZ<%*#BSRS+<1X2=($+(pS~q}|Fpk`) zR2%Q>hrZlE)g!gP=MNh>e{a%K16JrGJYKw8Z+4iUWu?GJl|*A){`wO@Y=M>Fr%~eA zI>qxhDh9Juw~8!lJXMOq74Idn47T=hGkHu*t*RWA?b5#5bAv{G#&|Y$A4Yha0Xsz7NpuWDJ8>6@${@$;k zqviPfu^*EFVb#jiD8W#Vtsf#c@*QrFPy_CnRyx3wNooL>37_qNXczK1I21Z#4|R8C zc|5Nm4r9uG1RuC9ozJ`ogt^77o8LK(RazSW{C! z)iNl9X(y9dPdZ0?K)aeUbK!Bg$5DImch5l;LHK2%J(*0S`J+32`rqIFKYz%7U}AEN zEovdRSJz;v<~Ao(JQqi3PDN`RSU$2=Yh6vw1z-L8E{2Tg*U#l%IkU$6R{Z~2fk6^^ zvv_!APY%TGk|Gx4g-M3s-DP$Ls^zM+ps->(-Azgz@4&Nfp6>k-_+Ldr705UcV!$gG_^r-qoIf71!!Qwg-|4!ig_<5AJyJ+tSWid5}U=!i9}8| zlJbUEh7I*5MdHRf4JDd#J*JN^;y(QSjw?hFumft~FLnb{@~oV&>a(8Qua$b9SX`WJ z#L#d9h}yNex=$hLwBE3E#nP^11aFWK!%bn11%HL|yC9itc-7?~qfS;MC;`YJ<)DXL zCt=5OCEllTv6aDZL_t-UfRaxi)RRp>X2NxYPDTSHw6>RT;SJy*Ba&8NyPh-oJQINx z?VE{?yV6?ViDU{cL9CSufft+H&mgJ%Q>Og;$Nv5Bo|p{RLgU1&Ukys`WLTPv5$7uK zCRfU71od0K+-#;Z;dqS?mmQO&ye`rF*Ty}AtU={}?~bjSE$|QAp-UTqNWs5=PThU9GhqK2@=}|_~3}>9;}b5 zeFJO`>^y$d?*-)A$A^ zzZ>ukwJL6$KkvK#n#E-Ljx>yJ5;x{wwjr2^b7jPjoxt>A16-7dzK%+ny}H%%P-ozu zngTq4AXA*Or{etgae!d1To5}mSbjU#iZG1})`^J{c6A5S!YA^yBj~oGTHiCl`osNX&YQ zJ{$W)_Z)QRx;A?sTXl6vJ>OJ<16Mgt5caeu5c_!O!wN-6yt8DTvp4^k)o_j>$N zEvPV9nwh=GV*vjp?}?>KR#<4UHMU2lSswq>OhZ$lz>j26Ms$o=L?Q~<~7WB4=dr;fSEJ#5C+-nFibq=i`&PzM08&TC#Ld2=orYFX|14ZF$M|p{KdK{H0g&rxs^lf}_lMUS+$)rbq}Xa5 zgB&k`l>eG0Bj-uX{YoGM1jGCz1;_4eYnrJ2Uo&Gn)`OCG~EZz$s?6g39T@=NYzXuj^&2A zgN1Bo?)n#I&it7bh2oQG_%hpz=bRv1ST%QG-`GTOeGPWVCOxKAStM&fhD7q{-6jj=MSsK01d9^ z4>C7Za{;qDFV6yw-#6}?Q&nTo*ugX&hYzu#xdtp5!k0F}!TXMqY~27!+PK++KV7}e#B?CJCpH6N9e9jokmg-jMAEWHVVZgK! zQYN2~;QL;RbMp!QcJP#Uo97+XXYNf~cyM&=h~bFX`J`X__pS7wSQN1@Ul1#jNzjn} zSZ($RMWp-^u{_37S=>mq9i>i&hSRgR_Q8g!NUUZ>(Pb)hYm+ahk!HhJ7bp`qgRAqW zw4E?tlG;M>YY8(^Ju&lNsB;+}5J418$3fl7e>qW7wX}g9&ejb@)}VmyHVo5mJ+Gh+ zY{5!`@xXiOZ{ADqFbZWCOIRjJCj*FeUcqza0Qxh$+FO!w7#`Z7dHMpC%9)9K--VLig33(+=tBLS%axWrTCEi zT&vjidCRMVHSkHmGtO*}8L1urXD&kSmp#_NFAKue^1dIETn##5NbmAH|5~T{yzU0fTI>k=}(U>zs-8~ zvmwa+XY%$)xJ9(WpK)}aV70Rvo0{?j38-K7Y*bE@1mt&&5Id-y`Wk;!UW0zA2J#Mqf*p%u>O!s;1yycmFdt9dny zo8k}0>MSatCND6H*0MJT#yzFydJd=Aobx&sPx3h=;iJJ;p^&LstK@&5E*hrl(&!AX zs46ZUt}v|Q5vDn{74xN9zKr8rltalZ$eo%ZkuQ&5sHa~36PuM`wN-lm_TO(pPJr4C zhpE~T^%tg8d5Np(n-qRuM-<-WN>37&g!_=%S<}9kJMr`fd-RHT60_QZ91}MLHZ>=+ zG1_Sm{#aKd1Z!pg^^YnMx4H(QwNnHmg|8tRi|>6~5ivqgqOyGU@^I#p+tu{v->f6J zLP3$)Wb6?*c73{UMXZR$wL}kfajg+WkljS;|J4|IdKS@J zFQn`%3e~y)z1752nA)jf5NTESBhKN{wtDhzfDu@TTjxe-GPoOulSR9At_XJhMs zyq_K2zj6W~if6Wx9Iy6z*rd@oFkv!<_goA3$69ZdF4EeOUWsQ?Mw#y1#dDGNiM)!r zg58Cca&YMtprgj2nU8uMVT>V&g*2}tUE;N)>#(DtS_*k`4?Fhkn}Hdjw$be7RG9Yo1;3`%bA=Elm20`=v81AOHO3gQj2}(tH`zCdQ!CyQ$j= zv1+dpWE$PU*iuv3QGV%@a)&`hDTWkQ%Cq*RNSw7Z zOa@_Pr}DtOMo-B zK3#&hVC(bu6PCU@O=Q6zRH!JfpX(L*=SdCa(+OHk{GR3LiX^q$--TRf3TJmri5p2? zSK@Gt^&3f-iInGNuAZ1EZ4fsWHR7iI_bZ7aAie3z9)0rds{kzH^vLS9h-AtPp{8KI z@UjR`jm8byD`7Goq#U2>*a4YR9CEx~Mn)CbOQIJ2rDodu4Q>*t8ofWc#TRIt^lvQl zq)bJ6OoGK(<5$FEhS^h7t^}Z8rZ33R7amn05;>|Q84lAh{3LvdRL%uIkXc;G9m4PP zk#OM+-{oJq67bUYY4?Fa#;wC?mks!x@U%YXSwM=kq|$kar^VC-4TEu*vRZ0=_)Q?t z+?gmY?);V-JD@~;k68%1bnwo5tKUOB%5Q;6cbu0cEz@o$k7|>+^=T1Qo;JCY`H=Ph zTZKjBFzYL23d#WyN3^-bRw0~tDlRM~eq?O*Df7c@ISpnNAL!LtiUzQg0pSp);tG;ipiwDJ@$ zdyc&h*6uW0g$7p2L$BX&0poGi^#sz`%GJBu*Xuv1R)UmNq1&a|NpqK6c{ALX81?tR zuk67^59u&fY`Ry-Mix8EswoJ#YE{-cuj=<#c#;W z<3M6Y9V}rm3;aE4G9*w^1zS(%G~)9Plg#ow&+(rycLz1@!a?})eh0ue|Eg={~Whs{Rku|Gch7FthX*y}?o7nbwL7H>6o7D%RcMGEkjPC8F?GXXRyL30z zC(g%I)xt2r$(Z3&9$;1m&-}m5n0#wqen6APnW=RVvw82FMxP*L^E$|tZKstvJ5=}S zrLLD7Gog(yU}e$1d+N1+F+mJExmXd}#PWBy?O4RN)S_b-xk3{RxrcXB19{lL(Xh6# zAw5q+Wn03>O`SYiHNp0Ef1`4XDu!KzKPdhuUbL2-(J z*WITSe%{(g>*4s$I9Y&rh&kK7j-HK@ueKgv$ga**! z4|PQ1pS=t_gT~0|-HJ2BiR*B5x+}7WF$}%ms*raH;?GBJ-F>7c!((EybucsHV?h7GAP*&2G~|UOY9}nq3>fAB?8J0HI_$=WNFS9FLnLJXSi1@3n(}RHmOFZ$1 zm_d(Bh`BC-PF%+6ab^Btk{NI~_AC1ws+)PYn=4Y&KUSp4KaT~C= z=C?V9v^PGt<8@M6WW;K3y$zW6^SVA({%{7S@8-VMLZb8<{glyJfWDBO+F-bIdgB0a zpfzhZ3!~Q(ph%hFqBSc~+TNw=<*)w+?{CHFzeytx>AJ|g@X)o0p|UYvb4byVFhF$m zuhlxnZ$Q8NiL8l<$p#`ARXPj6adLa~^-XIiK6?$Lf|2{|U_uTeyX;y)=M zVHKe{In+$?;${y2QwR^KB)gjuI~K)0ss?-kLZ$JVaM$_J%_$5f?v+L*bS6qV2WPQ{ zkC|A>oZ2VkD^$|{{Py8Pl6s5cq8P0wV}|pRqO&^Lr9lofIX{ZGGPV1%hC2g$1M zZEo&_{&`0X{h_&GM(eURZXD!^f6wqZ6=uQ}L95m=jDiuzSQsB%lwkg$(&RmN{&wj- zzfBa0BY2dk9sqjfsu`B)TR6RTw}cHC*c-7k){|*G8+*HIpbU6gVWxcy6u>%I_myt% zg1i*n%iP5qa;LjWF)eIX2<+C19m$~js>s482sppWLZxeFpc`6DyOjkqqHHl|IBrW> zv}i~?0a2Ky5SU+3d}1q%GJSxe(u3ieNERSmuQc3T)l?F^*Ui%Dueij)|A6Wd+vDXh z5RFQq_*}al>5EvY%?5`;1h?w@3H20(V%lP!!d((JCSBt17X)cEu?XL+09sA*dq- zl01B#6nYE1w<@7OIC<7^eL>E<(fD%EIMOpV?4Q>~+DF~|KiTvBe{H`04P(P`m;BqM zRddzt=wS`xM`Cvt#(faxuVFTTO0L-F zjx!73EJfRCB)w*g54N~Teu#9L?Bk~H&~13W`>270GvPe@Q;hqE(CHwm#~?!poHp{DxG`#r0D2coz+lE^ zl$fRra*@Z?rwWpZ*c8$gQkVlP_X8-7TJ5ljMH9&U7@*8~mrH7u$y^<@XMX^$XBVHL zFgf4j(#U-Rgki>UF1%<|&2Nxr&XAN5X&^1UOx8tK;m+@hUs#^Rz7z-=2T=$COl$l#&z;!%+CO)yd|ag3o2A@V(<2RFQNghVXvBSqVKe=;EsTE z(X*h+fia3Jinz5Ed05W=Y|1lz099u}9Eq9KH=COh=$Svku?Ot~c$tN9FDz=Z8zE)X zaCFZZ7$P#NX@b7k~>Fa^(UsSpV+I1>!bZI9{qn*opn^zUE8f`HocMVO_yv!1td2qNGm0sA_z!# zhjgdXp|pS?jdUX*N=cV=NQ1;#+xI)~`OaUCXAH-1Jo;m;b7|9t`t9x1Q{Uv~#>xHXy*)wHZaX%csmd?b!BWmL4D2xip4wqIODT+YpbT*xs z16#E+!9^cu_X(maYlogl9v4N!mXFp?89zVz_>`Ua(xjpx35)F|E|i%LK$`vHeUN+kf+ul=n-1~F4StU2Zz{Lw>eo}B3tZUFI_!DSg6;-ErU%A!coyvvWN~!lXGWWpJ$lBImyh8s7sVprM7r*cz=?;1( zhk|Z&*m<~DzlildAVOnv#p9I+xF9CLEJf2#^adXsm2L=~EJ9=xaC`nKo&iT_wdjqt zrDQ#1lTPI4TO8Bw)}gup%5Kj;7{RBSbRyLtHk6xyZMz7zy__Mj#GG$1wDox& z`xEDJx?r8LW!UBeic7Mf?$&gIn`I($lW)ejq^c?FX@1y`=g5ZO8$b(W$I9^7;A>=h zmI}%BuXfoQ2S#fEg4EF6!hF%_^Z~i`AQ&%ML^;VB6!S3zb$hL{uY`ldT;9RoU7L(a z^aNv-@z(~AA#1dCuEG@eBY+O74{~R0H-juiF!fK6CAytfnMOgcgV&igtu<1zCm5EV z2X<;3i=9ecnk$c7G*bM&?1G&qxL8LZ4tS?^zii5C-{T*c0M9Z0T4+%yQc6Vv|5020 z0uP96%rW(Z0l!rtKq(ZvfNG>r()_pn0+6in{6xHa?F=0{H@to7#5{s!Y3FT$e6D z(pLv5zG|>5j6(=e(OlV_z;lUfz6?j~V+gU3K#3Z-i-Hi2jOsNF!^E$j5};N+1EFcp zu3fNeC@qd11nGeZq~l{O@EC)xQF)%g(=uh7d{@qJDGPO1DWQ`o0M#>AF=L1n~;DTe?V5*wwdA%hu*{2MT*_`(Gp~_lsz6+ zC{If8iq}PLT4lHd(v$Gi4WwFpdAiLcgM$8o0YvgtE2MW#JZ=I_z7T0|w|}XSSlXdV zP-Pe#%81)Tu6V}|&ha4M=5Sg3sqNY*=*)jQg@4>2WKXQG96|OvxZ97>gZ^t^UV$6y zH5HzB15H*Fh(zP1N&+}&mJx_)Q8I#F? ztJ#uCkpHX?Ar9t_4u5iDVRLK>^-TdGQ3JngMnr~mwFuSBHaCpG!sY2ffn9YExxF!k z5sJ%zm7&+XZf3vA3bSGL-40$29iCSr@W)Y zQja?T`QRGOTn9l(=T6byp<09h^7UZO+ zEZ?UjE4|qq+hTaA`v@z|yb(HrhaqCKh0j&sJm|aGZeL|Dhk;6m+K}Q09viM-xXvPA zT_#AKD}W4@NdQ(G^2YNcFDC?x$C{IkO#%GJK< zxA~Y2(CJhx`yz9Yo-?p@__h6g7kG=+KJeDjH4yejvL2;EFFsI_ucA?h?9vp(9GCce z6HaZ&8pWHgddx;?#Cc10x3VMNmlqeSi|$s*WahKRz|_T|?akxzgx#WbHA$KMOHT+3 zv64-_5AKv+;d8(1#(!R$~|R=L6W9jGii+C?&n|HJFtW^aSoUvkZ#5hGSrQ ze+Du)kEGvrf5FVgK_J2t!9KQuu@&q|Y*Zo}X?&Enj3{l-q2vX~i;rLntbF?#jlL{D z!eG2byl((`ZG0y5tK$bRhx^m}4}q>;3LcL0uZBV?jEa#JxSMiF;cGnt@$JYqwU)~m zGTo1_D`%brTyg`hLkCt^XCTTk59amicP{Db@>WP>!gF1%SU!9YjV`9gLpB1`Qwh#W|!_} z`56zw?-9KRI8^ox3k;6d?hvA~FU;OnqJ?-Lo}pn8pB zu<6(iLFxC;4DgBnNrSEvTXa0wDXO35|CiwNwML)Zzqjr#6k+_6m-0NAqROTFH~%pA z4ltc)gEHQ9_kMdRGhr~W%@3?q@%Q^oV#+kC4;Y}22aue7x-gK!+yvGaYa7be<}M{3iI!c=!r6<2tCi4=LvT z=Y3dg4{4t$e=g@7b-UP^_mB`gh0`I(FpclCHp$wU_;Qih_cz zpJaVc(41f0gR5g%p%c^4My{`o4e28{3=%s>G_oXC^0VQNJ8FqFzVEAC`?FHOM%xMq z??Ss{jvq(f$-uLj_O=$={9nCz_--Gmtl2+)bq6Hu?c8r(z6<=e%z`!+t)@8Ph093p zFkugB-Q?;-SjAp;0a0v~&A7yqXzj^Q$X=&Cfq_TV`fS7z_fpf@ZZ*Mo zFN{1L>mDv0L{suJX~ou)oVE}%cB8&+TB2$}M%)1yh5fTkeTo8>{r9b3d9&-1BkQga zpX;G_Mb~y|H%#&AZ%j zcj*n80&nCjLkHUf(T$5R0e$NjFmgrA!J#!2x6M=Y>tyLnt7qkFWZ+sglMDEZRqg9vg`Uw!|MR}~Quilahyxu3 zzTbDQRfJzlW*jKZY#$;j#5S|S8h6C@h{*5iB2boZRZdd%0xp+N6`LO0cKIpO2IJ+M zV^qz+C`+eIzH{39SUxOPdnEMNy@`l|i$;4tWvp4YxySk(i$oFXFxh$a16}LBnX;ZPs=1XY~iq zvwwq6H8ZfgnYBrOuww|NQD{}a=A}@My|e^VYM{&7??;nEv2%B#x~IFmG=JC3<{QDa zMCZhR@M+s0oFIl-*@VGaz`hl@NV?-EvGbWqYX2;zE-{K-|ETsf{b$h_;jWE?$=e9g zc-pZNdI=xU5h91z1t)s#L_%(j-MH@5Z+Kt7SB6MDd9o1B7L)|(*>Y+pJ>5O4GjTcF znHn`rQMJF#0vyIB1}LfMu)@xI2f*1oGYZDBzr*KRHSd4Z?0?rq#NSH?Z}byv|AqN3 zl$lta1M}P~;OA<+=>9Db78n*9?nV;JM6E!l;lphd((M%=?xer%Nqqh_Df# zWE3^!>m9oQ9n$Hj*V4=P=i>Cm{Am;L7NimPR#|pEX+U-+VI2%2Lbl)SqMV~su>2Hl z!Xzu}b4GXWy?pcaR?TxS)eeVIJ;(wpwIwl89*B6H zsBEYPc~cs=PG|)_BsE^h;yW9ets_mFcW4V$6fYm?<#IIS_$V$)bp>9yJBl%@K9iQ3 z`_I+q%22f1@5u()M%f+R`*gWHXh@;_ew$I+gvJjh-km_ACieXm&CwfBdB!PDnRb|Y z2=+2fz!YlV@mnLnEV8F$QU(1pUVT0T_xZ-R3@e@@U}fQ)Z8`JOpH9b<@*u4wyc0~4 zvW$H3Efpsy!cKbbwZ23SfxZW^7Z7z-p$mux zhU=y5X0RDW{Tsl%eLOy0m&x5&=ypMqGP<)eF*Fl(wy}DbY!z&)gpX>UnAb1=2^d^Dl|X{=5LJk*MmaN^K?8$%7fGYu|v)v&Vm|D>gtEt-CG*_ zr9_Gp!tBBm<5Yr{q^hMgi)UDTGCAQZYcCtG{G(R)aZZ9;;+-FtXyTG#Jf%Wyp*b~{ zXc6#8cSMi0AUYqFb@>kQs2bEpO2jm%_tC!peIgK{H#enR&dq>Pg_4vJyg6I{=JMe) zJ*0duhBj9pW*41eSQ6SHPb}3Qgc6SCg-z6FJz~o_r96UI zqxhyfWHTW?LXvzY$zEcQFlJXW@a|fDW?Ix{5K>$OSzJ13*~C`+#hk zmXlwR>Bi|UzgPTBpVL(^=FjrBj`K)1$#yg z@d4fdLbj5vY@+L>I-!!>xXlz2`C%C`Gv2KFQ0`|TjwFu67m}TFX(fe_rPQB)Ul`cd zc&c?*JRiqFV~s(X>T7QnSnLeVh*u}Mx*ON+yZq|%EC)zqg<&YYohP>%Kc}C0JyL{& zpUFe!0LN)0Y{}!6P-HVVGp6hBPTD_Nkq?d0;_ZalX&q|<3>wsK-@MnG{eh3f=4fqT z8;F7mS;pRbU&wmUpOXqQepq$mAZ*FU@sUf)2>b!ona>`gP>*e417^&GR^;3njffg> zpSOr~)vbZTL=}>cZ{Z}E4y=`EflLl`oDv!N15E)7{sB9EI4TQH+X4GPPt|&;NIm9> zUHf!i{0Th{sryZM%oZzwZA30q_&1@E|Bb6F(kO6#Pi3VHyuyvXNBYC{pIW_s5V{;b zoWaG?LFn0|4lMfuPU`)5a`asIzO~7bf?0B_^MrxOofzchq21Z_%ueBGc&C%@D;_%9{%?5b7U|D_$_y`%>`vDc(CDKNFA}BK& zyD#>d^t}7Y0i2EEF_3u-ZZc!FC26(+b(5w^nRM-!&Uf5BP2yU!Z&}ks%mT1KfCv4| zvxQENu82>8mbG?J@B33XW~y)=SOc&w zP_y>h1&OoEq!s{Cw&tV7vqj`G%@b}i^o3Xs_m6F^`5;a|M|vi#l3;EtIac~@;n|HS z?Ir5Q_g-)|bDG^;C2>PBIa1vbR-!1xfh4oyO7@>()~ZBH?fNkUC_+sfkCVb^3TWGc zychawV)9c`Z!LNy*s>aS;`eXTV(CH~;PlYUR^havyXQvppUIGRD51;t1l%K2r~)Z> zY0#sszK$9vO=*ZjH(ES~6q7O+DX-Qt1skc3miXz<=kkdzi>C5Y>v_de-x_it1X;y* zL|Cri>ekJN9qj{(-(zb#!GnlQN{&!mw4S#pPY(4foo_4F7N(wR0l-noe{x&bumHzr zxKbtk=2oTB(QnKdaYM0Hw=9sC_~+nuaA%cOIUqX>a7Yyx-;jmY6K?+V=a;gB%ey#x zO1!K>9-D~UtubYho)I-E0_e>qCTR07F+?J-3Lp1y>5%$$q_XDr4Z0at<)?y^yl0|v z2Uy2NzkrrK!-~&)?@Xh3>Kr0i<>iPY!9+*J>9*+W{ss8QX}k>ofQa`)WJfSt6IjX$r}GPZx~Z2QsGFe4JVB5K*FK&ic<@j&@zWew z$JlD~eX~pYbLl6zM1^X>Zs<1W_~emkXNWoxjuAcdwQ#lJYnN~exB+EV*u#dZH+}%blbp(HCV;=+uky^H->z)^q#zQNc41 z7RAqtVvTHj`$8MoZROj*TRO7Swee!fE8qmr$KlPh2F@=4*kv)F@uY62`<(bHexY@B zUTNAZ3J2+YUuQqf_rC(7Q89qvX!(No_r7o7S5t7VYl6Jz1hN(*SotIr(JsHw-Tjx? zb){$~#7`~;URnx0RLBDJ1X!U_2ZE*Ag5kyb0q-+PeN0|cVRq!B;2m>7Xk7igTZR(t zQCzjzcM7rNEVp0Q?{t`0k?+PA$VklRhUm~Y7}BH;7FeSKEy*k8ot?R`dNV)Z>95v? zcYP7vcxO6mSAVLkXm|WVr*u4=uF9JqWs9Pv#g2RAX3MEHj%;?bc4r)vsv26-1{d;! ziWnJ5hb08&D7{9II&o8g`WToF46{UfZ^Bb{n*!vrFx1L_PGXkLc6AbMZmrUfW(cVP z1<+gG2_ODZv^cto0CckR4~DE1CaId8hI$C{RYeB686wOgqrUi8O&Sw!%wU`@-zCBi zQC5jIa&xBQ_@&x$giuir9?=e<+T)Hs-k`;-B!Goka@B3{5Hbcb|7hn9v=d+-o^;ve z=yv_{%cMpKd`2xzmfVmvT-3KM>ZU)n!WD_zh=KtA{3?M zw0Px!=uWTh1>nSPzwM00eQw0j{K4pdjSrSO+(rU|QgfHLH-BeUu}^drD~WtQ0g)0; z9F(z%Z}Ab`{xh95NV8#!mw-LE{5p9L05--v(50)L)O$^J<-?8OSdu2|rdsOCO|Ksx zuA!m7hXiFE@VP{#xHav9BQTGo4Jk+OvnodtQSu>p2fgr}P&c-ib%I;wU}Dijg%bF* z=84)A-S2y09PHqGXsD~VnRvP^(nnTaz8aDd-uLv8A)jYJ#12d6JOf7<^8*y;%+-~! zI7(1|5xNP?%QJU6ingD{-(b_ZsK4UB5FULQ9`T<~0Z~1Z@qaG}VT|;Fz(_#5_ zI%Sf6lgAt(%{Eb-`EUk|hI0|I`ROZqqai8!2H=*Pmr?Eoeuqg>vZfo5Th~o%tCJwr zu6YuwgU4(J()rIioKyWz;};PgS~Ub;XTsx+ZozNY6SwlTzSf7b@)L`L_UtLVC$8(l!Vzn+2Pl`_&m_h>Q&hY8L(O1acmYg|u_W8bP{CNGsgV1iG(w0vdC!r<6klr=S2jMbp!g zJi)b0uGx3V3)!6-A2a$j1>O6y(^T;im>{YDjQC<0T;Y^ae%)Z4%oz-Dd>zfxxpPmU z6toPeWF^so&jRuUWf2a(!iJ{RSW$d6cgp&{^oPAC1uau=r)J^*AA%|dy_^t=R^fXmsKzp z*>!%DUFHdyfh>n~7#i{^>H#wkI1S$lp+ftY*cisaX6a{RF`dV_w0|crC4R6FkLuDN z8$eaEIobTaTiS$N4P5vgrUgr|E>vt5YyqQG zS34&(PeV0B(2lgz5axrX{ikbn;=!mGiNcO#ZpUTB8kxQK_{_WQTYUcWAFb?WwlvH4cO{URCU3)Pn3VDF#bIr|L?LW-l*I?ST;+M2upgJAr$AJ~}oiM8~ z5c(Q+%tXqvf6NkeNOrL=)5sKrKtXLuJ3V)WNof-n--{Y6N?wmHs!QaE&sCSjjpsn~ zl_rwe+VGfCsRO=YW+XeN(I{$Z;Kxo;=PM7fPEisIgLnv7+)2S`QKL0hz157XB0WEV zzn!KHD7F57V!VVP*XB?h4(CUk8}uYiEPN1_GSCRc4>4O$HU{C#wuLHR{E%&4=R&p5 z0^;S@#m?#7A^a ze1gYz3tZX`ZCp+{%s3|FvT=8>Uo+1svL}GmShg>`b56DE=D{5Ktig4VFtYEV6u>%5 zz$QnDv0F6}%KT33&76fF@F|i}d}PNh0teHYt74y#B-Yk-&e2BNjR45(Z7WrDs>wo& z-5?G(>;cfpb5IutJgRv3(6A3vQtJz*Fg)1(7wNEMiVY@8i_qdRH6%gwZ-L?feib&! zx5IQ7q!=}<4_3Wew3SxKWPSfvw~i(Q#;BeLAV%e4clQ0r{6|iIjQSZ`0a98 z|Ndjj(?>Y##=74UkSptWe+(qRCXMA+ks5=0`L`MrqgM}J1Z<=*>r3T|!1#QxPTJ?N zf4@Fd?l=a}A(c{ejB$BzJcUc6+?bCB@9x6yTr=;!u~@w2I9&5#L31GLR#`LQgOo|F zK<2u-dg|fdKpR1AqS0uac?;%I`@cP2TrvC9@)A3t&-b+)Dv}fm>%ziYK

po43|W8{h?aWEV9*Pn+go|Rg*J|-{n9JwZ*Un=a0_HB zQC3)utDiw|I%ecDo!@*;%TI9XCr6hvRScQ?#U1i6K+th+NxZp0*|^0n@@`}QQZP2f zTGDzIs&M81bd~}=WrpCHMeFPY-jG$dmc}aj+pDp> z_Q*O`rPPaweF<&fo_8keSiH-L6lFpA7-i8=0qL|VZ#r;X2zV!6pSZB+r+efM9=E8x z6Oapp<|{orpftKIbR)fLV)-)ln!WBn$BC~LtPi5ezplgA)L9alpsXSM@afl=vv;x{ zkH=)h0)JvNF6CVeQI;-nc*HV`&`x3PygU(4t9< za!D!~oflBWjU0*oKqt{N-?|ISlJ`*OJB5fjly1q%8b5NDoGYo?`S~iY$><3;P=;# zIhFUwq{nc2P(;5G20=38cDL|&TgIp>o1mk;&(*{~xDa7dDqu}IVqP|mF&*0GhgmQG z3in!admI5H#h~Cq<0oBJ{@U(LV5S}XJ?MeV=M<>k5T%Fg(lHtlDJ$WhuA4bWHWk*}BqDew!MleY9ku9C}`KCMMMu2=) zJYkKx?-ANE87N<~f>Mp;G=ea>Ia$pBTSnCu0*=FD{@{r zUun;xFhnI@T6lLU5LVJGAG)3RZF}A?vb7!eFQ*8JYCd8T_nP=08UUsbF{zG6nijjC zAQbTbR3|2btHy*0W`RXQ@|^{Nf}S*2x(ytH3o!#!MI7btJ3_MI;m#R*X$BHI)uA?zjx|>ZBD8w0n`41lAr+;i0^#_4Oe2UrfPpY8PsNrp z161v(2|{}ZYVL|_Q}k<0)6h9!V%*ee{0r;d1zOG9g^vNRCy;Lx;jAbr9nH=!B%pG$ zdAqcXR7f4vU_P0mK^dIVj~$ zc}4pPSAaW-J%aX}3TrG0GJV~!xTJ{wDbMax(0BO;9Ks}*93lPUT$Ye~ZH9__94D@d zHkEA^(;NHCRd>(5{2s&gF~~KB4*%)|v0!YL-ilXmzab?y)fzlGR}Uj@rs^>b0!f!3 zxs+nV!f5EyEjydt0=d2|aq}V;#pPVHxGDo~h(7d}OGFZdXM^Guh>T2}ju(Oi`s#ea z5SVyN^09jUS(#)4Bzdp++r?w29;{^u@-&1~i^Lxu?#Ycu`^Eh@Fuk%29d4f|0f8LF z-A6II5y@bQqt9c|z!y)rNN_<2bR=o^8L!YSjC*Rv;UXGBU%eaD<1XsZH5SfV*JffW zcx>h8ya$X^cQ;T_XT2Mt2e#sKa)j@rn3%7mwH}vh%f)l3+sVC}UzUwLY{|nMQUs5O z|EAmjT`(v>fs6kOYR9WesmnB6B19qKKIspCW5ulW4ovJ2ijG(rnz~(#G8(}l=uMd( zD-lgxd*-qI0Ug}<;e>P|4_!_rg-~$JP_$d0;G|J3)30=Qy%%~zPZK7*yAY)0w{O(B z=8=giUMUW9-@G@qUX{(eO-Zsk2MmK(r({@Pl5n*tgNAeQ2D^MsXln0+8{&>GlRc&| zX_K}Z59XAvN%U0Sw96FdBDeF!stQOHxy)Xare zXPVe4Xnjv1Jq$epGaEbJ`FZG`KNGNVv?Lny5rbU$K{2rSj#s*!cU^gTnZ={tfs;Rg z`rriYH~4Z_2^;S2bUTPfWi`?5 z75q$^uR03(c>|Wg-eDzskERS(+fq5j>&=hb$ACu%-na>PWeeY6ycZvI7iMDfM`Sz| zFe1=m$_EQS$fg#vj1tQ{;FLl7NdE)6`JZ1VsKAh+xlxuATIU=bz#(YEH)4DWs;@zi z%GbV~!&Ts07TB0g$#!$~gwz>>#91kTgKp`307BSRy2XUJ0MciZz}?Cpcm>LUsa*RS zzapKUi{OA+%w1NcWEIyrY;#NXl!`Fs;i> z3QRlonN;!IKHM%a{L@TPM?lFLhkra_%qGD9*wq{bcZi|U;UgY3bcec5f}$h@Kd{(Y zFHMW0%&>kk>9jy{Ih@L0$wK0kF=WF56dSSZ*CQKdoDV^Op(8WOtf8!zx1`Yg31g*_ z$k`RY;>KOFWE(I8V@jbU5g8Om2pDa>fM%ANewe$HZFc~!BW@Zg!n{<61zyLicSc^H zR727unJDs&hgoDBxsS;U9I@G1FV_w|=(`!Q)5 zldRpZM>KNZE}^>n#s`(~rt`-zhpdfbCY_>V6aJjY>!X$62>UC@Bb$}|hqew-zXhbr zJ+~V7z%(^dV(zVL@gPc|84|Emltgv1x;xpQR#%)j6@}cB{`BZ!K(D=x8u1$ zSf@KY6eMtgJF%p0-c8@oPP})GpIuS=sq^JyLy4W0?J3vv?%uTz@RXF4NwO2KU&ZFh zp%dnQ9i=ffAg0pP$@3lWB_N!Vm zw%@2>l%TAk{bTn8K8KC>oC;Kd&Y#H)jZ6vB7Fr)*jt^{dgtGIFJKEWScp@6(VEQ_p(5>L=oP)qgITsh#3V8wtoesl(1W0GIaItgoU%doH zD3J;H^dc(;B)qGj(euTq1BBr-$TNBj%-Jz+pWq|>85X8pP)i<1AUA}zM=Y|&btoDy ziBL193VpuT@VJsyW0&qO^RoUU8T^-Og_N*0{!yJ8@IP30gJWRyQ)yjA%jYlSbt}4Y z?lwaQTYb?~1yJ*;x=z|HpbuE-Oat>$P5@Tavryl+ zp&DrkjcK7QUe{*Z!g2K9M5DOU^(1En|I?|`7X*g++Ra;Kb1l~}6ImSX%`Cvj+wF$$ zmkGne3bw~a3shr=<6YD}!QR$B9L+v-38;_e6$(S=HGx%Fho!lA(&^n~4Fr>SYD;>H zz*FBF@L;euf!yVDT*WOwZgALRE&w`0X@hqSkVN4LU@Oy*#0&s~cstfB!w*Jl->q-| z@LfOxBKfEam74ZLe^fIsPW3DTW&8A~=&%3a5HFaJb0H3&@~X!l7?+7_5D7hIVr0Yh z(zkf_cp<@HkzgUU6EfJ8@weeXX&QGFw`qev1df0Nb1bvX)X& zMM;oOD{euy-DLNOPY%FrnR4#2NEkZFbt;`_kL^V{obOh>;2y#25Fk{jo_>b=kEFTr=`}~a8GKhL}Kni{&(T( zxWE_x%rw~RvV7=um>u5vZxB^YbEHoC;3$qm`R-9qt@V#+DI*pfeb*b1ENIsKWiZKf zEtPC?I=wqn6Jy~R0jGthFTDXvHlIDSE&~3TQ;)&hXUO)ysWQ|j3y3CRyR$S3_dDRe zGi6V7l$`yw!b{`m1WGl^5cj^b7gl@i0>Dhno|o|AD3*4TBR$_rykP(;)+%U%myo30 zmk<+}#xfBCxHe{S(x!7_I58UoQeF)92_Y+-yOS1ML^f4jffH^n$IrJxB)56_ElqTX zb@?j-@0Sle7m#_hNm5^@0i}=)$S5BB)~^I$R;d>?^llXxzmZ&DPpg2bD`1qDac&fD zq_8^b+n$1Wf zuQRbE=2`=)3{97}qJtDr6H(ZaVANtfjiN=APp!zdzP)*TOU6Q~eH_EH1@AYZhpbF; zApv{y{RwC_c6A>JUrRF0o+^3%|2Ocz|AHdlU$ZbC4CML(v;M5fTx|HoB-jX^!|}lpX)z;wi*Dv zjaKFlQpW~F7>7=#NmYtkyrZBheQWPyT{>;gl5TQ9W~a)tU>5TXlheWOTvm3l+4)L? zg^&yzaDh@*-8%pP=t;Gr*seL|qI;}!9i_bD^0hj$KTEkxFyio#=A&;KI#Zd67%3!njeLett&zWQ4-U_~v*Fe#6@-DJ0mSMHn$2zV~oRww?!-_BM= z^7YQMyY4t+U!?+PLkrw*cl-;5rNxo|;G1CNRW8@AA?Dm6|FA!`y6&hkrD&$~`QX4~ zv^j4&?v?_D5day{-2B0;N$H2oC*0Tn}Xe6g>&A@xYQWo_0_ zR6L#_uN;TNN_WJrH-*I{OXoxklP(+KQdPk4|5|ke zuJq4E{KU}_FS=QJmpHK!fCY4EWm~0P5FlO{l|$i>MkpJsY12-|tCrhlLfineOAvp2 z>ob^BceqwPKx&KMB^#}q!sBmyt7e;@Z~eFQ%usi9hye^e4k%^wLy5oM_VB5)K5Yml zy8x>#F&0=MCKd(%yXX|-1){R8szj)u*VKUV9x?*@mWW~BYw~bsb$8=<21RN>zBsU1 zKSUUnUy1@>AUL<9LRAbMVrsE;l@HcpL znnxHT0s=E$DR(;L6D=`gKw9MI0D)-NM#zJxqqVZ~q`bUb^B-$7v}|b2QwCL#2nNX; z19Yi#{|CP$0DTbDvb7}By=wLk;vx|}@-%G0vM_t|@vcaW#)2C{N{yps z53fEC$sON3kZaZnq=d3wiBfL?gM~L4d-uJo;<*s_%m@pgE~F*kt*3^kUC1oxpP;r3 zV=NTwwKmiRSJf4TW1zmcJXsj1c`~m4t|NVJ_ysK4dXxQf)!Sohsm{xx z6e16|cE;?iZ@$CwWs)i)mhz?K!q_ItE#FRrIoIa*=S>(?Zv&@qrc`bIOkRX~K}DK@ z6Hb~KGhI)x&~`*;B3n0|iEL7@tB1$jWIvf12YF%;`# zKtY)q(K>S_x!z}dHiUtCl2lgv^}%V;);mrd#@Qn3Gw;2DNul=h=PJ=HB2{FZ))>GI zPkUx4y-eve2|xL^iHdg}E3#eOe|6ZF3CRrf;lN%LcEs>yZ8bU}2rq`d^X&$Faw8^l zQQ7#a->iuZ!4JK%Wak{l4q~`31b+;*{{iJ-$TzKF-9Cc~WG*TNDZO=8msXQ%3LM&G62^|Z1WhW~p*!;z8;($bt zjvI0ij;BDkTJ8*emrmB75oZA~7gJa)xH@ zyzYbS%iIW@F272;>3rWUKRj8L$lPmN<*&(oS(=Fz?JRs#Qy4gpD17w;Lfj=QKw+cd znj`l;nZo4-PncaF@k#5+HE9r!s&<&zcPdZs^L@Ja?jV#VPydtX$m3q#7^?i^2K&Z% z+XWI>+SQ+r>gK3P1I-4&RxY@;CgG`PGDIcpV3I3A`%i&5qFr91X@6Z*-wG~*R+j2I zT2qSk#B-1YXVpL@ZWsM=7dv_W;2n&JnLjuC-uCQRjfQ4tY|Trj3S197whC%Iy+b;= zg}6-6XCwlk;njb6R2MFv-3=W|>i!SB_`hGmA#<+#=Id?T$K)8o7)x9aBYuL0cE@Y2 z?hRQn?~qiF+sh3ymDC!oPC7q;rXVGNb5a+Z{gCBEk~hg4SX%$d>~iQ&Y(OjL+Wx(}%4vz$6=+!th>9!s~ms3y!jbKT>GwbIiONQVM0e4o)lhnCf^{z}98^w>c~ zZGpk%k@s%sTKlQYE5Z=<8E^rStxzKggdG)p4&9fwN}qBFj_C8CPo*hpaVR@(1PFq5 zik|m9L|HKs((~PF3Q^5cs0ywU^ngR%Lc~al0VKBnJ&n*Ox1f!BTk_lfs%Jh90p|693{FPx^Or|0o|Yc^4QulKaRAvC(kl7q4(D9F z(*Z``VEow{d+^(->p=H{?Oo2%D#za-L>{q2mx{8Mp(eqjlrNZhjMw+b2s@pX?C?03 zdGYe@L(QZMEr=16Oz`_u3)~m<{h9Zj2Vf;YAmhV)vVT0Y)Yw&Oa0xIdrsm0Rc}?VOJweT?Q| z#Z(w_YkxLuq*+4hx6mxYb&atwyv#^^)1CPwO=YaUGhLrHFVv;_YT0ox@nI0+9>&_`@9&8l8GUlKp0X&5Y!}&# zs1bh`v;5oSY?`o-ojmaHg3b;m%PEY!4OzVX{6cY`Qkxp{r2T4-gwnj@h)p zSvGwJxFSs?Dd}q`7+HqhR+x`rIza<(@O@B!05_seEAZagZ%~u0t}QI|XjanDy==?6 z^1MmcM(4a9Css~%3U=B`USID(`3R>-ZNG=#GkRLwp2Wk)~!5lb8Knt^{~J?NT2me z;y0o)p#GkMfwtNkW4;&tzR>DHkEWArP`XlfBa;$JwJrs&A@uT7U+(>dy{4skf))AX zXa>oJP5tK=_rD;^#$JB$4er@{kjvl-GQpg$^l{#n zHbR>}Ff%-mWIM==?0k5oRec8BNgi^H@R(XkhAyU{h9Eft$|ocxevtSu4);GV=2Fe2 z$QFhKJER41dn`rmHj7kQDZFaqIT?a(nII*O7RcwDR|@(ecco&E`j}#fK^IBRe*7|60p{j`t>k@tFUKWwOR)G4}F_aT-be*9&k zcTi_LmD~Bj;v!$yj^s8wm_vvRd+5#0_QsWXAC1s?ZRYJi5>k!Nhq`tW#``R5Nu@Pp zcf}t7xRcJI$1=s766sfFa;p9uwKKeE#E(J$R^H*Qdn_CdFFg;e-wb_NcGbnhjKkNu zw*HQb_8nsK3_}w7YfN&im9JCOIN5<-sNiU+Wlub<{@$t%=eX!4ffs46iR*?iZd7~gG3`wGut+zb|-tA%|$WqR=6gHhjLn=rDR|W~85KHAY!2(ijGs_xTxKB&* zw8vHdJ<))TQbkYG?1*XS)SCs1QNV)kmo(W05bKJOgrEs#7F~l;RQcbK9^fA^FKkD= z%A7E+J4AtOS&}GU0~cIimaIIzvey%wmQY6$_Cq#eRzX_8c67vC2~b%g#?8aFZ@Y05 zRQ!Y$3wGxYpTEcnLCc`7yr`52j2W%87{Ol(I*omxES;q6oLfpj64X(ps8Y@Vn#xa5 zsG?r!*uOmb*h)Ez119rj*B&gzuF@(S&tT)^9KUpz~HfsfPRI z`E3By*_AlZC;+0nsveaXI6v7AS+0SFyD8hWU||5VcnR+5w{PWT{Bh`M;|VP{l#rMx zq5FEen~y^NO5fou)O2DnJ|+H*pZ&rvsH$zV5(pa+|A(-*j*7DVqJ{xS7!ajV5a}+Z zLBgRCq&ua%qypJ`Fv(Mg5%!QW`9x-tEr2STYURq*MWC_92vl8olb?Joi5{r5QBQ#-&isZ&LNc zvWzWX^!!BGmuZUlgVQm`pvNh94;q#wq7` z&wlhnpE{X;!OvyuXok*1#fpa77iC}gH(mh%ZP2px)twk%!O4|f#`b9gaD-4{v^4T1g!fv>gKt?RZ-Ov%RS-iQI@aC z`tNO+{Mos{##pYb^@yCyYfqI$%eeihsb!6pMLC<>meTb4$i9PTH1}V6di&mPbzrC| z8@>64GncBYnI5d7aImQny6$wbx`IJsV0AwJG(n29TH-XnFQ^d}EY<(WXNjsn+Uf6< z59o-{-Aq{5FE;Sk;|2-p5V~E|t*zP*u;{GNq=`+TzyZ&zd-@gk;zsq`JROq1E`*x{ zvaPOsAn7*NuY6r{_Lm3F(W%Q!!@U>S^gf&>YmC5_O`26k(JZ{#e?uUk)}mYT$BRhv zd*Y{^qsKvM~SXL?Q}(?g5id2msMB3bm*)GK9G5yJw`cRqqhk@GsyK~atmw@ zEXN(2Ui+wbUYhy|;~uJ8RWq{diI}+?3HI!%kBpzxtKr@hp{DyV)S8v}^LWVCbIgvy7CdYTyi<~L!bg7u9+0~104ss<%nkpF$c0ZLN`IsfmkWT zU>~fy2-lbMdr461(H{iu5b78AUI^)ERz@$O%&zRdjPzwCk*3E(ny0pIYerEDmw^jp zZ^gx(!5ZH}{XR{_YsE_upEn@)>dD|#x0M{PtR^)@Oko*GFdLh3I)~qqDkqYm3%@&* z;@3-AKcw35P_ICi-0rN$iiHWysVv-vSBMkWj5+SVj9pYCzq&-PA)vv!24)5$job(HWusd5?J72 zBH)E}ChUWuIG|32E;xs`J27>|W4Gj?gDxGMZ8t12Pn!j7FO#QSNHe|Zg4 z>Je4X1nHZqihW;FlP<<)%7J-+G%D9^CCIJVQ*!yI8lKR*$GQju-#SuTMxKN#$>qNo zuqN*}@W1iSQ72;Mi9J5TyWA)Ou49MiJN5Kw3B>4k}Qcxh8i?6RI*cWqbLTp;BZ|?oMKRhUImNyo0T5s*K1e z4$bk%Ty=l-^&fAK)}`}yjP5sLG;7@p{|AtS=4CsOPGTta1cIJOZ!_Hc^u1{* zJ?h$}`$Y?xOl=ulC!yx{Z?G< zde6g0HJW~BuPKbKx$-IOW1`jnfIw?tcK;@~^j>eO8P zc85)e|9%>OzswX@*cL>G#o%M#fB864=E+)n`t|~Uy*z{gEL@PUsPaK}KzZG+psVu0 zR59&uZ~M!DApK-|nyl3yPp=V2Y*1s4riOL)5ld>6+E&R2c5qJpn>`PgGcVWtNmtUZ z#1xY(8!VMT(Okh_#XvzpVNuF7^Z;bRjzHHWZFC?Yon+5vnn$mIY3vjDQ+=s6n58OVdDSllG%mPRIlymB5xtr*{MTe;#43uVg8*}iE*Sl zG`bP+!`X>FepqJR`sLRXC@d5*x7SA@2I>6eji3a2e`*>qq#QfM3JJ)JVj$qUC0{{; zb7pR`ME9cW)tIQs=`*es1L8gwoZYRW3o>H86k+O1dcnUAp3(&$-8IP&a-z&4FDTjl z1B$Q{m^!HYl__%~YrR2~ja&B(c2PKU+$qp9bzYs4m*I|9=F-^**Db=zg(McT=jIA% z&_2}#0~%Sj5Gd$7$RWOQJC_2n6(m}()j`gPOcBB9y`$^?`~KT&zE?u;xO;-F${J_O zOlrK*{m*!#H^p(H;_ie0bPHeUp^p1ThvjPm9~o)&|7^kKqdWQQDhxaM6REyBU)6Xg zy}=ZW@0oaEpX1W%df~rb0LWMQZzn%A{D~8ppSp%pI3~D5%*0G{U(%f|&==NlgsXSt zFPj#DkKK0La_P7=F9{i6=;^JqW&HbA+Q&B9U`$_jqva9)28KVmm=XL7!?cfsUM7tu zhL{p59_+L>Z39WQX5qWb@C$CF&+S z*?rA5*dQqR{*KgS2(;L(d+mEBOS#7Y_Q(5ExhRxaTZW`(Q0d9eO(Iv*Z&Te@Sa@!p zPSWh7$k#9NiDg`u3sEy=w>{g67XEVPPRf5yW^LijIK2VvO}D%jXq;vI4AfA1WF!bL zM9Z)P5Gc~H;OZ9NCrcRr!F6909OZd`w;HBj z(+U9Zm!tX*a%cqJC5gdIv5!WUqac$p*stpfGeUOu`W5mm!1n>JuqU~j2Of|4GxR9l zq;Z(rQ!E{YeD8S&jAikSd+y6gify&n^+GiR<19D-=4R2^^yY0A$LCJK-!;F5q_dL` zNpKI5dwJnn!4&cQ5-}JS&^Uft*a#FvZ{7f+!{|U|tR|nTeo<^!zcj6}o-C#(! z)*5ElAYX~?{7mRp5RpjLe-v`r?){C$g2#0W@{w5*b&bIXTlu8`FP>G`(Q31^_OM>R zm+a9YQOENTG5maK8H5X(aNZPpi3v11ocJiu*;i$Hu;pTe1Lh%?+q;H&zR1=)g* zM|I0T`RPgIN$BZ^eD`DFKOO_jimTwI%iu{^9Of8<_4>_4Vyc>(F_y9AP>k-*`s#fQ zkjZ>OfCh{Iik}B-&$SGGEO73_;(&fnBvEq_apT|fIn)T_Zh}#T-p8uPH^`{zJup#P zAt--j@Nul0n(4yf9XaBta4f|eUSqzs`^t>c^f=glq0-0-a7}EzL|O#i!qnAHeJ*tzJ>aA16mclKRQ5fHmwjxsgA6 zY+f6xMIsyo$|S{6S`^acTPsAr3a2_l;`T%O3K-d=&Z8$o3e9EZNuwgM~)U~|7O%gF+&kXh}c3Fm6VKPQpiC`_Hp*-RO*P0vyd%_5&TE} ze63U?p52_vs28RtBk}p&!|Nw=(I@MM$&*iI`p?bE2J^Eck>uCSW__A$R}i0Y&IqQ* z`@uG*m=S<8{&YpC4xbFZI7=QVRMQPMx?QH4T(VXv2Tc;5T|TJyLJ&s@;TU}0u0~EY zi<_}ulfu!kMbfD>Ec4+6CTHKB-V@s7vrey$MAp!u5X4y6#8~JsHgj)*B-?Q}B|n06 zddw5c1$P1_b9G3i*ns}rK1mh}rwQ?4yEf+3nyG;@6amk-yQcN|dz2mIx%i2!zVd(i5*Q3v(TXk@_EP^Kh{5zxV-x z2@zUCoHi#xx2B(65Q$M-Mb9*@_~c}MlBGc~-p1Pebj`HN&9_72W*|bRr)JTKiU*Af zAv0&eW#?x;mVfgdg@yS02ZwC;f(V&ounlf1a+DqmhkhXwE6b%hK*JBN`4-m;6cTq9 z9>vXv5cLz%zGSKs@@{%FT&!|Amazwuc?{fbF=RlJw!h5~_vLK_g_Zf;K%6EJvSS~N zKSGu446e*Nx73M00_|IR<<)!6P1Nkfnn=XV7U66D9vD8Y!}(4!T0TtS(%*7^Uv>AD znMZLI(Zt})=;m&F4x=_~zYNF|_fVfF$HG{`So8H{|lYlXPux#l!DZJ=!S2x2UuxUb%L_bdf$~6T3^e7L6g}ri%o`_^ZeBvkyI3xvk@vC-*4GXM~@w4@FE7eiq;Sw>W*B?DrYd4kNWmtUJN?=JR?Rq9{g`>1x(BoK}Xo%vnG zAI&`iQgI_|ZgW8;5?8R-!-VU z40kd7A%C~o)zlwmSv*%`n<(hPy8At0dCfrLIS~b}{SoJHjfFxaubPNHpeht+s!g4!5{$Vx{m=PKsLxgfytvj8 zifTZbKZsPL{_8n2x5E&By~`+b9MB(-gS3j*9sSS~W{%kd* z{U&Xhkh;VS;-y+8tWEKT%8vK9*DZ`@AAp-67Ua6TH>VUB0o>xz>P>TZqQbuH9o@WV&jxz?%5Q1qTde z3vIrwm5rETnY~7MDOD znn!^C?}rMzgOYqf-ggB#WcL3xA;pkcJwO zJ~;q2-lnLY)S@^X5ZEX{QXW0>sYpq@rjuO&K^m~L* zo_qwm06QuX@z3kaP9|z4Nue&qE+F@lv)9E}!0rR{VXv66^g6hjD|y}wIuU&aY4lnX zaW)Z+PLoD&@`cwG#6mV_a@cLe@*ET756&cn!=@O_o2SdG$!J|bb{my?@B8pGW05v+ zayl&9^nK|z%}LgtJ~}5|gQEr`(RI+o(Z_p=x-VdvK0G}v;t~m)Nix0P?4+{u-Zc)o zNP{7tQ`(Kzc3NUiW^Sk&3r~b+H^u!D*s*c=vDpRsBocA{JAZ%u0ht<{^X}_dT)~8b z*nbC8iCAgQeDbTXID<8PKIt*QTHU0k49DhaD-L!9j9dyE8U#6F)$boT>}d2rY)=TK zw-xR~fLRT`J7%AR=Nha&VTI+=ciOIx=W_~RdR6FoZc)9CA?J|6 z(35IA^Dl~jVRVzJJjQy-;h9tQc0_YQzrT6?*fM9%MZdSx#qM&=u>s>Rt8F!t#5GBrRiWY`+D&7&d)tyGo5rpb9K^^RtQ0C4 zyYS@y>s7SHk*bq&kaD2NsHDI2?|VvYf#cIph_H!3YO6=B!u(17-Me>bt=9S(U2(Da z+kvn`oyM1bK5%I+t{AVKtk^#Q6aPc|S{#XBHuv=q7ZMt0KLZix=;w!LE=5jbWzy;p z#+T$n2Fhzf%#KO2P!h=2HMF#nT?A~yvGpqcDoA3wEl}Zg<$mt3JOzoM(Q+3;!gjon zpuO`(0*}GfY`SNyk^Gb?dzY}Q+}50~RUdwvz2G6;nE_9h{geLk|MhJVqgxxoPrQu! z)h-it4sAdA_pXG?MJ_t{R?hl&TPHg>Z3YFXLQ>>%%wn<%uV(B_3Ap9l7Wdt5Sk6$7 zH|BRiD8|#&Cgol>g)!su!OssDLVCf($})mQn^+!bJ84j3c*R1IiOr{vbnS}n59pFr+PsB<)XT$ zxjX}HdlvDt)^;kgepac!9+Bq%=Mk~6?~>ZYRC%XuJ$8a@ng1|oA&Oo%{qj+1`L%1i zjV0u|QG&=a#8gMSY0GZ+g>Su-R9Rj6=kh+hgw;2lTC1Ds{IWpv(rji45Yb8~`LGSQ z-hlz>fIJBvIv9~HjETclop}+^ta$`SuM~WguR4e%0aC*7U^*+nla&JVs>1KN^xeU* zSwhDKx2d-<0v=n3B0S{}$OyPsrp`xX5f8$nUHm3 zq-j4D<=Uz7kk#t}f=cCW0-Z=jJ>v*aj7~-07%TP@GMg%U_{D8-$NOdT+bXgrUr}Q? zo*0^sBiZd=P>CC2fJQ$&k$?|!?sJuCd)ny7j~o45&x=qo+EAIFf~npJm@h@ot?t^9 za^29Y)Em5?*r)0TmyOb~$|L-c|9Sji6axR)fu~$A$RdLwq;o{@pGql8ENb(#IT8Od z*`?k@8^*+J?%8mT0>I-jy_w5{k-3rqMgn0)Ftcj%V6%uOWy!9S)v6x8iXn*(%f6dI zef9|OFmc4fS0||vZ+!sQi)Ay?b|nVf=m@=TrlZMkXFf1J_Bn)yv5&0 zuL?US_EwdVDVtwllfd)TpkNxfk6!BdSr%bF!92D>F!7br*4Hc$7|MtX_`~uEv?^Kb-|jIeRmjs(|vAh5wcbH)d4DqEo7W~ zcy@}NitqQu5_W4=9kl|%;!f+P#u5w|Mh0{$e{THVxZ}lomI$CXRr(3UeYW!d>hM!7 zxo+ED5cz5ks4O60c-CIi1&nKF&Q#(9*yKth+frD4l{E5H@gru*ET&Pg z9rs4U`zGQr*nfu7_$fsdMx0cK>mxnf9u#Yhd~Wj+@C)P_E?*gnZWX_T%*bU5K6HUaT*J>GgDEcP z>wLvU$!k((=;rwO<%_WqD7$javmUV}@Ett{7XhBy-&u-u560M^i2J&B2Nj~3_Lx3} zdp~XMPI%_`b1m_1z+ee;t|+$v^PiQ3DZ*)*t7jv7kPqcZ$6w&p>kSLzmz6bBu(2;- z&w5fa6Z>UjO@zI5!@5Iq(9;94$mk*QF|g%KW2g0SGE;xEK!##0jlZ5C@HcF&exO*(?d?IXF0s{;=f*%6yHU z(JoaRNo0^+hxfAa*{6KzbC$6ek@O1@!Tx#W-k8d^q}p7C`$Oe7F<+UGSVBxPGqFaL2J|sSe-9cMG&am1vJ58z?XC>cI zVq{yTwN~o;aRqAk&k#G`2M0UhP%J;fL)w4H7AxXX)~hnm{A$0@i8g8j^dJZmm_h$@ z7p6Q$;h9QTYR_jt%i?hFWX%P0_9jI5}u8C>@G-y+9V;Q>GeaolcQH>7mY5R zN6{0*mw@g=(YoTCM3VpB{3r`nD{&uL+qfy`YgsK9In4rrFV;SZjd&sa>^mC`H} z!h&P)cXuTh+xc$XN77|~PJYd|VoMo?i*rWG4&q^mWgOqnc7vUw-m7^XVqGzpoHK(! z#)c~*%!1jH_lMk4!VKE`KpQk_r@}Il(>*^FwL@CE9qaR^^8(7IX8f2zhJlvZe7C6!bD zgsO9><*-vZQZa#cJlXY@`gh^oTtCF_d!D^&hcUZeTeA@8=O-x6_Cm057U5dqr6C`Q zAj1wS<98TkeyRj;*h(uynvQ-V?C*wMOo9IKM4}^h>VLf~26QHL-utBbS$E_rmcKFr z2UPwv4`5#^^Y!_IKtHmOGd*FvroK+StGTdQtU924KCo?+=2(aI)%Q+s2qoo#C7e>P zq;Ksnsk$HT(BAawYZ`!6;2HAusN1$)LyEs++nW& zH0;Ys$Nn6uOJ;Ej>Ssl^Y4vWK!3!h3h~H44Hvs7{>1ix@4_=J{%r_c1d(fpIw7bgG z{**(jC@PaeFh|>y=;jUbU2^yocdOl+bgcBO`+&wU@ECb^Q@32}wa4?lL$IV0fPOID zfj*Z72mL4ih6q^1iu7lRJU0vzMOtTIZMSkim3@t12^LDqzfAcN2kpn`{((PK!m?=~ z_`lwmSQ6()KVcK~rID*C%>nz3CCt*ncdKX?){YE_oWBM0@YHzgmNL{&uW3G27=i6~ z^E=Vcm;-ma@Dz;VYB=0{E{>FIvkAQy9%K``3_+Z&4e2*5aEePyKc#t+QnG8H#LZu? zRvoOoU(+GKPrslPymMDKqX5Wy24fj_#}i03N~!}dc4>cDBw^0TG=F~)?ATI!U3uho zGtHk|mmtzpxPj**1~%po?XTC*(!2tmGKzvR_rVXtNlL2kJ|2$S{%3x8*ii3XUJ!NH zzC6@3wD{T^P3Er^@Cr7f%3#wJ!&)Ncu=)ux2bUjvqMwj*>MkV?GAtH8BKIUY@n{(= zCq~U`=2dK*dw1wn4Qw^?F81)t(%jV`y!IE$q36Y?x$At*=caT}d}Xqf?bVGm~Xnfusz-DBq_7gb|{=gycK56$ZQVjf!q zWp?uGLBuA+&|S+~xOhd11C!J;u5%+t8n@2ZXDHO$yF!&jK7e1pxcv^csk}1cY;wNUx zRoukvyByzmP221#DZo!q0kHcx`31lsm&=ruJcPZ6^sb54T>l1~!!UG%4nSx2mi(D! z7JB7pmHM`iwVDJ)pN+sJyn@=Akt2f5zky`q^Dkch$xTj{ad2>e7vi~bb9G9SO3hWI zXPIfcaBbyU6|SCwr3yPFFc!mMb=ZuZq)K=OF=bxxIyzoAi(P~P7MJG`r@)vB4Kx~y zcR#PT%j(+N!*_2`HIx1uB~gnH)2-U)ZoPJIZ@H%NdjDyM6oWVJd?b5Zi#eCvhwiT# zS(Ox!bY>#s$dsWF$fW^M+`l_ia#;A}mzFwP_$XHj6*KccoMJFa4oS0SCqQ`}_v!M zfr_{xNzEo^JJ8{I)tIj~+r;X&?Uw&EWS43?I#%S4ktQ`tU&gK35jc) zzh?v2S&`dSnoa%YL9BOM+%5d!W`1_w%7k>t(=L_*pj?dwpMv6`o(1gtNq&WBTHfyV zNc`RdN!sf7O?E@HgEO(MFVc97Garl5zV-jGr$X$7)f$q9!X|d7#{@UsT+V&(I8kLd z%+HR6Q$Px;2TqQZ0|M+S{O2YP17wJ$;NYP*j}AvZe=%RU!uU^#X2wE)5$-3HYwAX% z^OXE`P2$U(4sao3_jx{p{&N0DT<_CE{EA;>; z%X0Bh?d`?f_k7>iMHQgO=v@!*szG(eBKZ3%THU`BS?tYN?$5+}-LU(oDUafY%U>?- zx)C0bfM_I1nMOg#7{A+hv2cLU>B7!a=H&0PM6OF73_T2BLp1(k;q-q8M3NVdxc2>2LRC&T$LRNe4ZLc6B;fH8c zjcNlh&e|FLHrbgdT4W1jaKHx&Uh*GU@QD(cSl=N*EQO<7jd)GG^AaPng3{NQ9sivO z002Uz;}=EmeEMiSrXa9Ub?l1iCNT}A15PTZZt|NanVFo$gV7&L)jz^DI>?AAy=9iK zW0JYeNcV(~fP5u2E--@;%6UGn)Urk3_?enAlQJ^OZ5h6i;OzM%C@$5NrBPX~>Slosy+McW_9qM{=E%w{&VzTddhPU8>0VR$jP zk7o6~4A1>Pued%U@EOP4tw@8FmpOv&K0=zu9{>ATSD-XFx(ukF9lPc~o*-2En-H>$ zMit0}m+6#irDSJUUEG8;p5Bs;@xgoZ#1D@bBct=qZxq0Sj35}LN>cgm>lWEO*8W z%a*mew?vI(30MG5l&ihUR#@WS zXttLY4KOC_4OmcI+yb8gK~PCyY*Qbhe}!+p+@5Ry2u1iJRo3%7K?l;6cVwT9CH8I{ z0dpMl=OFKFI77gW@DI%fth-~(k#(A!4OOLFW(57uPLDQV+lT33u?N;&BjRZ#mBDKG z+TLoDem6{hieAKK3OPYy@QV)q2UfXz?g>{zFM3*!6w0UM=JGaXe(WbS7#dxP%0kTE z=^ZSNd4Z0oi^hFVSnE5tbYHGzZRzJ%i>)O3U?CFx3VSjHiryH(AI2X_tyRU?`1kz# zuMH{a)uq?dWkg8&*Lg!{vkt?I_Z?{u3_;W&r|lfbY@>l$DmCmCqO<2RZ4-p_ z+YH{e^}=jg|EAvwhpphy8-khL+)gS0rger@k}GFDhVUlRhpWq#9o&+tV;hd(W6=84 z|Ba;&|@kbK;a78orP&|XX*gCY+&`2&QIia%X z8`I53zN>MCoc~;Q!6+0?hJ&2qS&gmTutX*CKhJ8wtWm5?LCtTCTQFd?cM+4j>WxAD=urtN3;@x8y{oK3g+;*!HnZoZq1psmL z)mPqJz?r9CXixfZyf!Su_la`!(a&D{uFU^sGj9-!zKmi&D4I>oOWwF?#Bw3&?}sTFSA@h8+5#VipCgVX^Dsx4+I}_ zn?|(U0r0?k-S`7hCIyuao#xjvc)Lqt5-$m0>)9$PD-(ED)=txGw@!8N)=E`?NhR)S zRb3XwE{TQj;XHJUDloTTH>_i(;5N81w#4wHClg6UK>%O6T-Lt_Ic1afTT=e5f2syt z8yxlDRz=?`cI%PK#FGIK(R)L$wVEAEdFwj=J~1}tQkKq#n=;cW8gt`7Bh4-(lsd&7 z0RP;@%8StioD-|_U{&Q0`e&i=2&liK0M73KBJzxTW%k0q0Ya?5p_U*?Lk(P7i$PS; z?utjTl!vOC`!Ygb!D%c1RJ~oQCih!e?-Rm)!aeU7mbMKr2-6>~6@(kWo{Ag`t$`0n zGgaqDKcs0YIp)hnlcoT7a+~rx{`{7m%h31l-<^pvc3#0y+on7RfI?!MFVPt;?Sxz! zFCHfbWQC$4G;B4_`jtr1sEGfg=)2WB%`)IrGo0i+eNBvPAXUy?yW?1G|Df7$oVk#5 zO^#tqkp!Nfh;(7?9*-A22>I!-mI;cY0V1(& z{N?YGzQ?;3zhG@dP{7T&$$3O=;~-%Qk{X@qx*rH%N9{_`Mf95tw^4D(p&Pnv#c(j4c1HyxMaBptp$-eImTwxPZ4EK(}G3^M!KKOS~+(#z*qtu=Cp3(MO z_1}S$A8*wWbNq9pu+Z+L3+Hq|i8E{lxi4%|H5A+?_g7LeVwJBjMy@IB1Be#Ih+VbD z-u=N+U&FSKQS`e0r@vqchuVZ#7{~*}mKiS~W=ZzpHDy2#P{6b{*g z)j_5Q+PM^@Q#ojAgR3GeEUzbSqJVj_Qcgm@t#0EnA6$fUTJf=8EA#Z^nxy|&(>BW! z6HBhh8#mC|Mr3G3@-wl$CbKVjqPB^@9LP&2O@D8`jBr3Uy7~!^rJRIQud8Ofc4jCn8^=?41dT z4%dn1+s}G_9me!`aJUo6sW9Mv zN%bcLzKj&ze*B0Nklw_s>Xnd(OkjAPkHLdL)IYV=nE)0BdP8rV5p)Q1p^P2 z$BSEUXZ--p2-%S8vETcRS)>?LbzohYZ7O*33U1<}rtYM^Xd#mLdzCJ|qigmZ&-7DS zM(K$?5&Yr*Z`tQXPH_kvJ2b^oYLNfc_k&!CUJmD^;}~uLHXo6#qCSl}>)UT-oiStM z)02ZCh95DOa3lax9Ma%dBsr0#Q8B(1wlI!=!#acEqJO51GEj2q-AQ;e5WxK?X;427 zK!IAeXwp*?jEd4yz%Yqmq-AZa7FOIoeuqt~t4oTs_rs2hBcXLS^knGDs)crU>~-1^ z(FK9oEqGOx`<)wlIO;RjxTNF&=3OGX7ztyfp!X2>juIA<1 zyT_4rzowH!+{_%fryW!r+3!SW8<1I0w~gK54vf#lBUj%XmvDu%$BJJxN~e>496Mz9 z=(_i{k?C`CKl`sR%KdkD)+y0{As^wZtVHb%{x+sGmJ@=mgPj@p)~@n#(dK^lrhH?|b24YL}gXaNQlJbwmB=p$ePunr`CO78Vdil0<_U>lt~_7 z-Qv(6kmCaYAn{3W3*nbdNBNg$z-@q;N<-~uG*-fxM3n~>sky4;wA|_KzJi$;!)bOG z+uyy6SstnE^wicIQLL$OOLX?COl!}z4QErNP-Mv6dXT1O9`U1p~SxZ40H8=gx0Mh}e+8 zhbhCTVovks`WdyoL7s(9v|0p)jKW}$nZjYJhGAtW?{2btOyE>Yz$roiaopgDpxH}O10zC*U9H@jw}+I zu34rwyIev-x&dp09zOU0uvfUfyEmNx#q@H+;g$h$L}Al}TCT)xr-`bh8R!whn2O|z zCJ~=Kx%8`bfd#1T& zVvKRR?qnVG&TCaqg>QM(aW-R_8;*L&#)URLX% z@hQd#onq<=djVR=!5C;<*o^0V82zYK!=?Z^l^e~U&u6E1+2xsxBIfSMamp>_YQOEz z%H669r7(SYvabT;`E(s?U_Vn*QkdVh`FyLG(@wsDvV>`jmf8)Qi0OJer$1!+RR#>> z0Wi*RfSxD$jw(on9{oDnaf`y#JGjsVMpuz%!+&6-V33i?Rm6_zBWw4jGZ=e(2cNeQ zh-=|tf!S)&17T}hTk8ArG2{g>!J1WKBBe$vJ;VPKxHVpR`Se%i$%!Y7k-x*Jn>y4= z+vNGHKQ}R1?Mq)C-%0>XV&{~@MxFQ6`I`lpi3&dW<)y5mBJjpJz9u#u@GA@jHrVQ? zqS{TLE68JR!YYK;#$`^Wjw=V$8Fs!T*DkLL#`dO-^H>E&{o5x?5Q}o;x@7Bh+&<ovT&)&}7bis42F6h;R z1-oJ-ocs}(G15p*B(3B36DH+ncDg87lEu9mqkHgOQ_<09o7?~&{ftvI93&MVuldeO zS2zTCUYidC2P!v*`HuVFK|q%{$8n+*KO#Zdd{0W~blNVj^%4K?LfVOzQAv$=qKW99-UlnL1r6GE6 zNiLt;^4XF`5132{YCCq!&rG-871`=KrW_q=qnpslY~5=t2@ILSwvG==?jak6&y z6MjW!uoE#X&Fx7ynx{Qrl0-IrxU%kcrgRkdgDet@ZdDAl0G;@IFJVn?##98XEHwI-4*DlLR4*3X~oAIul_06wAnRBelPeTIXSyV=O}Adr`^XzG>)6&9>?PHvh`EsnUWu@+B6KB z{~^Wf8wtwe^13f{Vt<9SJlnKv^+#E(`UV!{eSV>_T0emP-EHlHo-$miSg11cuj=o652SnJ(c2x!t|+H}y-L-_%PDs+Ki7V(4&)E4sCuw*aJ z!S4QW)qPNCGX1eucB=3EgCkSN>AK?OKj_A2*~XfEAT%5aWLIx#88_HVO9uR4)|)ZE zxB~Vm!Djv&Eah%$AVQsG3Vz31RV>e#M#PJIGlKMwN6Q2+3_V!17T=rg@ z6RN*l@t5bB#||rwLVe-JBSu<}lJ^$_NWEYa;W)7qPHZHL>CXQCVeO1TJMPBrm>`Jz z7}2QGnEELb89f0yn%jN&5e`~g)Q^e!((}l|${q9vYgk61bhKCWgx6A6XQrN0mbfR$z1ksYJ54v1+|cx+ZG*vQTk1W!xBl^G zPU+96j_tIQMWt7sm>J_m|HkRsMD&(BzG`WuxeJoz50tEn^$L!!V1y7n2uSTZjy5G!$vhukef9ZKLaDEi>xXHA-K5 zGD9|jbl$x1_*9`)g}1f$x^PsZhs<7A;=!qyc_-GVU)-~GHnBQs@6X^)$9}&_we+@8 zC*d~{%blE?Z#9bK~7Ie_3uI!KeqM?Yn7DP*QRQ@VA zaR2rzjp~s%C1-ZIY)`)U`!I|hI_h8MCn8(;{LVAw^M|FQhL%ZU1omFDy!-{a)zr4} z9+8VB|Ee9Q>D_Jfj=wF||M_zq?zr?;af_tp!G|^;j;kUl>x%s$;f`!7xro4}#{Qug z%SAKNt9gbYS1(w-y^=j?>2uuD$}#c$?vCH}3o6)upckZ8&6RpEbA8t2_s?m!8onS9 z7%RLmNa+2YDSU9z$25_ywed23)2h=|?)XERM;cozu>lGMH3RgEsBU1jKX#gkjD4I% z|60^uTXhG9ylo=NCiX&L|5PA#?rCRzhF3ei@Jr^$>RV0dLDYGI#Vc0%O$<-)<-X)T z4q(>$ZLN=-#{YUs+V{Hm;hzJ#U!d39fTg4FP(etuJ}h;nF$_pQ$+Gb`xh8Hb*osjF zZw2qj;LG%15LAFv=-M~b+YSy}1g9pjIfn-(NOJ1RJAqne&g@tDo2C-yhAuqiBa$C; zau)FxF@ZfY)xaH0=YbJ2kj8KuJ$C;2hUWVwCpoPWdx6qXs)prL;Pq9ububNNwQF?c zCL>qg;TCK7vmF|7p8@;4j~GTOX@Uv(xsOjXi}O&dSmXD?>nI;ZkO28p!?NI{+TJUB zQrsQc?3@;*o}#y9V~i*B-?_uXd8aMUy7O3Rk?yYe4EycGe-9*aA3t`J!Y27t#WqB{ zM9poxerk&AMe{E`z%T9eR!E>#C6d?;*5+G=?lLR`RaLzPZmLeM%OcKGs1u1C(|VmU z4fNC!-z$tbjT2NG=?=Yio*K4MLz`(#PcI#@X3jB>#W4#%+!2m?+}LY7a_A1JV3>8Y z??o%7pF2JZ9G?plNzv6O^A8ezQ`Q{rYlj1^)d151UG`Ox{-K9;PX`i{lB7X`&F@R% zelBlEM9*^JrfoClmj7#-PGY7K6T7ru3H6?s3^%OhXtrEf5^#_f=CZiJ<2lZci&vuN+CJ+uYG>&&mG-! zpPjh$s&4dloqP`e;tAU7`I!WX7auV~O;3ly(!rhX{)EM^VRH;C@1G21KYKk_rLSUx zVPCTGj|J-|>-d3d3g!Zp|dw3iE2C?2$0%J%OYy7aLt%17T6)XM?A6W0K&RnJW@%LCL-3Ev?A5i zXC1k*!{+xiI016xww&u0fH**VjMwwTlRXHkQj%nthp0pS(7TO}sWt*!931cN$xe2l zOMTyDYIJwJ$`zdN;Zu0ZCl^8*^`66#w*D!~9-+xULhD7a@#kXsF`WRssa5Px#l ze}V3G)wI*44hI@`=;<+r8~KG1*^mKrVU#8$XUp!cLpM!kYm9ADS5+8;O-xu@D=b{= zSIQ?|LI?XNROr_2iw*o)8XM|*b@s>DB{QsC!Wb&4L5JVH@{=Sc#i9l=lK3Ibmw4gt z={)OaBw>GC`gRH16lotYREI_1+CLpSenN%;%=&>C4d<7o^}pSJCqprXRf=mrEwNYW zA25f76eaExmG@en?NKe)!9ArfPQtwGXd1@hmk$DzZ{L1bFhArQ&~^TDwmLm74ln-h zfmTyP;#_CsC>S}Qy5so3SyZBZa7F2KtIDAXwTE%d_+pC-sy%^GS4M-6uJ#30a1x%` z=~bC;c1!SjF04&D&t)F5PBZdFVn{wN8Hl?HcLm>1&wE3!3kDWiRdO5gOG`>f-Dt(m z&tviElqG7DyzRhD?tTUmq^mw@#}CwcqCWYxK&x2U3&P{qZ-4R?e)1f5aYvcLVZ;5= zNRtJU2T=$pFsI_`>ys4=JzkCXcq*0OQn~cwK_)<2m6`N>VvTHNyi#9ee~_dpFy9qb zzwyW10&>WcRnkCPN@{HAPS8=#)4@(9?>JUQ^8I}ZMce>(KcOvk!Rx1utj7hRjwE5K zMi6hov49$pjr#GYc^gAApS9O#Lu{*2zpR1o+ znLX-Ng7d`UukdwGmG!v>7(RRW$)))=FLV7&yio#?VQ)Q)TGjc(7=M95Jl?@-?@&$i z_bpqTEg)fwxbG)5{AMWai=?%kYUR*){YO1fQOgIBp_V6&Br2O3rXOoQ{T>*&gTD*` znB}c%yv9x3dqHv2MxvjaF?va0vPiik{De>$U6uOtD;+<3ee*8{y(q^-K_%F3m;3YZegY~`VPsoD=n3z`>nq`Q6#r5cAw!-*0dd%RA4Syh8 z>@lb<5+BPB|H4ScmY*9=ik;IJ_mhqgiS-R-G!bQpq9?!c$X^$Wf1`CCCTTWW%=ZduyO+RJ$ef0<; z8h;@_{kWb=eTmou+glr5I9W9PY8oOfvu)|=;=a|^spz+(fJpOYja~+}7T?K_YuSn) z86`nrW6HFE?}w{JRfo2h7WHi1J@_oP^%E*VX(lD2llWO z@{739_fNMs0VV*dLCg$xCoP>WA-9ooPO4=`h=-K*wyoM4rl~c$*(iqgyr-C58o>7D zKevL;+^s73WosJgFJi$*hhBi$VkN#1&Cd+>gGZe5)QEo?{LXlkEw$#pj*}Xz$sL1& z-uSQwCj^^R&l>elr{zvKC-{AEqmtxug8lvIzn()y!s{=MkN9+A-Q3$T4Ux`RL zSR64gibz)@3!UYLZlf<7*Cl`HKl@(o=yuyd>KAQ4 z%wTrbwiLq!sxA%nk+(gpeC^yf+P{}U!*ugmSt(o;V?(_@-)6)<9Kyh#vY8w$^{Qu2 ztS#u&-$5eK1%w+M?`B!y-JJUoxlg}v8>iJFH~olYkJDn{mh3T}1owc=lOIi=Z-wG9 z{&Pnx$Sv%u%Wc$YJ!cP!N#4*eb?ke8A4jIZ;!;BQj@xTDOkWK@d}_gYWMVLx;1soC zB7#>r0@>T`|7YjnnF?kFMFS(UjGhas?D{Zs^5fg5am!p~*^DmjC*n?zL5hh7+W5X& z4+t;5b`WaA3ye5_($N3M)pvknz5nm$vG>Y~tc(=0GP6k;QARQ%**h!QkG(S@L{^Ao zlRe6w*?Shs$ck+L*HfMIJ^$Zzxm@SUIj6_x^M1d^{kre_<(2Jg6P0JP9hfAtd3(W- z?e)Guy_;es%)gdQ!(D}2%>@F3FQKaZm2(=-sz)# zxxawL`f{J*DwB?t?=)^MrJ_!Slf-pnGhXJX1r><7!|p}!jK}Mh$Mp-x>Cat&;T8!A z@z&r;sl|nZx9e-9^uM3|>PDf+g$$J6NiI6E#(TtH?CNas}4BvKLw zk<-d3o!}UG(WIWh%dfI8Y!SLNi|CJRiF%9Q&3Yc~-35kBF{TFluRQCmfYc_3i>KjF z9GKQjr!M0oLZldTM)`FEsU^K?Fnk_mEU1lscD+d^~@R~duAC#J_cd*1xYP+e&TK2sbgG{ zW4!Pyd7{BG-N%#Mg#9g^l(pBIFFshC-`;h)p{s-QGjkW1Bc)}up?rR#W;5qaS&4Fr ztVX#xUg_Ty4j{dBQ~Ow|1vH?8$qvld;E~1=0&Zxd9DTMAU}So9MVJTl7BLAYSlhpR z^>NykFJE;@kWH)aHt}sKt$FBI{@erZy*KXucl4YWbjia62`s*-@e@pMB4SuNH5Z51w)}QH3Fqng zZwdyD(qqQPPwOTGa7l4|pekm-h4mfY(fa|2;ilUzSxo4EfdZ1xxOB%i_b&?0Yg!Y zO5n;y%N^ZNl^+cdvfmskjlQaZJ={Ov?)8Sva44rr{AN|;;@H(<9hMNcQD^l=6_Erl ziJSdMF1HxI7R;uSln_=KS&9!LnYc@P8?$!370tsv!_a#S-7|$TLh}U=OQ5MDS&T~M zj{UR=EgABz`r5PuzWHe3R z>Xhg4Qr6%c9T};+xVZu+xT9lT7J?2SQEq>;yc{aOOg74n+&clt3F+3PwVnL znF#^XQ}xkTiAt6D@b$;h5VP-ZUjS5kY?L!>0J&qvwek%ubzZqYqe=KdiK@@Ef!G6t z$*)G>1L-$LYB$Zn_V64Ow0JKO^ajx@5^?xN$-f@WxPBq*D%EgAH}KfPWMyY3*ZrzV z{2{R`z`?LbkNyeCEn`Ig!#>k^kope2i0#-m*e})`iiV;n^j4K{H}{H~z~?;M=^S#-#B4Tme`KZ#i9EZt^8j ztN`3~ht5#dSeZyu0V}=N{-e@h6bsi977elRvw!5h>F6FDyNAw|^FxH+yXHu5dQ9jg zF>9|xPj{1tH%tfb?ilRB;QCb<^^?<*1%~V;`=4)lu3F;3C?Xvy)_zdRD-SsK!z_n} zPs`PiiYouTl?y{7rUoWasmy1XA{IV)s-~~~mXZP38K&cmOH~H2GN#!oO664|dg{+n zblqoj45JWqS$E6o9=2L-h#ZJ?E8^z4RJxsXk3OENd?Em9S6K~co8UBkqaQ^I#WE4t0CRec1HKcsvq_1O|l%_9p5oE{q6k#|+}3y@)%NZQMIgFLMz?yiiG zT*<`w+p4-oHGWT#Q&`qvC!!+PVvG6*GpEKWTI>fh*<>;6jFO=e9#m8VL>-$u92JH= zv^lkHjkGaKjRkrK50dn2;Fpu^N#K>e+Uw2-{vAv;W`lq`4({;)V2~cSXjZi(``FD} z&Ghr_xr-RV8FjCFx`A(p?KluTi<*?`MSNS;jeB`TQ<37C*m|nU?NgD3w5)@2=cutS zQn#EC3oN1xg@oT(>+?^HKJi)K@)t}WvI9xq@xq(@k$RpF8x3G$7jX==!+y*6hf1?I zUNg&wQZKr4%gFXuB=d~2#6h>80Io8f^VFGD;>{u3B6%F8gYZ(PXy;G%T4-U{NUx$P-MP9C7E#I_IfOB55EO3 zGs(V(d3fFpCLh)DeU7Bu*vB&S(pNvNDw;DYEz(}Md=t00%O#it-SP(RG%T=CI^Xa)a5#*511gW3vcaw|&z;?Qoq6m+*^tw65<=qRv+w@6 zJYamJX{IiS$MG+|WY1jW`F7ea!WpgJlqvEN;?7H&XMd`;VtFY##LfhE}%(}q-Ype)x#>~ur7!-%T|9WKb4uC6M@$rNlFEfwI$u>X?t0hxU`yk|7pn}I^GeovZH`lBZ} z+KJht9pEh4n@P3I#p^JG#ri)p+@q_5`9z{r9|}W^-YOf zrKmcowZC!(5;~ow7xoBj6RgU_;D82GBSQkC%uqn8R= zS;~Q$o!qvz6|ES6wF5mEqOWxD&R1UCVzn?X#XSYBt}_42zU*H$Pd9A$v&ddm3>6wk zk|jN75%h`>MKZ(7Oxvqt>-+A0XSt_6j8$IcVr$(buRmh z-aIAv+}6J81yfX({DfP({_;3(V1>p?!|oqLdQrkdJ2q@^v$=jA_(nMqUA>$aezjP= zmb~wu5s)Q*4G-QhP78(Z8pv%#aC=}Dgg48-#wJg^=HkCP?fNC6x_wJrFEw-Fx!}T4 zHyxA;svz2aCrDp9NhUbkHmm>fN^n)>`TL{3le<^02ZFi~G#nD-0eo4$xwHYe>yoUb zWPffcbu+e>_f2`deP5z;;xTb-U{>yKQ~zV+wU+QGIhDn~68aV%3p-;5@I0gF3+zb!|Y%kDC@z4Vg>Jze}`Y{@fZ zpY;9iN37`cY%;N1-cW4TWpk3;vS0=ipfPOnt6blZqeq~?O5>o$xk^&4?dA>2yE5mW zVE1}UQ4JjIM$ZQV9>d6Szv!$I#*xqcj|ehT-WAp-!M3G4$C37tQg;eN4t4_i+|&(O z&p(>DQ$ws`#MSBD3`Ks#zQkgXN_wEZXJm;3W?><)XkzSBWjy?s}g!|V(r!>EVlf5*TAfo<(Xc`RVy@zQojfJ#~7 z5l+!|tv&6ka4NKMVoR3*Ljp43Z6D%BoTpVRcjT>(h-iiIqB_-NPRwQmyzumpVH=v4HS%)6e5M&M3~KgS+jt`Nvu1&vs00KD}(P zcf!QT{KObt=l;v`7muzuNV~Cc?_WzQ$E>mZ7#BLiaAsc*G|uwf$M<_VA1`8Sna^Ek zSKd+w--lb3VLVj0)T$~f!-tbqj_jc6lV9bJLSXQNPibyzIrQvFB^r%^(WK4@;7a#% zUP}-&4a2u4HG5UG-Si9lYVFppiIQOcH#WVTgs_^>pdH5-amFwdD|~?ch?Qwx@mA^O z=1y5#8R3z4uTIqjkvP^$%o7sYucK12!c?%LktD%BXF}Df4*l%+yy&-12Cc8%E%1n; z{s-~fWI=WdWHekM&Fa9G@VCiS@U;2nsmd3!bGwvdPB>L&EJ)I7gmXbw^W(3YJ!_eD zpHzGY2?VOnYsN$Ky4l(%^MEugb#%M*VMT&&x(wz5ASgsd0er+-KT)&ptjim*MMmv>zRl4b8miD zkGbV{$H~KVr=G1zZJc0v=a3{-?{OeNm}zf8v-KQzCLG@k=NI)S5y7&>*NtFemZfNI z8_adl|56cx@KwcAK0Ahd2Cag*`k>SSDBn zk*A4OUy;w0S98JW!=-tG`C~eh#fw@W=kL3~bG33gRM02cq(fs+p>(9b`fyo=A`Abi ze8*rgckQsFFwdjEB~1~D@Xz;}(a3L{DH(-aCQRp$!mOz6V+&Mvu7Hioln8dH`(AGo zt}CJz#9`q=R%W&T@If5&;7oXF|S_v(Fb>^z0~Nv*J07)fR^ z#_$-ALM2@{(*&Z`h*WP;{Qh^57>p#h5|MEY3mhnwILe)6lNqgIDg#SG`93(EY365* zPxIHveZW^UCAq1$-Uym!$Q2J6KST9lk%S^MK}&O+k435xA7c$ZKaX8Ec*!va;TnI= zi6^By2>3RH6-ND!_s`~eB>B~l$PT==iDOTJR&_cu!;-b$QOTg=gSEMExyh*4(RHS%yYW1VQGP?7k&7z=Vwr+z zskpV)S*)r=fA@+z9v0gwq`>D4AG9y$LVvtmbq=L{I*+5L&=4wlTQKysn%O>B2ZY-cg+->_ORr;ZbqZb;-7k!KYNl&_=sWXwv!OuKdkI zm|yTNEZu^j*3;1(No6PBe2vsYGd5;lpZIUXB*i$7OF$UgMSzz(Y%ew5`^*BM zK6kN^zlhX>S!MJm@cy*fN!nX5rhYUZ#(mnCl45IYkOUJCE?lJ-mzlD5o8gjwhlDT^ z+bX8}7^IJOwAC`MgoqrXvL*>${PdZ~@4k&9h`cb| z_-*3%Flk9ejU)_FVi2Gy-pbkfXgzg1H3R= z2joy*KP-4?S|#^owe*xdK$UCB?@uU&Plg)g8YM_EngqhxYTbOxD*$ghPO98;Sc8;; zOJ|7raTDZc%jg}nDJI!wX$FjlbDaFYeO5LJkf7frwQA4=`vNhW-lZzw)!l-AZ-)R~ z`8<`u**h(4mI5**B)$(q;V&Jn{dOX1ECVR-K&0 z+o}7uYA??6vl!8km8?aEO^B|s=_^dh;_mSpCMeqvb@*$zY{0n`j#lw4pXqf@y~@un z=3vj&f?&dF=0t)>MW+>Le!{hx{$MH-l|nWMj4oAF#d2?==JsWN*IyHlml)?n5V7p4 zdsE50kHE2MPV(%4cnklxZHz-yqj~>_{BHXdhKx(4!HQ))O^?#8{@#ROKIEka(ygay z7d-5F)0w43pWvVz8?_4bI8=gFFI8)rq9o>k>?o~Q+Q{aG|9&ZQiRWG=F89e@f5G`6 zWf;ZrK2L5fh)pTGZsk4h z{aeER890($!tk&$kDv}6tGoO0$KB9#xBq^;TlVl6$5b|j46Jm_UZT<9upp42Ul?qr;7WfB8rG(*1L{nP|JAB2NhG>p*(^R66m_G(`rS{iqpSX))|CYhN zMTEeWHH0ztPw=4{g#82GV1nZW``v$Y5p5>gy&V$SJ;2S@2Hgfa$ubz}tpDUG9EV=< zGt9tM>>UlWDH2^7O|Xid6KuFzQLKuBaH}6B&^SIaTk1}1of~J3U!e@+cUKI4OJOu^>iZQ{HFM*Mdj zRhpa+elqKt7U2Wayoj8(B!X5*Cb20EOeaZ%!5UC}tC_sniiKbOSlk5f>}dAv8~k=z zP&y~Qk*hTYV+$1&?2^6DXaWn3pg4R%v)1VUpu#q#jw#zPHRKUJR#F%*LVJ|gMOu#_ z2SD|oaE%6mJ!1}g8z=1O8)mPSH=3a2KFd<1biF)h-xck=2lA zdB=A9Kh$B=>u=n_s9+_vVc>kcIcWHZNXMA5l8Nz@=sWu<5zUq14)xHtFPyuf3Cb^Cie<-BHV{wX;u$oM-d)CU8Io3iKC#=?WrJ z2~bjc8~j>n-3&QgyzI-5bjMk~3dY37KB}H>#^~Q?Yowh=dEl68GJRGaBkKU-J%JE6 zjMv`4xEwX8MQHYaT_ak`dspDV>8A|wc9XoZnn-dm**co*=GOIT@Z$T;NYH$DGfOdK z`5_J^;$wN_n}(>adx(E(fq$7t!4gQ~4%M72+rik7y@>yr&PjbF9XlyVVV1p;l5xg5 zrN^7@&)PUbJw{xX*WO5*_Se050Q3nK;9NKa(>6NOLB%6E6b^N$ zY&4UG_GPLa3?OsgK^gC5GNY^djuO9IY_v6mdrhVh&E$Q0tO1L0-xHm}>>-#QEr0b} zaio^&T6GS1XE=dT(I6~j#+ApSnu9?Xf->|kqX$m#?=QJPpCZ#Ix5|#QvUV?i?D03? zt+C<2K>PdCW)395BYFcz-t_QU)h`}G1Jd%LEI7?9Xo9KTUaxzY!0v0P!P{G>$Pi5& za+^~N#eKE$5|dmAO-UQyfHVIpjP}Wlo+i;3nrCzz08+YSMC&VkyH=wfk?OgZ7+v_1 zzu{~X{YK?2!{}gLg!T1whzUtqq^~gwgP;)~0BM>{9Ld-PbX_BGKZwgVY<5OX^`ZC4 zh7~YAvG-7h7ux58-`qrzWb9ddoY#Y~6X{u^+WZ__U;LlYWfU!iRd@T@H%TM&<(`g| z@tbAd%y-9ai-nECaZ#ZSzDg7fH-h9WLVJBv2U;5D$E~Q#r5Qyr*pw5bRBvqgXYsp; zJOR@T-LIVOpmIlbun#@9G0$VL>~ z6U+tsms(CoP!bYA^@j#v~ zi?JG`s35b8wys42P1`j1#LA!a|9go(>^S=cnQ=xL>RhqYypV?5%YvEy9#do`TU6rB zGAsY-^AbDys$y$=jOa*hW_yqU!rP$&hdc{yz+hda~5m_|DpuRCMP(1REKaOCb7x0y8BzdrplJ~$(tA1D~Vxu5)?b< z@?7|vn*TiMO#!q=DUXWxGZ(4Q(o-;ujz}`_19FXKpAnzGC-G0nBT>Z|Tlh?*n-Z`Q zpc&}BU~z@|oe(Ylbx=1&Nc7P$Q$Do_dfMd@&YLZ z7atB_R=Ar72FfDyMq6cQ&-1UFd92^W5gmqKn~u+QTP ze{g@<;Zr%smLs~##Qm38Gk>1`zu%4g&*_>$5iu;+p{~I(@V6`>NyQ-YRCrN6Bf95A zRv!l$eGeD4I0YqqCoBUA;a>y(Vi;#D#p{OQSx-NLV-yd26b+>jNNikG5p{ad_E0D? zE)wkf*QKu{@ZmHIL73XF_oDV3K^d8>uBrV7X8!#wQj7=;f9Jqw8RN5DDs~@Mgta@R z{uD^(5L;4)K@o62y-e`sw|G3l`zJB9L*_sXTqcEbZ{x|JNTPTTH}k=Ykkv>xvm;p% zhlgf*#;s^(r-Q|pVw$q_IIh0u3ZI9E6Ljgp3{@>dy1G4%6N+t4=4$LjDBM_M)*E~izTPc=1rRdJ=d=`Xz0jg zoU!-Bft!FQ8%4POux<1w4>&jCoS}ES`x6)PhV9j~8-HW;zaLO8$wxR*bWk#gGQ{hG zlIJ?f2#un1)W2q^Qd1 z8BUo5l(Gxb7vN;EV8O|1u8d5OAAzEc$!bas|8Hpj=kNN6(TqEs`;l`8Pf2JJOGl36 z@AH%DCg04#e@Ad_jL@;TkWi(#%ZC8ND54QmKO7<=<9zvRSNJQPd5dkQMMXXKiv);V zu7Pxv{5Am7uF;KvfOCLAmTwmbcumZHs9#C~Vn|z?^T*4VWWn4hdOlR^|Ky{OvW=~x zemse8n>Cwdky`(JKpl)duUk%TvE?5KR?H}55QyU2hCmRjKu68ttrzpay)m)Oarus> zvMm??SXBawBTT_@uX#WagK_RU!euoRSeOO+Rjv`VOZbRl<#Vf|@r6XKlont$7cB6o z?9bripXI$8JdDNAR8_%{s!)^wXXffP1LFXhvn-vXAaNeRp>@4Fi+Obxa zdu3=SK->p}#ZZ5F6I1G!h$5qEc~^h%_u_u2bS+?Mi}U2E>6oKd*)# zh>!vcn*1V)JPxPC^>rg`1F#r)a~mrNI-1`#`R@-X7vv*cd>HE;>0((PG4usCEQbBK z)TysccIh9}NnUxzb)Npn(f6|3gDhJYKXfOd;o>xNBZ0!#H3xHDCj%x(qkk}|5T)c&uNy>u@NYnvSfCILS+$Ls_P-efA9A%vRRp9DaA~l5(k&c4{@M zn7oJ5Y5oor?W71Z7t(cllhNIv`jN5@Zgf5pFv$RgND09;aq$%G2aIae*DcShB=9h) z34&q0qcE6FLzfgeFfh<vrez7m;ewzY#CRph;y zwuX{Eoq5KN@@<+p0v1oi4RNF872lmPf)5wQLO-G0zt4_$h%ZWdC5t}&kxwZNb!2EG zDo5-f!*Ed;4`Tr;z6>vZG`u7Q@rBeUlu8V5|-?!^1c&r&8 z3Dog>u=Iv26deGFEC?(X@x)_E9|5_ufa}y+7D#ka5jaHTm@Uz!Z&?n`Qth~&}Tod^8`+qfj%Y7&!yGpO}=E%zqJ zsGavpo@}AkkWAXvykDz7`r)t;!rtkFhJInfr*}B;+-%FlKApMzI=+3u(5aamKDgUS zB8Pal>xHDbU%t?znOLYhlAmEHWFpFx9qtBNCIW|1jRLK!z36Mme!BR9^a{^|^9E9c zh}}z@5q2K3C>4R^7n!1Az-`F!rEn;_+6^Dj*6)=U9t?|Z)Rrp49j*+0GiY%Zms@7z z1ujip_Ztp}l{LS4!h+VHesuKDzDBFnVG12D2{qsjkB6@pzJIU64b1N`zAGf=EJ!H) zZj7Y7ub5DTH+kV*4gw8hMb>AhT0(+d=fOiXUz*OB%O`V^W-bLE~mKoTa$#e`9!f?RrPL(tu5?kYu)SLo? zuY?^ciphR(O<9Bu5u4t*IN2|{o{>;8M+L!IxDlm|iAFJ}H*WovP)q z^4KJLeBPh1*!eB^fntzIrA;CDe}3P%zVk(4>+7+`yZpXq3Ixz)lU1T}g@hcEw@iF&M&@iZy*&;aS? zJL!f)F?HSLt2Q+({C*s24tO^XzQ#HL!M6)c!(>;XLACXAtAOp%>Zt8j@e@7wDlpqB zN`E@!Bs+wTNgV$CPCoHEaRCDymM^3FcuxK5vdW7u%&&IUKWgyND%sS}e+@LL2^7Y-U{#MdMHD>6WH2xPfQ%Hn{O#4~f8X^?v%q zV&sr{BK7ZUa@8Iz=R)f^&F5i@LOhH+4nOrx%o2bY{nZ%n(GluTxnZr%3)sLiri!%l zHkfi-eNX=3370S|5MBh+GV-INOmXF1;Cple9%=%f)6v9Un|>mba}x~CI72m!AP-D+ z+d!-x+VwzxG+Rx${BW+gjXrkvV}{-`F97_(*Z# zjx|V=-FDyTxYyIR*`s60W+R>U0)Yw3}=4fyF*IoW|?NE0wf}Nyy zdD!(NQ(>B<=v-jz9nsqvNCJcXk5tEV7+tL-T!N-E@>DJSrxV&w&trsA%++Y;(@kD~ z1(&+zQCo38^y^L*^}J{RDm z|32=2z9IEsa{hI|V%tP8ys_)rYUL-S!)Iyij%(frT~aC@+d%4#)T}=}Tm|poH`?M9 zIA1R{KILU#9g&vucMPzDfv7;P)GJp$v5{Y+rwR<=?|&5O6${=)qGIajN`w_vnDUuR zt@rC%X7$D8EzF@m(e`)9MSXK(Et0rsb>FI_HWvehLREndU)J+Z*1hdjgEJk$F+)E! zPp8(_52en~#gXfvoV3RQU6*z4KF?hQCbgYo#gFl4i<)&cR!YCtXfF^)PG8T52lL6R z%!vREqnhMadFMU5WDm`tS2#IF+V8gr`FG8-Q@kcx=(WR9pV3DO7h`QU`TEX+*!I`s zAu85m?3jI>|NAk8F>p-n)6`fUKsWiFvH4-}fwTXHz7CP-3Ff`WUkhh2stoRJ7pq+}Gq_Z;XeNKSL7HcZg$h;YkTH0i zUW#AeTj|#vt6P?ZJ94EF{r>Qv`3mlh# z{N8FDoY|HYFI-AaZ4%2ywpZ57pkXH%9_|V!bSynPJ?esnux9!zYF1VOlAeq`M_}-T zT3vpf;X*kU?8xfV!~(CswDDl#p#N~ChF-exsL<=SabKIfKo>RU-_MC(X4Uv}h~N3c z>c^Try6NBb(a0*%qMOsiN3Li4vaGwQx31)ei*;x`5GdjEWq6o^y=joKC}#`?U$cweC5m#lf*~B$@ua)N%MRnBeUYSZrO__h8fCwXXD}Zuz`1U>5&OS5fWt^a z!U*g7nv8=QLApVeDZKbX-N^Yi1(3pj+9}_BIr*9VJO&==+wzSJOkO>38+8C{f_CtB zB&%`wmLAmVO9L1$Bk(x$yluhOoVn~hru0}n7h{H2C(PJstU7~9FBv(vXwbn108p?! z-IOb$&BO;ioEGu*68>54dh@L32W8KJ(YNW+#oVp0~eiW2M#~cNngi5%(!(`*~7ZuCnM@1K1yjL^8k1orr6f54Ej>e}|5LzCi?h&bBTzeTT$# z4vF2oQ~P-UoJES$UL_Q;SK9>XWAba~J*(lh{&aV273hjY<(ZVGE9nvCa~h$78ap-A zHfJy*SbUoqiHFHI1q3c5p~bccT6Q3w-*<_H-GK+6RE)2RviCx*SDep^N`;wWUo#m_ z%<~H-DEbYt@V{QxEcPL(L6d}MHpZ>=EP1Z?`Hk&ck_pWSMLf?G*M*QK{Sd+$^=}e< ztEo>~Y0^F8jI!f<=zD}Sz2y*DyhwYL3+2zit0Rz$SI8-3fzUB`LX=nn-Un(DtJ zxsTLcL@D?1JZpBdQ29LT&9KMP9-5XH45WgZ6x-Y^J}ImHH>otTl6Xno9%bbB3(E~z z<=k~IqjP9y>d#BwIfIJuahLn<3ai*!l-Tf5&s(${=?6wL7=2XRpX|a2?;`*yHEBK~Y7t1<>+z zYP8J&>5F5@^Z~W6v}`$bx8cbch!9FWh2g4~3}z*aIdpNg6mU@z#BjtPr0ad^AG`i` z%&HTM);h`}<7_6yGr&b7BT+~6zT!~Un62gf1g*7(cuH533x9-^I&OX}{+l4WJkJ@4 zp$(eZb!yv;*`j|-=<_T^#xd=*k}ftG_{~*~<$TWn=OSD&L~HQVNj{IkFZ>|s!vICF zJqSOjFcv0)rl~LT7)te?{B97qu8opUr2Cy0R>!wF9&I>frP6o0i{!d?vLd!#rvAW> zMsGG#(UZzg$8o7U8d@MOT@xhE?0&0I$&TP7A;Gi#Hp#%17*kloNds47*OE73KJ;qA ztreM$l^$ko&M#XTsNghnv&kS$QD++S*v1t=(5KzL6Qn5jyc+K#WlCrq^r~&P?X?cGCJS$Pywcvtwk3rlw z+WpB5qj^N#|j zvFoOH+apFmDxv!taIP_lHH)8miw$Qdh2ac^x!z(*03n*d%&nt)q}ly_8SlQFw@TxS zbkgBt?wd7^AT0?q`P6>Ciw#v7n402yr3!S=4?Qs8ByR$X_L2Jn{X;0F|B4M47Eizh zZ9o2Ur-Bz@Ia&pE{ijd37332X()KU}T7C&%v|XSoj^{cklu$YPq@m?Vq$$fdY`9T?sEpzzi7FhoqsVg~Z&xIIc%DYTz3f5DD#v=u~r?9M;;!f2*EkgwpHVc){*kH<<2m zvp*!i_VX**$%8jpA(-DpQ6c;9Tj)UWPsY6b9&^00v@}JCPb=LRm*)Y+1UdfA_MEa0 zACv?rDs9YizpO8*6^%s zNj*DWhUoF39b{nVb%znGks?dfRbcdVOq*X!QI;`f<8+ZfRkQ}zb;2TNvDEUxXz?Sq zPR%O^`r&&`wzd;$^pxiuLCV$%(O&i)SdnT?Oiai)r=I@IIoZ&8%lC*a;CN;X%}V-u{Xvgir>fU>teU>ll7i%6w8w_~Y?qc@aSJ6Mh&k3g>=*D2_I;;- zjNX*DR`#xMJRlNR?96NTok7QQ@8ieC)?EE1P*!&-NNkf9Tiwha;}Kvu{2atrTD#1l z7U9@ZkbAZxd3FiZDfVyUU}H&$2M{v)M!57U)lN6gPE`x)_H-UpJ6TlBu&rI#g9b}W z5VDk7UC)c739CeU?uKKBXOQ=d!Bw;Pi*5knMaQv6K^>B(TTm;#e})gKwwrg-G2@iI=_~*nLu2%cO~Q``Mwz|2NfSG)}NGU-P_2)fV_tJ2TOz zcOYD<7&GL^NLn4&+-<7tC6UuJH6_Eq+yCukg7J(46N=mbJQ7v8aR#r7eZ;+HKEdu8 zgxHfDz0B3K&7iFya<)6kmko{=s50LLF6n>b&?fOU(QMZQM2!{p@07@?k70i(e)S$U z?0&cbD{LHKlXLK@#gRh=?^wJ{&N}Wl6yo%`XT|bF(TggR2j#%%O)jCsko??Bs0Oc~ z!`ezUQNfSQ0h9lC4~ow2D|EttgxY7hIHaz+o9fUvBtP53JErB%ERic`Cx|^d3GJz0 zsYnhg;RCt7V(eLh2B(r=oo@>wXAO%liwCKgdKGG%G-obttwkUza;3AGE*3i-Lo#=m#|o7aRfABx{M&ib;ft6K0` zD$xFCl_R!>V7ZinOW<9;H#Qjr!#OPfX=ia585iiixGuh7N?>~1-#i#^JNa5GA|BPx z_g`7A&7%3h?wq-(o_hF#&grvszon%5CmOTl`B`&MsOyZ7*p%UOZbqZ`rgg7uzA+NY zT??5-Ej+FXomLYA%;or1eLUI%7~}4s(^v-p{H$HyI@XRpr!8V~K z%C`DgiW#4>^Mx#th#Z)7-Y8BZ?5Z#t*X6GJInceZR%2qHex6P3yW^sn$*Mu0 zpaYQOy|7}H1E%K~K-FyxY$`s}Zt4d`hRB9?Uj%1t7tGN)N0yWi;KD7L^^QYFz^{4g zE>chz-@gpomG**S+v%(U|9fHQeKdLcYtom$7eJ@a4f0ii;vK*Adg;QR0$_uNv~b8J z64>Ik>3Env(tVoPp4}EoaDn3AkQ&ukJNuo1B$=2(ImW`_nYq3xi9JrfcS*j=8KvNe z*QWWC{F@a2iLE`XMLZWb3falO#33(uTIQ;#r#?#`4KH@Hix_A}0OKq?$G~q)DfM;F zyU9Zq7i--*y0qs;-TdUDOFxBxNmY)JQt8patD+f-`+D*=U>n6R?1(kD7};9D_JJlb8~L$;o!E} zD6ND?WsVT?WLIF5N!M<65-eTYtLlEk7U|A5C!j#$d~(w53sV4|vXqOGsSX=;zuYc3 zkxpm@Rc`)Va<+C&w-Ewn&{h$_sV z8yYrMqlO&+xhE=&Jf&PSw`pH<-J>>TMhxTwqVlUDMgMHK(0hG$u2^8JKkZzN$WI6- zcYoCG^nR{L)Yvvlan^b(VZr96-aarzrSp^HLGk(JL_XNldeHye>qTNjotZ6>8D%X{ zFH4~83L&~4{orA!py^(VyDievi^`9y^jzY71Hv=?Jh3JMm~FE#Co+f}!%au03V)l4mc=T~MMyepo7PhR^SGq$zTG9SDe zB84zdVs_PslF6HZS7*NSU#f9mKN(|jWelLOgVIE7s;Bw zlIdOS7OaJRjfA+$FWwoypu5Cz;$6f^$8OXhwNA}|$5DA<@OPo7!p3t#%iJ|`cAT_c z$KEjb2;H`;mmeuLkJwX~C=Re|rTkw#h$Gfl~=*2%V637*!hD(^k)qCZT{|>!(KqrgGi=qJ2|Tc| z?JSwt)>w5J+mFY$%UaY8{m2KMD7fIG2R)*H!`wa=t_W8em)p7rAY#5jyE%4Migkq49ugxs6KA zPwEsApCVm~rfYAabgEm-_MeMD$ZSYVAs=RH@;Pm53;_k2nu@TUP`IAzC>F;W?k>nK zIJGHWRS0~&ulGd$rI4f_?afrq3U;?li*~=&n(XxsUH5x}>Ht8xi_H6z^Og2KRSIL^ z0;n!h>tH9#JkzmFqZK0$iflnPx6HA?n>biha-$oT64edRG3f*n&ztB;5% z-vGnFw3}Xo<@NR)zUB6sUOAos7MuCu^Cg+oK?Xg9)=k?xn)lu0j&4Ly> z+~4tj0daMfB+>;T(;{H`_tg;@MSoIWPO?x)})z0K&80& z>s8l&m$A=e@sX0+P&_CXzxYh8~*mQSI=_Eb0; z(QNfIcmF-p)GV08aA)>}%71)!J-L(JknvFegT%5Z1J~w!LCud&`Zl&+sH~DFwHfufR}mv>{{Iuz|zvA46*WR)vJ?Q5qSuZ8E2xMH^=T zqmfSR!mN#)mYbtY<~9G_BbmO%8^oV1p@;k%w6N%KzU4h$+NWbWZ-2WwcVt>H@cBnQ?%zRry@WPBK8sOX z@nUzBRdC4M6aK+|!$7^+9dpJ&@o3uzs?}GXi|=Wh#}!QML#5XV2KKU(uv?N%fFY#{ zpnF-N#t&mg_{G*lFFs;e&g+GE8LS=F0p_u^cFT#Hyz5P@Mz!PmMOZG10{YYSVe3|O z`hk&I#-4r(dkcNgI$U%F!lFv=W(P7RX0wtk?2L==i%fKimk#nKtlmZCx+L<1QT#iy#$JIu(&imX8;zz+=ma3&TT(a6m%o+y|-6$s2wVZb=2kr>C2NU z^L^5dXC-5144l)rreHmY98=?Jis}0$P+Fh0^qhz``f9>F5&l z5AzrX+IZ&gNwd>d8$-G3vNpwi)-XQ69j4$PoOY}NxR4Nm^Fozzm~DLq4f}qdp>lSfji=0Z`s_6wN2Vc2@!Tq9qDW9-qsU`-$F-MZ45y(BZyZQT?MGXJ~8*6>0bpsX~q-*R8 z8q41olF6*8c=fo+NEP$f>gY98yv+HLsd_wNf1%)Bs9^P;lD^*WGTACaC{$?NIM#L= z#}h~qGC?RhDGEPLuh!$=^DH;?Qy%x&=`jKRcVBnl3>PEQg?&R*AOz=jU+vFw+20s6 zstE|(fO60O?r6j1bg^@Y&#afO*!l{*h)t@d-FLhvxvHLuYnoTHD0?ZaWCh$iK%n73 zmNoc&Z2L`G@{fn^nX^5BTDL^%E6YFaTQrL75QN+M5G~^H;ao(l>Mi2Lw`hd(vk}>^ zSqHq|+?pkR@qu*h;_u)mh(RgI!MCooRpDBqg+6;n>Efl{C+rKfkYFAmQqTzZu&0Vo zO?T}XCRC~WlRwz> z^8RL!B%OX%+5xJTDm$fjU;7PR!C)#x0u2~goMVleUdLY;+VXaAO@sHwZBKPb6bkqN zhyH9t%g1~lWn7z8S0`SkD2~DJe?bqOA2Nw{k@V&)?C5f9hfVHr>dOEJ;&DlWDyHSM z?nxXx#96>|KHJ7dlIw6hPyF$^>lvM|Cfr**L@w{PUES^7l(FWR90k^ z9obu1A>*Wu5FwO(WbZ9|q>y>+y(*iijO_3G=)QA~`+1)0cU`|f?)y5OPM^>F{o4E0 z$Ep_Rdk`|oNpuiLd3%C3DmXlo3D z3d!ox*eU(d8NwV?G284OCgKZeKR`n+n-7RE$&L?$MIIjw4I(Y!QBOvG1IPK9>u84#mD|txSC0Cbt<4)$1D_n}+sQ6dsm9`rUb`d&^o{3_1F;_tm z!LUb_D~RyOw0GX*s`OkTF{dl^nzPnhAF2RWr@-8ZcpHFSCB);kVsm6@U6XF%GH%!# z)!taWX`|{P56z>Bm1k_rH#GeI?4chA4qZu4@G;Urv>`Wi;m9 zl)l6Kprw5`a*qr}v;U;gC27tIiRF?}nQ4fm=*pdUuM)r!cDCz7+Q=m~m37`-OdA42 z->;W}q8As-33I;E15o6^+5MwSm=~;XzAC&>cQbK^l)o<;bc8^URBP$DQE6-z26r3T zpf>9UNcY!(&?1Dl!*?48r1eG|UQc)`H@!6^)Gu&df za1xG3VA|W2u2=18<5F*gcc%e+TWcBx$EEUnN)1$QeW#2*==e9VPQrF^ML&#VItlxV z=VnAssG!&_!G#;SL@B0EivyxESub$pJDuVIb=LoaJ3|4cB*LJRofyynp|9S9(B*LD z8Nr=%f`T}kvN9}*u?JNOpCq&h$gcu5l#ERj^spwS6P0a!UEa$k{RG-p?pP#?zazzPaAy?YRFWZLTAh zR+s;G4XXRi3=B$*1Xk5xl8cZq4L+*O-Q0YCOFl?s#RZ`pEhMk2t7{6%b73PPpMfQF zNzVeCOzh()0-nJ_Z;;LnjF0U$ zHeI$lP%h*nb6o*_DKDA(LNn*21g3UE?k-l^|;kjT> zM`B57so9(X<#^2|UY%aqA=tLT<(+R^{w_I0QW5x?x3`;R4p>noLD>r425y}*ALEMv zaIJ5!xV+^HcfG9O6Rk^c)2AoX+mbQ)Z!t-R6Kr_!MYUIPHl>qUY1~^$T@kxexG+L| zoCLKXCOdNmY?4b5h2Dk%X$LoOCm^5Kq z(13Ro9awA?6$S}FWZsH-FTm3LpcJao3An3m_YX(HRtB3)+#*r2HPp;h15(4O{RpXxb%Un79-f;{Ghpvm;+=3WMr3g3RDY*N*nBTM+^99G)r7pbSprc^e3S! zrmeTHtVBewPx8-ynUh& z88Dtv2TZn8HkatAGM+ol%)GfDf;%*1VL2IQS!fh*)I`o;_@VansBN)|b3gJ4@Hx7D zn)xKYfcU|$uIP3cXt5IjlbDLN;pDqp&4DD54;rNNdd8iGOH~)?sHL=D89 z%L6{sw;O$ZRoX0hxcEkXVzu1xW7MCj9W@__t4iVl$-Sz3OLe$hsibcV=oEWyFeG>Mp5eWoKc~B%K$F^ygW84B zw$0?RK!ON#`2}`ZKA9itE;F6oNPCn1HJ7VQ$v*&p#R3&g%d0!^P<7aG2Z4mO^-LoN zFyD|YH(B-4ZX1Q$NCiUEZh29nqqcfv@)M%YOI&DD%5#tbq-*3=97_A|CV5%=K24`x zwT;JPeGE3~Jo!>9T6NuZTzPceJl|q+qrIARdrE4fPa^@3)O1Ox%^BYp`NMlXnfX)5 z28`?pOG!Lw4aG=PNV+W1v$9?+d{w`I>!ZV$P-*zCk=K{K!FI745GO$9&nTv}?Oi<*LJQ-_Cw*&!Zz-ofI~cza;9_`@3%1HJ4d1F?-49qvexvqWLZs z$~YN+9G}^^;ilm~{auWiNBv&Lc>6job<5dpxpY^3+wu>E3Yf@ z$Q}3MmiOE%u`5TE4Uk=HiX_|nb~#-(qOZGynqTA)ScnAX-o!=m4s?Yur`y&^&|BGb zrRs-RSzBpL>Rj4yh&6de95hS5nXFTE`I+b^AyMT4Int=ih9AEyXLQ@bjsw9?t|g8%Vxdf0) z?(5WexR?+4yq_kk)XOXXLU?(~_tf>O!qsnJn<`6;3adpyh}}AfagLKAyNP@(nZ-V~ zf|=QTsKkJ!7>-YLt_k*)?xYK-;@!e&y-vxc9Hg|6u$`S9=z9o{-k1bAHQA$}Fqn=^ zeT7#46Fi{+>zSpIdZM(~UdaJSc*HG(+4aLJbs#Sq$WKI591Uh*Pj(qe^*)5@O58gK z>Wb7F`JfyNl6DO^lRE?NKX&GVHe7gFo2W*hO+M~jlxDsyy=_|P!}addQ%}AZ#$asY zBaoP#6^&B{4vQtjTL&y`Z)tm9vP(B<(>_;toM&ayGx?Mal9KwT-)U3o$^F4xFi}Z< z9~AyoE~;+XdLBpN^PmCB@#}%mYYz{YLH+jmVhF6?X7~s?9$u>GQJH)EZ@>Z6eDWB{ z_~=6IX3b=R;D^%I*u7jZbh4bv6n)~D9(R30aIvO7AtnfRSl z?8)*_VPSfuUt8NHJMmX`#G4?1*30A>I6*Yv4XuK2vz9$Br3To25R* zL-y@7dWz#s4@SE)M*Cacet1sqCNM9DKP+1fAq{5nnC{S|-me8MVTBC4z#^hm(OR7x z>Q7=I9GL`>hyB$}U_H*A{!Yv+J=Rq=H4;Et_=UL(a@uv&PQ36}R(8+CgaB$-MVGuv zg&+v6`_DU`F|eBzybm@``&0vM)LU0**W71(F>W*_6W8eN5@@g~f}QP*JW|p6Km&#( zy0Z>(gVckLv$-`Mc~!{L+W7+u8RvPm!;$R9ES!6SmX(@=6JY(zXW*!RDuS;s4XQc= zRr>Ngji#pMg-;dw^Otn$;JC}DH-9q+_a5{*i;mgGMAi9jsnYRzeN(8ZD0Q;U&aGt; zS{^L4{$r#7f6<0E`dx$N>H+@*RMosya#-9YL!w9YDAcA%IKOt!Zl}0qNZ{4w5*5>1 zgV~=pDG%Mn$`lQ$Pu-;6djHj2rgj`lTjY}9mpESPrAh8)@wf-IhpCG)&4QhiV~Q>$ z+_S$NQoho9l5j_Y@4ipgs~h+QZa74Ibs`b?P9L}Q>%CbtMOTHQrMs_Zn-TL4@kJlwQ+ zUZJg>$BkEYQX)?;fO!c`wCaA)6?}<9Lk3&J!Up1~{v@i@q0`g!GrvFWjJ^&w7-w&p z`(?EWZnXNr{#&PQ+f9ir$qw%)y>v4(VvqPsS8L->rE+(8@0r)(-K#q6@FvM~J&fI_ z*IzkU64hs3=IJJj`0hg<4v)Uod{^)jnO0%EIx6Mg2lqAmg^wqsoh^Ov(97B%MCNp` z+dh!klr#5cu%dU5;-=D*tsPL^&@m z6{SGoS|dG__@)_!T|u)FxJ9z6mTNgS&-;*KyXT9>sY_;}@9Czm$Y1-{PzBK7gU~yV zsz))oeea`dN60XyyHwPCc#agd!Z`HQ&UqpG@FSb9RgE?=9aFM zJCh$ltw~(`>fYs%ikGcB)JR|W!;zg|@lubqvL`EgD_$rbsc3rgw{FGKE}n(itSdWv zNV>hzm3K*bcX9a;4L-xY4m5q_1hxe~0yg?-9_zH;_vDBQ-`J=a82IS-O&fW!IkDSS zA;g|jpeKiTA^fb!?vMUUXD&BG?z|8GEx!AM`qF8qFKpdQHx%|R*Spd}QBY)5g(dH*p~Gre53r6|MZI0(Uppqf@gHD4&;vtpJRiuK)!vv z4wa>Pb51JmhvD$XTPPwL2US!5k2GgP0OTkzW7d}UjM`;6L zK3>p_^BU!&dcK2Y9hF6(d^i%+fGpzj{pS)Jl*A(P94Q`TGe>$DbH#Chw6|L{vNN-XTYsX83? zqVe*caijTGdY5NVH(@F9XBcHmXDCsy83?1%LeAfK849}8o{dz3VwBII$&ULpqa!pU z0Tkqt(}5}D`W_3Zw*$d&p_WE6nPvSO91xNVMzYEhPwCX7zQ5^NgG?CoXo$U$SxWc1 zo>LqZZ>+kn=X@AeDE}u4qN!%IEu!*1r@_o-wwY}4b8Zh$Vz*C;lG84aH;(nO*>Utq zkv%jnI_WExm6y%J+vPnOS(RrJZ?9p&cBqD21`Uqdd_bE@r_Z~wrwr1Yg|8Gq0pC)w zYzaL~hzR6jm_vVcxslO2%#q>3(kL8pgyfoe?Rj2tF4-<>>dZ;8Bu03@t7tQG&e&a| z3ApO>ti|wE!20;*k{3tC$1p;IoVNl(I)Yt!P`ogo&m!B5FkSBp7kB+})Z5$rg4UcK*z-W&t*=Co4Tmxk82CHpFDhs%T%74N|da zL(rTdvOHqBe{2AxGK7*zMw+q(sXXbccE85oOk;uLmGz8$60TUU!&V3@m)H>B-`C>j z#HGqAX|wp%Hn$P$@G`3~@i;Wo-tqf;$)n0QwQPt~rS02;hhk-&A4*=} z&r~c0JMkeo;!*pYx>t1&TDkNsYY&{y{KB{|z!*p9J#)D2oA3nb*fE}>i}41*1Sneg zC+4?w4z@}Rx7QxOG) zdmNhb4jl9){j4^(wCv|Q=g>@A>2bSR<5OGOzvvBRL3Ke`kbykd<07^E3jZ@Lp++~4 zV6paY_VxY9qO(LO5^NH~`x!EP@MG$&qLGihV zPrd!hf|xFsCbihVUofKL`t%P;BOt$4Y23G+&p2qsHNJK)x*f7sJpDhsdk%t2CLFeq zPbwZZ5Rv&=HLask&eUAb++_kiq-%h^TOn#?NnqoWm*{Ep{$5GW4V-ni^g+{7#&?s+ zI)$9>yz+*U^51gr{{wSi1CKFm&~FC-gKvSp#gPR@ta8YY{V%~DkuJS9VpR3bg2oL~ zOUo%ie1pHVM{ijF^z2xpqO-HXDY`xA1l%9tH$$^_z;UGo?`Od|3);EDvcs~L5E%*u zm)${;x$l`M6J=3xilpXT=Wl!U*NKBzh3D5N=iC)|)zE6ckMMj5;n?MkezWNxXQ{w$ zJQN^DPjf6=G_TbcUD6gS{1!ft#39lHdkS5^5jp!rczo1L^4rIQ&KmoNF)anL@*eB1 z0%>5@8oA|T>@G_8eLc2fpT+N4T(1_MnuvxPCwbypqk#{_r}XlOd1KCZSO?4=QVIt) z1^Y>l9|R_Gy>o0o79N^+!M&H~dh}t$J5KyC;>dmNm>EltUv@CQ$nKFz&v-BIIoB}S z5DJ|tFq8vjx+0QrM_lIP--u@LUJjl1ju!L>DhoLYk@Wv7kWiFpio2d)p31+Iawk^W zOr)%>COk7qi4BYc`A2?`b0_(da2R6~SHL3*8g*D?vk#P&t|49?86#z(_ZL_@AD%N$ zmXecRCkvAvm7vuK08bT(tIw)n)3{Bp^E0B#CSfd#sI$i>d9d;xg67}n{&@qU8%Ob6 zyAjm=w8&5^O`nmOW$aLYOr;a8ztWb(!$lJgsMjNOa9SzaLmPOx zGaAI```CmWHJhtcG|=BF$MV{RHIbOOVV~k7d7n|`2gr_RZ?s3tq@|i{{!o(VIxU@d zP|GY=(;uD^c~-wyC2zR>^49JnH@B-5+QpajXAN)hqsc>*aym~GA^Mi})ICQvS#w!W zGCWaj1cu;}r)lV^c!s}GnXYi<^N*x2;vzou)qV3 z9KCBKr2-1O$5&od@MVE}?3Bc$v^>710fPw+!lK-TZi7BdZDz?=~5)>4&(SG z`Bc#!Wi3|-9ug+B_d?;6BB9ftD;Fq9PP~e{0xIuEm$Tsw^;1bP&8fL_{5JB8P~a}y zZVshj>ZO*v=zopnKK8zsRmMS{b6s9tl~ZhOqx_9Mm6XYg|KK);R$EH-+b2r7DE5yR z9{-HXiTbJzI`c#)Y3VVI8&oqw;AvqT&t#tro9=CzeUtcLX5(ITX{~s{i0&-sB5AMW zgllEQ74!Qj5bstIhX&8k>`N)EZeCVY?PxC;wPTAt_f`8As)kmr5u(HypxmuAn)OlQ zd0i{dW!Y17QT>jBmh>l3{2V8XzMXu}p&NI#OvY&fN>l3ELO>v9sS4a$T*^XZrH>^XS?2d@n5^Pvi!wN82^mK_)bdzra4ar7f{LiKCXKr}o2m zjBCa1s5Wd5JI}@%2)a@}(i<;0c~g%S$kC_1U|(@V>Abqn`?v4!#|M9}$!)QhT;D^` z%zX*JE1>MQEbuE~*nM^=dP`XfAW@AD)_ytKy^B%Q>9+ZV@J6b~eX&#%?SL6Uu9b%Y zaFh6CItyo8Ms&OD=u#E9_~tDujp}GN3J`m6wX22ohptDbMW-Bg(Qki^jo#IP;s*8=k zBy&Dq0RNSwP-mb75E^eoS-7DWW!3zwFup|G#;HD?^2(N5ChXkL$d8;3PY~u--bYStd=3T*vS!BO4hKV^aM9KB(bdmFHDd9x)?qT`|4~|4FjGK6;da`XT-f*A z7f-}C8=K_)qDo`bM?42$5OfZ{u$>BVe_>}KuEh1eDN9Kgcm^*>WUm2Vdk@!odQ87c z&F^e2&(GT-XK_{2em5#Iath8(ULH7(to#umMm1QVE|~xAu`7f6e(`R-O|K*T&QGZo zXe^%Sy0fdPAHSUIw%MQ#y?L1nG4J%40z!!LMW}lhUr}uCW@~zADl2LR074c))7i0g z?nsg^z7Ln7R`nKCnyKz9EV}B>5p{xUSwi(dpZAzFA10eFDgw=P+FHADWn`X-f}u8= z^s@0ak{Z!+R^7@<9cvg7jjs_0{YT^-hJ(W#B{|!~3PNm>3TJIN)E`>f)#0KfpXQ@s z`kmx+;tMV!=jn>fyW37b75PqB`(OqPfc8(P2mF}yWRIfeTMVAb=9a%7KS=gmaHJGl z{um{>Czs~zQtWIwx&wu`i*47sQTIWivbEtkgD)Zd(&DFsyKRXr0&|{7=}f+^Ym?l5 zu&CyK<$t9%0yM!)=^dETY-I&=R>!mOXXYK;k*cA%?~8KDPDJ8L6eD; zm4ElDq;{#i|5Yc_H0=bO*U_4{x){I%q!UOxe`hbv7G`B_PEzr+BpHs{T&HD!#qS!BuKm@JP4 zQzvA9w)N&G`7)S+jV>V&G&mwkjpidEHliASxMJS~W-4{)uV$rozNgyh{@_RnhRMWg z)kj7q#muF7ktST_TzO<}{ZiRs;MH^QW25-!+?qE4X|_r@p=)>`>&$UPGNy6t+Fj`T z3bY<1^N5c5Et#+U6U_m_J}R;SZW<6}oe=vDo7ietv_EZ{81oSTMy-7H7b(r~%(u`% zm#HhKd_r0w0keSv2waiUNMjUJTdg_)b$@{JShFgWMY@tc| zQ_Eo12m1Y4c!03deCCevC4vA)y}4yKM!*|hoGsfVncb#t)m)@Z+@pgFj zyB$efY|L=#r*VmsZUi}ZE+E*{4bu}55ig?h#HPF_iTbLk_iCvb&%j2`WJEd%b5Or@ z&0_M_#gt(zYVLs9cpN&<92+j+;II*L$MFb8iL9S?PEvNa&Ec&+q>P2GIQk7&OCQ-2 zM$=i(1zlG{zabW^i3ZGhDU`YIf$4Ba;8JLOs)9dp{GRkI(p7ozA%P_~t58)I11x2= zW{L5FTRKH&$H%3c6{C#@7-s;(ToeJS({{=i_%LWTR2D%muH+=*FArk=7wRiM-^0d) zHi7`CL(30j@dB8%!-J&In1Wlqd`5`FBQVw4USFZ};}z`Gk%xZF6PowITNi!sN=|6G zdY9q>7Y*7AARdw9yENvEE2~we#Yl4U)qtNZpOQrjoOtT-^U*%3se+j25Naa)4LrlDzqwpRr2R3`44o%{*pTTBON zgV>YdIcZYNCxAPR|1z<`6hhx|t4kgi-UcD-aS?utLMZ%+#@tpP6c1Dm)x(2D>OtV6 zhLEq9m3?-6?1%2I;dD*Lmx^fXZY0-#tWatSqqhNs*POtV!@ph-N;KoHTP^lYo%OdNrlu7xqtF^EasO|h7)7Bk_%;bB7=|4-`z zs`pQaZ7~nX>e^m6oMDn8^G>!g%$JpH&kJ)B&efsQ{beo1%Zhdkl!jGf1C*$U^VuF9 z5?C+~nl(;K7TH!Z%NK5qJtKNhzZ$pnA3|1j;>gA1!>afK%;CqJMVqOnt#rKHG}YugUjZs`IUNq1!*x`!b6p zi;Z*RV(GQRI!gjTUnv?Q0NwGh0zJeLaTyOopsIa^Ty}VCJ~9!{NAh|2tmJ0ih2;%% zE4^8@%{)8$q-!K|i{I#W=#5<5BPT{J%iyXmIAZIQ!ucxN1TcF7zG(KJolHBG2BsTv zdu}tI6+1pS`qz4g(VwrRbX)kj$2V7DrSd2)@q3gyT50`}83GUnte^i0VGdDcR#Q;# zuL?uK8oUN1GTmC7@Q@m3_hik{f&2IRBZz)-*nMqjH+QeuV#=dq2%#Nbmk0}%x!we? zMrvm&!CY11j3!szRu>-UdK9;*u<>3^#fJEVYCZB?yyY%f)P6a3@UrfrlkgBlN8EK+ zI1B!tUbSOpoIy#ou@B447;KyI5LRs;&bh%Vy#B0av}0VE;I_r0{wnSS%f`7VsT6-a zv6OEbJ0qQ?Z67O1#AxZByu=f?G-~AZ4_F`LeyBymLQbjutG31S5IOa{ZfRFWetZ-}0%Vy9%Q7uMdC2{M9f*^_Q+PAOtLl+AW`zd05zVJ&ZYVXJL zmzEU4#f4?m2hzn4(~9^SPbXJ8+c^j4&5A;Zg`ib-lRkN{N=HyrvB4uRF%v{$yq z;|*}$_lsGZKDCs4npil^?X}9JKe17AV13o-K8dZwUDHrJ3F6-cw_cG{PP`PQS2Y?J z3X^|jGxfBIu-dJzrHD3g;j1+X-s5^QBkc=Y=^+CR?5%38UiO@L$`QZ!_)n2kop7xz z$pLH9fAdRWIL0>k1Wkejh|4mG^HL7-e?x@q(-gT&FL@kBH1yoO-JRa5I=DgBckk7c zGNRmGS^vbd8L^~2r;#h7M$BQF`AD92kr(xqGi*VJ_wKh5yvqt@cThDe4m zEQKL5=7#hSH~GV+%#Ggor_wy3o4k@w&q}5x+eAE^@YF7gNQfz9CkZGFhPDLFgS~^!2UU&;2sxw^;!g;V2(iEodmuG=fZSO!6>I!a_@Vf#o|qp{tnhC!sV=S&O1VreT z=T3_~rXLFpKSLVyAxmuRjvu{9K?9k60hWJ_yax%7EpyJe6ilo5$|+ir$Hoinpc~7y zV|08p9WFiZQz^S$mGjyYHRx}A+*ry!@hno!XiYPlFlEJetImpPDpt6*!bh_C_8Sp zydKbH&}|yNjSA?aOWH-0>0h~1Zi#97Bc)&|>Dd~tvEdYyYa`AV15cTyFpnW+9cDcq zy-*~1nLD&XII~*p#Zqo;5kGlD__FQjc5UgcSJLCsiYhBc@p$Yx3F-M2OrRI&7As|r z5jL^Sq~JX#{@&|1XK}p-W7qN{m!QFAlOK(^oos)c@}c`9#@V7b_G@RnRz|n!^RP*!eoZJ8@^1EY3+`c^CV4cUe2uhb zWgK#Dy|iJLtwv-?whMcwXUlLWVNq*2EO4|_kE_T&x;Dz1<%1_Z0&$JdMnt<|LiidH zO?hav;rsIPuT({mT-ocz?uLG9oD1fvZIbSC!h*`YrnET>G`Ffl$dTmmKXpCFP(_Ra zXTupxn?+|*kWXes=+HS+J--FAz7X}_Ia>%@c7~P&kKU`4V+HxQSF=19@BE%nESD!^ zDq~|9L(>>mgly z!aMPvK{00e8PA--*EAamQ&)usskn$xnGN(n$a5kvy11N95A$0Wj?nX=u=EVx-wM{c zFC(sUk19COrTH_ZiGTICLip9JW+}NMD;LclWgz4Y{fm@|*n0cTW9WyGXPOfec>zqAfNUq?C6 z^5(h&u|vjXT-Pwe;VJmaM(^+qhc6Dj+qJ2+TRCB~QCW00B|Z)|b+e14bBpW5GL{Wb zX-7mIr|-B6_orz@#8BTNj-UX=5#}kY}2PgqPb#wJYl4}9?`hIX6izg>jN;(IJr z?W@!uO#XNe+gx_OY2avR8k{$JCc~(KaLIY3cgxjhBvF-#w=KwyceFnr8Zynvr~!)g#>S0m>R^2Ozy zFHO^uHRh9*EqCtw!J?x59Mgxyf?dV+_*QW)`-IET7Y3zmR7zRO;1!_;Ek647U9||h z`}!TrjPMVoFC;OE1{+LPCTB`~?f)EukfC0L68>+Hc`3S*{!|%V_TAPXSYZt7tbZP* zO-VPxdE@#~J@j8#`$7jHrsnfUqfCzl1qeS<++s9joBq5|xyTqS4|MkUC^x}wX^-9E+Z`I(*g-V#$f#IR7 zZJ3OHB?fVJaD_%hUTZFKu#qP509_H0G)Nm2U5cW5XHwZa3!D$Ho_LBRHF(Nx8HpHB ze~5MxIov%c{8XZXXcQw*=xsKz)RBNGt7TPaS?5`paYW?$|*9+s>2xFY} zOpC65I;Q4?$$kBcePHE-4sLj!f=8# zVKw*6c@=>|zZoqOfjf6eUFcfA&i@2f{zEG1SkZL9c}_@3zEG+bgh|hR_2%(+3m_Z^1ts-njVdW720bi%@Yolb zC)pcZ$sDhi5CiB?CZc)Du7vDuQ!~d*H4r}Rr&#l&QTK+!A5&hn+VD?#A(-@Mn(kjn zA_4$-ksq0*4bZ_0+o4%27AmiV-o<-E=mKCPk1eCkx-jzXEN6P4$_x_YtymCIHmEtA zew)Seggcz!E6jV!1WDW4vICxR4ZFr(1=(2ok@RyZ?@10qE(4`1vMcU;fG#AU286$L z=onRng{$QZyyHaZLD3`&A}LeaUJ}yQdx8Y1s?= zqeCAcCnEX`TbWSs6D`zS*e^D3tACdxM5@~k9sX^gc50#;|vlJYYnu@a* z`y|gVEV3c>IuppqCqXUhuS*WB1HClzVB=4r?=j1XRAZ^GiR|4}K8=13L;~*KrLP)7 ztOKV>OsA}7-V#y+vTcS03|?7!&6Rr)O64*p6N8yrX8~$9nvU=nMrS^dT0Ic{JWzWx zrJe)|I~R58$?o22l}M(eU#xH_IqLjB7G?-X;71dRn-F2%lMPbQ6qa#4KGAFWwi5^MeC`38nUDV_!vc5x!Th_`M(% zPa15Ff-myMpU6Bq5|X!1nI@r%xzL`H0L`1`*rQtRx1qm@>h$1YLkvo`seJ9co{UXG zZH06C#8Wb|fB~a5GdG2SXnwgs&w(AT22FU6dNM>ppcdr!M!J)uK2ASoN=QD>X#cLiLP0&$&&ljRuAKsK?Lg&`C0P7) zd{U7dU*?w3@WMjU#Wjj}s{xasBOJ%bBZLvt6!&~YlY_6=_}Q49jE60v(t(xR8I~9l zO{4^>c%x$H=D*Os0qa~c9_ikS`q(ZO5}AofY~&eafRY!#I=zcTI}i)99uhA;t^2^B zFNe5L@<>c@(%dS&)#FHMvY|WD^HIbZlOrr8P=TfI=Tl|TAEV|5Ar`#j#!N?Y0}IAO z_&5NYgjoE}r&rHq(P4-MNky4Jeeaer<9gg+@sjnrGj^3RYLE-U3Z*iG|(^nfH` z?-R~+yc#BJ#YZZ@)y zUyKnTK$bwfS!5sO6UVFQJ>)@IZX(+Cpwy&5Yu;Hi`8>z1_Cg6AbWi{;?%+Y@>hRvE zIi^MM3Fb;$!2d#OP?UVC%p{Svh9w13M_Z#yI#SlmNBd68fUfoowr-fp=oH98%46Mfzl_?J*6eDZcAg@1q| z`tzZq1ngvjm)@;xMlzvD#hQI;t2UGhXLctc_l^|zN&UM6OaRTv2nuwWj0Mw#eKe#S z`&ySC^+_3px@@8$U-+2;MDH{*?5Ed~b{roXL!ypqjtcojhv2!O%M(}`?HY4Vg)Y%( z)TfuL6Px~tOmbK-dIIOAHUHT8m(&h*4qy&?+I36|21x-K*L@eIi7Ck}DY5f=ALkhq zl6@(c{s+pYRKU`zjPCCIBqbc`dnuhb8Xca#&|X5Mux#%+`XluI<<=F>(4lenp2w>z zc=ZRUZ$-glsKaYj4Wz#G6$nsa+TSc{6ieT~A->BaCFAjsIT zA@nixMrza;??Wg9Rl`jg(vL}dIjACi&de+8Ik?4)D2e2&$Fi{7j~6+-`i*o;@f$i} zph)~RAUu)YZiE0jppa3J&S{S2gdSd5y>s@iK%BY82rjQFaXbV5ml%aZvQK&Fm(&@O6kE^p|dd8T}02WM_03ayW z^a5tVK>?)Gkj|FH?=X~v74-^O4AIc)bhqnyrb3b*28`!fPob*=a!XEZ4|B`3@;LM+3PIGpD_^NRS?TxyM`8O~i8}lNnupvmmuVLU}&xX2L$Yu7Sl6 z{)Q26UemEXuRzxSfGGS(pQ({xGe8HTFOEICl()KIZ5ttk{$7-hr2(15K!OVxFe86K z1iagVAeW_rSDm+x!0$qPi%2NBIj)kKR1C8&G=d6Y0S}ks!?b9>QQo3AWGggtto@>M z6u(0d!zNGBNSb6Hh>d-Gp)E~^j#ZS^L5+}CliePcK)-8Atgp<0hq$n0*?rf?1iA-J znk^{oF7kx_X3EulY>tp&mp8Ea*mUAj;V7Idz)#qPS7D@s);L1tnk}fJ_eI{R)4W4d zLUVWR$VfV$2w}(+VCb*`7LXkp#!z-8$xDmCVcnD4=(sZTloh3hED`mqFe0Wxo{M~HFP{-( z(&`z=dOH%|r3}MFu;W58;|Q0!&ut~%I zI(knM2q!4j`^?YK+r$slR50qC1xLw9_hqubzP~{1Y~Q;P|6M;6B~#I;1MmTe;PL9T zV)#IL?8xa69UpzyLE@iBKaaUSwdD&B3u$%UlO}#{LUU3*ZIBy{4R~-Dli*YqKvQfw zLj;r?^8V4&P){P~xAx}HZUK+kYgfi~BAVqmAhoH7Q-$H`+JOCH6cJW=bXCzS{O#3e z;t0y-;BDvK(e#!`Iy61&lcVbuVZ&-;Z*pOM^StW~M$rJb@0lq%M);-$qm{ipH zvyN>?WIxVT;hyi0xN;pUM_?8{u?D_2P3AJ!`~F%R^Ar$oBfoJCaD$K2=3JvFBh6JZ zW|;`$LWMsbx|s3Re3%N2U{Y~yjZR64MVLPzz_GqiFKWW?a)IHmP(iR^=h5+gF)_ng z1R)GrsnTH@h#gBkNhSu}A+>PIuk~w%i0lr_LG*>ix*HHY$NXmnA(jnGYrG0XDL4z~ zWpd3^Km>8`!uyn&Uc1^gVZWaEh1Pury9&84^Lw=CzRRpm+uKQa;>Gp8&{x5!XIgl{ z=?pJL9lNR-Sx~dVm&7u6`u(mM|C5K&QKx{|@ftZO245U+4~uqHYasfyS8FK|%G%q? z_w3lHhTeb?ocGpSTL~M!i+BUKz0LO=N(T-f1#3b(dnq+iG#SEJg!j>@Lwoe=wFUeL z6da-*bKuZ>XVxS;EUWUFGItqpE|OP=wthS&W}wmn{`ygh`g0KBq1XqqucxjEmPHyg)XWX&2XngO$E(w*1fgqWe&iH3of6*&RnsKTFt zYkDSPEdvkr&y$gh5FLB9?&p@-%w~&|P*`}wD!|{`e8F)+AS@hGY0k%lBh!P|Fp7FB z@)pWG&g?2XV|g70Y7BodTD4c7k13edJesHZhnZn7pqUJsI3jZOZ{-jrfWT9OwU7D_ zkNz4Vk5g`L8+J-ww5kX=Sq9tl#jC(jRCiG9F5-OmiT&T>q3k-#p@6y5ocAJqkKjRm z=2ne)Uy)DAdFsWHFm!xo<44q*o4);U_i|KUF+Zv(s;#m0Hf!kp`iY3ei{meAeto-D zNtYU{&WclRet^&*$ zNyv^XWT{O6q2uW{I}R|yZV=6j=X#G;s`oe;4;Fo61*$Y$noD2UwO{z@u{Ir1V%d+m zH2t8c27ibFC*}@d3iGQxeFQK`qQ~(BQbO}-ebwg|^%=y_Tx@7C&iberD}wch_`~3= z>xK9w;l47}&$2)03s?S8IDhbOsRjFH(?kmBY6GFZdY2Yxh`TaR`cKJ)C(0i1(n0`6? z4)#k7-Tw2ibN96PB;MT60jUAoG*OtRAH-8y;F&HX~Dg?y8!XOXIJS(HIJ9XQye27*-_do2o z5RHMoK>6Z-2nmr19%j4PC_4k<;(*-0N)*4rry|&!(6YRF zCyjHAzxP2<*s+lJg^uEv`<&{%0;abbxu!6}`V8q>FNC_F-&hMJJ2qPGtcsxm3@W7T zRWPUk_B|!xeSc-5SpAoL$HRw^c#y?t*aY3k6tsv)iyDZbtT?J;(;N8#T)u3rA$f5H z75u-#D8GpfV}!gsj6D@3e{6Aplk?kEHa#>KJpK+G2=)kYGT+T$7xIiiRJg=Ag&ap$ z5pn*mo`>0su+Za6A)f;ybi|nz;z|vJPHh;{%S| z2DDptC1qL72IU{pHwWmlsj0w1w2ey&0?<$A>q|Xv^j56XPkCNI22Vxc3HaWbpPTQ$TfNv==&rRsKx6JPCx<0OlKC4%Nmg)5Q4#% zIQ8NkXWp^dp{j^u&uP07!P+6tv@y|vxM*kA_qCTD-NP#OG$U~A&_>DbzqYUQ8EiP$ zGTZ5xaoY$0?ci{WDtwXa^&5#G@Suq zlV?HppiGYS>Exfs|5}U6@lfo0#uplHLf8PMqmO{DgA9Q1@ed*i%Y7K!G9?uv!*7E=T)TasLZOl(~ z35W^@@=SKe%To_@(5*ET#GDmiAwz~v9cNGaS5SmEfLC8v4F54sj$kcP<-0p=ceD6j z?9Pe5>AMCAKve~^9=W#%c+RR8U_ zJ?5zYeu8m=t#?TTq!6uRJ@KbC>KrrH5$-}>c8g^fdPV0aU{>c@JddQ=J6ku zL9oO)9C!70OuOW#+`Wlm87^wkvEznO8Wy}K|LhRZFM5ebL+c#;Dxk{K46no zJJ{M|cK{~h%n7~~{oer!gfS<@+jjwr*R}rq$WjpoRv6Y-gE5!SD?2VW63^xocM1u2 zh-k3p8#>rm|G4oBY?w;3w@+}NRfz9W#?e)P)&!$w=egwn{_qa5 zQNAeE8^7?08N}MF_kOm?Q|nr1rnZA2ZTb<-9~~fI`&t#9km;@Q%eN&L*nfcqbfw=% z&HVls1Y$wzfFP-l{zInFr}aaHCcz$Nj;U5@(M^#CzpBBugds-PA!Bo}o^9*!9fQ78 zhiC_1aXUqad8Go5^zW%tPGIj(%zdBz(u^CZMW`|655nug^cZogcOk3vXL2-~$4xZ; zeLGZ%aGri8`IR(J_V@8h7vV*-e6a28VDhDKrzL0Nlb=Pct;IcHUYSMHD`p3<4O8v+ zE?6$o0mi>BY~iHsw2O_}nm6FmuKQD^q}TLRg4u;RVaAyvU`>52SnvG4??Ln|7%sPq z+PS7=aJNxL1}?wxFPY+Rvv@kx4{ z9AmI%T(^JHXVT}e2DQ4o1I+F-N&qYB^f$DO8WnkSO~Lj9?c$*aK{LAVdHDd3$?P^S zXXn@QR8-w$eE1BtXyx;aB0pm3W$A!!TQzFHHH|2kS^rj;96>*q2Vnps$yT-~EC8P~80Ij}bHy-$G^E!5b6H zjgtN7oK09`oK0jK)k0(LLB&OqGg0h0?lKr`E^LQO)iOoMt*%Og8VLy|#;Stx=~qoq z18#kW$6ZG|7Rtb)*xg@R2~l2v`h#^3oSmH|-f)$F{Cxed!HiVHmijnwzcd`a>bbL! z-tqrvd+&Iv|Nniw6iP|MD6%TaR#sL>l)blOX79}rl2DluvbW4*9S6q>A$uKrC-Ye0 zko|o;z2@iheSbc`??1nPx}8wYc|M<1GW@n2|1EXPHvjhi8%ZnjbOe;XPn=wuYM90W7 zOvpDA-D{*62HxSFAo5kKM=>ZG{Ki7gL|lliK&7?`)0sIEpWotxMre#9Vwz-vt{SBC zbRhfXsY>nH3jI;wmJgSJki?oPW!qGL@(-PNKhsP8Lj&~KpT)O>%zNj-GqHP)K$k4e zIF$czvT9pLsXcE1{l-}2a6>$ueoEK3g!y@m+YvB(<*qy241z4iU=lGMRX-zSpbKe? zsnX=mi75iHPrc^DqQaWqAid;}{BX88B#3=k0Si|l$&+$8i~nJ9bQub}I{6_Z+m<8A z=@)QObwgBJS+NZ6t2m*{)LxlqD2c4aQ+jJJL6sR&Vf=l#B6D8zjRtQBglKH<4t}Ho(7^|MYnGec zuGpoK!Z#QZL3yiRA6ycYscmAii}zs;O=K?l5<|nEh>hOyk5Q?atX8iY@v296l?tv^ zq~~vrdNBx|e{1F^^r3@X|CPf!Fg3Pc+Xr5Vvm;bK67B77u_nv*j7myG-2gAt}^6Ywo{3(`wVl$D1B(IxT$N)S?(D~$O z*VB)H_(MS+&QC-zSi&;ue*g+)>cXlT8*>9os+1}q>U)Qz! z9!}rovK;FH40qumaJYJ!`%R_|Gri!!|AR_epOK;WqFY6MP7c1^ag6N?4vMOwx^fNdrEoBn&Mtb zD*sYQ;kifo zou}kxv@y56um()?fC$ zT=>`O5OpGYTR76d)3GJE25e)u15+*((pD#P8b00N#i^0>yHID*vo^HuY?}!P3d%+A zFV1YM>)4R#O)Sp1qz8WP@pB@EU6t3`VXS^tFf|2ESq45ODb7YQLh6^z!M7sOAtEPx z-S@j|e5Jc!uuBZoe(QB7leF-=WhJhreLFm|0eJS7A4m6`lrM3#{W){tTLDg4hZ~Nk z#TGi{wJj#QOg!9KManqS8zftZsn5883hqVqFE!0{Xmrv!_J#Sb(V!NS~nW#;3}MSY|JHDlU> zFV=k-jYx0?9`mtWU60d$<~9ZpvsB8u`HcQ2XrNBMU-u|4;BzQI|z>dXnSR({}nLTE>6r<3-TWhxP1%M zAvJrrA^o8F;fEZZAP!j(5D_MLP*8(0+eg(qVt1n|O<|sGOkcJ|B8ik0l?yK~_ugU3 z@zfg%X=81%8iKj;EIWuZt0MMDQ|JENI7zujJJJ8PYxpNkcaaup|gA4E9274XJX>Eb?f z+KyK=z(IB_50Rn=$nhicfUC%NUuG6>R3MNVqMxX83&Te|Ffj4OC%-_>vKQyP=IZSH z@`;;StO@yot7@I{0<3R`?6ICDY(Y}yrK^9p)I~hgzqptv2#L6Uuj%w)Kq6kF!(M@k zib$}iUtTaagm?6NvMa75{0mj<(^Muq%$py@Ei_>9Yz2FOr|u};(lQo7h^F*&0#UqN zRt06zQ)ouGic}MGxFJY{#g%V>gEba7h;y1)HI+g|2G>>x7lqj<6Ye;LE=<{?`Ieci zjZ4PYO8`$PL|b`*8*~&e`bgI_)}-UB@3|1$U^$)v)lA+fi(0h!z{b_l&|_ymB{n%{ zV~0U4Gn_Dmj`hRzU(PT#H}AbU3a9V%fuB|6`592cQR&ms_F-px{gDczp|-D=2;$KP zhu}Co)TkNKw-=Wa7CFy%K7O^rqHfu1r6Jmq*loll#=lf~s_F@afS!23Whs~fmrMGi zSEYSrPkGp!Rn|dv+ODTp?`}}n#hPV>g@9TT)nxFsIz_U?7|WU6HkQKQK81D1=(>E{ zS1ea&A%FT9Ie~TOEBB)cy!sc8AYXKm^yQqVQx|2}%0Bff3eLbunMvVW<4)as3+N}a zmL8ewY&iSW8BJ&7=)d7tRWN8xW%~}_E$^`U?3sNzT54O_xy2bIRzPg)({{YHxC{um zIxY|HsRO%osFtBW*;D?J$z`!aVE&2a$e)6Op>0uQR>)5Q`0h|%v~)LC2kA(BKdtVB zhxPg63i{!f_N!#FG4KKY)PO6=KF1!0y&w&2#KR_H=-$#e8xN~q>Ybes1fi}}-Gs|p zMrOw9!bBL-=KL1AVd37e1~|n3ftgaI@g~i=8nDRj7Y#mk)E3FH;a3+&@j!U!=1NQc zh8gX+2dLp$Q-;3=G*lb#ADlsR)Uc$B zPrrsr*&^$09LOwS&Znv_7CVzvb1fq0wr4tV!(R#3-`scdsHI-3OJ`u0$1%-Hc*kpF5kA+S=%{VTo(%>;-*ST-U@qeJ=C}*rd5wsQ4pSs zF|%{ej5gNUb*Fu`A!bMsKUb0S6er}0|C(s>6)2RGsP`G>YnF5K)D@TQ@&+5yFqMW? zu{h@^%p;Tcg~nupzpQ6Y-wvu}uUaXoQyV#Jc1FIu zgz1hAx#SR|*eh)(@uAyXc(ENojqv~v9UdH#Skc{r#1dQ7H!M$|Q$@zJWuqkJ-_>_N z#(++(_ht^l@ILcutXRQDnBdxDW7=uKs%C}JBamIo$ZHjF%aqe+zdxC1Z#jAj9L1Kc ziP0EKapTflH6?DZ%~nQl5&}!Hq-ac2CVG*C(FNzz@v!B0Bi^T6@uDQ{kUYNgHbVn< zpB5!x)e3gOo%BghV%@xt8&BEKo%^U=GTfe9EkVjA0@5#U;ha-{3aHQc4{$x>MEY)0 zd5xcmu?oY^+`yPa95pyZ0s zhG-pQ0Ma;Ro!P4vBfzjlNO_|BI>$KZ zSR5sURW0(pEWL-*jtWXV`=p4H2^b0(-@y}^IzKNig)fg&7G7e1k``x}Z zT;i~XipBIKfq@(M{vjNJRfM@oziDssihL8^GCmk}9}cscS)cYJ35~L%WhI{{Y(M@L z<`c{o!!rv4=63gSSPB-Enj4VE`a1Pjy>}kq3PqdQWx4Yz(xa>uNoqh;5~^4(uZ1v* zLXZ8>8~lxBnT$^Tb!X9p)qT{a`Uvm4uXa2%dP7$tXGVe=hh0~j(x&hZd!N(dbhgKV zb9K^YgG--h8By_sX+V(dD?2+q57&fI&09N?Z{09T&pmR3v`Jcf2&}(F_wQitz!vH| z`KDG9XbbdnmdshUDfD0UVnLKx=?XZC`A~Kt0=Bs?gu`IgEi4_H#vwdL%~43s+(3!Z zN<%w@Cn|ns_z6Rz&pzv0744B7(|kTCaL9yk3|X&{L@d7v(%)xB4DUG>*xBU=;GAdg(il>8{G*ter=ifhME(~=)Cl{k6~@`_^7 zdMurovg{qE!x)T;M*2=W!6Es8UAztIR~{62PBCXiI|$$=*#V?4E{S;ffwq2KyUnzx zlHRlb!{d{l(=)Soe?s@5oEhI(=2&X8SlhA(UL znLo zARM~bMU~`e0wj@k;dJ>Nk=5wvj=1fw!o%_#mUl;M7?a%ZsK2Fl8jJ_9Ur@A$Tj6pV zX3kP4z+FX-N3C~rG=TbmTiumgoWu&`kloVJ;45wre|sM51%MzuB$cv_3`wxxHb@8c z+%FY&a3^|qVTaEh1XG_Zcauar)hsugb629z4shb3wdSR9Pi%9=g~Jl9l37|yYw~`J zan?H!=@DMJt*Ks&L%tB;Q`dE9YS`nO4d&X1&5o)+?Mmg2s14FJX}|xeSfU~g`sy*g5H2X z*G1px8`I67j($g=(?BoDwrevm$Gk#$U4DoZMP_^d(wOvA|2pNtUg!~1kq{fx=drRR zn^kPixT_Ru7m;>^S#JyzIj*p9Rj6MCJ(`&kwVxpctj^b_`y2xn@hCmy!W%;>%rrfG zy!~PVVtaf*epv5JqKgNp`J~sj;4DNsVhy) zmpU$^IXr(pxg`q6s;xbt^6q8|*_nXZaSb>5#JXx4RjnetC3wm>u za+V=v9VB(3Aa4?Fo@FrxGs&+;4vOJ~Y)5_JG*z+&{1gIU0T|Z`*g_FVPSL79mxd~@ zB{-qQa+97o-(Y`%pm|36X-?Y0gk|xC@mU{G8yBmshF_QOPSJx8$W?9-qf)Nos;}#@ zyBEfH-aD@?|C&l==dSD@i+{XaEhcy4_pjN+Ivz}hxqVxnr7ImXQDJdK@zjU(zlm-k~ zyTA)f+6rRl(2_AN;5#sG1(W$2TE(TxJnD++H_P?K(_ft=W5!5m=3`XpdOYOej98T}%ry%3D60l~;ao!kQ@x0N2uQC4jwh<@sj}8Kh09w98{x zo4FzXn0`mF5kwBT)aWx$cAM%C-3Qh2YIenIyj4(T<7tFb!D_vqZo9xRYcBg*5llq;PI7F`sd1BBNV=J4_bU-Jo9?=~tvvBAoctQ>`q; z$AcuPPe!+d-A^P$AT|fceE3^kRGtZ|XX|{+Pz{^SvKiVMW>~e#+=GV~67KgpJ)JO} zmkKGePox?pA$HgV%n75q{#FuvndwA9xT8W@d4#B=k2nRHVc?(rC5tTB1we>d1Ludiq zhK^p53`ks7hwlGTCAmdla4hPkU8UzN7R5t6gN|L1^qTRcsHno^|K%P zZEqrhDs}Z9vDkH(hyXRUo0; zhA)y7A#!(abnATo`G4V8-xLy6M_gk|F4H4K%5*op^ng(hS@ z@t+`;A#DyUsD*|!O49}x_BrANc>y;bXPul3Nn(V6d0nW$a<+2b!B!-y4n$s>Xp3N< zh8!nkQxH$kUPz{eC!5^7vN{X$^+z&i8E};<^~+46!#biFye?d(@CQm<1uBO zLjC4N1qNw)SKb=!HZKv6xH)FY5(je}ML9kMXQ#e(C5TLY6iv~UjdSQnEqkmq9H_7E zB4$nO3~qTEb8cl(8h&)PT|BS=8+xa%+t3>+lQDJhq%1lXD1S0*p<|C5e7ow^v{(bTf-bmc?}Ns^&VKXgz2!$p!t1K*l0Nl7E0Q{(p)$Lz zq{fneq0A_Y+O@M9a1Rn{94@QMNPB7sI2qD1qvirx#T2L&baV;v8KSL5fVtkf|4MHu zPGOhu%}Y-)?tL3-IS;vpu?sPC%q;|pH>qA*JIg17yt{coxLqjxQ8$F^f~dFcgC{a= zrAMrD3~c@1yIUbb6r!}#T1KB#@Jiu6tOa^;(sXWqu_^q}bx5bOD{tL>rT*p20737~ zL~(R-#3$W_nCOAsel-o4iE}q26IAau?@(TgxADC59{6Et7fhn)#jW|A71-Kq7$+hH zf4sYUh_kU*%)kAI>_=OiZjb4O6((8x5Asb>`K#o_K1Dh2esxhk`(^Nk4jPcy5cVrB z;`DJP1YEa4wzlNh6-Ioz;lb{ie9SH=;CWnwJBA-2;%R66LFvvdqMf`UfcA9(cpPbo zv!Q#x9N(?6OV8*z+?~J^*wPl~(0nU|`c31jy}6irqxXd<8I1*@u{h>vpJ`vhX4|VB zptO^$9gs$!b!a5!`}Tol?zYa%?&V4eyR5~mZ-C{GKz>Np$IfPjV77)c*R1imR_!l# z#e1SMKP~m9KcA1FGMo|Mb5>Dy$>FYS8}u-%8fVi&;?-IQRlJF$}!e{y42UFd*N z`@J(yWO&#Cx-@Kr(s45mbPbyOVzTI#zUs?i^RZ-pX>|OGr#jMh`zHF0YTD3C=4o)M zF%uDdb{(}iI&-|?Qj4uvAWj%WMRdW&@?t&aW6e51{ccXWSZL)9o2{S;&b!SZcrU=` z$>^kncl-B^LaPHUgbWucOf#c5`NmjerU$m5oCw!THQ}Zt`t>qbhvKJX?itH(X(q5; zzXlVijhm4=sd;h^KycC^QsRy4h5BK1+vDTkvnuCqAv*whlMo%P( z$$DP|wa);*pv-nTwPZtuHb2R&N8g6m1QX}bk6?VXE5jjK>FJGS?(`f>GseoCjX=g? z2@&(86NO0>x@=8k+XhvQ9M&xE>4RroFZUZf+Ra<828bJZ0ps(D#rOB|(;lRxYD?9a zf2WIUBQ~|C<}16c{PnqZ?wmuvE7oJBZ>O}+W5C=H^8nQ6hJgW`fXdRyWbKO26j(2L zrM_Z+wRjaP`K7j0nxX7Rf9J3{^X}FSw!pu%){AI7qP4-s1#{)_4JpP%)$fDi2KZ`N z1(InHze;n$0hikaJoNsMaGZ$!^w)-c1RsE(>XcrKax=NRf;i3~Q@&T)V%Z4tjyd&? zD^njm2U_pPcYqlC=9f0~NiOGlu^@|-@uoo7IyT>K!PhZhp;;o1I}fab&WhK zfhyBAzpo^nZ2~Ebul2IBH^){weTt1}zs*TVdX;wTPvkkpim1YSSITI($PiMx$UXz7 z0t~oev3*hRo5@f=CK|>x79|OL2AGl6)PV~Vl;w93KS(lg=F&XRvRYk}Zb1PSMhjyB zg@5^Sd`J=@Wg9EygQQHe004%zEYkagqbA{2rGOB@k37!v9t-fJeiN&OSa@TCqbzW8 z*BkZ3-A|d}BwH+`VIXskW{?lbI=ei_g%<5B|jA7`Y@mNVpz?%>Ft4U`|771niaX zJ-`lc9RGd)Ru3v&s^zX#IRe;){5{ErcXkN(+S_=sTrwwvhLoE$T6>}nw3re>;1OQ% zKC?-#W3BXwHt7ZYMZC$_jN|ezy_(QD>g#FbxKaB?;*bEFY*R;AVWsyMG$OO1QFf8? z2x0|P7FSX8`>06U~3QkVO)80=~2ZT=F) zXHfhnxv-S*`9)KMZLu2lLE_l6=LksKa+;}JR+k)Av7aonfM%zgA7won(urKH3LS%E z3(T@Wk%1gI^e7XsWv)(YLHrC=FM6FY{m0+#b62k`+PnRRvwzMEjM54??%xz+vB3#Z$Uy*;)P85-uN9tq z65LDmap)yG)OItK9k)c3E!4(AU@^V0j4f)WwEEU2 z-EJ?8RCCMLYChMosd(r|@?^skaRH6(7g)B`D|i|afo%@8J%d}H60wLX*;)s2&kN{f zWn^3{?y5W3et0gi7KyHPLK4Jzz&eH&#Ne{@!0KUE?KY-3mu(b)bzkC&ovZg$43X4S zh1;HYajQ|+MzuTp5%Yd`PFL~0TJQEkk*}<|fB)}knUw2b?S!|r?~e%#yMi|DO}RuN z+&=LWW_c_MgZDT`XlN7HHU>WbA^pWoTZOuyKyb1m<7kMY1$8yKAE#tbKVnR)QzMhT zPHps5q6(3T;qt?yM2124CzXIP(K$k?{lHw>@^koZf{cJ$RSM#=M9UoOLP8d8ph7v<&kXU-qD-l%_1|FQb*x zCaj>YN<{N5U5m`9H?6~Y zw&!y$)%{7lw9=!k2Q2{AOmibm%9niL_9Z2XD)je|`}iTBBDv*lw(Ak_^R4>^<>W1L z2TI+>8lgIaF5D|0n!g2S<>7}zW3-hHN+)hGdSmX0lqRl$@VK-_j^Wk&+G&4uj z!haDySIO;%^07R)TylA-Iu)2O*CKj1T>Q#IsC!L)2UL*XRzT{b|I!B45dRm^2_fHp z=5Dpnt_kGe@eEvbf3nOAM+!m^b7TLwHxJi^Y4Iz9DS4PB`^xR~OWT<9zHRzljWPleNg|)14vPW*OnHY^jAIA>UrHy$3C+QzrMsx z+}8O+&7Ok7gNNsT^@o&~Mf4$pP?buaRa$$oeEdRTBzZ?l(0z@J-HlZ)XEFbQ5`&^;8=2dY!%!-Lss{*uV0sU>c%qwXpgL_|-(x zGYJ`U%lvZfxnyq2((o5mK(5DF(C4^E^iAW^yuGqwF3lvLOrLoB22NW$%vx%p$>K|* z`fwunaAUN0io>tL(>*UBUhh|nbE$5P6RUA-jb*HAU`Oxjew6f*kDxJib9R24r?FWB zr+45n8UatnbNyxopQDFR)Hr&2^Qw6job}540q^Cb#^dC}Hga zPZpwiA|y+QWYo%GTs0Zs-iyZ?PNTTW@B7oj+0XB?<~rD($A#5G+mz}=dX`r=zKbH= zv#WGuKXWMJSiz39S^f9)7W>&DJ5sioh4QlKA-|148u$x4V+)`W37W&dg%Bm`#10ke z7rkh&n*@OMJfyN64Xdt|U>P#F1E+aVBwBX@YaQSccjxC}uK=ZtTo{dT!-)SbWv+5f zg@lS;q=4SMb{FvRcq%pg13(smEqv4I1U)`g>#k0_%4ikjExQ>mP#v3EB`e`js zW9yfN?giR*L%ug3+ovemWl`SeS@r%D5XW$;96|T_PtWb@ zUK4tqV1pVty!2I>&s&uOU#p@Ii0W6{5Dz#kbjZjkwX_J(z7%q3PIg8M-M+6pnsq?o zJFnogufnOd2|eLda+~>0#VnPPChQxaNTb5Nx-O?hTTO`N0qD_!q~gh<6@v*>$wE-V zJM!2)`g*7U)X`{3kLt=BzVi3f7coVTo#j#U`z$Tf*rxo-`MBN{%o0#$4!;6A%3>2B zjNw#Nt;DM52rM%&*oP{ znF~b&gofzR6w{W^iW{#8-t-~|$EZ;2DAu1Pg$~P~g&npeAsHsZ$qQU*8|t~iQL`O! z3kay3iZeXnUH*TZ=jN8sAjkfO;;9#)Z$2JWa-N)FACGuS;Xgxjp(L7&br+N(=k0>@ z`e_aW%W3s;bjodKq!v6UE)3h0Pb82G4_KOmBiaSA&;~6OhjXBNm+>JhlkJsj;Gh58 zsP>GNgXx|4+3OyCppP&`ib%2jR8vmSL~de%Tqwri}>XBB3x2a7womgxeLO4M)-$GN>U@* zLu9WbO<484yay=n$8B!Y-GIgX)ryLW;_Zk00w{Y6=k_#ncaIm*RkyNg^?E{TXT|(O zd&y{Q`Nn4E_`mu6;o6)no?w^p`|eZ9^Vni!Xs$A+nTi-ao#*mr(our$mcwTBhud zf`I9%ns$GvPIlDKjaX!H*@Ui8s;q0mksn)gbZ<-BTmA{qpfiTOft6$d3aMTQi0q1+ zy7?CpFrF{|py&i$UtfQO5pCf)ub5=`{Q8IcrxpV#Qs>9ru<;p=s(;v+_$uR}XCQe> zY*$xTnglMKi!H6z+_2>v02k+ObEpJm!9H_T{)`Ci3-$oeT?l({x>SMx_G>*p446Ua zT3l&xo?LqpY0G>b-xlhOfLQU$eXbhjje4~;a01SggaDXc|8cug`$c^AY#H?samsrd zTJ_c7zaO}znqq)AFy@jjH72r@5qYRb+aN{{1}roct$VQmWYXZKmR*$8Kn8|@5a>?U z+TTA+0Sn~r)s8==5shpd*d8W z#ImAY5cIk$_opzQu75?Fx~N)o5*el`Cr@X9b&6*{pXnV_oPdBpTrQ73(p&Z~k>u8; z2d{^5qUw#aOXZS-&Y;4L1=T1&>Nl9Ov0w*Ub`0T|wp85#td;Ecj5hoCe$KbxP1-2g zX?oCx6Pl=P@lZbC?3d-5t98vW^-43bxaq5uE=gt*vwXILEIiNZ5iaG+79(3Fd_?87 zSM=0?Sf$hg;!Vz0LY0_@hkpMlJv+z7-oF|0JK-Ka+e-_B)Wp6Ge@e%DahQ!X4F4e1 z6e-uZaJ(2o?ziv5a#G(qceBMo`gC*he;M+3vQCHa-mRARJi2}CiG4A~8TnQeO9TpZ zzRBr&KjDLspres^)uj2DNb7ZcJDJ@1wb1uCZ1B^)Fy>ARn}w<8%)dOOObiVwHal;l*S9J_O0)~M3;vW^{) zc_ghq9e_Y+p)sz2y1JR8o&z8!=sH*sInlCINg}3sF&<4lH8=*2AfYjy3^UJyjd@Tt z4ws-VX|DsUW7m4x&q-(JtcP`C3gXcyC?JH#g5H(`@|T>=4Uh*EQD*1#Kg<)F3~wNy=Y3@Y1!H)(k$z?(~S0WFk(mg>~>9$z&&(mnmRx{a<_Eb(8ew8=C^4qs-Hk z0*J*8vMY+7x=8hGaC-RcOQr$iq))4Coamwp!ou1xTm$`$zroPa`x&4bt~@II6FbLG`~oCcz; zNBwv$x1D4S6Th|+ESYJ?_+NdX^V&-Cz~Sy6nrlUqv$nEPNBwWf;uS?nez?>xq948# z5%3TC47D%qPU(wa7;eK_#;hA9&GhLSJBNFr=tZj5_3Bt7oo!E)XB7tL=h#q)X3ARS z0LO5b3)xS=B9XYG%EadH8tJ5dYb%+Eg}c5B>@ayofn;+R0#2<%!rh)!ChH@EQoTlr z{V1K`lJax|LP5Rk(s4Q!lGZH@to}aAMT7ovgXFO%B&&JWXl!xk-dndLpQ(5a34s#3*abIXTy=NHS)6IfefXEO@K`RJ+f@Z? zE_n8B!M6rQnm_VZ@!Nq4O(XyESgGa@0bE?_RKsZ4hCa6ekSsXZEP(SqSok+YKf;2@ z=SC(Qg~rZ_&Dj0km`QrmB*T!%Wf(MFLmg3o$@Sl}SXhJF08NCtT_37&YN1RTY$uE8 zXS+OaW=X5Nz^RdQ_9E&zf!W;)6Lm^OZaEnNRQ0Gg8WU8|yq!$JvVIUD_N_kIWo70O zLtf!k9e3fHDFZ)Z>4N2MyG%INTgF|)#SWo*_sqmrzl*uM@DXjw0kDj)WQH_i!~fQZ zS?+K7H(*)A2aTq@ojRQt#DH`UgCe4<5>9QkfY6IJQ{86j6JfC4ZSWurNx%+We*KBxBAfc%%BHuGwyoh*G zAha^=ee$;584%0jurp>$9#4hcIsi2v+g3giL@vxZR9k^sZ^Mq2+eJ^Xv8v?FW)Wks z!(^^m9ehD-C{5F;uj5x@!>z2!N{V3a^SPc$uLFckx}iZP8p~Un%$)a%FnQV=Z~Ifk z{2r~xXkYnw)!YKQ?}cZTH6YiS%(}|av`xfd%TDRy?wu6%UAQLo_A6e8AI}u(FDVG) zKDhxl)gA0F-bQwp!7&kH78w2ABqOkq=-c2;r7-toOk>iU_wfwGRJ{4w-nn^rzE&UE!Ra`R!ixu^S$DVYM}i@|IRTAF6bTOX z$kh|D(Y9(zA2&g$0g-5L<*c;j<#d`men8kaI`bq^Y`XW+pSX0MLl`B;Jui_Q3L3%N zvMUw2zph;b259!3=5^Bl7V+?$Tx=v*zLyfOu5}K7rRDp2+8O{ZShNIPGz#ao4s+(* zm6bJCA=P`SkW2g*56h)%YZhV?ccMp+bzE--B?!@k6-!zVmY z)p9pX=sL?LhtFP+Q=<6LN|5B2c32cfdZ|2i$DI`B;W?QqzXLv6!wP(L4>E*og5HYC02xG^_a`Zs z60jABKAyK6X~UL>BNCl$3`N}3PE-n2EI-2Jm3wjm_+I431g!SCywiTdQ9kb26`0xU zQo5t8A@5b1pZCcLKHEdDn6m5L^N}+4!lu z6Ys9w71Btl)?UNXd*rB}oa#IVb*4fPvpGW4;<)jg>GVtN6<-B^Py5wCVlPy6?HK0F z>$jpX;3tBlRe&bg7S-$IiopPzdB$s7*;w28?`Z-Z#*7QpN}rm}0}Eq~uU8}ttb-hk zbtI_9u78DQ`bv6;c zoIAb_YF6f+8C<;Cqr>zw9bmjeS7~G?wR{9D!H`w_v@}O#39mx7nU781co}5mG`D6&+j6=tCYrtHhz~rONhs}ul*0Yz4J;Rtf0J9r*v9NI&XcxvR6MkBXALIUZ7rLJf@MC{ms!jh~OkE!M0=Ik%b_7DWy(3 z14w1277K4F3x~$)-S&>=tC0-#{cyVDjQ=uII%(K4Pl4(F`LvciX9B=7ecjO4uMm`}s6LWg&ge`@NOY?!!80 zqWh$Fz%35|#1GXtqwzvtT;P5N_PKzf(q06MX!m8oc`0R6c@ z{NJyT2#754T*>DDAFj@SdFwV1aI8b44mqcu&WiqTe}G33aQ`l0)I5K$$p7WXQ~vRE z`!B!ppZoFeS3PK;6+x~Qr_cZUtN+tKL}r2yY2%?e!u0?BuFsy=hLYP(SJ3~?nYe8^FVM@9+dul(CA@_)RR|NlPRS>_YDD*wZ~Hplk`iq!UCDn5kb0MnO*zRT1C z7)Ze7FEk0Q!cOL|ZknYux0X*Yf6p3p#SX5Sp8Urv2BI>7K*|yUoFdY+Pwgjw-j==1 z7@BD}mpohB822k%k@0k-9zh2ZA7C>NO2bYs3h}kyQzyq}l3_G9G+x`i0oKVOz?sVP z;-cUjWR?SAymGE9OJ9KNj-?`VHXC5f7vR z{VM?Ef*f;9!4y3w(A2oRFZ&EJG_u!+0YcRY^a93UuE`8Y4cVbV*LYw6GAjG9iS*bC zU@0Bn-rry>Q1o{L{Cv&`=$kqqwjSW`-cDd^qni7(Kl)pt_}2yUmAuH=@htu42s^X^ zi0WL-F53nunzj7nne$sfk3_zzgN)h&gMn7HU}PfAME3?fzkkY$gHZqgl&h42JFa0Q-Vo8Z3whBN)2_8v3yhjhpXvk#sAHeh115 z$}VZI6Q7e^pLsB|#RtxZ>OH8ur~}1!e=xP+p!51<&BhSKHZ~BkdTPj^Pvjo>q#dB5up_-0^@(&G7z}aU>ttu~sXN-) z65-I+O5OviB}{^IpVT~XPvD}mK`LpPl>FOu@ZW1k@jcjYq85w)xTG-@J?C1WYM8`G z@vG?LC9wi+0TR$og%`{X==jkr#z_BV0h7}t-8IzM`#OTt?@4t@^);mFAgH0w>5c>5jfagA2+E)U6p4*-)#il@-9M3(w z>jn+ULS{F{ds4TEerN*KOQh)#tB9R2jI8&e`bHaQN;?3!cQl(y7mkSjY2PbBvA2K$ zjKIwJ5m4Fo#p+AVk6IPl(14-W^w$Y=Ojn;9Yhnebzui#S_6)c%|LZRshJRl_Xh+0o zL#B~zvU{iBuwmKw5yY`i(y$8#Ms8er`59=ZIss)mzpzwaOQTM>1lG`Q)q9exe!YQq zQ2@niFo%l?Gd*}`2)LLKn@0uy?(T+Bsfb^I_tV}7#0Qr!X(CW1cU%!%3BUP$+9Rp* zcprNdxC^iF1Xe;niC!vFy9lk-9@w7n6m%g@rw2!%>L+wJRB+9Fli*RdERMzjDyP9r zDnIICrCC|BbxEu2^XGwFba&M4qqsIPTsHm83jkbv2#G+nfL8J-iAxVa5lFRB$0WSR zK+-Pg1Q4VH0z_iI1gGB{L~{t0nLPg-X&vLo4Og63RubhO_Syga+JJQ`3`&CRFT7)? z^@I7Pv$*s2V7VRmUgoIYcxsruy>SE#l6H0a)P^E^nwMDX5{=6>o?g!ZHg76iyzXSrLqdx)M&v`&&;ymYHP<1_B2%(vL9vSK% zu=^Nv%IT#<{X}$1d3#_SsYgVTaz1^MNk+KrB(DG!H#PPq)c=jCZgQCYM6%}^{|Q$X zy&&31nlje}kOAF1+tM_G#Qte|(xfj^^^_W<#O23pQ9u~*oYo8J8nQbaGH;CsH8u)NpgjRo%K1^`+{#-01+ zfH{YnB#@Y8Oo>0G-`eE^T}`A&^*cPtG%0VXF^#ZG0s*;I0hiuq0hQ74Z7`+QyMk8u zGN7`>{YnhN;Bm?v)I=9C9!6bee=-{mmldh5WOiMeW!JJShm`S)5Uogqg7lHQ#Oz#n0;9t&8 z?^7?pR%gHwz(A@?lt9fEjOg#4;~~A5Za}Jm28>>C0O^wD60lw15jp;CY+YWOR9H;r zWV+}?e#8pgB4XzCPxdvsA~gS7-q_#e+5;$7baPzu5>y(b1YKMdE1+o$@Vkm>d;`DN zqliaINw0tgB)kl}-oMRVh;f`L^jAtUprQzz#~j5HI0H4YYaWsC3iqEMpBv>_xQAwB z%m_o8($nLctptn<#Y4;UXA%A~J>5Q?CQ1JgkP@VA>fR7o)0RKWHp2}XTZog>tei1< z1K`6tz|lcsRMa}9(VglZu$x~?Vy&Kn_P?%rQz5zq|zq+{9DQN-%4FYE>N^n=IVKXMQQwOIQyb+pVud0XEy8J?4dP@HR4Z1k!y$ z!M_0udpea=?_ymU-GLxMFf`oCt$Li z%rcL#71}eQ+8jY7yr5%rgoQl^bsuo;b$uUo3DYfrX3Z*Z{P)1~sS1nS;HSs)?Wkbou{^u>C zrwo4+_kN@8f;EPZRbN6I4mbr_*M@Kqu>BIbcOu-WCKfBY2EooA|1hoC;;i zo2BKtDL1+SDjmdQg`yF^P6DzY0r9-D?DJf_K za+r@MBXC#GMducsl;%eLaOpp0$}ciJMSD5utO!o0R$63H~| zm!+J#=s!pAXq0N^e3GNMEa}k{_1Lw{tI+MdILS!YN2{}#X#?d&oP*f^xn|{`LttnG zA)v@ppJ5+)%aHO?P2W{#J~_r!b8G5q({A+o`gvrENVWKKK_~$5XS^3^{Ug&Ez$;L1 zX$xs;BU4Y9IA-#B|E@0q6-C?0zZ5Jd=hc6>f=*o=(QQQg|A{vl+zU>>GK=S%dw zFCU&)a!o3<(r^PNd6YT%mwya>l30#$xmBeEU7y1LqA{FL0bx)Yev)NB%5f16t?xk{ zbxm(_bJc=!nBR$O4_pQ_NNKt0dX``3M4si|g^r#{*P-{b3+XoBKEM7nYIlbMBLSX3 zPoBc0Ml6c!krPP&MZEe%)Wyfo$U&3C3!IYLKfMGL;Q$6m(Q}U7?s8k_19g8P91*m6 zB{9_z0ckL@YWH)vOZNgJWNhye-lu$B*~|;Tfg>)9CRYr=^-WY$S+wm)nX;P1V&QZO zA_FiTUwyg;mK1FfJ0#_~*%#=9KWG7jj`4>%B8w+v{=Gs?jjtU=UG$$r(dBgZv*zJN z`Z@TU-;A?tA{b-$4B!9tvp}pL8H4WRas|Er_Vf@>(0l<;lKY_VP-|2`wcm2hNFRun z=PBhLEyKZ=?2u^G_S-G_>hntd4Pb^G)4xiA{lxm8e=!2v((J`tx_$WznEfBad_XKQ zmLr&LjmKUFep7rAAp)A@!u$-r+UF(Bnwfers zuV70Q@X_DuN{x8&Xp2u*ftuW~c{i*wU2$oM2gb^OPU%I2BndGAy~EC)1318lK*by& zKAwePRUWHgH`9p8Bss=1YUah6VIlRP$#)9ipSRVc$=bBIW|*%a~Dc`jf*Or(a%5)Nvr9WImyta@|MHWLRrXUbhJ(2eD>dk#E2^??as#<$U`}^_G#| z{31xljk`t&zEyiE9wN$m+@h6z8gX__2UWY$nf9wb*N<9yzy8qIH3WttGgpOp-5O++{oI#G-!t82!D{{(GO8{HC!E;6W)0W-J(6)=&(_x(mvmY4WZpdMkEi zMdjDIcFOBUY0W5}SS5R^p3yVS@G41{4eJk~J{gVeGmU zY%*hi$uQ@^Fa%j&?=@e{o#Sj zI#`%>*!S|S?8$;p8!q_XPR#3)3d^j^ttiCqSEzaT!-Pb&lAl+=Xb|GRR7L0Q0T_Hs z86yh@xN5CpDh7qbwKH>9!AJ4qB9oG`c2eT(IE-7E*BEAkyROp^jr_?;&)!pe1dB(l z{MelC_Wk7JKjBVOQt)>5_>~GpVbcv`if7YWc_kK0L!#w_0Wa1%jynMB5o4s+ET_)o zV<>Qu5%{bZVi%jq@*?D4JF4Kbl_fb8_b1fCcK&(7p6cE@!ZR{a0* zg&pNqu!|U#Z#uz58jnXBuYiZ19(*tLGg8w#EK{S%TFDdK6&oC?1=k2R`|whKLScI6 zG6gf#GC|l8bk|U!flsFriZ*`CE`yALPVM6%2^kZ+2zWDUvBf==V<%R2CuhO0!dsx~ z#%_Q;%};P?ypx9?VP~(yE$d~X2_oc@A)7a*mHqK-q28Yd*sgSv=g;JR^XiOaigu*= zC>MDHr}6Oe>qjY;W3{gL^4@ndD4N@Be0n*STXpfNr@dl5Ex+Prl8MtDn!9&7)U~Ew z1eF%?Hb9SFd3~<;aqqRok59ekFRe^GZCH0kF)M2&@n$n<;QUZ*SC2_JbH2QH`e>MH zu^dhp}p9!103Rv{+Gy7KeAZ zuP7bun=$%NmB+a1Yr!(^=PYPnJRcbicx=_mNeT@5}<+CSaMw8xGq z$38s#iXtw{B1Cf!s>TDPc5_`)_9kP!FMXRS}!>JEnZuGn)RFEVFk}f5)fz4`A|%puTZi*Y+R{8Nc60A~=Z6l* zFthp#S@NQeuTFG7kb71n>N*2_=Hf{C)x5(R#2jQfqyCk`DRvqEb3?)6B`C4F>hnDj z+&_SLq-`jCKdIni-yCb7`s7Zu-I+S^Sd{gNhxB&!#Kj*kD{B<^i0J)>Oi-xVopdr$ zRI$VoD`($Bck@Lf-m6$))ZB{OrRe_1QXp&o$0h+TPjt@+qDD%&%K^rkk_@eswtK zy>qF8-LEob2$@fOUmkZP6A7s*q7$;Rn4wfA=7GZh;w+R_`RrEZ=lBU?7l(__!E)-j zD&b!^3~h(vlawR$D;Fk=&wIkuI;p&y=k_8W!rGk*o~ZbT*K0P%-Qnw9VQtJpJBoDRem-$#p$y5yr;}- zlkT7KIyRiL0G38wgL&t4{8#bxDs5F0-^a8#e;8jvSJ&=VJH6=vJ|U06f%RJ}9GdEP z`MZ!=d1;4DCRaE~3h`g4;Xi?r{n)V0xz<2@a!l-CnWap-dwao!!fMO2ud_g45x8=s zMS`4BR7HfLvVl03!qifoWD2j9;dJw^xXW7s1Chr|RBA+OcTQcl9*nLgsf*`2E*HgU1+?0H>RCJ<0i)d#A6u4H6awr>3^GsaQ(*tZGkREINq4ZTGk^UUBCej?-p z_zFG+YdreMhUg$c$Wq~Y8$m@dByJdXgx93vp(aagixhs0rdm#~pU?wFXx2QgUo`$T zP~&2}@c#L;48vMPi}`9)@uhc0g;8(8IEYfxVrAmzXQMBcy3|ZI-E1-`JwaJ3cOpkI zl&H$%qZ$_{ev77*zmp#*3}%6Oj#mmNGxfRf`emlfh$!WZ!eFkZUHZ|(d242Rcc{C5 z8Z3o*xr|(>vvYWvVHADjRMGvW^O@3`iQGD9B0j6xu^N{Ump8mt?*vRL6s+%C)4{4{ zJ6Ic9`vf)X5{iv6W^{-o_UVr|MJ7;jWG=JUZ1^J)3c5>zE?y6_KK51Ke6n7gJ8!e0p z(;}wYxX>^@l73+*{~#io=|s2aJ_2*-JyACqjO)WVqo&C*C3%Zq$BnPG2FY{RB(jq{ z#}lHkZ1ARL$T}hGH&AsZOeRg%zl!ETOI7d%Cr9YX`nWvfvw3Y^8HumJ%bljwI@;_&3xf;>rl&e@{{E~0V1>x+f(Hby88?qAdx;1Jneq?E|sNZLC=&o&hbfHdq z?dO|dxo*Gqp*>%gci=JNT!xJFGD==avolYj*h(^CdwTgbyFu-U2iv!3JhxT>?oO^^ZxhY5YnQz?32C zX)%Gt^Ncdu`-w#X)vdZ zBNn>o%bZPz+{AE|Ac8V2t(~(D^XmQ?yE8v;k@}a^uCoi1n7p|Dw2*hAr}MOtFn*z5HONw}DO9h~B2suj z_`S=R?{v)!ba#$^vj=&`(Il*9yavk?~2v#Jp;RVtxhFkjF^8E2N;plvV_d_P97irZcBjykD@%@~tzJle3B?01 z{Gc~J1RZ={_BqZij>`$;a1$?ig(7u;WA-Ceo=4Se5r6yjvt$j?ugHTUtOIC@$Fzig-` zLE6XJY3jvu4E;BG?$fu6o0qvSmdc7*JP$B>0KJ<`5gbs>Y`$#5CETji(QNT+*2(qB zXc@g-Xh^4JJOkxso^G}hcaT1Pd(6>P@N@60VyvI6ODr|7GjlL1F?p@eBA7S)v-R93 zOsQG;QoLf#J(^t=1|kL)#`cvPqA5=ny#Sr&xzDt8JW6r$lZIyVSy}(ZQNYu-oHpNT zt%p~^-z8a>zN0ZW_7IVjPKLOMB%|&&pr4QdNj1c}E|xF<;F$oe=-B>i2~=>eHr6AOlE&J7YAS2W;kok}UePBv0$)eAZ1X(n$MpDKP{jvXVk-Ifs(P zaQ!)?QXEixNXg8cIKV!*-z3&GwR(#~g&crQC5l3|INr-Doh?WZu~Rc|MM2^&_yJwa z;^8?k=cB1_tfSHXX4^8t>%R82_K`(t=!-Ior~0r7NuFWp-(j_EZ69f8h<6)OWD~b= z0p6~;a?D2bOWUh7d>evt0%X1pg&wB%{i|7T`c~v zDIdoFJP<4Ib;bcFkFSw=#M9~KhQ zytKFbi-{7v$)%c!iC(E;XUY=QrFUKg+eqw(NF^QEak{kjfyn7a;Pg0Jzw`U5l=PoN zU%?$5L);t<-Ch)(`>HDwEnvbLooa!meZ0qON1&-RWYFG0Rg~XT0)F zO3{>;T8=#6o2F>tWs>m~FOtoB{T})m-u5$2c!{Mm32z=>FqgE<)1p5omxNJ65AZye z3Hn$Q@=+Oe<|gkQ`pxO_9CgaMHz4yXQ0C5FbYx`;iKGpq5+Q5k@}p)Dy?^$ml`$Q& z36-%l19l`7SR5mdd2#{{nM%=Fk(8Pb3*YV!yj9n};)fO1dN>kWDL;r&Gaw0?8BPp0 zx;J$90D<6?ruE(Tk-J5FD+MICUt>)=$1n?QO&hFEqlY( zZO;v&!BoOSM?TBmFYuYEX^9yPm+sCw+myOkYj3Sx>A&HiI7-KV=EhEa@>sbHR6R6x zW93_4=pWIOd$ zCQo3O>%UvSa_^(Wun<_ce=A!J);*GeG`780XVec;>@E>QS2EK79@?kql8GmFp5lhI;~hqC zz#r$DxIo25U-0lp6VsI;A?;{0|Dz<{EpL0m1sZ9kn8p{$v*ZFXU@&~c;4N^Dm(MK3 zgpl0DcM!&VeGHp+%tIG(p!1yVqBe-)$+|w6`-bn!sNLN4hbSQw^iyh^%+f{eIWIhr z-K!pc1r)4P-eKN0G>utqiih|$T$t=>XEAsdF$7OlhId^YXZGiCnf26UF~57ETAh@hl$L!6~n|m*M~nE zn?BsUW^G9Dkvs=(2h+yg{&zomqE~~ig11WuXSZ?0YRb*$!`F>_)ylrh!kAL|I1 zRcoAoAiH1;Sh>6Dt39#DX=>YMUM6Yp_#>yl&b*;|(m(QXr~=w9f;!UI=-K$JDECBZ zEZY6m@zZW!ey)NrEDjg1#19rX1(NBxLCQMVf;L>aiVCi&@eB&{&PL0Lvu`Yg6E$Mauy4;`guvWiO+tlelf>DS766z_vz=x z{B1tVj0cR~QfWwVOn^hK7-A>!PjJi@ArM0Sjg8j!eY#+37GIGj>6&qw=cAZ?&#beF z^&Au43>g5cPoq4st$YrGN@gaE8d4HOBfDzNoTs>5o|83lSnrrA-XcEzm8QdJkzpj6 z-?;ka=t_nBYL#okT+ zfo51iFRBtQ7ly2XI;7Kiuqg=qnp*Lay)}Lf3@e3c z5ITpI0l+h%@@=1g*e?ZO1{)6_eET5*2XYaek7^p>xn$hODz&Wm-J`Mq#+3F{XGyqV z+C%XgNv1bIt<($Ej?P>x8UIx@FKvM_(a=f#ToOs?=IEU>UY|aDW1_?yUy;b-!&NH= zshMobM2ZZtC(d|&{x*hAd-*`lyjY){7jjI_8&fW8aa1V29WN@(H~4W#2;m_!qc;o4 zRYR6$I=JMf%8Skqd}G$KDeThX_>79A8n(iTiA%!jXv(edr}5FuB(com;BD(Rd_`CK z@eQEch<${WxwI?HyMg7f7q!BwCqY{gnkSV`Z@7z{qYta`U_^tqy)%t(!;}kB&em`xwmt?6fbpzl0U%+Ta0eZ+wqc`YVX(_(!nGNV7yQ~~vX zG9#wBSc*Jw)~ibjyS7jFO=-Gx}>nF^#; zLPuPVI|onv266!mrpxZpX#M(eHzym7f%1!I;nul_WkRYkN9eVwpy6xz_%wG#$b09( zEx0xijC9ZS2ICWG^S4sd5LmGaJ6#di#`2JSE7VRG?NBz&x49LZywjU1bb9XE!k}u! z`W~1{49wYtu-I@-Q{#7#V?H><3&XW`2y>=wWP=w%>hC&AQA5=cv^aFKSw&{F#pto# z^xQ{4IO0X2=f^i`DV&v^(Ls2T(Zb{vG!{Zod+W8Z;Lf!|C_(COvg1kp02-16O!a}_ zu_s2XDyuelgL!_jV`QAHq-c>(>BBygM17Ha9G@Fnw%5>$f5cQje4l zlFv`>S=-NoxKSXcnO3tRL|CX}NeW_ONZ^29KXD-WTaJIC&4b`f{3o@letj)JvJ}un z32syKjn~ylXVV1Ep(?fTnpnUGLx_}=^rUp#`{#j&9uVK8gLPYQsI7qw@fpBk{h#i{$aBPUCe*I1-+6&mc*mwAkMeg3F zB7taE=4z?!XPae9!v5qh19b_JFSG{Em{=3>6vm)^<1j-ZKYPq>;f5h;8Ty`OU>)u{p&x z0EM6;R7Ts_8`_aFEbTy1lD}Rrm~zUT8=BIO5#<|7P}sH^E|4(8Qj;}udoPax!|~nY zyiJLCz_-h$4c>((S;)d1vDb(uB852z$jE)&v!p=_^*-?D5ojR!nC$gnTH*d zaL6_4e+j}Kn&6q-f*F?pfMDT#al7hIe)>aaziCmfA@o! zKwj5I{a-*MgTCBHB{QGE!M49WNP>esQ1S-(+9BOt)@a%rV5{@k)kymY`y~`Pb47;y z$q-+-R`cr^8dG%o>U#@tusOakRG8@EPlrdqU?=W2VS+HsMPASiN{13fv)pK?%+eg3 z=0UF1T1*`-q7qgs_rgo|2F5&y54L-$2DHn-9ra@2e z0#$?3aa*DYv3mL2{Q@`Jf3Hk$mzY#Ovb|*Ev&`6AyntMN@ogvMG+eylr+BL6)ZY}N zKt_W@pCsqYL_c^6Ev_ZQHc?8O;fc0$qmp%!Y0x&IYC`zfDX;#!fB0C10tA>Fa=trA zQwoOEIKJP&RB!!?O@8BEx#)wkDkZsqZ-o)hmW@Rx@VQlb)+|4U0`#nD@vzMDQGSee z`sj5Cd6(8_yQ6>I@Z+hVExFQp=oQj6i+UXsPl4?LO|Cs`$tvezzPZrUA}JO()K@|% zD-B{9tX1wBV&}s*mqz%Dw`Z)Tx)oFd1>6PAYfG%Z2wwp&WOB9U&AGYo%1SF%CqGK) z-Knw~Y1@Y`IP#Xc8mM0{TXseR~3L@UV$(Fnx~ z_dEBaKSTSB<8-3yknDp7_vZ<7}nyT&Np*h+T1sgpCTg<0yF& zC;XhdkjQyETSeAL-zG*FoTWaZ%p|jor%Ye6+F65*@(eS?5zkvrvgd}UzOeqV3zH7# zBk$a5)L5Dad|MDE&>U0??8DrB>rESxtQm?GyW-&xlAHKnX&4T1&#!q-ZDFaS`SP@q z*R!sDrh1FdtmwWx6S@(xamVZS(QPU2?L1dy*WUVepM)gGHe|GTE6mJ2sFh0jIM4A$ zD&~A3X|v@|jQ<<*6M|Eta6qvW4mTXNBP#xCrpOeVtijqrnsm}Y4IF1PKIDX@)@abk zx3Vu>sSx@b9Ws&VkU4+*`2pFX!H!?=+Syis5q7%mPMGwY8pkPz37AQQF6PHasxf?7 zv(ov)9;+XYut?BhjTjdb?%n??IWj1E?2Kow zHze8O5C6VyG8%;ve<32`W(F}%PEOvL(us&jQZtBk!7}(0{BWkdgc#Jh;8^7(zRXTI zAQ==GNOKR{jbHiOs$);Vvz<6T*9qo7MQO-q$ELBn|6I4{5+HABxJ;NWZRIyZSQ@?0 zr0$o8C$;e+$VbX7y}l5}-l>^+a^jGSsLkl|Tgh0>+iVr$I_{~+20MXNIJYAIR)Y#C zW~9R5;5~A%(6>&Z5K5Bb@=QxOHxoUwh*bROGOOo&pp~j)$npVWmPn-Fs}$9a3;nZW zZ6olMEIIhCTbdjBN32Ik?o0bsHbE-5pP6!}NDd{K}d6M((wZuH;qRz_U%KsqOXSKMjv^EYke zx%R(Qh_E^8cN%;-8SEk;F!chBW;vcO63Wkh1b&J;9Pl)P6Au3rm42XkAnGqLiQiv8 z*0&@ua$~tZ-(P}AeisZqG^GZhN39_aGzC@Wl9hPHC|@vDt9 zBOZ}|Lk2W1!d#3tpcefsJxar+b<60o{7AIG{=WFlg>z&OBK-U8SY0X-*#axxxSl^6 z0u5Fa$!t&ZxqO{_+=w7Sd%JKQ1c2Rn=TdW_wOkav&*|UdLm?V>+07w?D&T}Hn~IIg z1);^2Kt|8^UrWJ^g$UW;Tcm(RmDyJl($LT(eMOmN4$HR~=)ucqq^C(wC6mqt_#T>bd{>NThIZlShG@9EAsh)*9&gHetwWIa z#vY4!D+;4d==qW3mFbOU6Uts+A$`bfQ-T zemk5y8mL?OPp15{aPI67gm|R;{`nh7!1>QFEFF7r1!bYZc+-n};!X*i+}vZ|SD+7_ z#HJiE`iX-BwoiyJC{4ba#O-GjDqe`PD6Wq6e_pp5>U&c}ELZwJZHE8)`TM_1LlQ-@ z>kbxA8HNS8D-i^N+FNFLPWM`x?!36SO=$T3k_H(dykf895fGUU~`3| zIgTHIepYNfcf4eodVljDl=?R`{}ZF*70|Q{^4IHkfUigNAH7yLAdRg^=}zZ0bEkf9w1*1O%6zKTW*P8~*di|NenW z2d$MVc*zPDY||Ri z9Zk=V)W?I5FR+Z*L2Y`D4^#5lERGO)Qq~D5Jr13=sXaKB3aCm|;J7M%aI)YY>pUYb_x7ct?t=kr@d8;7Wjc($)Up1zyND&7vtrc*Ij zCL4@?Qur*7N%UUlL-r%oZ^vFW{sHsppx=sKT+ zvT1kj#qYhqr$Dz=L_(=#Ay`%VNufNuW&GOp+{iw_S zW;u=m*b^`|oSq#n@8QCu)ca61kfKiHOssHe=OJJ@P#lAWT_n-#anoRl2PXOo_q{#Cd@m z{c`H(|15(Wl5=lUZ5#wH{a8HAk>LV8Wm9aa#HGul71_WTsvc&N&ea|&Gpo^3yM2tS zqx&7eEMES4k)Mow@SIm?;r}VHYDO@Oey%@)Tx###6UXG_IeLCf61?h&qH`YO=e4X@ z+=|~T`RF?j7ty;Lz_4K*OoADcc0Q806nr!kKKaEpRum=i=?QmboNFT*jw;-E2Y(#OZF;jLyGl;WN_W|19J9MFh#l?Ji2>9+MXYX%P(C+5q7hAiQDtUjgN29SLG z?l}cP{%N^tA1gfJ!JP|2-R&rcib<3*5&j6SyF6_<0xyW8-i_Lm{iKzm{-S(WGJh&L zv#iOq2O-gUGvsRaq+cgN@>l%npSlq~9}^bHVd>9b1I~EP#zR+D!ISsJdLlNcNOzQ6kpc$8-07wbK+>N{7&fF!+(&M{4)?H!K3gg_Q@pM>4 z72pf$?MNT{g1j{^tX1J`dAms=*^?oEEtRWoMDryANs5;h^?AqY~wdjjA6_6^Ackxh8s;whGRUM?3x^fQeXQ`Vz&ri=a`@MX0h1H6YDq;@qNu=@H1^TBPpwML_#bx!u zpm;$`>2?y52Mh=TW9bJ$0G~YoR$pekQlU6c6) zgv!8foJVQf!Q*iw(1ujt-d~vtFSqq`I^k&Iwz7k?-F6rYomPEP>NO}8$uKQ}cY(Rk z%9QreRz2-Dae46RNkANV3$BF1bh*Wj{s5U#z*H9>a zNo-{~aw9Onqn3-g4}9{-`ZryD2*)Frr7ojXdjrr}nBw@%FOa#NgneX8u2_1bGmTI6 zA!@3cCWV+C;awng@(JN*E<&As zVXbG{a9@SdlLERd6EG7H>J;rGXTs`QusC0(8f(_9a_kp`9CeNFQCPZyA4yI!ynJe@c%6oHCTyASS=VE zO)^1Kvi3f_!h$3+@@qMqixj}x0qfU~7V=uVKRA8WYLZxO74|W96k+zR+*gm=ulHaZ z{~z@}do-SvGvgc-rE1Bp+aO#;jYI&jsyY=K3vHENNGMFbtbkn-LNr}vG_|#ppwl&K zZ@^-U(tIvIv|{npQqQhVE91-NMBNa zE?T8ENcWKnsR6@nNc>#|dN~QA-f#cxFl1+@u?2rkZjYu<1d2@)2J_Hr1QZpto#k~AIy<6j&+FYJ!8P9bP zdVJ|-umK{fhfHR<`MBg2L?JX(6f$-v<(Ks|4yzBSYm#AR`4SWk(oD4f1{MH{$~XpF ziYa27YQ?AvDB-p}FaRUW{yrTZri8kC9#l*kXO+9h&jMgbJ$*+Fo-7Dq34`Wi?&E~briMroAhQ~B!W?i&-> zDy!R!h*Ux`_TnaiGO5ry%y?!*16CUb6)&~U4}9_%DoQi{)eI5OS&jc>@8HvmsNkfi zP2d&~hGR^TG$dHiplL$Qh*$xpfKtHFI!T8OtHA?%DmN>8c8LWNU1x?m2u?xH>y%Ts z^ySO}_0ULD@4b)!7|jwoo_uL(Y67aI=XPPdBYgn3*?1IjUAY8u**H}%(7}R$*GIsv zPo>0+C)*L~LF`=k#`eJXwykx=dyzq5sjJR?)+I|^6n12GEP%E+Aj+)kPgZJIPZEyhL*&+w z`3`d)y)b9XkFZN+lb}ZbJg6P7z=^;ZntrYI?@|2~vvV(VfKX654LpX*kDqp5_d%~4 z2?B)u@v9tO9z&)~aD?|T)A>#Ch7i-Y=AtV(enHtY?fk7ULC@{>;7DxeS1+4TRRS&+okj7kg`rN92KX0g&U1xB`87U>+BX#8cZ491w|l zV0Et#qegS~?Vw(>7rlg2un`ajAX%uF6Y;@nP-+Vh9HUYGw~41-_P~vDKgVFB92vyW z?9eq%h8DOK=IQmHRd(`XrKFtmo7v0FpbQ{9tB~3Dil1N+QCkn@>l9x%t8-f(inV7Z zXM+T0(-RsQlB*^Do>-VjBq9Tkm|snEu66rAPZ<2U`vTtkBgBw109GWPG{`qOCi>8L zD^nFocUTP$WS<^j6V?op)p1Ml%v0Z!k>~mli-SU*kY5V8L2ZuCq8}jb(0U;2k0v^N z=!gIn1BG?M4^UL(uVb=#V#C3R!Y8LE$!)B<5W35Qx!3fqjiMXKrphcpf3aLuP{$TO zF%nsOykqt7n5<8e+hoIHncFXbS7b^y1tD8ySTflphH|g4w2F)Y%If9I&nZik4iq<; z()LSi9J-PMf6Ghc`Qkw$#Iy}`U(Y;H0zF5peop6o1-Ylz)3~)R8y?UwiZ6`UmAlR!oS}$!CNO(Y$=ltES9P@Rg7y%sxI?_t+{?{v8yldg2e zR|1KlFkc*r#2PfVU5x%!?m)qNO9{0amkRQb?o63Fx|8}U))Q7W&PIr`6h^pt^rYB* z&rqROyB-I~ggHh)0ZXPtVH;ARpQqK3Kdo~YVyYSvQ)M{1$65Z3sewpL4V+KnKs52# z0-oTgMd+;Z^Sf4uHs^kYo^mb(AG4aOJF7HTpKqpUmh+Jv5hz$ObF=AkM~YinqjdS8 z0H^WV(u{A7yI?$M+1s$$UxSt6i>Dchq%OZFq6`grZJ}Xgq-2#DkCMG;TGbpN&QR|B zGK{`x+aDLdYz-qRw?6&CUHzp>juLq_;srJ_6;=n^7&RI_u4JYBZd zqh0yn7@Hgu#v2U#`T3yON8->sbOq`r5N4j>mBdH1;(XQS#FaT}otN{Hk|a2j9jMb| zZ#TQsd1%6GtCwBgQ`}(uq~6SgU~iIW;5tHDI`wY7<6ruZKl%LS%IvGI@`~J;B zDKr_9D#RTMte!U*c{IK1<#YqSeP@o~-I{G5m`c9Cl|KZrhKZb%s(3`xdUX{nBBbf{ z)9T|^xhrdC!+7{ESDBMB5${9@m(=8;eoX!z-(kF-^5Hc-a4&P^SrN?MK3^VgU#IVW@ zR@cs!AZz=9N*G^2g6IU?SBIUp{#V)y`p!LZ5;d&T4S0u#g-^3@gwp~TYlE4sRWX`}rW z%tlvRF%GnANH~~)nByBqR|!rN(+2a?71_X)4CgU_DL1J(6_n0?H_w~ACa zzGT2gY0kBmUn_yk>k{Z6z@qMIPm$@9e9gy3C*-f2+^e1ty7;0oSLgUv(@S_EX0?xv zK<%jk(XYhFV@iRQtDI8Vxb3Bw@?CaS@3O0uC!|;b)h5I33JV!ZPpvVRGS7Iajl^YW zEM5wNC-EhJ^+4{N9nq%J%zuo|K1#xCJbq=oR%hgwx!Y6K)VL%x`3!8MlSnrW%F&rry2TunsaWcV;Gv|1HRooqNiRic^qhKjWza z&t49yjJ)?cKLJfP(BGAG9ji}?`hhy>A9VR0A{v2`JrCw>E2YV^bqouk1_!yF{-^#d zA&3Y@!!)tRM1~!0je6GPiH4zRCiRqGVi~`XE5x4oFOvP11q!Cg?*UUZIn@lnPqU)w z&CcyGCt|i~Y2Por&^;Yu6jLeUiBb>-HvQSKy+3HnDvbIrUq|2)nfHgBVbw(EU&wMRx&!_%_ar42mgzu$t!Ep5 zY-2K(v4os8))l>xa0Q4w+qsi1qms87V^#VVk{SUo|AcD+YczX(osr-1*yQWVIT zm*SpeH4Fds0S4Web`!28p_M>z)l9tRKT)BPltp*CFj3tLlFr^u@ z&|J%{Gcc*acE5z=KW?P=ZN}{G9^h|j=A-El@Zv#8)sPbN1BIEFd|F0%DzJ`Qfacb` zqX~@Oe3^V^>>pwI;<_o>QNfZ%O1+~T)@x+3xvO!UPOc*sl-4@^UPlE{b7Y zGteJ9wcj7Z;*jQ!u$90;V5A~K`T1YsazDOkwA%U^a;G@`$O{D#`#oc9D#k5=-&nAY zy;X#4mXC}8E%csgqDA-Nj6#d0wa*ri zAUeAud_?37tRVasElQW90@6x(`4!+tH1p~EoTEpJ&buDlh!rBb%R*)sfrLb<)%LN51wJ0M)*Ok^h?IilbVBn!iF=(ET2yo zZNJ!2PtfYa@Jh&{UN<#-z!Szyv}S>oUR+0!ieQaeDdoP!Yu}U986g@w>%X2mC{NB? ztwtx|JT#D8qZQ`7g8``-suE!u0v=pggFPbvo>HkRriVCq87gOfl71Crxw%tWCMVx| zAdH7oJV2#U`fgtEPoT>s!C#KybR}*qp;Py7bl?8==0c58@-CD&nq$=~zJXTU9v~>KS6pj1 zBOe@ik-yT`!6@$U;1&`?uw9%LZj;iMogpIZ9b_}GyxTk@6)1+4j{f^+a_Y~jUnVhv zKriSX>Bc4O4xgjuRs0O7nGMHO>!Y8`mt&FfEu&56K0M_J&;yt|cHv#67G78TwaSg1 zYEbjAqbXWDey3L)5kX=70wS7ov=LUI-z@BXx?Er_?lO7PyHwDbs(mE^7K%2>3xr(v zKMUVMB8}}m0C~2*c09fJAqqVjbQ*T=F33EuTm^kV(oq>Rxmbo(rS2HN^tSeM96~T= zq=&RtTwa3-s3xcjK0rrNToGH(HdiZUVG3!syqvVr&~>4uc#3QS)q-Z7sdze zv-BT-|9+Kh?MH0LQN7cX-idmy>2J$$l>*ZSEgH=+fMnH8-+eSZQmKQm;E&n+-W;4g zG7O~@sZs=DmQbjnvA)_uU&eTQ9by5a5q=)*uL|4Xqi?KatI_C~@PAu4a0-(VLc^!C z^_r|v#fPI)truZ%$|!FfYI{vYk~?D4zS?z{gA3yG4JO~}h{{!(%l2Pl(pH?WUmIEg zb9`C2NN99e6F6$`1(*o;+b&&&{(Tq7Xm)*OW^5qr?o~I03LhW)T2Jyc`E~&f`>~Z<u_fQnn!UbzF z)c|Quc!6zAihmK!$F|;+COHO;)Ckl5ye&p&xmTz7fpIbBr52MXlPpD&(8c#dE1>Ms zZcw1BbZ0RUsfUgmK{KxE531Zx1JoKghVx&56G=!*7B2QVDgpj6qY7{I0Wr%1A%-mmWgUF`{wVxJy94~N(OK1;jFhS4A1lZH+%5DhB#le zDY3S%zw?oG2;EV#-#*wi+E9|XQS)45%gwwrpdC4-jOl1@aiZ2%$y=Wje-t&U zxYhYtRsE&dv1?L{Vyle#`Ui*PLn@rn50Fb{_s^&ZdK#MzI4FM)nP}zC#9-+eFxi|D ziZj-m{(`q%h6WC^eu_3|){;qp28jMKbg zi-qXIv0~C~*YR4er!!D7*1AnP+ge-DJN}0i!v4ml6%LpT1ovrr0x-9}X^}nz2j!XV z>;p*BlO|NpnBLPFBBl_tuQJVb_-4r9II1ev*TbX)e}Lba&m-koO|cD6+-H>DJ@tB= zL06r8%3SiM*|R4AJ6!b}8N_f8ZK3?#3jh|hYV(uzl_@uN;Uq9@76|vX##I6l*{nnm z_8oV|dEg=TtoN<4Q#;(apVy%Xp9SSdGUN|3S*!cmsG`2L>B#W@w$o88OqIMF%;60FIN1#( z(9V~&8?Agv++-ko?SvVS-8+OokTPKZ%Qi{j0}W%!)|0C?PrpBLEVvp6IOVS5G|-b# z@?Xi_gQkCP{eN_Q2RN4P8@EI#GMdONWQ0=LtL&LwWQP#4H=z_lHksLbMCcI@**jbI zNO-Jd@9(_z_Fv!k9Y@D|Q13JDbzSHATj$$TWUwt6W0E?(`Mb(akLfDp`Roc!pHguy z@P{q$+U`iu`ysXA#{OA)j;`4$_EiXomstde(A>FqHE&TG%w05<4n8w%_^CfOyw>t6 zPc3djU_pb}xU5>h*lV3v=6!?2iL=3Vz(g|c=ykc!fMdF@uC7d(opj*rvF{q2{>4hu z&`R6jmV4Bupd<4&k{+Dx*qcdAxM!HSP9z3wiX~2>miKhv;UWVyNxT!N&xXey9$2oA zHlZBc!k+qp3sGkMRj!nP_0&rvt*URtaCpRrF5AmXT!m$9XVN&>;?TJqga*kRi(VFw z?xcXC@Q`orYE{&*U#Chenr#rypK=qH`X!nyiXOx&?Eke61JtrAPe$;3gcJg6?GTGYmmFYL|%$EDGbK zWFFJt+fH#kR2m6&;l+auoOJw*)Qk%iR0>bpqPt{2CACawif0lm*< z&LEd$TPOOx@9SBDe5!vLR*>b!?%Xf(_%l;QhIcsb34#?@aYyJGA?pV!1`n>~vC82B z7U+G*BC+%tdldn!PJBWzNZ&vW;hPBBeO(m%cZFs5QVTit$~t zJ;?_1Rbv$zasCtQ*e5{*SC)mA)DZn%f!9e}Fi!Q%jG zK07HSiH{jVDh8bpwa=r^#Rm1Kk+1+*+8Z|>K2LmvDDrLCNCdoqZ<}Di5iUJ>@btbg zj;}>$^EB9v5vqCGK!_Sl~WvXEg0Vt z0MJPf#Xtl!-R8yN?l056wZM;`q>X4K5OLexdD2@-5zq*CbZ=iqTbCI038@P?F7(Uq zxF_Gx%21%P=tk4DtnN|HfOHJ@4*VIxD+HaKFQ;FMMyf9+=w3h9vT7dOd|W!ydi{@t z=zPIwjcn$-4{u7 z6%+u+)GY%kr{|pbbqK2<_QLap&6?oejcjbIa}EvbaP?gT5UEvYu*mh5$>+kg2w-xO z%XRaaC&i>kg3}a%*K7|MdUrPlMc=6N6U=Nc1&RJslGu?*QBsXQ;rOS_r=1~~NIS1% zlLzX(QZA?Y63A7X5f3XO8JPe?d+T=_SLd9LDtYTod9d0~)XzW^aGtHv9)JL6)CInP zE>I_yAoFqRC>|oKgQ{$y;e{)kR_U|t_|o!MF&OQbbW7D zvB1|r8$>5jNYm2Ljh0~F_k|UP;A@5`h_UHmiPqQ+9!~@xl^S8B>*ius3GPWkDQNU? z4U0mdQOkBmye3~|X~VakmnZ5&AJz-@ALF;xZ%!D#C{uIFuSxcSDc-U$Mp4J}I|w=V zwgQnSugK!x**5S7^zZ(bBq3f%Nz&=c2Q>gK33fr%u37TRMaO+`Bl6z_;n+ZKXZ%wG zu+&DlfV5qHy>hO5hD#K?hU{161Nn9Nk%fHe@iXmWiK}vj@~HlR@V)9SG7ZoN#a2bt zrtfvyi~O;NXku>%H#X2~9*C3>~ z0(A>^Y3rBdEvx+0Z(^>2#Qe~BeRSwNnUoNyqeI+sov;1bzb@1M0?E#M^{&gWako;T z+zAB`>-6J8Lxv6{-3Uqr6@L&#se#1lNp8b!0vz!qr`4(75dS`p+10UWfn=(b^3Q!n zFRzmlY}EkK&H#0DcOZUFH==oLj^iy>=oE7zh=U{yzZ(aLr`B(JBuj;Mi`cYtv)I8o*Q$+=dh*UXAm?F5B&;&_&dr^Hw8Qd%cpdU3HT9tS00l`ny3= z$t{@v!9cv+Utu>t#o=A}{LL9*zyWpu^pQ_EKS75*c@^E<^4R zn1DXK#nquxy_xT;zJa+LKCLnQ0}u<^9sHNSsn0fYEdq-D8$gNcB5tQ1DMJz&RUE9# zvcO7&a|+>;^FxNDrw0N^B$bY8@_?!+xBxXe7ONPk>;?kRdH)Z>xQwtcWe!%D)nQi<;Zd^lBh64aR0hM(R zUs>1=B-mQu#1Y@)ye!_f4gSL&qU#SXHe4ccYy6#Ob}zKbo)N2NF=>blw+mEZWAt1$KLK@Ctrh znNiGaSebdqb{{(*%J{Aeq-WlVTt^%z;kUnEBoN~~Q5Ee!5muWoPK97Y zg~nj1k0E%*3B~uxW3bdbZF<)>%c^r0FPH|F-8=H#LsWLq@=8Piws z#4+CYg>#_kR>Xa>is}C8Ye5t~bSED(Q4*2E1bc;g_ur7;hcjC@NBRS9QJL*9Ktt+D zZ=5|Aqx|{kTmo8w_bd=r5`_psHEx-~~%qGcz)MGDWvA%Fr@4aQgubF<* zgvD|8mspAnaK=xYlDXi+$A-V0dnhPM7{JGp34h9xN%`n zJV+NA)b|>+T-xeP{@0!6%FnL&|dWG zQinBh+GPT5Iz@%VXC8h@U3|GF!~@0z#AOdsC5zLEU8&ONzL%JWt-}N~=24STEc^)y zZ_#EDr`Ub^HB#sjqXVH~u(tW0mLYWn=FBmI6uppD;<(^@8IScP+yf%2 zYydrK1@bn{yldZ+MmXtKYao+h!)vUP_n-X8=vH{rXG5!;R`I+dzr6X zzOnjku(-%K5I)7HrYh8Y{U1%_suV4#&|4ho+!Py4pu+S1^&C-`KCgB<5$4HBeM2of zd>!lqX8pP3!I?<=3knV_V*g)}Z*Ya2eCbLs09WkO5rFKL9-Jo{3HsaG<6uq_&6N7v z`s2ueV8e^d(D}IZfbJQ9x?9+sW^-W1`^^hwv+4#lhr}dWO&{kBL_3Ey*> zB1U#yvm}%7GN;H+k`KNKfzK%?j&{h%6x)o6{}Vj=MhJV|o$B90*5l?riQq|Vr2H?x zi#2CRC~l?hfB*P92`(2dME$(U==cOuG}UFWfPQ@T{Z?4Q@nVj=435eK8~;Y^}E<-8cE0^M=fJ35(8nu=q7Gz2*vAuh&)9REHa$css^K@oIoBNbtli2o{_ZW~YB`JcbklbwK|_mA#-4k{iXr;_927l%dlMlpn97-VJq zCl~#%$NTROPur@^|Ni{ zKJ?u?lE)|Mvim9=txUm1@S58mD~;UlY?rLNEHC_Ur%p zkbe&6|N8PG?{JV%I#|;lz58@u15&Uo&uAO^Mj`=8-}XJhn3Nmo!fyn3-i>1AA(Fa} zDv0*y|Mk|$U#9Y$@QsmLRjEGvU;l^553)k8yZF6a_kjwzf#(dWFImBee_iJP`-fF| zLtv5N`2YIye~-3K5YUji0|pT+h|iI++>@fs+KL{! zaG48(cdvh3nt?efdlMqVwhgLi-a>k8P>;me8alWc?b zAq_Sd=wSP<5yL~)r<^Jkk~D-^&Qk#e7VR@aycJD!h%=?N12GGL$+6#cLS9R~`Hk+P{!P%%GDO@YWHuM##%NGbF z$NtdtZ^lXi=iZEa>-^Mfxv-H-)upk1pvmy&MHW@dJ>m zcwK@>YUHl)Cr#Qz{YpY3xpq+MiryImAzREH^GLKM2(XX``p}M+U)nvZb>S+>5V(RT zuOcq;)`lW)CK6tU#po=agyTu4(u#i*#;@UdA=5OFu?e8vKfV1ty4MY?v=I^C!9oxQ z@->{1Q>@1yE=%o~bu%vLd6CeF zCg==uv#*o^^N!T0em#5sh_Q(9E8`6LwPWPOaC_!JTwgJkZA*WaJ{y7s-VsZF0mL^M+TI} z@Y!dl!qb-}|MA$qSI6cBn~4;NOpt;#{ibnqs5s0mz5#QtqM13ePCHqd_UFdz!!sKx z55q_`Y@q3S^fRl4#$#I)Nj#8*9HCqXEpW7ahFUKT!X1BjnI$id&2R6)XbR;bjEYh!Qci08eh?uWqO2P=;FLHY}f$tTikjM1CNV z@)smWtp8Rr*O&m$g2BtJb5@3!2_I~GH#F}dBP zwf`WZN2J?;0^cAae?SjF6s$dI0**PVS-;UTbb|{+pK|i(PQFLteRFQsLSc9EE@1cg z)0_|kH{)}34)m5}mwB8*5Q(emA^>6kCKE|oah|eMx0*y5{%0xyKI;jltz><{=eO=! znzR*y0ll-52ShhBsicole=}fcKjR}5qh{$d(;Z>yrIP}uYRuudD1ZDGT zjzrDg5_f1dXJCC`6#SEOBaOo;8Z@f~0e}~-vN=T|3u$Ot0C>)x3!J^Dh$yXsjIi5Z zp4bHp;Yv=;yRY&S5gb_`Aq1)ZVQrqUwBXr=e=utiHl!tDi;sM}kq1KeW8ZSvQE&*D~Q1L9vN?$O}2LQ*eCySVMJ6D)tVosIYc2-*iTmTODZh55ie zLHPO5yOKtUzaQ3*xJ_~s^@ENzHh=&284f(0*PcIm-|fwLZZZ^l8rB@;B7Z4XPGcU| z&0MG?myQN2&N(f#I9pFB-g;$u`;8QH+U+pLZd!4AvSiUYiL}S}8ir#hC7J{rLM;97 zWj+pFh*0(o?apoPrDa)zk>0aqgNWt|QITFixrB+M+vwjbkjW7kaDXU|!KvhAHm(y$DhhQ(nvRNnbk%^I>0l z=Az&UgXPl8Dwix&5+yf%o4(n?kdQPEkJW&A$ogS7KIy-b))Zt76D{TB7l_~u;afbo zNYeJeK}q%n)}P-+^pdJ!5L2YkMwTMc6|4MU62*`GKIHU*76R=Xxh8_<6PS4G2J%2_ zgM?Q0if1W0xW})%oFZ!KC_EIcapox)TF?*2tPc(;^EyJm(cRYjD82g#*1-_J$8yn$PPzOpn<)p{C7 z+BQ2zW9nx&FBaZM?TxOCeJ`2Xo2wA~*{4(BI=(m77g+B^m0q{vMYBJFHxDyTGCi2;?7A)t z2(CDH>N6_BoUG2AB|C_uO~=Rao2TBc#!H8&W^Qq;A4wA-A1;PU?v8C$Y1qfw*Aj83 zjvsE8G3OfaPL|YP)FR}MO_&~q2exmsd>OUS4g0)+{kc0sG~9}SZ`8JIj2|wIyL6ac4Q~pjQzM2ME2Wh@RBV4aPy<#iY;&F?Zdmm5W5is8Kl# z>T3t^K;Tu6>)2WXgU~6riIX&Q?{x{q)O}0uPuG&?zA;?w*RqQ50O3u3THW0sLzBuJ z|NIcfC$vFbUF~}Nm16ItN_Y=Tr&nhD6x|H`*(6i0Tkr1=4OB&X{0>V@{k79q;Gq_8 zp8v6hh51nYm+V9K=HF91n*qA}^an69ydX$F*zL#F*N6H3_5p^vIBRfKJQb|l0#>r{ z1m{ErHC)Mp#=FX`sTT}9gI4@=@=@8lm&SEh-ANzi-TnR~tON={^~Ugor$WdB1h?VE ze31o8Qzdijs{C&|E4x?M4)Brt^@z-=zc`_?1&z2;l`(!MjtF$tx$AgdHJ1tGu{-Bj+u5pyixhQ^IGoCJ<{ml z*|$6CR@1K6tvd5jSA%#AhF*VgJ6UE{T%7@8o%W;8@|A7$UOBbuTiUyS)D)z6f~>@>%84No5sCk{BS{>!$pY|FYI>u z(t4ZdyF6{j(NFfUpGO(1^v#I)HweIWsmxjh{o1;E)%X6(8MeHM=Kz(u zaPds@=&MgK7ra`1**k6J)=(>E<9(t`qM7Y{yWHYH+Dys9MF@>=;TGG=SJ^-KGD&3h zbS5^k@5#-_x@foJPlBx>l^u0L5kA@!76k>e=*dC~HRs))#IdKc%f25jEX1GGrAVBy zyd3$oi`OaZW`Wz)FGC41k-L#HQE;bOk-9uid02NNTZ1SrU+~`kQdAtW43tTleqmp_ zMG4XlLOx%Qg9Tly8LZC+!@~>b#$8uOOHvu|>OW2KknU+7L=Wgl%r3w{n{k)JwZ!oj zFW&nczjKQ_%yRE{2=T11-!wQoHuqlgta8@oXt$j4-Zqs5#cUBfUbh;WSH`n3ewe*V zF=0KG>C==;O&og%`I@oX@lss<2KVRmt1R9FAIa%PMiM45UJS|}0))zJa5e@b`86(G zi`I82i#xD)IJn_nD6*K6cE)VSMAp9c2{2lgm5W9k=JoO+-Qt;09((?}_f}M|6!&p7 z#cIr(@9h`7b{smQ6FrttlO*G68=kA$5sB(?{YtY1o(JZu{Q0(Htht3>OMdb$HX-GL zdAxnUX|Ck#N7h)skcaoN-kA#;Qkn4I$TOMPMtfWj|3F3e%`D$IFUw z-Vg5{H}5d}C>9X37PPx*hF8$8&RHudyxS!an;ghXDcxz$=1gS!Y?-(AD_+{P9L~#w z)nLzO$TH4a`q^@rI>@$Wn?F{eS@BodlxwzN@cfsCz%~5ql<97p#fcVdh8gh=6vo|I z*6HVZRc)vSR$WILQE^2n5nScB-S#Y6;~u44I6-4u%45F(R`mntcp*gWbi9(|g%hcag+?EBf>!9N4Ypa^rB+^Sr0XC5Dn&R+99-aXq= zMXEmEe7*@0v4)y3G8?vS-g~KEmiL1e_Hz#oPY^qa>@79vtOc%=K$)&F$|>iV-i0e_ zw?{I+F3TY7o}jL2n${q>8t?JlDC!Po_(GRpy{yjf(9MZnbGz?_z;?(JIacvP-e zZ>gB+7;KSScTokHILFyNDOu{_Ehq(-bC>soP|8pi31*M;xl6Xu{ERJ7yYyJrMS)yf z=?7&Qb0qI*XfcDTwxQUTZf)dzu2$QcZQ|Zzz8~-IXRmM7vZstYRUR4}+o0J$F^xZ! z-;qA@@~n;TX&A3Iq+W~8-V5s3RYE5=yPHXOuR7-I?nlx{`sFiXX5f)koK0}GtVFFm zx5NDfr7X%kjwiVJkpv3UAV{K387@;|(cJ=~w}gaWFPT!WxZTZ5uiujBh<;!{&Dr)6 zGJo}577b})r(DgXW$d;{297Y=;cU%c8g91p1dSI_PYd*mbCa#QUmVqz&-gL%?Ywr> zb*MisOC;@`{=uno`StNtkjoTV*NRe!NZT(_ZZU0BSy!WEx*N|R91<=B8&ie`dbh8S zKxKU+gCv^h#NNWQOs${SagwdRaaYdEGiSKwTS}}i2y^c$dCbe|zPE9Rn`#eIDrpor zUPDB*C(2+;dBBa1{L!FZAT+~2{I22==E`Ht79>xb3_;@)z^yXPI31~F`DoNBsM;If zsYx;DZX`K(`tPt7W^57+!!B*`T84`sB@rH#-GfROGbbPlnKQN%D1x|G`RRn+NAO+V zcG5>r*%!EKf4LDEeOH($Zg;1CC0Ji(gtLV?moh~Hy^&TOxcSDq=OEC^ZMr&kl=p_4 zYiRyJGzjqu5^Y*`OQ2>`9Zu1B-YU;M8gAq8I)DB(InVE=t12F$6*n=N{WpdpDzHr3 zgjs&~^z>Pw@Lx2Of8NzBH6h{dE`wSW?zjcHl>wDAyR)(TKc2gEx{j$7z&Awu;9Ihc zZU@t*$FRdPAP6-lTmC@d!uetMD@L!W9He;7r45y=Q+%2pT`fDqihDwoeaK1 z%9F^2yYarguD5f2m0#Tsp80L|Hx`dLVxN7KzMx7f`j+%@5p*i5Xi~n$fN}EE|CF$L zGI~{Mgzc9upT}0g?izoZdZz6XIz6JB?Z`{j^xN(Rn(xy)+~4U5*>)96rCPM+LREbw z=%CJ_(1xqO_QsE@F1UAlOrFtwI+H=86~f1->?(=s6?L}@&Uh7hPwzdpvadq?eW&6= zoS!|s(^1fFc+KK=b=uqA=Nu(FC~I^ULwk0*FWYAD4_8e~%B*((c7V}-{jg+qMc*C2 zY_^12$gejY+wCvA+hCX_FDy_FlN=x2 zur}oJXkBOJT@xFy1D&%otMW`#siYkZiF>o82JwcGxc~Z~PG!rpT&;sR&Y>$7!Z3}G zvi)u8%BKOHzzu(onOgep-m>sBRP{T3;KcQKgXSBEG^NphhVsbA$U zo4j5T6Fu+FG9H(Y>hsID?|^op0VUqDwVk`p(V;J?CNs)dQW5jU=*v#Km1W5dR2q5d z@V7aCMidNvQdkx0P#ZG3i;8Iyb;tvYSY=oKYk?nhOU4&;Wp!4|xLR(tSu@Jj=B=&l z438f=u;R7c?q~k8;(4@J_PbWxJ$A=m<4ZD+#j}+-zYL6bQgN&Q+%iMQ9oq>dYt$TB zdhZ3L3evD|?PBieD;tg~xeGkO_DJP!tyMQ}H(P3M<#Xk7KOZJDB^|ZII~!*^MAKfW zG5glpQT<;vL(i+M&sz%;0Vc|{3aF`<3`fm&2g5KZU5j_rdEpXzTbxw^27652uyU4V zW!LO)blP!c5E~& z)^Rrk&j=i@23yTAt@BGRxzKQT*j#*}&OeuQp6%wG#N%b-!U!&9yY@0c5^f$_|1F8( zgzY+}Qc3123G{5&Vcdn3h{RLuK`I6NiAOt$4K@bU<8T{eQWS_f8&DCJ=M$cPB~Y=* zMmH<=m(tUwRgJGNG$M$bMQ0gn)FvwERx| zc3O4Eo@$h=)N^=PS3Ig82puT*A)i{I#qO42@P2|3HL=(#XB^0>8&{aGxZzDpFO{tO z8Izr(Ns800qcCg!Y*@#r+nd~Ys=^djmQd4Wo8$q;=rC~vL$)FIHPh^ z=-kbU&-9#$9pq($1_AlDFTW5`)UH062o0AMm?8~y?wdX8aICVGHr9?~bX5EHa10Ek zL)`B^2&UkRG-)f`P7h|%agB(>uDI!utGn!=XBLX@lwV*=c(ihKxPmMY^xGfSQ|$!O z1m>DYH{EQ|7`3}q3;7lK+IKdFR(y}>USGNOjm1%I@L{48i{}Hs+><@bt98$rNzl4U z)#U?Av%*KtwFcjMI`p46kel32cOB|U=npd)*|Q(Hox4&DCF?zV6|~9KHM}CA{>P6h zZ(6wOq!1dvad>N(=szQ8!+F2N3cv>Q=*A+)Xyq1}unK{>_jV&Sgw}!C=aWbW(&<~} z(YJ9pSOGmbaslL8y&nDcnK)Ux_G%0NSZwucu|Jg7K zt8st#^fHfcQ)j;8!X~_uA#`s!l75Ei-a=pCB;W~UN@dqBAfQT%V(DGRcJ9;49gN>H z3pDCh+K(}&`26xIe+li1sXGPE@+~n7Lp~}6T_y*PRcrC#DqOeFc7mDw@YNvTo|QHjZTxU!G)e16(slq+$k?fgL|@khSJ!di-+ zS&pyY34s~# z8A`g@P2cbO4;4wAMW)`Iy|azYbAahYQU7{@CwI`QsJ{{4 z@cN9F-rcQ%S2wJd{(`_E~U{gX2fN>DO99xB;1G7zS1Us zNq!zRlh*fhbPRZR{I*;+ntV01EIx%-*k5)!$9}OKYJP6Z{@J&r%qFFmAGr>3wPaD) z={Nn&0_YmH%FvPJ-b)u^`x4V^&x)&Fx3iq|5f*iBiO)T2_jNey8_qIob}b*fk1=A_ZtzJcW5n>qc$t>voRV%>mP{kl-e)T(*YDi zjN$&S?7j(b&YhXIJ1R#B2a6^dae_?s@?*d?7NQi?G)1n&+LrOA1v4hI?+dTRbf(5B z8Dwt$ASA0_wTZeFxQqY$`hon)GF&7AAFm37+(Rnq{2T}f>JvFds^$UfGh^pVdE zlREiC*f}VU!SKS{*xuxk5}hRoR>--4ZGH0w{fezjPYHMkiTe@|T*YbiR>l^k+3u#) z`2J7s9HwF$8n3Jaj*P1oo{JG8_mS%-VW@h%8p9*|^FuYKVsE2L>T1cdo$T^^%ZX#K zGduR-Ny#e)FGmvX4091!RB51qK0zlh3gkhhI(>JpaBodZPq{QR7w5!kZR{Eb@iHO^ z*SnkOSC(JoZ+QjW1bRJ#XtJ6c);?J}zTcxTJ%kh;2Q}OhND0!Pxu9_q zfG(yK*|Z;=a)si|_{kc;8ys?m?b$1w7=PG@k2MG=JHzok=_Kw8KJGu zikd)yst7*NAJp?5RF_pw*7A2LBMZCI#yM&q*KvHa=j48r;oF|@>A@4;u4%{?f6)mx zy2RwYcTIJw7~`y)A*X?>SSKNA`*u>|4szbwth}~sv^gph6taRZ)p|k!qbwLrm#}&D&joBtXgOrOg|T&0>6O$+g10$C>JGzyxw0A4w*Vi_s$1^A>T~D#l7fU`t`JuBq$hf6sf%O42o=-Ao zjK*og>twKEpr3IjeHG={M!sX34!NxH(;~xoizmZbcwFJs6xqS&0@>`NZplrjETp-_ z{IF!iSWw*-h@akiMzt`tO*8sj+j%A;u+PstBjnY!!+NWNIT-|&V>e!XDwpdGCyNRj z6*E`;D-t@b}y{V{=8sOqN4( zZTXOS-Y0bzsN98~f6noOca$MAp?6mCdDj7%_#eS||ppGNU_ee`$WWAZ_FN z=nJ!8;V7MHf(0iEkh-0f8Q*T;PN!<(G+m(he(5alk9QKqI$NlLw5TsX80NXrJMFzD zA*0iPn;W?Frcg&<9W-mv7RGDO2CtlPa+xw+8Y(~_1?9%=VECBd?4Nkw+r zw)zC#ZALWs$$CD|CJd_`jjnib_Z26I6{Tz>e1AW(AlI z8a?+V`Cw+5$`d9Y@Z8SY)86Ni95}zB+=8Xr5?7dd!gAG7mxHvhjHDILvH@k?_>+Cc zISuxT#g8oTqRGpQJ!)v@owrzCub61LYQGNu>^~;cK8gsKs&b^gS>5Hix&K;tpRRJ| zhLx$?wdfC`$p&KBvc+hJyL8FWyCG10Y6J0w_OYGR0S^1nvE2=?UNhDK}KEd(*|En}r0{7_7^xPF$|&{hl z*gf_)9D5BFPyQpZs6@b)U6rqN8fbX{bv?f;a7qXES8j}ITdoMR)YHH_{ZH9$V=&sH z$hf$`Ro|>k6S+Xs+7mT|J}=!AkbTrvf;Qn|@LPfq15Smv7{y!l#CzZp{_KBvL)>f&!*!Qk zN5`g_fmyi}q1pu#h5UMB4`c-T*rPure*3!nEPZnTWsth(3dGw;1X-z&0wEfR(R_t# zvIq2%GT(P`Q#S>sz8*l4J3zfIa}03flSHd|A%}f3N*5I~WS-1p1q&F;_PtZGi*(Nu z$Bv9O_pEqQ5y3V-hYl0zG0CHTh+)~RmY#mh>><2~QKmH}Sqfa#i#KA&E&A3dV|Mnzsr1y(r{q*0Ha3@?%V)D-1LoQ1i zx>fp&adhXND%xE?x-&V)Z};<4oWaJMm^2P|)7`;Fp-7D4kyT;MDN`WusEjuo$Ulf= zE%YmM-hblgXX(t>3N-8pFVFhk+}9MHDgAV1W-4=8V)O9y_%rrlHo0eJnFQyh`_O|- zh*m(|izuy))H;tDl>rij3qP@o?vl}R24FkJ)EjBtJ}y%(`-!9qA)8&|-Jz!`ENS0^ zfq9jnz0Y=iw=*!qH(9Y0$Q*RhVmY%G2Pp}uyKjTaymeRB`gDbmyeUt@DE|yG&D#50*RievX6h?3Q`EKEB6E@$dCu_R_ap-*S;D z>`gx=3A{u1BJczzPRy$nxTv9(BIk;`RU*?Uxn*y!37r#nd=g^)L1#(;lkaFDbjkjk zDBgu)$4?m?7GWNv8YLOAy#vaO{&O}<;7gPPT0x^z#8+B7vBSX~;YOa&@&1f9>;}+H z<#9n159u&Bwq`z<6zzZ&RnvZ8*HL`v0UiC5rpzd$evt1z_kGPdF?(3iaXTc~?a9!y z;&zHKagB4R8t$VB>QE+W5^|p)~X~!miJMJ;MIbRk2(D`VnB!b^8E zzS+Q9RG`_HOI5#K1wMN9-CTEX==^svK3M@UU{&oos5_%^u)Bx=fc3=`zh$?>5KGkY zr0MxKwLRdl7KmAvUus%@aQg0IU%S}I!-6Ot4P&MkL4JEnL!=vSN~v+e6^01BpDLVY8gPQ$dC}fA^Gf92stDOluJBD}9+K zPXj3V4qBkQjn?_)eQBN2$WNH{pu>OMwySepE@EzWH#JPbtWC(9ZyB)P4DfbcsOWi< zu_ep>#84v5ws1~M@S{#!y|PRi^aB9gln*d{ z#MK`bH#XOmN=IwmfN6zFfmz}2e8A8?SibuTzQM*X~W;?aiXJIvbKNni+5Fn9GseaIOP9$p$qq2AQ}-kenA0wsfLSwi2P|2F}FX=j9% zOs6&=bG(4PSKNQ3Nv@mSnvN@>Jb;OqUv}Q$$?WXd-%_lBnS@(00l>K}?I-(pDOH1n zcIVDBZ|y~t(_Lon;J4b|r1V53il=`@Hp#$dpIT&>jg!z%CY*;rZ$+$~v+C-IEhAY0q1A@+DszxWhwJy&>=l0( zcf!V6nY3l2acJzrB@(j(k{Zenoj{YM&?i3?`}uh;s}W2|KVt0Ch=r9<+iO`jvv~NT zEYq>pqVUxN&LMsh?k96`bu@kKnR2X^U39^iFYPKr&;b9GXPNCmpc z3s*r)$huV0qo6%qFJh;$ISv;paQ9fyW!1EcXPx2%SwjleVQ3A%XRZiE=+c*?qb5ks zAD@51rQOzaLH59-QK+wAWn%@P4pZP2Gd~{oMU7kO<_7ck)aq+<97`kv={kz1o5$tv ze!pf>+nsdgel`=1(QYh^i^z^BKC>RrzJil?<$6~O_jW~ke3qA+ z&zDcTn4C+);Z_$v*)({ocgPL2F#n>If3p_5UuVi{5)%7fD4jXNt+q76ic>!D(gOLB z%_WMoE-gef$7+&j{x-3gsl@X0Ok~4t-V_~HVWP3i!jYssbuK>+OSForc$1PNtei@I zx3@~f9;3=X<9)0t*V<2f30&P(dxn}%4|HDVLkk>kPg71excrFS%Y$Un59T*{^->~y z%r&j6hVJH}e@;y2oATi`R9f<$ygy#rKdeL1K`yJgCSlU~#b0Vz4#@091Yc0?s%*T|X$07gz*>nn? z#LJV@G)GQ`LcE1W6D4V-vm!ZOJvZ9%rym;6^YqM9tk`m@EaX|ao7=kuS&;5Yb?gqd zXye^xGu-zf7IK;}vx1XWZ#sRMET6Y2$$qV!TVrhp#CKyY+6J2kL7+<=7knO*5oaS~ z9mCrMNT&uGWe?MX)Wn%M0k|`_Rpp^}q z{zoRC?hfktiza^tUxn#oe^J_eVbZ_Mu*tcJ%u&VVm17*RZWWAkm ztcK9qKG!#^God%;*1-8W%xmZ|gpN=NyW}}u;<+UN&9P#@eF zHH6b68AH?v2)XNGm+z=>ay;Q#maZ)K>}vw!H@LIQo-U=lj&$Ezv}XwWglY&!!=EU3YwRT#qjz zRT~M`G++NrzI;HC^`b67TAGLyiPhv9hWs6QKBx*<@hpq|sA_l($6lWHS|yk>HgN0z zBa^yH@(`N;B6QXMkGgNR!6lI!p+4t#(~|>NZC~2%P7-@QOuXIx#O!;+SJgIVukRze zd!jQ;w7c#C8;??ngAV;J{4MGppL=K$Z=Vfmvj^4%WLZaR?M2~Ko8Qn|mE)Z7PWNdcym&Lrk~P{oK%tGzur>D9_< zs+H>rvtks#)YKoP@g!!x1-Rg$VPL~-NTh)9tM8e?Rg1FBg_Kt}vzo}ccis3Bf|FxE z1-rul^jF84dfr;G2pfLkzJ%ZHefg`&vA!;suBWZ_9N}?sacYhVNP}5wi(T48^ir7?*dd+ZwvqEOo`iFtJ6kBO?MtTbUb8 zHT?GjxqGf6indscP*GJbWZQAi#Exy4)yiJ~h0x{9COt6m$cp{aLdvZmtFCyZvzO<1 z7504Ql=gt8IQu}m+n}%=?qm-PJ7G^RQNT}mo$#(q#c5OEXc{SxtXFb?dTwWqMlp&{^ z#)VrxdsCVWKi$c8S-0;!SD13T52?KXkt*~9iOxXvsb4Xv_xv~W#9sUExlZCnZ(J*2 z!XY7{ILWhrG3mXDE1+(Eg$JOou}-P>y#fvNbFs*e<_TO+yG4#fpPNCZhNJMPe){V?ykn(cbCzuQZO(>*9b)#fdI%hjb+zK! zPya~R6kZL-?6UO)=8=uZk)K}LoKhXR=#gNoqTox z;jy2=Zdbolwyl28BX84Z?`Qy@(iGjr9t<(Jr=3~ZNeV*i}NS5kM@~XOP z4S14gu^AN>J_XJ62spBks{TgKClku-p1GoJh}wbn4C`TW60berXn57CIcIzv9g^_Z zDJNJq{FV2N~|p;KpCQRAOW32ZJ=||R)potG@q^hx%)85#N<}@KYa&t3Mgk=;3Tcu2+aG?d;?))$ zH3gRRWb;955gN;6$YXMUa=cgaJ)zyMNRc0RjJ+*W;;G>6uQoT$)tusle+)a`r0nOh zJklGAuqnOT2W(<~7~Cqe-u+OZMpMdVTlq7j~D`@jv@7uCn z5wG6Gd7E$Q^O+5sj(~(hzwv?XexiWnaJPq=Sn}UKq{&!a$$5g$!>mD|YC?R* z^#cVzjW(A7NSEY8O8DTFjb3%vzuw&x2!!?n=A%IvuT$SUSq~J%N+zn{(?c)y;%<|y zgQ{?xZyz12o~d$+yuQ_hQ11r2Y74_*9l@^t9-)Z(+?GD72+c(|X4A+Hcj;7H{^lZK z_Zo%g4~<11R#TPt1o{f7vT7wEbT_Mfs)rg7&rArr6fpdazUnnEWeY#wiMq~8@)3^~ zbQ~9*J^I44Orbee`$Dhfn}Q9Tv*!pO6iv_WoP2H)->q^H;GU<@-dAxR^1k$m+CHAM zP&MP5_NpA|LIg7tsq(kU@B5>u=oMph}j;uEJ>_^a{p;6Wk$i}3D~CwB#5r168*2RAf!Oc2OQGjSDJ zDy$7pGs8V@&nx5^+PMY|=T5lRS1oy22g)yY#hR&YAbca^h5BN0 zpSs&V2pdk$t5*s47*W5!Ml-eh$n5ZrCe`%Z^xP(^M)Hhrqf1AEw+B^4TsdCn9hP{u z$AzAoz<^+*N$jR{yu!COuAsX?LL+UBeMkHR#8qEM3psZbIPi-*;k$tUa{GtMm0<38 z>JP|nWwZ~tt|b;Bo1gK#sDP`?6FF~kUGv3CxI{8LnNG%9xy84wT{w*Ug|r{ia+}!u;vZ9GzdA zH1|Uk)$W8AjDV^ilMe5nd;Ksk*cMmdL`|*Q<~96};~*33&as+>)v#3Ir#!D&SN{z! zk;7WokQ6>_`nibwhuMKd84g1;Ex(F+y<^I)!)G3GF6{t&aws4zd9b&gKrb8MFm$qU zd&OLVzu4MO+BkH%+Gd;lR;I50&tA8rndy>|{Gy7-8r~+7b7!Ukei2@f2gzY(Qp!Qz z8&U)aBD^dzb)Ga$@X&qXlzbh&;5)R70p+fOOB3i zKLM>Vqfl4X-b_O(*rM9ER7TLmePkO1!swdjex$5$|c+8;aYtHVkY&5$`X&yv#>1EwiW^090c(yole^i zkKPlmITmLjRZs59{Ia-NYUo>wiyhnmbtU&uw6(vFje7SEgsk;tia1&Q{o3G}e+ zL_Hn-fNoKRZ6#~97EcA*?lOeyRz6Q44WDmV-kc+pmKPLkpm6==U&+y4W?z8yd3d2p z*+I6n%?M3-ssUP#hEzHT5a$TZ#T$OA&4DVdIrb9l*IeO3(1N6tJkh$g-lH58JlsUMf%YE77rFLG#lX6J4qaw{JTM4YHUl5xB%|4w|a7{7ehSI67zEWxJtRB^r%R@n&3q-m(OD^%{0)28 z0eYM}mhM@anq9r5xI}8{bS^&c$#YiON{Wl=Uo1ZnqBhrm7Rfg$*EG>%t#Vfzn;QXmPkSd9AWG>%R`D5_2^KR2#=P_GHiUX@ zN1zNjdM6_wK4L+6TPB;IKVnk2kEv`!L?Q_jQtgvvlLodkv=|z|g!_gZ4K#5yrU_^S zO@Mw$$#7bAc70CYOMKBmw#kAw>NG@TnS9qED+{-IaY&H2V7Jj!=4SWhWk4jFh1)Wz zY6F`Dy<&>jO`1)>VB{;zBc!{rh`{|x#?!D9tW9&ms7!5Pl7vD(kxYUNk~~ENJPHj127)ut#o!xPLpJE zNQWMZKe^)EXEvT7+yeR>Uxh{~&{55M1*|V*RWM~_B`^Csx`vtf=^I1s%{}!xh*6** zv(fkab<_RtjpaBUiL`_Ay;6+dp3>>Ag2_EY}K}EH9^#R_dpdGxQW+A z7Emji#Mhoe%7vkfk@5zFh4U4qo{ZiG6U>aM7>3)?i+41v)HNvJ?D!_ zJTg{MNGOn+AB0)koBnRyVg`!1BT17<^^NPZ;`Skz!)XWuIRPGRYf=~>Y|lo*v~cNY zX_#L9f*n&j_wb*Q8mz#?C*s6R8X(h+g@+Ojzflc3v;>qv65z;8P|C`=mW-Is#l8*MJ%#WxEU>+rNlkJdVk5}rJ1whH;K7p#@2#)co+2lQB_qq|z!P+O z3s{^E3660iU?<h{_g6yT=X46X_`r~jYp#ywr4~+>w`5hNPPgX^zH_wD{;>)S~5EqeE>p}Elwt2z$Z12!5d?57|bK2rL}kIy%j#DUSW$iBV%I^r%m=UVkc0cxPZZPvhU-4zIv zAGf@C88r5l>&wDCc_9V?)l=a58TT#hZE1l*7poffm%u)B(VJ^niG5Oo*^lNg1b1lS zxs_3eMfbG<^&_q;0as-B{2eRzKxQWMx^IkTP%l9L~Yzy2$ z4EKBCt^n5z+3l^PL*8L2FM-2VpxSn#>#s>fJF6{41zL}~wkOZX4ZY%$kuX;b!T@W% zZo?Z(xiFX!d2n0N{XVlrzzL~DOUmlyy`AoTv?$k}=rqcJzV?1I<>DF&4Fx*eVPQ?+ zE;t3$^(~F}@1Aq6*`V;rLiQAaIsgG!8;23S&&dV`c@7X%+ixje71 z&H%ts(t)70z3LRmictIlitMSU+hzwqQ%qBat~lLQb~OGf4r@E40uM)yogMd7s^JM9 z%Np@~wlc;3`GFX>iK9mTZc&c`GN_`wEg-$6*=vl(5&HfJOa=jK?tQcw}%;jfA@Q2cVxQi0bA+@;1fXNsYN#| zAu7?KAHhR}uRwYmXuQV>m=XriBp6==MRqz+lbii8*lSbP8Z(t`fCXJAZ&fO8bB(rx&G80i)I2 zaxSjLrei<4V}ZMb0D2rr$oZ9TH$DX znAGnaNF(=TCct5d|1t$IcQh3oQaIX_jygB-{QlV)UWqv! z*`Lu{uD+IzfFeFjkOTRtx!ddXaro}*fAYGX5&D~rc&_<(RI;DiWv%=B3cudsH!-0m zmJ+60+w!9JSMLtwsVNCu63j|Dced8JJBd0Cnyi&}!Pw3eJA@`J=Z)|k6oXZN|H$$UiK?O^b)M7)5vPFrtUjwi4+in_*c z)gRB@elAQ@{te8GnvwO-rkRpOjRx~2%e?h%DfGB3{O^5LjVW=jD9Ee z&^e?2Ln+!0pRb5!7G&*Imu=fyX?k$^YOse`8|BL#^DV!oc|6|+y3W9Ck$z<@O}xbw zB=?ENJEFD6+qoqz zrI3@)1TGh0q$3JR{Li?(1pFBvJ9~$SXDfbZDS~?1@#j;)BK8}vowyv(Xn)tg|F0$r z)yZ2|nMb<2QzZ5jEiY(-gD+6C|IZg8rrisAKvD0fhZ38+0y;rl5PE9nM@$oGwm zjNd1iP}#`uoBCv8DT8bUDtoR@L}QjsAb$#_F00pn`t+Z9cY8uD5-k`?IvRrHT@a{$^TxBHFIc$JlAa+s0!+2PZ)^VerPZ>_a;| zdoOV7w{a6V`$j%k>?EsZGw&g6L41={uwIJ8@-)$gH^sN$upW@WV;gU*+?-pMgY2PC zTvzyBR?lE6k6x0+Hn!e>7;o%(AuL;D*@TQJC7C*L2H#{}XV9z7>BG2@rAtJiC_)M` zEky}Skek+PD@HDXVH98-pgxNd1a=k>+OGrc&7-;Tcz!Wci3Ca5<>bhZ66|x6YpBlgTZt z^&yrqey8&G{$$BxS$UzeL>qXaNt0@KBYl5L5K}aNy?0Fj15s7B&MKiS(&E%?rBj=- zg)`Sy$(7$Q^%Q8HjF38DPg=5gXq+pQg4vVqqB^uw{xs1W?=;!nA;cAbXn;Jo1FLl#G~(KX^whND+o@}ii) zx#bTZmez?5sK>>#ue@jv`S&J8mY*iI-${zyiSTc;4Cqr7t4? z{ds(BuTQ?Aj=G}z69WVE2op3iE?>g|?#Y8td*!l;@ zO_0&!85MdPXXMz2(V<4TzR?O5!_ZDo#_ysH%Da}oP9&n}s(yrzDiL%vQ>6^dz_zYv zUEButbv}`#obVxb^T=0-CT=M*$cIuMbj&qdzGjh`J0Co;um9i&E*rL*2ujYo&CHZ{ zb;b*LHa(Lysjz&Td1foOO!f~kx?70NabjdwF+52T>>w>~6L(rurn~K2x2B2K!SDYFc@4(V-&Kcy zsV<_g5@6uKPF@zC^<;li*QQ^J=A9;05wEZb|XhMQ`^`J6;ywac~&nJh}GGMV(Cx%m4 z7{9x_jyftV*%K#=mVW4z#4shf))if22KI9Ic z3rbZOzdN(Rm6!jx`r%& + +.. raw:: html + +

+ +.. raw:: html + +
+ +Osher, Stanley, and Chi-Wang Shu. "Efficient implementation of essentially non-oscillatory shock-capturing schemes." J. Comput. Phys 77.2 (1988): 439-471. + + +.. raw:: html + +
+ +.. raw:: html + +
+ +Van Leer, Bram. "On the relation between the upwind-differencing schemes of Godunov, Engquist–Osher and Roe." SIAM Journal on Scientific and statistical Computing 5.1 (1984): 1-20. diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index 27d7682899d..3b88e07d447 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -1136,6 +1136,20 @@ Particle initialization Resampling is performed everytime the number of macroparticles per cell of the species averaged over the whole simulation domain exceeds this parameter. + +.. _running-cpp-parameters-fluids: + +Cold Relativistic Fluid initialization +-------------------------------------- + +When configuring fluid parameters in WarpX, you can utilize the same parsers as those used for particles. + +* ``fluids.species_names`` (`strings`, separated by spaces) + Defines the names of each fluid species. It is a required input to create and evolve fluid species using the cold relativistic fluid equations. + This input is used similarly to `particle.species_names`, but it is specifically tailored for fluids. + For fluid-specific inputs we use `` as a placeholder. Also see external fields + for how to specify these for fluids as the function names differ. + .. _running-cpp-parameters-laser: Laser initialization @@ -1499,6 +1513,37 @@ Applied to Particles and :math:`E_z = 0`, and :math:`B_x = \mathrm{strength} \cdot y`, :math:`B_y = -\mathrm{strength} \cdot x`, and :math:`B_z = 0`. + +Applied to Cold Relativistic Fluids +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* ``.E_ext_init_style`` & ``.B_ext_init_style`` (string) optional (default "none") + These parameters determine the type of the external electric and + magnetic fields respectively that are applied directly to the cold relativistic fluids at every timestep. + The field values are specified in the lab frame. + With the default ``none`` style, no field is applied. + Possible values are ``parse_E_ext_function`` or ``parse_B_ext_function``. + + * ``parse_E_ext_function`` or ``parse_B_ext_function``: the field is specified as an analytic + expression that is a function of space (x,y,z) and time (t), relative to the lab frame. + The E-field is specified by the input parameters: + + * ``.Ex_external_function(x,y,z,t)`` + + * ``.Ey_external_function(x,y,z,t)`` + + * ``.Ez_external_function(x,y,z,t)`` + + The B-field is specified by the input parameters: + + * ``.Bx_external_function(x,y,z,t)`` + + * ``.By_external_function(x,y,z,t)`` + + * ``.Bz_external_function(x,y,z,t)`` + + Note that the position is defined in Cartesian coordinates, as a function of (x,y,z), even for RZ. + Accelerator Lattice ^^^^^^^^^^^^^^^^^^^ diff --git a/Examples/Physics_applications/laser_acceleration/analysis_1d_fluids.py b/Examples/Physics_applications/laser_acceleration/analysis_1d_fluids.py new file mode 100755 index 00000000000..a376ed872a9 --- /dev/null +++ b/Examples/Physics_applications/laser_acceleration/analysis_1d_fluids.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python3 + +# Copyright 2019-2023 Grant Johnson, Remi Lehe +# +# +# This file is part of WarpX. +# +# License: BSD-3-Clause-LBNL +# +# This is a script that analyses the simulation results from +# the script `inputs_1d`. This simulates a 1D WFA with Pondermotive Envelope: +# REF: (Equations 20-23) https://journals.aps.org/rmp/pdf/10.1103/RevModPhys.81.1229 +import os +import sys + +import matplotlib + +matplotlib.use('Agg') +import matplotlib.pyplot as plt +import yt + +yt.funcs.mylog.setLevel(50) + +import numpy as np +from scipy.constants import c, e, epsilon_0, m_e + +sys.path.insert(1, '../../../../warpx/Regression/Checksum/') +import checksumAPI + +# this will be the name of the plot file +fn = sys.argv[1] + +# Parameters (these parameters must match the parameters in `inputs.multi.rt`) +n0 = 20.e23 +# Plasma frequency +wp = np.sqrt((n0*e**2)/(m_e*epsilon_0)) +kp = wp/c +tau = 15.e-15 +a0 = 2.491668 +e = -e #Electrons +lambda_laser = 0.8e-6 + +zmin = -20e-6; zmax = 100.e-6; Nz = 10240 + +# Compute the theory + +# Call the ode solver +from scipy.integrate import odeint + + +# ODE Function +def odefcn(phi, xi, kp, a0, c, tau, xi_0, lambda_laser): + phi1, phi2 = phi + a_sq = a0**2 * np.exp(-2 * (xi - xi_0)**2 / (c**2 * tau**2))*np.sin(2*np.pi*(xi - xi_0)/lambda_laser)**2 + dphi1_dxi = phi2 + dphi2_dxi = kp**2 * ((1 + a_sq) / (2 * (1 + phi1)**2) - 0.5) + return [dphi1_dxi, dphi2_dxi] + +# Call odeint to solve the ODE +xi_span = [-20e-6, 100e-6] +xi_0 = 0e-6 +phi0 = [0.0, 0.0] +dxi = (zmax-zmin)/Nz +xi = zmin + dxi*( 0.5 + np.arange(Nz) ) +phi = odeint(odefcn, phi0, xi, args=(kp, a0, c, tau, xi_0, lambda_laser)) + +# Change array direction to match the simulations +xi = -xi[::-1] +phi = phi[::-1] +xi_0 = -0e-6 +phi2 = phi[:, 0] +Ez = -phi[:, 1] + +# Compute the derived quantities +a_sq = a0**2 * np.exp(-2 * (xi - xi_0)**2 / (c**2 * tau**2)) *np.sin(2*np.pi*(xi - xi_0)/lambda_laser)**2 +gamma_perp_sq = 1 + a_sq +n = n0 * (gamma_perp_sq + (1 + phi2)**2) / (2 * (1 + phi2)**2) +uz = (gamma_perp_sq - (1 + phi2)**2) / (2 * (1 + phi2)) +gamma = (gamma_perp_sq + (1 + phi2)**2) / (2 * (1 + phi2)) + +# Theory Components [convert to si] +uz *= c +J_th = np.multiply( np.divide(uz,gamma), n ) +J_th *= e +rho_th = e*n +E_th = Ez +E_th *= ((m_e*c*c)/e) +V_th = np.divide(uz,gamma) +V_th /= c +# Remove the ions +rho_th = rho_th - e*n0 + +# Dicate which region to compare solutions over +# (Currently this is the full domain) +min_i = 0 +max_i = 10240 + +# Read the file +ds = yt.load(fn) +t0 = ds.current_time.to_value() +data = ds.covering_grid(level=0, left_edge=ds.domain_left_edge, + dims=ds.domain_dimensions) +# Check the validity of the fields +error_rel = 0 +for field in ['Ez']: + E_sim = data[('mesh',field)].to_ndarray()[:,0,0] + #E_th = get_theoretical_field(field, t0) + max_error = abs(E_sim[min_i:max_i]-E_th[min_i:max_i]).max()/abs(E_th[min_i:max_i]).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the currents +for field in ['Jz']: + J_sim = data[('mesh',field)].to_ndarray()[:,0,0] + #J_th = get_theoretical_J_field(field, t0) + max_error = abs(J_sim[min_i:max_i]-J_th[min_i:max_i]).max()/abs(J_th[min_i:max_i]).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the charge +for field in ['rho']: + rho_sim = data[('boxlib',field)].to_ndarray()[:,0,0] + #rho_th = get_theoretical_rho_field(field, t0) + max_error = abs(rho_sim[min_i:max_i]-rho_th[min_i:max_i]).max()/abs(rho_th[min_i:max_i]).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +V_sim = np.divide(J_sim,rho_sim) +V_sim /= c + +# Create a figure with 2 rows and 2 columns +fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(12, 8)) + +# Titles and labels +titles = ['Ez', 'rho', 'Jz', 'Vz/c'] +xlabel = r'Xi' +ylabel = ['Ez', 'rho', 'Jz', 'Vz/c'] + +# Plotting loop +for i in range(3): + ax = axes[i // 2, i % 2] # Get the current subplot + + # Plot theoretical data + ax.plot(xi, [E_th, rho_th, J_th, V_th][i], label='Theoretical') + + # Plot simulated data + ax.plot(xi, [E_sim, rho_sim, J_sim, V_sim][i], label='Simulated') + + # Set titles and labels + ax.set_title(f'{titles[i]} vs Xi') + ax.set_xlabel(xlabel) + ax.set_ylabel(ylabel[i]) + + # Add legend + ax.legend() + +# Adjust subplot layout +plt.tight_layout() + +# Save the figure +plt.savefig('wfa_fluid_nonlinear_1d_analysis.png') + +plt.show() + + +tolerance_rel = 0.20 + +print("error_rel : " + str(error_rel)) +print("tolerance_rel: " + str(tolerance_rel)) + +assert( error_rel < tolerance_rel ) + +test_name = os.path.split(os.getcwd())[1] +checksumAPI.evaluate_checksum(test_name, fn) diff --git a/Examples/Physics_applications/laser_acceleration/analysis_1d_fluids_boosted.py b/Examples/Physics_applications/laser_acceleration/analysis_1d_fluids_boosted.py new file mode 100755 index 00000000000..4f4ae2a812a --- /dev/null +++ b/Examples/Physics_applications/laser_acceleration/analysis_1d_fluids_boosted.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 + +# Copyright 2019-2023 Grant Johnson, Remi Lehe +# +# +# This file is part of WarpX. +# +# License: BSD-3-Clause-LBNL +# +# This is a script that analyses the simulation results from +# the script `inputs_1d`. This simulates a 1D WFA with Pondermotive Envelope: +# REF: (Equations 20-23) https://journals.aps.org/rmp/pdf/10.1103/RevModPhys.81.1229 +import os +import sys + +import matplotlib + +matplotlib.use('Agg') +import matplotlib.pyplot as plt +import yt + +yt.funcs.mylog.setLevel(50) + +import numpy as np +from scipy.constants import c, e, epsilon_0, m_e + +sys.path.insert(1, '../../../../warpx/Regression/Checksum/') +import checksumAPI + +# this will be the name of the plot file +fn = sys.argv[1] + +# Parameters (these parameters must match the parameters in `inputs.multi.rt`) +n0 = 20.e23 +# Plasma frequency +wp = np.sqrt((n0*e**2)/(m_e*epsilon_0)) +kp = wp/c +tau = 15.e-15 +a0 = 2.491668 +e = -e #Electrons +lambda_laser = 0.8e-6 + +zmin = -20e-6; zmax = 100.e-6; Nz = 4864 + +# Compute the theory + +# Call the ode solver +from scipy.integrate import odeint + + +# ODE Function +def odefcn(phi, xi, kp, a0, c, tau, xi_0, lambda_laser): + phi1, phi2 = phi + a_sq = a0**2 * np.exp(-2 * (xi - xi_0)**2 / (c**2 * tau**2))*np.sin(2*np.pi*(xi - xi_0)/lambda_laser)**2 + dphi1_dxi = phi2 + dphi2_dxi = kp**2 * ((1 + a_sq) / (2 * (1 + phi1)**2) - 0.5) + return [dphi1_dxi, dphi2_dxi] + +# Call odeint to solve the ODE +xi_span = [-20e-6, 100e-6] +xi_0 = 0e-6 +phi0 = [0.0, 0.0] +dxi = (zmax-zmin)/Nz +xi = zmin + dxi*( 0.5 + np.arange(Nz) ) +phi = odeint(odefcn, phi0, xi, args=(kp, a0, c, tau, xi_0, lambda_laser)) + +# Change array direction to match the simulations +xi = -xi[::-1] +phi = phi[::-1] +xi_0 = -0e-6 +phi2 = phi[:, 0] +Ez = -phi[:, 1] + +# Compute the derived quantities +a_sq = a0**2 * np.exp(-2 * (xi - xi_0)**2 / (c**2 * tau**2)) *np.sin(2*np.pi*(xi - xi_0)/lambda_laser)**2 +gamma_perp_sq = 1 + a_sq +n = n0 * (gamma_perp_sq + (1 + phi2)**2) / (2 * (1 + phi2)**2) +uz = (gamma_perp_sq - (1 + phi2)**2) / (2 * (1 + phi2)) +gamma = (gamma_perp_sq + (1 + phi2)**2) / (2 * (1 + phi2)) + +# Theory Components [convert to si] +uz *= c +J_th = np.multiply( np.divide(uz,gamma), n ) +J_th *= e +rho_th = e*n +E_th = Ez +E_th *= ((m_e*c*c)/e) +V_th = np.divide(uz,gamma) +V_th /= c +# Remove the ions +rho_th = rho_th - e*n0 + +# Dicate which region to compare solutions over (cuttoff 0's from BTD extra) +min_i = 200 +max_i = 4864 + +# Read the file +ds = yt.load(fn) +t0 = ds.current_time.to_value() +data = ds.covering_grid(level=0, left_edge=ds.domain_left_edge, + dims=ds.domain_dimensions) +# Check the validity of the fields +error_rel = 0 +for field in ['Ez']: + E_sim = data[('mesh',field)].to_ndarray()[:,0,0] + #E_th = get_theoretical_field(field, t0) + max_error = abs(E_sim[min_i:max_i]-E_th[min_i:max_i]).max()/abs(E_th[min_i:max_i]).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the currents +for field in ['Jz']: + J_sim = data[('mesh',field)].to_ndarray()[:,0,0] + #J_th = get_theoretical_J_field(field, t0) + max_error = abs(J_sim[min_i:max_i]-J_th[min_i:max_i]).max()/abs(J_th[min_i:max_i]).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the charge +for field in ['rho']: + rho_sim = data[('boxlib',field)].to_ndarray()[:,0,0] + #rho_th = get_theoretical_rho_field(field, t0) + max_error = abs(rho_sim[min_i:max_i]-rho_th[min_i:max_i]).max()/abs(rho_th[min_i:max_i]).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +V_sim = np.divide(J_sim,rho_sim) +V_sim /= c + +# Create a figure with 2 rows and 2 columns +fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(12, 8)) + +# Titles and labels +titles = ['Ez', 'rho', 'Jz', 'Vz/c'] +xlabel = r'Xi' +ylabel = ['Ez', 'rho', 'Jz', 'Vz/c'] + +# Plotting loop +for i in range(3): + ax = axes[i // 2, i % 2] # Get the current subplot + + # Plot theoretical data + ax.plot(xi, [E_th, rho_th, J_th, V_th][i], label='Theoretical') + + # Plot simulated data + ax.plot(xi, [E_sim, rho_sim, J_sim, V_sim][i], label='Simulated') + + # Set titles and labels + ax.set_title(f'{titles[i]} vs Xi') + ax.set_xlabel(xlabel) + ax.set_ylabel(ylabel[i]) + + # Add legend + ax.legend() + +# Adjust subplot layout +plt.tight_layout() + +# Save the figure +plt.savefig('wfa_fluid_nonlinear_1d_analysis.png') + +plt.show() + + +tolerance_rel = 0.30 + +print("error_rel : " + str(error_rel)) +print("tolerance_rel: " + str(tolerance_rel)) + +assert( error_rel < tolerance_rel ) + +test_name = os.path.split(os.getcwd())[1] +checksumAPI.evaluate_checksum(test_name, fn) diff --git a/Examples/Physics_applications/laser_acceleration/inputs_1d_fluids b/Examples/Physics_applications/laser_acceleration/inputs_1d_fluids new file mode 100644 index 00000000000..73fa6b7283f --- /dev/null +++ b/Examples/Physics_applications/laser_acceleration/inputs_1d_fluids @@ -0,0 +1,72 @@ +################################# +####### GENERAL PARAMETERS ###### +################################# +max_step = 40000 +amr.n_cell = 10240 +amr.max_grid_size = 512 # maximum size of each AMReX box, used to decompose the domain +amr.blocking_factor = 512 # minimum size of each AMReX box, used to decompose the domain +geometry.dims = 1 +geometry.prob_lo = -120.e-6 # physical domain +geometry.prob_hi = 0.e-6 +amr.max_level = 0 # Maximum level in hierarchy (1 might be unstable, >1 is not supported) + +################################# +####### Boundary condition ###### +################################# +boundary.field_lo = pec +boundary.field_hi = pec + +################################# +############ NUMERICS ########### +################################# +warpx.verbose = 1 +warpx.do_dive_cleaning = 0 +warpx.use_filter = 0 +warpx.cfl = 0.45 #Fluid CFL < 0.5 +warpx.do_moving_window = 1 +warpx.moving_window_dir = z +warpx.moving_window_v = 1.0 # units of speed of light +warpx.do_dynamic_scheduling = 0 +warpx.serialize_initial_conditions = 1 + +################################# +############ PLASMA ############# +################################# +fluids.species_names = electrons ions + +electrons.species_type = electron +electrons.profile = parse_density_function +electrons.density_function(x,y,z) = "1.0e10 + 20.e23*((z*5.e4 + -0.5)*(z>10.e-6)*(z<30.e-6)) + 20.e23*((z>30.e-6))" +electrons.momentum_distribution_type = "at_rest" + +ions.charge = q_e +ions.mass = m_p +ions.profile = parse_density_function +ions.density_function(x,y,z) = "1.0e10 + 20.e23*((z*5.e4 + -0.5)*(z>10.e-6)*(z<30.e-6)) + 20.e23*((z>30.e-6))" +ions.momentum_distribution_type = "at_rest" + +# Order of particle shape factors +algo.particle_shape = 3 + +################################# +############ LASER ############## +################################# +lasers.names = laser1 +laser1.profile = Gaussian +laser1.position = 0. 0. -11.e-6 # This point is on the laser plane +laser1.direction = 0. 0. 1. # The plane normal direction +laser1.polarization = 0. 1. 0. # The main polarization vector +laser1.e_max = 10.e12 # Maximum amplitude of the laser field (in V/m) +laser1.profile_waist = 5.e-6 # The waist of the laser (in m) +laser1.profile_duration = 15.e-15 # The duration of the laser (in s) +laser1.profile_t_peak = 30.e-15 # Time at which the laser reaches its peak (in s) +laser1.profile_focal_distance = 100.e-6 # Focal distance from the antenna (in m) +laser1.wavelength = 0.8e-6 # The wavelength of the laser (in m) + +# Diagnostics +diagnostics.diags_names = diag1 + +# LAB +diag1.intervals = 20000 +diag1.diag_type = Full +diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho diff --git a/Examples/Physics_applications/laser_acceleration/inputs_1d_fluids_boosted b/Examples/Physics_applications/laser_acceleration/inputs_1d_fluids_boosted new file mode 100644 index 00000000000..f9437716f66 --- /dev/null +++ b/Examples/Physics_applications/laser_acceleration/inputs_1d_fluids_boosted @@ -0,0 +1,79 @@ +################################# +####### GENERAL PARAMETERS ###### +################################# +max_step = 10000 +amr.n_cell = 5120 +amr.max_grid_size = 1024 # maximum size of each AMReX box, used to decompose the domain +amr.blocking_factor = 1024 # minimum size of each AMReX box, used to decompose the domain +geometry.dims = 1 +geometry.prob_lo = -120.e-6 # physical domain +geometry.prob_hi = 0.e-6 +amr.max_level = 0 # Maximum level in hierarchy (1 might be unstable, >1 is not supported) + +################################# +####### Boundary condition ###### +################################# +boundary.field_lo = pec +boundary.field_hi = pec + +################################# +############ NUMERICS ########### +################################# +warpx.verbose = 1 +warpx.do_dive_cleaning = 0 +warpx.use_filter = 0 +warpx.cfl = 0.45 #Fluid CFL < 0.5 +warpx.do_moving_window = 1 +warpx.moving_window_dir = z +warpx.moving_window_v = 1.0 # units of speed of light +warpx.do_dynamic_scheduling = 0 +warpx.serialize_initial_conditions = 1 + +### BOOST ### +warpx.gamma_boost = 1.5 +warpx.boost_direction = z + +################################# +############ PLASMA ############# +################################# +fluids.species_names = electrons ions + +electrons.species_type = electron +electrons.profile = parse_density_function +electrons.density_function(x,y,z) = "1.0e10 + 20.e23*((z*5.e4 + -0.5)*(z>10.e-6)*(z<30.e-6)) + 20.e23*((z>30.e-6))" +electrons.momentum_distribution_type = "at_rest" + +ions.charge = q_e +ions.mass = m_p +ions.profile = parse_density_function +ions.density_function(x,y,z) = "1.0e10 + 20.e23*((z*5.e4 + -0.5)*(z>10.e-6)*(z<30.e-6)) + 20.e23*((z>30.e-6))" +ions.momentum_distribution_type = "at_rest" + +# Order of particle shape factors +algo.particle_shape = 3 + +################################# +############ LASER ############## +################################# +lasers.names = laser1 +laser1.profile = Gaussian +laser1.position = 0. 0. -11.e-6 # This point is on the laser plane +laser1.direction = 0. 0. 1. # The plane normal direction +laser1.polarization = 0. 1. 0. # The main polarization vector +laser1.e_max = 10.e12 # Maximum amplitude of the laser field (in V/m) +laser1.profile_waist = 5.e-6 # The waist of the laser (in m) +laser1.profile_duration = 15.e-15 # The duration of the laser (in s) +laser1.profile_t_peak = 30.e-15 # Time at which the laser reaches its peak (in s) +laser1.profile_focal_distance = 100.e-6 # Focal distance from the antenna (in m) +laser1.wavelength = 0.8e-6 # The wavelength of the laser (in m) + +# Diagnostics +diagnostics.diags_names = diag1 + + +# BOOST +diag1.diag_type = BackTransformed +diag1.do_back_transformed_fields = 1 +diag1.num_snapshots_lab = 2 +diag1.dt_snapshots_lab = 6.e-13 +diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho diff --git a/Examples/Tests/langmuir_fluids/analysis_1d.py b/Examples/Tests/langmuir_fluids/analysis_1d.py new file mode 100755 index 00000000000..2d1a8f69d1d --- /dev/null +++ b/Examples/Tests/langmuir_fluids/analysis_1d.py @@ -0,0 +1,137 @@ +#!/usr/bin/env python3 + +# Copyright 2019-2022 Jean-Luc Vay, Maxence Thevenet, Remi Lehe, Prabhat Kumar, Axel Huebl, Grant Johnson +# +# +# This file is part of WarpX. +# +# License: BSD-3-Clause-LBNL +# +# This is a script that analyses the simulation results from +# the script `inputs.multi.rt`. This simulates a 1D periodic plasma wave. +# The electric field in the simulation is given (in theory) by: +# $$ E_z = \epsilon \,\frac{m_e c^2 k_z}{q_e}\sin(k_z z)\sin( \omega_p t)$$ +import os +import sys + +import matplotlib + +matplotlib.use('Agg') +import matplotlib.pyplot as plt +import yt + +yt.funcs.mylog.setLevel(50) + +import numpy as np +from scipy.constants import c, e, epsilon_0, m_e + +sys.path.insert(1, '../../../../warpx/Regression/Checksum/') +import checksumAPI + +# this will be the name of the plot file +fn = sys.argv[1] + +# Parameters (these parameters must match the parameters in `inputs.multi.rt`) +epsilon = 0.01 +n = 4.e24 +n_osc_z = 2 +zmin = -20e-6; zmax = 20.e-6; Nz = 128 + +# Wave vector of the wave +kz = 2.*np.pi*n_osc_z/(zmax-zmin) +# Plasma frequency +wp = np.sqrt((n*e**2)/(m_e*epsilon_0)) + +k = {'Ez':kz,'Jz':kz} +cos = {'Ez':(1,1,0), 'Jz':(1,1,0)} +cos_rho = {'rho': (1,1,1)} + +def get_contribution( is_cos, k ): + du = (zmax-zmin)/Nz + u = zmin + du*( 0.5 + np.arange(Nz) ) + if is_cos == 1: + return( np.cos(k*u) ) + else: + return( np.sin(k*u) ) + +def get_theoretical_field( field, t ): + amplitude = epsilon * (m_e*c**2*k[field])/e * np.sin(wp*t) + cos_flag = cos[field] + z_contribution = get_contribution( cos_flag[2], kz ) + + E = amplitude * z_contribution + + return( E ) + +def get_theoretical_J_field( field, t ): + # wpdt/2 accounts for the Yee halfstep offset of the current + dt = t / 80 # SPECIFIC to config parameters! + amplitude = - epsilon_0 * wp * epsilon * (m_e*c**2*k[field])/e * np.cos(wp*t-wp*dt/2) + cos_flag = cos[field] + + z_contribution = get_contribution( cos_flag[2], kz ) + + J = amplitude * z_contribution + + return( J ) + +def get_theoretical_rho_field( field, t ): + amplitude = epsilon_0 * epsilon * (m_e*c**2*(kz*kz))/e * np.sin(wp*t) + cos_flag = cos_rho[field] + z_contribution = get_contribution( cos_flag[2], kz) + + rho = amplitude * z_contribution + + return( rho ) + +# Read the file +ds = yt.load(fn) +t0 = ds.current_time.to_value() +data = ds.covering_grid(level=0, left_edge=ds.domain_left_edge, + dims=ds.domain_dimensions) +# Check the validity of the fields +error_rel = 0 +for field in ['Ez']: + E_sim = data[('mesh',field)].to_ndarray()[:,0,0] + E_th = get_theoretical_field(field, t0) + max_error = abs(E_sim-E_th).max()/abs(E_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the currents +for field in ['Jz']: + J_sim = data[('mesh',field)].to_ndarray()[:,0,0] + J_th = get_theoretical_J_field(field, t0) + max_error = abs(J_sim-J_th).max()/abs(J_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the charge +for field in ['rho']: + rho_sim = data[('boxlib',field)].to_ndarray()[:,0,0] + rho_th = get_theoretical_rho_field(field, t0) + max_error = abs(rho_sim-rho_th).max()/abs(rho_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Plot the last field from the loop (Ez at iteration 80) +plt.subplot2grid( (1,2), (0,0) ) +plt.plot( E_sim ) +#plt.colorbar() +plt.title('Ez, last iteration\n(simulation)') +plt.subplot2grid( (1,2), (0,1) ) +plt.plot( E_th ) +#plt.colorbar() +plt.title('Ez, last iteration\n(theory)') +plt.tight_layout() +plt.savefig('langmuir_fluid_multi_1d_analysis.png') + +tolerance_rel = 0.05 + +print("error_rel : " + str(error_rel)) +print("tolerance_rel: " + str(tolerance_rel)) + +assert( error_rel < tolerance_rel ) + +test_name = os.path.split(os.getcwd())[1] +checksumAPI.evaluate_checksum(test_name, fn) diff --git a/Examples/Tests/langmuir_fluids/analysis_2d.py b/Examples/Tests/langmuir_fluids/analysis_2d.py new file mode 100755 index 00000000000..cf5d2fb44de --- /dev/null +++ b/Examples/Tests/langmuir_fluids/analysis_2d.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 + +# Copyright 2019 Jean-Luc Vay, Maxence Thevenet, Remi Lehe, Grant Johnson +# +# +# This file is part of WarpX. +# +# License: BSD-3-Clause-LBNL +# +# This is a script that analyses the simulation results from +# the script `inputs.multi.rt`. This simulates a 3D periodic plasma wave. +# The electric field in the simulation is given (in theory) by: +# $$ E_x = \epsilon \,\frac{m_e c^2 k_x}{q_e}\sin(k_x x)\cos(k_y y)\cos(k_z z)\sin( \omega_p t)$$ +# $$ E_y = \epsilon \,\frac{m_e c^2 k_y}{q_e}\cos(k_x x)\sin(k_y y)\cos(k_z z)\sin( \omega_p t)$$ +# $$ E_z = \epsilon \,\frac{m_e c^2 k_z}{q_e}\cos(k_x x)\cos(k_y y)\sin(k_z z)\sin( \omega_p t)$$ +import os +import sys + +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable +import yt + +yt.funcs.mylog.setLevel(50) + +import numpy as np +from scipy.constants import c, e, epsilon_0, m_e + +sys.path.insert(1, '../../../../warpx/Regression/Checksum/') +import checksumAPI + +# this will be the name of the plot file +fn = sys.argv[1] + +# Parameters (these parameters must match the parameters in `inputs.multi.rt`) +epsilon = 0.01 +n = 4.e24 +n_osc_x = 2 +n_osc_z = 2 +xmin = -20e-6; xmax = 20.e-6; Nx = 128 +zmin = -20e-6; zmax = 20.e-6; Nz = 128 + +# Wave vector of the wave +kx = 2.*np.pi*n_osc_x/(xmax-xmin) +kz = 2.*np.pi*n_osc_z/(zmax-zmin) +# Plasma frequency +wp = np.sqrt((n*e**2)/(m_e*epsilon_0)) + +k = {'Ex':kx, 'Ez':kz, 'Jx':kx, 'Jz':kz} +cos = {'Ex': (0,1,1), 'Ez':(1,1,0),'Jx': (0,1,1), 'Jz':(1,1,0)} +cos_rho = {'rho': (1,1,1)} + +def get_contribution( is_cos, k ): + du = (xmax-xmin)/Nx + u = xmin + du*( 0.5 + np.arange(Nx) ) + if is_cos == 1: + return( np.cos(k*u) ) + else: + return( np.sin(k*u) ) + +def get_theoretical_field( field, t ): + amplitude = epsilon * (m_e*c**2*k[field])/e * np.sin(wp*t) + cos_flag = cos[field] + x_contribution = get_contribution( cos_flag[0], kx ) + z_contribution = get_contribution( cos_flag[2], kz ) + + E = amplitude * x_contribution[:, np.newaxis ] \ + * z_contribution[np.newaxis, :] + + return( E ) + +def get_theoretical_J_field( field, t ): + # wpdt/2 accounts for the Yee halfstep offset of the current + dt = t / 40 # SPECIFIC to config parameters! + amplitude = - epsilon_0 * wp * epsilon * (m_e*c**2*k[field])/e * np.cos(wp*t-wp*dt/2) + cos_flag = cos[field] + x_contribution = get_contribution( cos_flag[0], kx ) + z_contribution = get_contribution( cos_flag[2], kz ) + + J = amplitude * x_contribution[:, np.newaxis] \ + * z_contribution[np.newaxis, :] + + return( J ) + +def get_theoretical_rho_field( field, t ): + amplitude = epsilon_0 * epsilon * (m_e*c**2*(kx*kx+kz*kz))/e * np.sin(wp*t) + cos_flag = cos_rho[field] + x_contribution = get_contribution( cos_flag[0], kx ) + z_contribution = get_contribution( cos_flag[2], kz ) + + rho = amplitude * x_contribution[:, np.newaxis] \ + * z_contribution[ np.newaxis, :] + + return( rho ) + +# Read the file +ds = yt.load(fn) +t0 = ds.current_time.to_value() +data = ds.covering_grid(level = 0, left_edge = ds.domain_left_edge, dims = ds.domain_dimensions) +edge = np.array([(ds.domain_left_edge[1]).item(), (ds.domain_right_edge[1]).item(), \ + (ds.domain_left_edge[0]).item(), (ds.domain_right_edge[0]).item()]) + +# Check the validity of the fields +error_rel = 0 +for field in ['Ex', 'Ez']: + E_sim = data[('mesh',field)].to_ndarray()[:,:,0] + E_th = get_theoretical_field(field, t0) + max_error = abs(E_sim-E_th).max()/abs(E_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the currents +for field in ['Jx', 'Jz']: + J_sim = data[('mesh',field)].to_ndarray()[:,:,0] + J_th = get_theoretical_J_field(field, t0) + max_error = abs(J_sim-J_th).max()/abs(J_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the charge +for field in ['rho']: + rho_sim = data[('boxlib',field)].to_ndarray()[:,:,0] + rho_th = get_theoretical_rho_field(field, t0) + max_error = abs(rho_sim-rho_th).max()/abs(rho_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Plot the last field from the loop (Ez at iteration 40) +fig, (ax1, ax2) = plt.subplots(1, 2, dpi = 100) +# First plot +vmin = E_sim.min() +vmax = E_sim.max() +cax1 = make_axes_locatable(ax1).append_axes('right', size = '5%', pad = '5%') +im1 = ax1.imshow(E_sim, origin = 'lower', extent = edge, vmin = vmin, vmax = vmax) +cb1 = fig.colorbar(im1, cax = cax1) +ax1.set_xlabel(r'$z$') +ax1.set_ylabel(r'$x$') +ax1.set_title(r'$E_z$ (sim)') +# Second plot +vmin = E_th.min() +vmax = E_th.max() +cax2 = make_axes_locatable(ax2).append_axes('right', size = '5%', pad = '5%') +im2 = ax2.imshow(E_th, origin = 'lower', extent = edge, vmin = vmin, vmax = vmax) +cb2 = fig.colorbar(im2, cax = cax2) +ax2.set_xlabel(r'$z$') +ax2.set_ylabel(r'$x$') +ax2.set_title(r'$E_z$ (theory)') +# Save figure +fig.tight_layout() +fig.savefig('Langmuir_fluid_multi_2d_analysis.png', dpi = 200) + +tolerance_rel = 0.05 + +print("error_rel : " + str(error_rel)) +print("tolerance_rel: " + str(tolerance_rel)) + +assert( error_rel < tolerance_rel ) + +test_name = os.path.split(os.getcwd())[1] +checksumAPI.evaluate_checksum(test_name, fn) diff --git a/Examples/Tests/langmuir_fluids/analysis_3d.py b/Examples/Tests/langmuir_fluids/analysis_3d.py new file mode 100755 index 00000000000..0211956859e --- /dev/null +++ b/Examples/Tests/langmuir_fluids/analysis_3d.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 + +# Copyright 2019-2022 Jean-Luc Vay, Maxence Thevenet, Remi Lehe, Axel Huebl, Grant Johnson +# +# +# This file is part of WarpX. +# +# License: BSD-3-Clause-LBNL +# +# This is a script that analyses the simulation results from +# the script `inputs.multi.rt`. This simulates a 3D periodic plasma wave. +# The electric field in the simulation is given (in theory) by: +# $$ E_x = \epsilon \,\frac{m_e c^2 k_x}{q_e}\sin(k_x x)\cos(k_y y)\cos(k_z z)\sin( \omega_p t)$$ +# $$ E_y = \epsilon \,\frac{m_e c^2 k_y}{q_e}\cos(k_x x)\sin(k_y y)\cos(k_z z)\sin( \omega_p t)$$ +# $$ E_z = \epsilon \,\frac{m_e c^2 k_z}{q_e}\cos(k_x x)\cos(k_y y)\sin(k_z z)\sin( \omega_p t)$$ +import os +import re +import sys + +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable +import yt + +yt.funcs.mylog.setLevel(50) + +import numpy as np +from scipy.constants import c, e, epsilon_0, m_e + +sys.path.insert(1, '../../../../warpx/Regression/Checksum/') +import checksumAPI + +# this will be the name of the plot file +fn = sys.argv[1] + +# Parameters (these parameters must match the parameters in `inputs.multi.rt`) +epsilon = 0.01 +n = 4.e24 +n_osc_x = 2 +n_osc_y = 2 +n_osc_z = 2 +lo = [-20.e-6, -20.e-6, -20.e-6] +hi = [ 20.e-6, 20.e-6, 20.e-6] +Ncell = [64, 64, 64] + +# Wave vector of the wave +kx = 2.*np.pi*n_osc_x/(hi[0]-lo[0]) +ky = 2.*np.pi*n_osc_y/(hi[1]-lo[1]) +kz = 2.*np.pi*n_osc_z/(hi[2]-lo[2]) +# Plasma frequency +wp = np.sqrt((n*e**2)/(m_e*epsilon_0)) + +k = {'Ex':kx, 'Ey':ky, 'Ez':kz, 'Jx':kx, 'Jy':ky, 'Jz':kz} +cos = {'Ex': (0,1,1), 'Ey':(1,0,1), 'Ez':(1,1,0),'Jx': (0,1,1), 'Jy':(1,0,1), 'Jz':(1,1,0)} +cos_rho = {'rho': (1,1,1)} + +def get_contribution( is_cos, k, idim ): + du = (hi[idim]-lo[idim])/Ncell[idim] + u = lo[idim] + du*( 0.5 + np.arange(Ncell[idim]) ) + if is_cos[idim] == 1: + return( np.cos(k*u) ) + else: + return( np.sin(k*u) ) + +def get_theoretical_field( field, t ): + amplitude = epsilon * (m_e*c**2*k[field])/e * np.sin(wp*t) + cos_flag = cos[field] + x_contribution = get_contribution( cos_flag, kx, 0 ) + y_contribution = get_contribution( cos_flag, ky, 1 ) + z_contribution = get_contribution( cos_flag, kz, 2 ) + + E = amplitude * x_contribution[:, np.newaxis, np.newaxis] \ + * y_contribution[np.newaxis, :, np.newaxis] \ + * z_contribution[np.newaxis, np.newaxis, :] + + return( E ) + +def get_theoretical_J_field( field, t ): + # wpdt/2 accounts for the Yee halfstep offset of the current + dt = t / 40 # SPECIFIC to config parameters! + amplitude = - epsilon_0 * wp * epsilon * (m_e*c**2*k[field])/e * np.cos(wp*t-wp*dt/2) + cos_flag = cos[field] + x_contribution = get_contribution( cos_flag, kx, 0 ) + y_contribution = get_contribution( cos_flag, ky, 1 ) + z_contribution = get_contribution( cos_flag, kz, 2 ) + + J = amplitude * x_contribution[:, np.newaxis, np.newaxis] \ + * y_contribution[np.newaxis, :, np.newaxis] \ + * z_contribution[np.newaxis, np.newaxis, :] + + return( J ) + +def get_theoretical_rho_field( field, t ): + amplitude = epsilon_0 * epsilon * (m_e*c**2*(kx*kx+ky*ky+kz*kz))/e * np.sin(wp*t) + cos_flag = cos_rho[field] + x_contribution = get_contribution( cos_flag, kx, 0 ) + y_contribution = get_contribution( cos_flag, ky, 1 ) + z_contribution = get_contribution( cos_flag, kz, 2 ) + + rho = amplitude * x_contribution[:, np.newaxis, np.newaxis] \ + * y_contribution[np.newaxis, :, np.newaxis] \ + * z_contribution[np.newaxis, np.newaxis, :] + + return( rho ) + +# Read the file +ds = yt.load(fn) + + +t0 = ds.current_time.to_value() +data = ds.covering_grid(level = 0, left_edge = ds.domain_left_edge, dims = ds.domain_dimensions) +edge = np.array([(ds.domain_left_edge[2]).item(), (ds.domain_right_edge[2]).item(), \ + (ds.domain_left_edge[0]).item(), (ds.domain_right_edge[0]).item()]) + +# Check the validity of the fields +error_rel = 0 +for field in ['Ex', 'Ey', 'Ez']: + E_sim = data[('mesh',field)].to_ndarray() + E_th = get_theoretical_field(field, t0) + max_error = abs(E_sim-E_th).max()/abs(E_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + + +# Check the validity of the currents +for field in ['Jx', 'Jy', 'Jz']: + J_sim = data[('mesh',field)].to_ndarray() + J_th = get_theoretical_J_field(field, t0) + max_error = abs(J_sim-J_th).max()/abs(J_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + +# Check the validity of the charge +for field in ['rho']: + rho_sim = data[('boxlib',field)].to_ndarray() + rho_th = get_theoretical_rho_field(field, t0) + max_error = abs(rho_sim-rho_th).max()/abs(rho_th).max() + print('%s: Max error: %.2e' %(field,max_error)) + error_rel = max( error_rel, max_error ) + + +# Plot the last field from the loop (Ez at iteration 40) +fig, (ax1, ax2) = plt.subplots(1, 2, dpi = 100) +# First plot (slice at y=0) +E_plot = E_sim[:,Ncell[1]//2+1,:] +vmin = E_plot.min() +vmax = E_plot.max() +cax1 = make_axes_locatable(ax1).append_axes('right', size = '5%', pad = '5%') +im1 = ax1.imshow(E_plot, origin = 'lower', extent = edge, vmin = vmin, vmax = vmax) +cb1 = fig.colorbar(im1, cax = cax1) +ax1.set_xlabel(r'$z$') +ax1.set_ylabel(r'$x$') +ax1.set_title(r'$E_z$ (sim)') +# Second plot (slice at y=0) +E_plot = E_th[:,Ncell[1]//2+1,:] +vmin = E_plot.min() +vmax = E_plot.max() +cax2 = make_axes_locatable(ax2).append_axes('right', size = '5%', pad = '5%') +im2 = ax2.imshow(E_plot, origin = 'lower', extent = edge, vmin = vmin, vmax = vmax) +cb2 = fig.colorbar(im2, cax = cax2) +ax2.set_xlabel(r'$z$') +ax2.set_ylabel(r'$x$') +ax2.set_title(r'$E_z$ (theory)') +# Save figure +fig.tight_layout() +fig.savefig('Langmuir_fluid_multi_analysis.png', dpi = 200) + +tolerance_rel = 5e-2 + +print("error_rel : " + str(error_rel)) +print("tolerance_rel: " + str(tolerance_rel)) + +assert( error_rel < tolerance_rel ) + +test_name = os.path.split(os.getcwd())[1] + +if re.search( 'single_precision', fn ): + checksumAPI.evaluate_checksum(test_name, fn, rtol=1.e-3) +else: + checksumAPI.evaluate_checksum(test_name, fn) diff --git a/Examples/Tests/langmuir_fluids/analysis_rz.py b/Examples/Tests/langmuir_fluids/analysis_rz.py new file mode 100755 index 00000000000..108d054e75a --- /dev/null +++ b/Examples/Tests/langmuir_fluids/analysis_rz.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python3 + +# Copyright 2019 David Grote, Maxence Thevenet, Grant Johnson, Remi Lehe +# +# This file is part of WarpX. +# +# License: BSD-3-Clause-LBNL +# +# This is a script that analyses the simulation results from +# the script `inputs.multi.rz.rt`. This simulates a RZ periodic plasma wave. +# The electric field in the simulation is given (in theory) by: +# $$ E_r = -\partial_r \phi = \epsilon \,\frac{mc^2}{e}\frac{2\,r}{w_0^2} \exp\left(-\frac{r^2}{w_0^2}\right) \sin(k_0 z) \sin(\omega_p t) +# $$ E_z = -\partial_z \phi = - \epsilon \,\frac{mc^2}{e} k_0 \exp\left(-\frac{r^2}{w_0^2}\right) \cos(k_0 z) \sin(\omega_p t) +# Unrelated to the Langmuir waves, we also test the plotfile particle filter function in this +# analysis script. +import os +import re +import sys + +import matplotlib + +matplotlib.use('Agg') +import matplotlib.pyplot as plt +import yt + +yt.funcs.mylog.setLevel(50) + +import numpy as np +from scipy.constants import c, e, epsilon_0, m_e + +sys.path.insert(1, '../../../../warpx/Regression/Checksum/') +import checksumAPI + +# this will be the name of the plot file +fn = sys.argv[1] + +test_name = os.path.split(os.getcwd())[1] + +# Parse test name and check if current correction (psatd.current_correction) is applied +current_correction = True if re.search('current_correction', fn) else False + +# Parameters (these parameters must match the parameters in `inputs.multi.rz.rt`) +epsilon = 0.01 +n = 2.e24 +w0 = 5.e-6 +n_osc_z = 2 +rmin = 0e-6; rmax = 20.e-6; Nr = 64 +zmin = -20e-6; zmax = 20.e-6; Nz = 128 + +# Wave vector of the wave +k0 = 2.*np.pi*n_osc_z/(zmax-zmin) +# Plasma frequency +wp = np.sqrt((n*e**2)/(m_e*epsilon_0)) +kp = wp/c + +def Er( z, r, epsilon, k0, w0, wp, t) : + """ + Return the radial electric field as an array + of the same length as z and r, in the half-plane theta=0 + """ + Er_array = \ + epsilon * m_e*c**2/e * 2*r/w0**2 * \ + np.exp( -r**2/w0**2 ) * np.sin( k0*z ) * np.sin( wp*t ) + return( Er_array ) + +def Ez( z, r, epsilon, k0, w0, wp, t) : + """ + Return the longitudinal electric field as an array + of the same length as z and r, in the half-plane theta=0 + """ + Ez_array = \ + - epsilon * m_e*c**2/e * k0 * \ + np.exp( -r**2/w0**2 ) * np.cos( k0*z ) * np.sin( wp*t ) + return( Ez_array ) + +def Jr( z, r, epsilon, k0, w0, wp, t) : + """ + Return the radial current density as an array + of the same length as z and r, in the half-plane theta=0 + """ + dt = t / 80 # SPECIFIC to config parameters! + Jr_array = \ + - epsilon_0 * epsilon * m_e*c**2/e * 2*r/w0**2 * \ + np.exp( -r**2/w0**2 ) * np.sin( k0*z ) * np.cos( wp*t -wp*dt/2) * wp #phase_error = wp*dt/2 + return( Jr_array ) + +def Jz( z, r, epsilon, k0, w0, wp, t) : + """ + Return the longitudinal current density as an array + of the same length as z and r, in the half-plane theta=0 + """ + dt = t / 80 # SPECIFIC to config parameters! + Jz_array = \ + epsilon_0 * epsilon * m_e*c**2/e * k0 * \ + np.exp( -r**2/w0**2 ) * np.cos( k0*z ) * np.cos( wp*t -wp*dt/2) * wp #phase_error = wp*dt/2 + return( Jz_array ) + +def rho( z, r, epsilon, k0, w0, wp, t) : + """ + Return the charge density as an array + of the same length as z and r, in the half-plane theta=0 + """ + rho_array = \ + epsilon_0 * epsilon * m_e*c**2/e * np.sin( wp*t ) * np.sin( k0*z ) * np.exp( -r**2/w0**2 ) * \ + ((4.0/(w0**2))*(1 - (r**2)/(w0**2)) + k0**2) + return( rho_array ) + +# Read the file +ds = yt.load(fn) +t0 = ds.current_time.to_value() +data = ds.covering_grid(level=0, left_edge=ds.domain_left_edge, + dims=ds.domain_dimensions) + +# Get cell centered coordinates +dr = (rmax - rmin)/Nr +dz = (zmax - zmin)/Nz +coords = np.indices([Nr, Nz],'d') +rr = rmin + (coords[0] + 0.5)*dr +zz = zmin + (coords[1] + 0.5)*dz + +# Check the validity of the fields +overall_max_error = 0 +Er_sim = data[('boxlib','Er')].to_ndarray()[:,:,0] +Er_th = Er(zz, rr, epsilon, k0, w0, wp, t0) +max_error = abs(Er_sim-Er_th).max()/abs(Er_th).max() +print('Er: Max error: %.2e' %(max_error)) +overall_max_error = max( overall_max_error, max_error ) + +Ez_sim = data[('boxlib','Ez')].to_ndarray()[:,:,0] +Ez_th = Ez(zz, rr, epsilon, k0, w0, wp, t0) +max_error = abs(Ez_sim-Ez_th).max()/abs(Ez_th).max() +print('Ez: Max error: %.2e' %(max_error)) +overall_max_error = max( overall_max_error, max_error ) + +Jr_sim = data[('boxlib','jr')].to_ndarray()[:,:,0] +Jr_th = Jr(zz, rr, epsilon, k0, w0, wp, t0) +max_error = abs(Jr_sim-Jr_th).max()/abs(Jr_th).max() +print('Jr: Max error: %.2e' %(max_error)) +overall_max_error = max( overall_max_error, max_error ) + +Jz_sim = data[('boxlib','jz')].to_ndarray()[:,:,0] +Jz_th = Jz(zz, rr, epsilon, k0, w0, wp, t0) +max_error = abs(Jz_sim-Jz_th).max()/abs(Jz_th).max() +print('Jz: Max error: %.2e' %(max_error)) +overall_max_error = max( overall_max_error, max_error ) + +rho_sim = data[('boxlib','rho')].to_ndarray()[:,:,0] +rho_th = rho(zz, rr, epsilon, k0, w0, wp, t0) +max_error = abs(rho_sim-rho_th).max()/abs(rho_th).max() +print('rho: Max error: %.2e' %(max_error)) +overall_max_error = max( overall_max_error, max_error ) + +# Plot the last field from the loop (Ez at iteration 40) +plt.subplot2grid( (1,2), (0,0) ) +plt.imshow( Ez_sim ) +plt.colorbar() +plt.title('Ez, last iteration\n(simulation)') +plt.subplot2grid( (1,2), (0,1) ) +plt.imshow( Ez_th ) +plt.colorbar() +plt.title('Ez, last iteration\n(theory)') +plt.tight_layout() +plt.savefig(test_name+'_analysis.png') + +error_rel = overall_max_error + +tolerance_rel = 0.08 + +print("error_rel : " + str(error_rel)) +print("tolerance_rel: " + str(tolerance_rel)) + +assert( error_rel < tolerance_rel ) + +checksumAPI.evaluate_checksum(test_name, fn) diff --git a/Examples/Tests/langmuir_fluids/inputs_1d b/Examples/Tests/langmuir_fluids/inputs_1d new file mode 100644 index 00000000000..48fda36c61f --- /dev/null +++ b/Examples/Tests/langmuir_fluids/inputs_1d @@ -0,0 +1,72 @@ +# Maximum number of time steps +max_step = 80 + +# number of grid points +amr.n_cell = 128 + +# Maximum allowable size of each subdomain in the problem domain; +# this is used to decompose the domain for parallel calculations. +amr.max_grid_size = 64 + +# Maximum level in hierarchy (for now must be 0, i.e., one level in total) +amr.max_level = 0 + +# Geometry +geometry.dims = 1 +geometry.prob_lo = -20.e-6 # physical domain +geometry.prob_hi = 20.e-6 + +# Boundary condition +boundary.field_lo = periodic +boundary.field_hi = periodic + +warpx.serialize_initial_conditions = 1 + +# Verbosity +warpx.verbose = 1 + +# Algorithms +warpx.use_filter = 0 + +# CFL +warpx.cfl = 0.8 + +# Parameters for the plasma wave +my_constants.epsilon = 0.01 +my_constants.n0 = 2.e24 # electron and positron densities, #/m^3 +my_constants.wp = sqrt(2.*n0*q_e**2/(epsilon0*m_e)) # plasma frequency +my_constants.kp = wp/clight # plasma wavenumber +my_constants.k = 2.*pi/20.e-6 # perturbation wavenumber +# Note: kp is calculated in SI for a density of 4e24 (i.e. 2e24 electrons + 2e24 positrons) +# k is calculated so as to have 2 periods within the 40e-6 wide box. + +# Particles +fluids.species_names = electrons positrons + +electrons.charge = -q_e +electrons.mass = m_e +electrons.profile = constant +electrons.density = n0 # number of electrons per m^3 +electrons.momentum_distribution_type = parse_momentum_function +electrons.momentum_function_ux(x,y,z) = "epsilon * k/kp * sin(k*x) * cos(k*y) * cos(k*z)" +electrons.momentum_function_uy(x,y,z) = "epsilon * k/kp * cos(k*x) * sin(k*y) * cos(k*z)" +electrons.momentum_function_uz(x,y,z) = "epsilon * k/kp * cos(k*x) * cos(k*y) * sin(k*z)" + +positrons.charge = q_e +positrons.mass = m_e +positrons.profile = constant +positrons.density = n0 # number of positrons per m^3 +positrons.momentum_distribution_type = parse_momentum_function +positrons.momentum_function_ux(x,y,z) = "-epsilon * k/kp * sin(k*x) * cos(k*y) * cos(k*z)" +positrons.momentum_function_uy(x,y,z) = "-epsilon * k/kp * cos(k*x) * sin(k*y) * cos(k*z)" +positrons.momentum_function_uz(x,y,z) = "-epsilon * k/kp * cos(k*x) * cos(k*y) * sin(k*z)" + +# Diagnostics +diagnostics.diags_names = diag1 openpmd +diag1.intervals = 40 +diag1.diag_type = Full +diag1.fields_to_plot = Ez jz rho + +openpmd.intervals = 40 +openpmd.diag_type = Full +openpmd.format = openpmd diff --git a/Examples/Tests/langmuir_fluids/inputs_2d b/Examples/Tests/langmuir_fluids/inputs_2d new file mode 100644 index 00000000000..d75fa0f3393 --- /dev/null +++ b/Examples/Tests/langmuir_fluids/inputs_2d @@ -0,0 +1,69 @@ +# Maximum number of time steps +max_step = 80 + +# number of grid points +amr.n_cell = 128 128 + +# Maximum allowable size of each subdomain in the problem domain; +# this is used to decompose the domain for parallel calculations. +amr.max_grid_size = 64 + +# Maximum level in hierarchy (for now must be 0, i.e., one level in total) +amr.max_level = 0 + +# Geometry +geometry.dims = 2 +geometry.prob_lo = -20.e-6 -20.e-6 # physical domain +geometry.prob_hi = 20.e-6 20.e-6 + +# Boundary condition +boundary.field_lo = periodic periodic +boundary.field_hi = periodic periodic + +warpx.serialize_initial_conditions = 1 + +# Verbosity +warpx.verbose = 1 + +# Algorithms +algo.field_gathering = energy-conserving +warpx.use_filter = 0 + +# CFL +warpx.cfl = 1.0 + +# Parameters for the plasma wave +my_constants.epsilon = 0.01 +my_constants.n0 = 2.e24 # electron and positron densities, #/m^3 +my_constants.wp = sqrt(2.*n0*q_e**2/(epsilon0*m_e)) # plasma frequency +my_constants.kp = wp/clight # plasma wavenumber +my_constants.k = 2.*pi/20.e-6 # perturbation wavenumber +# Note: kp is calculated in SI for a density of 4e24 (i.e. 2e24 electrons + 2e24 positrons) +# k is calculated so as to have 2 periods within the 40e-6 wide box. + +# Fluids +fluids.species_names = electrons positrons + +electrons.charge = -q_e +electrons.mass = m_e +electrons.profile = constant +electrons.density = n0 # number of electrons per m^3 +electrons.momentum_distribution_type = parse_momentum_function +electrons.momentum_function_ux(x,y,z) = "epsilon * k/kp * sin(k*x) * cos(k*y) * cos(k*z)" +electrons.momentum_function_uy(x,y,z) = "epsilon * k/kp * cos(k*x) * sin(k*y) * cos(k*z)" +electrons.momentum_function_uz(x,y,z) = "epsilon * k/kp * cos(k*x) * cos(k*y) * sin(k*z)" + +positrons.charge = q_e +positrons.mass = m_e +positrons.profile = constant +positrons.density = n0 # number of positrons per m^3 +positrons.momentum_distribution_type = parse_momentum_function +positrons.momentum_function_ux(x,y,z) = "-epsilon * k/kp * sin(k*x) * cos(k*y) * cos(k*z)" +positrons.momentum_function_uy(x,y,z) = "-epsilon * k/kp * cos(k*x) * sin(k*y) * cos(k*z)" +positrons.momentum_function_uz(x,y,z) = "-epsilon * k/kp * cos(k*x) * cos(k*y) * sin(k*z)" + +# Diagnostics +diagnostics.diags_names = diag1 +diag1.intervals = 40 +diag1.diag_type = Full +diag1.fields_to_plot = Ex Ez jx jz rho diff --git a/Examples/Tests/langmuir_fluids/inputs_3d b/Examples/Tests/langmuir_fluids/inputs_3d new file mode 100644 index 00000000000..f92ccf0ed60 --- /dev/null +++ b/Examples/Tests/langmuir_fluids/inputs_3d @@ -0,0 +1,74 @@ +# Parameters for the plasma wave +my_constants.max_step = 40 +my_constants.lx = 40.e-6 # length of sides +my_constants.dx = 6.25e-07 # grid cell size +my_constants.nx = lx/dx # number of cells in each dimension +my_constants.epsilon = 0.01 +my_constants.n0 = 2.e24 # electron and positron densities, #/m^3 +my_constants.wp = sqrt(2.*n0*q_e**2/(epsilon0*m_e)) # plasma frequency +my_constants.kp = wp/clight # plasma wavenumber +my_constants.k = 2.*2.*pi/lx # perturbation wavenumber +# Note: kp is calculated in SI for a density of 4e24 (i.e. 2e24 electrons + 2e24 positrons) +# k is calculated so as to have 2 periods within the 40e-6 wide box. + +# Maximum number of time steps +max_step = max_step + +# number of grid points +amr.n_cell = nx nx nx + +# Maximum allowable size of each subdomain in the problem domain; +# this is used to decompose the domain for parallel calculations. +amr.max_grid_size = nx nx nx + +# Maximum level in hierarchy (for now must be 0, i.e., one level in total) +amr.max_level = 0 + +# Geometry +geometry.dims = 3 +geometry.prob_lo = -lx/2. -lx/2. -lx/2. # physical domain +geometry.prob_hi = lx/2. lx/2. lx/2. + +# Boundary condition +boundary.field_lo = periodic periodic periodic +boundary.field_hi = periodic periodic periodic + +warpx.serialize_initial_conditions = 1 + +# Verbosity +warpx.verbose = 1 + +# Algorithms +algo.current_deposition = esirkepov +algo.field_gathering = energy-conserving +warpx.use_filter = 0 + +# CFL +warpx.cfl = 1.0 + +# Fluids +fluids.species_names = electrons positrons + +electrons.charge = -q_e +electrons.mass = m_e +electrons.profile = constant +electrons.density = n0 # number of electrons per m^3 +electrons.momentum_distribution_type = parse_momentum_function +electrons.momentum_function_ux(x,y,z) = "epsilon * k/kp * sin(k*x) * cos(k*y) * cos(k*z)" +electrons.momentum_function_uy(x,y,z) = "epsilon * k/kp * cos(k*x) * sin(k*y) * cos(k*z)" +electrons.momentum_function_uz(x,y,z) = "epsilon * k/kp * cos(k*x) * cos(k*y) * sin(k*z)" + +positrons.charge = q_e +positrons.mass = m_e +positrons.profile = constant +positrons.density = n0 # number of positrons per m^3 +positrons.momentum_distribution_type = parse_momentum_function +positrons.momentum_function_ux(x,y,z) = "-epsilon * k/kp * sin(k*x) * cos(k*y) * cos(k*z)" +positrons.momentum_function_uy(x,y,z) = "-epsilon * k/kp * cos(k*x) * sin(k*y) * cos(k*z)" +positrons.momentum_function_uz(x,y,z) = "-epsilon * k/kp * cos(k*x) * cos(k*y) * sin(k*z)" + +# Diagnostics +diagnostics.diags_names = diag1 +diag1.intervals = max_step +diag1.diag_type = Full +diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz part_per_cell rho diff --git a/Examples/Tests/langmuir_fluids/inputs_rz b/Examples/Tests/langmuir_fluids/inputs_rz new file mode 100644 index 00000000000..2be427cbce5 --- /dev/null +++ b/Examples/Tests/langmuir_fluids/inputs_rz @@ -0,0 +1,72 @@ +# Parameters for the plasma wave +my_constants.max_step = 80 +my_constants.epsilon = 0.01 +my_constants.n0 = 2.e24 # electron density, #/m^3 +my_constants.wp = sqrt(n0*q_e**2/(epsilon0*m_e)) # plasma frequency +my_constants.kp = wp/clight # plasma wavenumber +my_constants.k0 = 2.*pi/20.e-6 # longitudianl perturbation wavenumber +my_constants.w0 = 5.e-6 # transverse perturbation length +# Note: kp is calculated in SI for a density of 2e24 +# k0 is calculated so as to have 2 periods within the 40e-6 wide box. + +# Maximum number of time steps +max_step = max_step + +# number of grid points +amr.n_cell = 64 128 + +# Maximum allowable size of each subdomain in the problem domain; +# this is used to decompose the domain for parallel calculations. +amr.max_grid_size = 64 + +# Maximum level in hierarchy (for now must be 0, i.e., one level in total) +amr.max_level = 0 + +# Geometry +geometry.dims = RZ +geometry.prob_lo = 0.e-6 -20.e-6 # physical domain +geometry.prob_hi = 20.e-6 20.e-6 +boundary.field_lo = none periodic +boundary.field_hi = none periodic + +warpx.serialize_initial_conditions = 1 + +# Verbosity +warpx.verbose = 1 + +# Algorithms +algo.field_gathering = energy-conserving +algo.current_deposition = esirkepov +warpx.use_filter = 0 + +# CFL +warpx.cfl = 1.0 + +# Having this turned on makes for a more sensitive test +warpx.do_dive_cleaning = 1 + +# Fluids +fluids.species_names = electrons ions + +electrons.charge = -q_e +electrons.mass = m_e +electrons.profile = constant +electrons.density = n0 # number of electrons per m^3 +electrons.momentum_distribution_type = parse_momentum_function +electrons.momentum_function_ux(x,y,z) = "epsilon/kp*2*x/w0**2*exp(-(x**2+y**2)/w0**2)*sin(k0*z)" +electrons.momentum_function_uy(x,y,z) = "epsilon/kp*2*y/w0**2*exp(-(x**2+y**2)/w0**2)*sin(k0*z)" +electrons.momentum_function_uz(x,y,z) = "-epsilon/kp*k0*exp(-(x**2+y**2)/w0**2)*cos(k0*z)" + + +ions.charge = q_e +ions.mass = m_p +ions.profile = constant +ions.density = n0 # number of ions per m^3 +ions.momentum_distribution_type = at_rest + +# Diagnostics +diagnostics.diags_names = diag1 +diag1.intervals = 80 +diag1.diag_type = Full + +diag1.fields_to_plot = jr jz Er Ez Bt rho diff --git a/Regression/Checksum/benchmarks_json/Langmuir_fluid_1D.json b/Regression/Checksum/benchmarks_json/Langmuir_fluid_1D.json new file mode 100644 index 00000000000..454afc73bcc --- /dev/null +++ b/Regression/Checksum/benchmarks_json/Langmuir_fluid_1D.json @@ -0,0 +1,7 @@ +{ + "lev=0": { + "Ez": 123800860242.32724, + "jz": 48549778373280.2, + "rho": 344278.3437340355 + } +} \ No newline at end of file diff --git a/Regression/Checksum/benchmarks_json/Langmuir_fluid_2D.json b/Regression/Checksum/benchmarks_json/Langmuir_fluid_2D.json new file mode 100644 index 00000000000..8afc645639d --- /dev/null +++ b/Regression/Checksum/benchmarks_json/Langmuir_fluid_2D.json @@ -0,0 +1,9 @@ +{ + "lev=0": { + "Ex": 3790984950257.2373, + "Ez": 3790984950257.2373, + "jx": 1.0093528745824196e+16, + "jz": 1.0093528745824196e+16, + "rho": 21098347.344494 + } +} \ No newline at end of file diff --git a/Regression/Checksum/benchmarks_json/Langmuir_fluid_RZ.json b/Regression/Checksum/benchmarks_json/Langmuir_fluid_RZ.json new file mode 100644 index 00000000000..2613f971f91 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/Langmuir_fluid_RZ.json @@ -0,0 +1,10 @@ +{ + "lev=0": { + "Bt": 3.64571737083816, + "Er": 1300317001574.1577, + "Ez": 1815076954086.807, + "jr": 244937533603400.0, + "jz": 342537403264912.5, + "rho": 10096982.130957149 + } +} \ No newline at end of file diff --git a/Regression/Checksum/benchmarks_json/Langmuir_fluid_multi.json b/Regression/Checksum/benchmarks_json/Langmuir_fluid_multi.json new file mode 100644 index 00000000000..1e5d41f6677 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/Langmuir_fluid_multi.json @@ -0,0 +1,15 @@ +{ + "lev=0": { + "Bx": 20.814308179643078, + "By": 20.814308179643234, + "Bz": 20.81430817964295, + "Ex": 82994788022201.23, + "Ey": 82994788022201.23, + "Ez": 82994788022201.23, + "jx": 6.343089944595606e+16, + "jy": 6.3430899445956056e+16, + "jz": 6.3430899445956056e+16, + "part_per_cell": 0.0, + "rho": 694435081.2275198 + } +} \ No newline at end of file diff --git a/Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid.json b/Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid.json new file mode 100644 index 00000000000..2843e49ce22 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid.json @@ -0,0 +1,15 @@ +{ + "lev=0": { + "Bx": 14264658.38987597, + "By": 0.0, + "Bz": 0.0, + "Ex": 0.0, + "Ey": 4276056420659746.5, + "Ez": 762168740318568.1, + "jx": 0.0, + "jy": 7.47674123799233e+16, + "jz": 4.817762115932484e+17, + "rho": 1609691680.1267354 + } +} + diff --git a/Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid_boosted.json b/Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid_boosted.json new file mode 100644 index 00000000000..85919660834 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/LaserAcceleration_1d_fluid_boosted.json @@ -0,0 +1,14 @@ +{ + "lev=0": { + "Bx": 7268241.144300916, + "By": 0.0, + "Bz": 0.0, + "Ex": 0.0, + "Ey": 2146970168314705.0, + "Ez": 370041408451418.1, + "jx": 0.0, + "jy": 3.4818196503065956e+16, + "jz": 2.3196405062669472e+17, + "rho": 775945622.8993922 + } +} \ No newline at end of file diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index da9d8398869..e4c502d858c 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -1096,6 +1096,78 @@ particleTypes = electrons positrons analysisRoutine = Examples/Tests/langmuir/analysis_3d.py analysisOutputImage = langmuir_multi_analysis.png +[Langmuir_fluid_1D] +buildDir = . +inputFile = Examples/Tests/langmuir_fluids/inputs_1d +runtime_params = warpx.do_dynamic_scheduling=0 +dim = 1 +addToCompileString = +cmakeSetupOpts = -DWarpX_DIMS=1 +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 0 +analysisRoutine = Examples/Tests/langmuir_fluids/analysis_1d.py +analysisOutputImage = langmuir_fluid_multi_1d_analysis.png + +[Langmuir_fluid_RZ] +buildDir = . +inputFile = Examples/Tests/langmuir_fluids/inputs_rz +runtime_params = warpx.do_dynamic_scheduling=0 +dim = 2 +addToCompileString = USE_RZ=TRUE +cmakeSetupOpts = -DWarpX_DIMS=RZ +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 0 +analysisRoutine = Examples/Tests/langmuir_fluids/analysis_rz.py +analysisOutputImage = langmuir_fluid_rz_analysis.png + +[Langmuir_fluid_2D] +buildDir = . +inputFile = Examples/Tests/langmuir_fluids/inputs_2d +runtime_params = warpx.do_dynamic_scheduling=0 +dim = 2 +addToCompileString = +cmakeSetupOpts = -DWarpX_DIMS=2 -DCMAKE_BUILD_TYPE=Debug +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 0 +analysisRoutine = Examples/Tests/langmuir_fluids/analysis_2d.py +analysisOutputImage = langmuir_fluid_multi_2d_analysis.png + +[Langmuir_fluid_multi] +buildDir = . +inputFile = Examples/Tests/langmuir_fluids/inputs_3d +runtime_params = warpx.do_dynamic_scheduling=0 +dim = 3 +addToCompileString = +cmakeSetupOpts = -DWarpX_DIMS=3 +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 0 +analysisRoutine = Examples/Tests/langmuir_fluids/analysis_3d.py +analysisOutputImage = langmuir_fluid_multi_analysis.png + [Langmuir_multi_1d] buildDir = . inputFile = Examples/Tests/langmuir/inputs_1d @@ -1741,6 +1813,42 @@ compareParticles = 1 particleTypes = electrons analysisRoutine = Examples/analysis_default_regression.py +[LaserAcceleration_1d_fluid] +buildDir = . +inputFile = Examples/Physics_applications/laser_acceleration/inputs_1d_fluids +runtime_params = +dim = 1 +addToCompileString = USE_OPENPMD=TRUE QED=FALSE +cmakeSetupOpts = -DWarpX_DIMS=1 -DWarpX_OPENPMD=ON -DWarpX_QED=OFF +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 1 +particleTypes = electrons ions +analysisRoutine = Examples/Physics_applications/laser_acceleration/analysis_1d_fluids.py + +[LaserAcceleration_1d_fluid_boosted] +buildDir = . +inputFile = Examples/Physics_applications/laser_acceleration/inputs_1d_fluids_boosted +runtime_params = +dim = 1 +addToCompileString = USE_OPENPMD=TRUE QED=FALSE +cmakeSetupOpts = -DWarpX_DIMS=1 -DWarpX_OPENPMD=ON -DWarpX_QED=OFF +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 1 +particleTypes = electrons ions +analysisRoutine = Examples/Physics_applications/laser_acceleration/analysis_1d_fluids_boosted.py + [LaserAccelerationBoost] buildDir = . inputFile = Examples/Physics_applications/laser_acceleration/inputs_2d_boost diff --git a/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp b/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp index 73feef4ed00..c36e31c0f93 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp +++ b/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp @@ -7,6 +7,8 @@ #include "Utils/WarpXAlgorithmSelection.H" #endif #include "Particles/MultiParticleContainer.H" +#include "Fluids/MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" #include "Particles/WarpXParticleContainer.H" #include "WarpX.H" @@ -41,6 +43,10 @@ RhoFunctor::operator() ( amrex::MultiFab& mf_dst, const int dcomp, const int /*i if (m_species_index == -1) { auto& mypc = warpx.GetPartContainer(); rho = mypc.GetChargeDensity(m_lev, true); + if (warpx.DoFluidSpecies()) { + auto& myfl = warpx.GetFluidContainer(); + myfl.DepositCharge(m_lev, *rho); + } } // Dump rho per species else { diff --git a/Source/Evolve/WarpXEvolve.cpp b/Source/Evolve/WarpXEvolve.cpp index 067cd9a11b6..467a4268d97 100644 --- a/Source/Evolve/WarpXEvolve.cpp +++ b/Source/Evolve/WarpXEvolve.cpp @@ -24,6 +24,8 @@ #endif #include "Parallelization/GuardCellManager.H" #include "Particles/MultiParticleContainer.H" +#include "Fluids/MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" #include "Particles/ParticleBoundaryBuffer.H" #include "Python/WarpX_py.H" #include "Utils/TextMsg.H" @@ -1031,6 +1033,12 @@ WarpX::PushParticlesandDepose (int lev, amrex::Real cur_time, DtType a_dt_type, // of the filter to avoid incorrect results (moved to `SyncCurrentAndRho()`). // Might this be related to issue #1943? #endif + if (do_fluid_species) { + myfl->Evolve(lev, + *Efield_aux[lev][0],*Efield_aux[lev][1],*Efield_aux[lev][2], + *Bfield_aux[lev][0],*Bfield_aux[lev][1],*Bfield_aux[lev][2], + rho_fp[lev].get(),*current_x, *current_y, *current_z, cur_time, skip_deposition); + } } } diff --git a/Source/FieldSolver/ElectrostaticSolver.cpp b/Source/FieldSolver/ElectrostaticSolver.cpp index a6932f223e2..216e80f7494 100644 --- a/Source/FieldSolver/ElectrostaticSolver.cpp +++ b/Source/FieldSolver/ElectrostaticSolver.cpp @@ -7,6 +7,8 @@ #include "WarpX.H" #include "FieldSolver/ElectrostaticSolver.H" +#include "Fluids/MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" #include "Parallelization/GuardCellManager.H" #include "Particles/MultiParticleContainer.H" #include "Particles/WarpXParticleContainer.H" @@ -209,6 +211,10 @@ WarpX::AddSpaceChargeFieldLabFrame () // Deposit particle charge density (source of Poisson solver) mypc->DepositCharge(rho_fp, 0.0_rt); + if (do_fluid_species) { + int const lev = 0; + myfl->DepositCharge( lev, *rho_fp[lev] ); + } SyncRho(rho_fp, rho_cp, charge_buf); // Apply filter, perform MPI exchange, interpolate across levels #ifndef WARPX_DIM_RZ diff --git a/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp b/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp index b502620641b..36029e7d3eb 100644 --- a/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp +++ b/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp @@ -10,6 +10,8 @@ #include "FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H" #include "Particles/MultiParticleContainer.H" #include "Utils/TextMsg.H" +#include "Fluids/MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" #include "Utils/WarpXProfilerWrapper.H" #include "WarpX.H" @@ -30,6 +32,13 @@ void WarpX::HybridPICEvolveFields () // Perform current deposition at t_{n+1/2}. mypc->DepositCurrent(current_fp, dt[0], -0.5_rt * dt[0]); + // Deposit cold-relativistic fluid charge and current + if (do_fluid_species) { + int const lev = 0; + myfl->DepositCharge(lev, *rho_fp[lev]); + myfl->DepositCurrent(lev, *current_fp[lev][0], *current_fp[lev][1], *current_fp[lev][2]); + } + // Synchronize J and rho: // filter (if used), exchange guard cells, interpolate across MR levels // and apply boundary conditions diff --git a/Source/Fluids/CMakeLists.txt b/Source/Fluids/CMakeLists.txt new file mode 100644 index 00000000000..a5f28debbbd --- /dev/null +++ b/Source/Fluids/CMakeLists.txt @@ -0,0 +1,8 @@ +foreach(D IN LISTS WarpX_DIMS) + warpx_set_suffix_dims(SD ${D}) + target_sources(lib_${SD} + PRIVATE + MultiFluidContainer.cpp + WarpXFluidContainer.cpp + ) +endforeach() diff --git a/Source/Fluids/Make.package b/Source/Fluids/Make.package new file mode 100644 index 00000000000..96bce415c4a --- /dev/null +++ b/Source/Fluids/Make.package @@ -0,0 +1,4 @@ +CEXE_sources += MultiFluidContainer.cpp +CEXE_sources += WarpXFluidContainer.cpp + +VPATH_LOCATIONS += $(WARPX_HOME)/Source/Fluids diff --git a/Source/Fluids/MultiFluidContainer.H b/Source/Fluids/MultiFluidContainer.H new file mode 100644 index 00000000000..94d95d179e5 --- /dev/null +++ b/Source/Fluids/MultiFluidContainer.H @@ -0,0 +1,78 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#ifndef WARPX_MultiFluidContainer_H_ +#define WARPX_MultiFluidContainer_H_ +#include "Evolve/WarpXDtType.H" + +#include "WarpXFluidContainer_fwd.H" + +#include +#include + +#include + +/** + * The class MultiFluidContainer holds multiple instances of the + * class WarpXFluidContainer, stored in its member variable "allcontainers". + * The class WarpX typically has a single (pointer to an) instance of + * MultiFluidContainer. + * + * MultiFluidContainer typically has two types of functions: + * - Functions that loop over all instances of WarpXFluidContainer in + * allcontainers and calls the corresponding function (for instance, + * MultiFluidContainer::Evolve loops over all fluid containers and + * calls the corresponding WarpXFluidContainer::Evolve function). + * - Functions that specifically handle multiple species (for instance + * ReadParameters). + */ +class MultiFluidContainer +{ + +public: + + MultiFluidContainer (int nlevs_max); + + ~MultiFluidContainer() {} + + WarpXFluidContainer& + GetFluidContainer (int ispecies) const {return *allcontainers[ispecies];} + +#ifdef WARPX_USE_OPENPMD + std::unique_ptr& GetUniqueContainer(int ispecies) { + return allcontainers[ispecies]; + } +#endif + + void AllocateLevelMFs (int lev, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm); + + void InitData (int lev, amrex::Box init_box, amrex::Real cur_time); + + /// + /// This evolves all the fluids by one PIC time step, including current deposition, the + /// field solve, and pushing the fluids, for all the species in the MultiFluidContainer. + /// + void Evolve (int lev, + const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, + const amrex::MultiFab& Bx, const amrex::MultiFab& By, const amrex::MultiFab& Bz, + amrex::MultiFab* rho, amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz, + amrex::Real cur_time, bool skip_deposition=false); + + int nSpecies() const {return species_names.size();} + + void DepositCharge (int lev, amrex::MultiFab &rho); + void DepositCurrent (int lev, + amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz); + +private: + + std::vector species_names; + + // Vector of fluid species + amrex::Vector> allcontainers; + +}; +#endif /*WARPX_MultiFluidContainer_H_*/ diff --git a/Source/Fluids/MultiFluidContainer.cpp b/Source/Fluids/MultiFluidContainer.cpp new file mode 100644 index 00000000000..234cefb4f07 --- /dev/null +++ b/Source/Fluids/MultiFluidContainer.cpp @@ -0,0 +1,73 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#include "MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" +#include "Utils/Parser/ParserUtils.H" + +#include + +using namespace amrex; + +MultiFluidContainer::MultiFluidContainer (int nlevs_max) +{ + const ParmParse pp_fluids("fluids"); + pp_fluids.queryarr("species_names", species_names); + + const int nspecies = static_cast(species_names.size()); + + allcontainers.resize(nspecies); + for (int i = 0; i < nspecies; ++i) { + allcontainers[i] = std::make_unique(nlevs_max, i, species_names[i]); + } +} + +void +MultiFluidContainer::AllocateLevelMFs (int lev, const BoxArray& ba, const DistributionMapping& dm) +{ + for (auto& fl : allcontainers) { + fl->AllocateLevelMFs(lev, ba, dm); + } +} + +void +MultiFluidContainer::InitData (int lev, amrex::Box init_box, amrex::Real cur_time) +{ + for (auto& fl : allcontainers) { + fl->InitData(lev, init_box, cur_time); + } +} + + +void +MultiFluidContainer::DepositCharge (int lev, amrex::MultiFab &rho) +{ + for (auto& fl : allcontainers) { + fl->DepositCharge(lev,rho); + } +} + +void +MultiFluidContainer::DepositCurrent (int lev, + amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz) +{ + for (auto& fl : allcontainers) { + fl->DepositCurrent(lev,jx,jy,jz); + } +} + +void +MultiFluidContainer::Evolve (int lev, + const MultiFab& Ex, const MultiFab& Ey, const MultiFab& Ez, + const MultiFab& Bx, const MultiFab& By, const MultiFab& Bz, + MultiFab* rho, MultiFab& jx, MultiFab& jy, MultiFab& jz, + amrex::Real cur_time, bool skip_deposition) +{ + for (auto& fl : allcontainers) { + fl->Evolve(lev, Ex, Ey, Ez, Bx, By, Bz, rho, jx, jy, jz, cur_time, skip_deposition); + } +} diff --git a/Source/Fluids/MultiFluidContainer_fwd.H b/Source/Fluids/MultiFluidContainer_fwd.H new file mode 100644 index 00000000000..a1a55c3b450 --- /dev/null +++ b/Source/Fluids/MultiFluidContainer_fwd.H @@ -0,0 +1,12 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ + +#ifndef WARPX_MultiFluidContainer_fwd_H_ + +class MultiFluidContainer; + +#endif /* WARPX_MultiFluidContainer_fwd_H_ */ diff --git a/Source/Fluids/MusclHancockUtils.H b/Source/Fluids/MusclHancockUtils.H new file mode 100644 index 00000000000..d0c37f9c58f --- /dev/null +++ b/Source/Fluids/MusclHancockUtils.H @@ -0,0 +1,484 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#ifndef WARPX_MusclHancock_H_ +#define WARPX_MusclHancock_H_ + +#include +#include +#include +#include + + +// Euler push for momentum source (r-direction) +// Note: assumes U normalized by c +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real F_r (amrex::Real r, amrex::Real u_r, amrex::Real u_theta, amrex::Real u_z, amrex::Real dt) +{ + return dt*(-u_theta*u_theta/r)/sqrt(1.0 + u_r*u_r + u_theta*u_theta + u_z*u_z) + u_r; +} + +// Euler push for momentum source (theta-direction) +// Note: assumes U normalized by c +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real F_theta (amrex::Real r, amrex::Real u_r, amrex::Real u_theta, amrex::Real u_z, amrex::Real dt) +{ + return dt*(u_theta*u_r/r)/sqrt(1.0 + u_r*u_r + u_theta*u_theta + u_z*u_z) + u_theta; +} +// Velocity at the half step +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real V_calc (const amrex::Array4& U, int i, int j, int k, int comp, amrex::Real c) +{ + // comp -> x, y, z -> 0, 1, 2, return Vx, Vy, or Vz: + amrex::Real gamma = std::sqrt(1.0 + (U(i,j,k,1)*U(i,j,k,1) + U(i,j,k,2)*U(i,j,k,2) + U(i,j,k,3)*U(i,j,k,3))/(c*c)); + return U(i,j,k,comp+1)/gamma; +} +// mindmod +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real minmod (amrex::Real a, amrex::Real b) +{ + if (a > 0.0 && b > 0.0) + return std::min(a, b); + else if (a < 0.0 && b < 0.0) + return std::max(a, b); + else + return 0.0; +} +// Min of 3 inputs +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real min3 (amrex::Real a, amrex::Real b, amrex::Real c) +{ + return std::min(a, std::min(b, c) ); +} +// Max of 3 inputs +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real max3 (amrex::Real a, amrex::Real b, amrex::Real c) +{ + return std::max(a, std::max(b, c) ); +} +// mindmod +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real minmod3 (amrex::Real a, amrex::Real b , amrex::Real c) +{ + if (a > 0.0 && b > 0.0 && c > 0.0) + return min3(a,b,c); + else if (a < 0.0 && b < 0.0 && c < 0.0) + return max3(a,b,c); + else + return 0.0; +} +//maxmod +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real maxmod (amrex::Real a, amrex::Real b) +{ + if (a > 0.0 && b > 0.0) + return std::max(a, b); + else if (a < 0.0 && b < 0.0) + return std::min(a, b); + else + return 0.0; +} +// Rusanov Flux (density) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real flux_N (const amrex::Array4& Um, const amrex::Array4& Up, +int i, int j, int k, amrex::Real Vm, amrex::Real Vp) +{ + amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); + return 0.5*(Vm*Um(i,j,k,0) + Vp*Up(i,j,k,0)) - (0.5*c)*(Up(i,j,k,0) - Um(i,j,k,0)); +} +// Rusanov Flux (Momentum density x) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real flux_NUx (const amrex::Array4& Um, const amrex::Array4& Up, +int i, int j, int k, amrex::Real Vm, amrex::Real Vp) +{ + amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); + return 0.5*(Vm*Um(i,j,k,0)*Um(i,j,k,1) + Vp*Up(i,j,k,0)*Up(i,j,k,1)) + - (0.5*c)*(Up(i,j,k,0)*Up(i,j,k,1) - Um(i,j,k,0)*Um(i,j,k,1)); +} +// Rusanov Flux (Momentum density y) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real flux_NUy (const amrex::Array4& Um, const amrex::Array4& Up, +int i, int j, int k, amrex::Real Vm, amrex::Real Vp) +{ + amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); + return 0.5*(Vm*Um(i,j,k,0)*Um(i,j,k,2) + Vp*Up(i,j,k,0)*Up(i,j,k,2)) + - (0.5*c)*(Up(i,j,k,0)*Up(i,j,k,2) - Um(i,j,k,0)*Um(i,j,k,2)); +} +// Rusanov Flux (Momentum density z) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real flux_NUz (const amrex::Array4& Um, const amrex::Array4& Up, +int i, int j, int k, amrex::Real Vm, amrex::Real Vp) +{ + amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); + return 0.5*(Vm*Um(i,j,k,0)*Um(i,j,k,3) + Vp*Up(i,j,k,0)*Up(i,j,k,3)) + - (0.5*c)*(Up(i,j,k,0)*Up(i,j,k,3) - Um(i,j,k,0)*Um(i,j,k,3)); +} +// ave_minmod high diffusivity, sigma can be between [1,2] +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real ave_adjustable_diff (amrex::Real a, amrex::Real b) +{ + amrex::Real sigma = 2.0*0.732050807568877; + if (a*b > 0.0) + return minmod3( (a+b)/2.0, sigma*a, sigma*b ); + else + return 0.0; +} +// ave_minmod Low diffusivity +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real ave (amrex::Real a, amrex::Real b) +{ + if (a*b > 0.0) + return minmod3( (a+b)/2.0, 2.0*a, 2.0*b ); + else + return 0.0; +} +// ave_superbee +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real ave_superbee (amrex::Real a, amrex::Real b) +{ + if (a*b > 0.0) + return minmod( maxmod(a,b), minmod(2.0*a,2.0*b)); + else + return 0.0; +} +// stage2 slope limiting +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real ave_stage2 (amrex::Real dQ, amrex::Real a, amrex::Real b, amrex::Real c) +{ + // sigma = sqrt(3) -1 + amrex::Real sigma = 0.732050807568877; + amrex::Real dq_min = 2.0*std::min( b - min3(a,b,c), max3(a,b,c) - b); + return ( std::abs(dQ)/dQ ) * std::min( std::abs(dQ) , sigma*std::abs(dq_min) ); +} +// Returns the offset indices for the "plus" grid +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +void plus_index_offsets (int i, int j, int k, int& ip, int& jp, int& kp, int comp) +{ + // Find the correct offsets +#if defined(WARPX_DIM_3D) + if (comp == 0) { //x + ip = i - 1; jp = j; kp = k; + } else if (comp == 1){ //y + ip = i; jp = j - 1; kp = k; + } else if (comp == 2){ //z + ip = i; jp = j; kp = k - 1; + } + #elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + if (comp == 0) { //x + ip = i - 1; jp = j; kp = k; + } else if (comp == 2){ //z + ip = i; jp = j - 1; kp = k; + } +#else + if (comp == 2) { //z + ip = i - 1; jp = j; kp = k; + } +#endif +} +// Compute the zero edges +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +void compute_U_edges (const amrex::Array4& Um, const amrex::Array4& Up, int i, int j, int k, amrex::Box box, +amrex::Real U_tilde0, amrex::Real U_tilde1, amrex::Real U_tilde2, amrex::Real U_tilde3, +amrex::Real dU0x, amrex::Real dU1x, amrex::Real dU2x, amrex::Real dU3x, int comp) +{ + + // comp -> x, y, z -> 0, 1, 2 + int ip, jp, kp; + plus_index_offsets(i, j, k, ip, jp, kp, comp); + + if ( box.contains(i,j,k) ) { + Um(i,j,k,0) = U_tilde0 + dU0x/2.0; + Um(i,j,k,1) = U_tilde1 + dU1x/2.0; + Um(i,j,k,2) = U_tilde2 + dU2x/2.0; + Um(i,j,k,3) = U_tilde3 + dU3x/2.0; + } + + if ( box.contains(ip,jp,kp) ) { + Up(ip,jp,kp,0) = U_tilde0 - dU0x/2.0; + Up(ip,jp,kp,1) = U_tilde1 - dU1x/2.0; + Up(ip,jp,kp,2) = U_tilde2 - dU2x/2.0; + Up(ip,jp,kp,3) = U_tilde3 - dU3x/2.0; + } +} +// Compute the zero edges +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +void set_U_edges_to_zero (const amrex::Array4& Um, +const amrex::Array4& Up, int i, int j, int k, amrex::Box box, int comp) +{ + + // comp -> x, y, z -> 0, 1, 2 + int ip, jp, kp; + plus_index_offsets(i, j, k, ip, jp, kp, comp); + + if ( box.contains(i,j,k) ) { + Um(i,j,k,0) = 0.0; + Um(i,j,k,1) = 0.0; + Um(i,j,k,2) = 0.0; + Um(i,j,k,3) = 0.0; + } + + if ( box.contains(ip,jp,kp) ) { + Up(ip,jp,kp,0) = 0.0; + Up(ip,jp,kp,1) = 0.0; + Up(ip,jp,kp,2) = 0.0; + Up(ip,jp,kp,3) = 0.0; + } +} +// Positivity Limiter +// if Q_minus or Q_plus is zero for the density (i.e. component 0 of Q_minus/Q_plus), set dQ to 0 and recompute Q_minus / Q_plus +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +void positivity_limiter (const amrex::Array4& U_edge_plus, +const amrex::Array4& U_edge_minus, const amrex::Array4& N_arr, +int i, int j, int k, amrex::Box box, amrex::Real Ux, amrex::Real Uy, amrex::Real Uz, +int comp) +{ + + // comp -> x, y, z -> 0, 1, 2 + int ip, jp, kp; + plus_index_offsets(i, j, k, ip, jp, kp, comp); + + // Set the edges to zero. If one edge in a cell is zero, we must self-consistently + // set the slope to zero (hence why we have the three cases, the first is when + // both points exist, and the second two are are edge cases) + if (( box.contains(i,j,k) ) && ( box.contains(ip,jp,kp) )) { + if ((U_edge_minus(i,j,k,0) < 0.0) || (U_edge_plus(ip,jp,kp,0) < 0.0)) { + U_edge_minus(i,j,k,0) = N_arr(i,j,k); + U_edge_minus(i,j,k,1) = Ux; + U_edge_minus(i,j,k,2) = Uy; + U_edge_minus(i,j,k,3) = Uz; + U_edge_plus(ip,jp,kp,0) = N_arr(i,j,k); + U_edge_plus(ip,jp,kp,1) = Ux; + U_edge_plus(ip,jp,kp,2) = Uy; + U_edge_plus(ip,jp,kp,3) = Uz; + } + } else if (( box.contains(i,j,k) ) && ( box.contains(ip,jp,kp) != 1)) { + if (U_edge_minus(i,j,k,0) < 0.0) { + U_edge_minus(i,j,k,0) = N_arr(i,j,k); + U_edge_minus(i,j,k,1) = Ux; + U_edge_minus(i,j,k,2) = Uy; + U_edge_minus(i,j,k,3) = Uz; + } + } else if (( box.contains(i,j,k) != 1 ) && ( box.contains(ip,jp,kp) )) { + if (U_edge_plus(ip,jp,kp,0) < 0.0){ + U_edge_plus(ip,jp,kp,0) = N_arr(i,j,k); + U_edge_plus(ip,jp,kp,1) = Ux; + U_edge_plus(ip,jp,kp,2) = Uy; + U_edge_plus(ip,jp,kp,3) = Uz; + } + } +} + +// Compute the difference in N (down-x) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real DownDx_N (const amrex::Array4& N, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + return N(i,j,k) - N(i-1,j,k); +#else + amrex::ignore_unused(N, i, j, k); + return 0.0; +#endif +} +// Compute the difference in N (up-x) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real UpDx_N (const amrex::Array4& N, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + return N(i+1,j,k) - N(i,j,k); +#else + amrex::ignore_unused(N, i, j, k); + return 0.0; +#endif +} +// Compute the difference in N (down-y) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real DownDy_N (const amrex::Array4& N, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) + return N(i,j,k) - N(i,j-1,k); +#else + amrex::ignore_unused(N, i, j, k); + return 0.0; +#endif +} +// Compute the difference in N (up-y) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real UpDy_N (const amrex::Array4& N, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) + return N(i,j+1,k) - N(i,j,k); +#else + amrex::ignore_unused(N, i, j, k); + return 0.0; +#endif +} +// Compute the difference in N (down-z) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real DownDz_N (const amrex::Array4& N, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) + return N(i,j,k) - N(i,j,k-1); +#elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + return N(i,j,k) - N(i,j-1,k); +#else + return N(i,j,k) - N(i-1,j,k); +#endif +} +// Compute the difference in N (up-z) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real UpDz_N (const amrex::Array4& N, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) + return N(i,j,k+1) - N(i,j,k); +#elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + return N(i,j+1,k) - N(i,j,k); +#else + return N(i+1,j,k) - N(i,j,k); +#endif +} + + +// Compute the difference in U (down-x) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real DownDx_U (const amrex::Array4& N, +const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + // U is zero if N is zero, Check positivity before dividing + amrex::Real U_m = 0; + if (N(i-1,j,k) > 0) U_m = NU(i-1,j,k)/N(i-1,j,k); + return U - U_m; +#else + amrex::ignore_unused(N, NU, U, i, j, k); + return 0.0; +#endif +} +// Compute the difference in U (up-x) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real UpDx_U (const amrex::Array4& N, +const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + // U is zero if N is zero, Check positivity before dividing + amrex::Real U_p = 0; + if (N(i+1,j,k) > 0) U_p = NU(i+1,j,k)/N(i+1,j,k); + return U_p - U; +#else + amrex::ignore_unused(N, NU, U, i, j, k); + return 0.0; +#endif +} + +// Compute the difference in U (down-y) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real DownDy_U (const amrex::Array4& N, +const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) + // U is zero if N is zero, Check positivity before dividing + amrex::Real U_m = 0; + if (N(i,j-1,k) > 0) U_m = NU(i,j-1,k)/N(i,j-1,k); + return U - U_m; +#else + amrex::ignore_unused(N, NU, U, i, j, k); + return 0.0; +#endif +} +// Compute the difference in U (up-y) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real UpDy_U (const amrex::Array4& N, +const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) +{ + // Write the correct differences +#if defined(WARPX_DIM_3D) + // U is zero if N is zero, Check positivity before dividing + amrex::Real U_p = 0; + if (N(i,j+1,k) > 0) U_p = NU(i,j+1,k)/N(i,j+1,k); + return U_p - U; +#else + amrex::ignore_unused(N, NU, U, i, j, k); + return 0.0; +#endif +} + +// Compute the difference in U (down-z) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real DownDz_U (const amrex::Array4& N, +const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) +{ + // Write the correct differences + amrex::Real U_m = 0; + + // U is zero if N is zero, Check positivity before dividing +#if defined(WARPX_DIM_3D) + if (N(i,j,k-1) > 0) U_m = NU(i,j,k-1)/N(i,j,k-1); +#elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + if (N(i,j-1,k) > 0) U_m = NU(i,j-1,k)/N(i,j-1,k); +#else + if (N(i-1,j,k) > 0) U_m = NU(i-1,j,k)/N(i-1,j,k); +#endif + + // Return the difference + return U - U_m; +} +// Compute the difference in U (up-z) +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real UpDz_U (const amrex::Array4& N, +const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) +{ + // Write the correct differences + amrex::Real U_p = 0; + + // U is zero if N is zero, Check positivity before dividing +#if defined(WARPX_DIM_3D) + if (N(i,j,k+1) > 0) U_p = NU(i,j,k+1)/N(i,j,k+1); +#elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + if (N(i,j+1,k) > 0) U_p = NU(i,j+1,k)/N(i,j+1,k); +#else + if (N(i+1,j,k) > 0) U_p = NU(i+1,j,k)/N(i+1,j,k); +#endif + + // Return the difference + return U_p - U; +} + + +// Flux difference calculation +AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE +amrex::Real dF (const amrex::Array4& U_minus, +const amrex::Array4& U_plus,int i,int j,int k,amrex::Real clight, int comp, int dir) +{ + // dir -> x, y, z -> 0, 1, 2 + int ip, jp, kp; + plus_index_offsets(i, j, k, ip, jp, kp, dir); + + amrex::Real V_L_minus = V_calc(U_minus,ip,jp,kp,dir,clight); + amrex::Real V_I_minus = V_calc(U_minus,i,j,k,dir,clight); + amrex::Real V_L_plus = V_calc(U_plus,ip,jp,kp,dir,clight); + amrex::Real V_I_plus = V_calc(U_plus,i,j,k,dir,clight); + + // Flux differences depending on the component to compute + if (comp == 0){ + return flux_N( U_minus, U_plus, i, j, k, V_I_minus, V_I_plus) - flux_N( U_minus, U_plus, ip, jp, kp, V_L_minus, V_L_plus); + } else if (comp == 1){ + return flux_NUx( U_minus, U_plus, i, j, k, V_I_minus, V_I_plus) - flux_NUx( U_minus, U_plus, ip, jp, kp, V_L_minus, V_L_plus); + } else if (comp == 2){ + return flux_NUy( U_minus, U_plus, i, j, k, V_I_minus, V_I_plus) - flux_NUy( U_minus, U_plus, ip, jp, kp, V_L_minus, V_L_plus); + } else { //if (comp == 3) + return flux_NUz( U_minus, U_plus, i, j, k, V_I_minus, V_I_plus) - flux_NUz( U_minus, U_plus, ip, jp, kp, V_L_minus, V_L_plus); + } +} + +#endif /*WARPX_MusclHancock_H_*/ diff --git a/Source/Fluids/WarpXFluidContainer.H b/Source/Fluids/WarpXFluidContainer.H new file mode 100644 index 00000000000..28f37ae393b --- /dev/null +++ b/Source/Fluids/WarpXFluidContainer.H @@ -0,0 +1,186 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#ifndef WARPX_WarpXFluidContainer_H_ +#define WARPX_WarpXFluidContainer_H_ + +#include "Evolve/WarpXDtType.H" +#include "Initialization/PlasmaInjector.H" +#include "MultiFluidContainer.H" + +#include +#include + +#include + + +/** + * WarpXFluidContainer is the base class from which all concrete + * fluid container classes derive. + * + * WarpXFluidContainer contains the main functions for initialization, + * interaction with the grid (field gather and current deposition), fluid + * source and push, advective update and updates for non-intertial terms. + */ +class WarpXFluidContainer +{ +public: + friend MultiFluidContainer; + + WarpXFluidContainer (int nlevs_max, int ispecies, const std::string& name); + ~WarpXFluidContainer() {} + + void AllocateLevelMFs (int lev, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm); + + void InitData (int lev, amrex::Box init_box, amrex::Real cur_time); + + void ReadParameters (); + + /** + * Evolve updates a single timestep (dt) of the cold relativistic fluid eqautions + */ + void Evolve (int lev, + const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, + const amrex::MultiFab& Bx, const amrex::MultiFab& By, const amrex::MultiFab& Bz, + amrex::MultiFab* rho, amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz, + amrex::Real cur_time, bool skip_deposition=false); + + /** + * AdvectivePush_Muscl takes a single timestep (dt) of the cold relativistic fluid equations + * using a Muscl-Handcock scheme + * + * \brief Advective term, cold-rel. fluids + * + * \param[in] lev refinement level + */ + void AdvectivePush_Muscl (int lev); + + + /** + * Apply (non-periodic) BC on the fluids (needed for spatial derivative), + * and communicate N, NU at boundaries + * + * \brief Apply non-periodic BC to fluids and communicate boundaries + * + * \param[in] lev refinement level + */ + void ApplyBcFluidsAndComms (int lev); + +#if defined(WARPX_DIM_RZ) + /** + * centrifugal_source_rz adds contributions due to curvature acceleration for a + * single timestep using an SSP-RK3 timestep for RZ specifically + * + * \brief Centrifugal source term + * + * \param[in] lev refinement level + */ + void centrifugal_source_rz (int lev); +#endif + + /** + * GatherAndPush introduces the Lorentz term in the cold relativistic fluid + * equations for a single timestep (dt) using Higuera and Cary Push + * + * \brief Lorentz Momentum Source + * + * \param[in] lev refinement level + * \param[in] Ex Yee electric field (x) + * \param[in] Ey Yee electric field (y) + * \param[in] Ez Yee electric field (z) + * \param[in] Bx Yee magnetic field (x) + * \param[in] By Yee magnetic field (y) + * \param[in] Bz Yee magnetic field (z) + * \param[in] cur_time Current time + */ + void GatherAndPush (int lev, + const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, + const amrex::MultiFab& Bx, const amrex::MultiFab& By, const amrex::MultiFab& Bz, + amrex::Real cur_time); + + /** + * DepositCurrent interpolates the fluid current density comps. onto the Yee grid and + * sums the contributions to the particle current density + * + * \brief Deposit fluid current density. + * + * \param[in] lev refinement level + * \param[in,out] jx current density MultiFab x comp. + * \param[in,out] jy current density MultiFab y comp. + * \param[in,out] jz current density MultiFab z comp. + */ + void DepositCurrent (int lev, + amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz); + + /** + * DepositCharge interpolates the fluid charge density onto the Yee grid and + * sums the contributions to the particle charge density + * + * \brief Deposit fluid charge density. + * + * \param[in] lev refinement level + * \param[in,out] rho charge density MultiFab. + */ + void DepositCharge (int lev, amrex::MultiFab &rho, int icomp = 0); + + amrex::Real getCharge () const {return charge;} + amrex::Real getMass () const {return mass;} + +protected: + int species_id; + std::string species_name; + amrex::Real charge; + amrex::Real mass; + + int do_not_push = 0; + int do_not_gather = 0; + int do_not_deposit = 0; + PhysicalSpecies physical_species; + + // Parser for external fields + std::string m_B_ext_s = "none"; + std::string m_E_ext_s = "none"; + + // Parser for B_external on the particle + std::unique_ptr m_Bx_parser; + std::unique_ptr m_By_parser; + std::unique_ptr m_Bz_parser; + amrex::ParserExecutor<4> m_Bxfield_parser; + amrex::ParserExecutor<4> m_Byfield_parser; + amrex::ParserExecutor<4> m_Bzfield_parser; + + // Parser for E_external on the particle + std::unique_ptr m_Ex_parser; + std::unique_ptr m_Ey_parser; + std::unique_ptr m_Ez_parser; + amrex::ParserExecutor<4> m_Exfield_parser; + amrex::ParserExecutor<4> m_Eyfield_parser; + amrex::ParserExecutor<4> m_Ezfield_parser; + + std::unique_ptr h_inj_rho; + InjectorDensity* d_inj_rho = nullptr; + std::unique_ptr density_parser; + + std::unique_ptr h_inj_mom; + InjectorMomentum* d_inj_mom = nullptr; + std::unique_ptr ux_parser; + std::unique_ptr uy_parser; + std::unique_ptr uz_parser; + + // Keep a pointer to TemperatureProperties to ensure the lifetime of the + // contained Parser + std::unique_ptr h_mom_temp; + std::unique_ptr h_mom_vel; + +public: + + // MultiFabs that contain the density (N) and momentum density (NU) of this fluid species, for each refinement level + amrex::Vector< std::unique_ptr > N; + amrex::Vector, 3 > > NU; + +}; + +#endif diff --git a/Source/Fluids/WarpXFluidContainer.cpp b/Source/Fluids/WarpXFluidContainer.cpp new file mode 100644 index 00000000000..758e353208e --- /dev/null +++ b/Source/Fluids/WarpXFluidContainer.cpp @@ -0,0 +1,1353 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#include "ablastr/coarsen/sample.H" +#include "Particles/Pusher/UpdateMomentumHigueraCary.H" +#include "Utils/WarpXProfilerWrapper.H" + +#include "MusclHancockUtils.H" +#include "Fluids/WarpXFluidContainer.H" +#include "WarpX.H" +#include +#include "Utils/Parser/ParserUtils.H" +#include "Utils/WarpXUtil.H" +#include "Utils/SpeciesUtils.H" + +using namespace ablastr::utils::communication; +using namespace amrex; + +WarpXFluidContainer::WarpXFluidContainer(int nlevs_max, int ispecies, const std::string &name) +{ + species_id = ispecies; + species_name = name; + + ReadParameters(); + + // Initialize injection objects + const ParmParse pp_species_name(species_name); + SpeciesUtils::parseDensity(species_name, h_inj_rho, density_parser); + SpeciesUtils::parseMomentum(species_name, "none", h_inj_mom, + ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); + if (h_inj_rho) { +#ifdef AMREX_USE_GPU + d_inj_rho = static_cast + (amrex::The_Arena()->alloc(sizeof(InjectorDensity))); + amrex::Gpu::htod_memcpy_async(d_inj_rho, h_inj_rho.get(), sizeof(InjectorDensity)); +#else + d_inj_rho = h_inj_rho.get(); +#endif + } + if (h_inj_mom) { +#ifdef AMREX_USE_GPU + d_inj_mom = static_cast + (amrex::The_Arena()->alloc(sizeof(InjectorMomentum))); + amrex::Gpu::htod_memcpy_async(d_inj_mom, h_inj_mom.get(), sizeof(InjectorMomentum)); +#else + d_inj_mom = h_inj_mom.get(); +#endif + } + amrex::Gpu::synchronize(); + + // Resize the list of MultiFabs for the right number of levels + N.resize(nlevs_max); + NU.resize(nlevs_max); +} + +void WarpXFluidContainer::ReadParameters() +{ + + // Extract charge, mass, species type + std::string injection_style = "none"; + SpeciesUtils::extractSpeciesProperties(species_name, injection_style, charge, mass, physical_species); + + const ParmParse pp_species_name(species_name); + pp_species_name.query("do_not_deposit", do_not_deposit); + pp_species_name.query("do_not_gather", do_not_gather); + pp_species_name.query("do_not_push", do_not_push); + + // default values of E_external and B_external + // are used to set the E and B field when "constant" or "parser" + // is not explicitly used in the input + pp_species_name.query("B_ext_init_style", m_B_ext_s); + std::transform(m_B_ext_s.begin(), + m_B_ext_s.end(), + m_B_ext_s.begin(), + ::tolower); + pp_species_name.query("E_ext_init_style", m_E_ext_s); + std::transform(m_E_ext_s.begin(), + m_E_ext_s.end(), + m_E_ext_s.begin(), + ::tolower); + + // Parse external fields: + // if the input string for B_ext_s is + // "parse_b_ext_function" then the mathematical expression + // for the Bx_, By_, Bz_external_function(x,y,z) + // must be provided in the input file. + if (m_B_ext_s == "parse_b_ext_function") { + // store the mathematical expression as string + std::string str_Bx_ext_function; + std::string str_By_ext_function; + std::string str_Bz_ext_function; + utils::parser::Store_parserString( + pp_species_name, "Bx_external_function(x,y,z,t)", + str_Bx_ext_function); + utils::parser::Store_parserString( + pp_species_name, "By_external_function(x,y,z,t)", + str_By_ext_function); + utils::parser::Store_parserString( + pp_species_name, "Bz_external_function(x,y,z,t)", + str_Bz_ext_function); + + // Parser for B_external on the fluid + m_Bx_parser = std::make_unique( + utils::parser::makeParser(str_Bx_ext_function,{"x","y","z","t"})); + m_By_parser = std::make_unique( + utils::parser::makeParser(str_By_ext_function,{"x","y","z","t"})); + m_Bz_parser = std::make_unique( + utils::parser::makeParser(str_Bz_ext_function,{"x","y","z","t"})); + + } + + // if the input string for E_ext_s is + // "parse_e_ext_function" then the mathematical expression + // for the Ex_, Ey_, Ez_external_function(x,y,z) + // must be provided in the input file. + if (m_E_ext_s == "parse_e_ext_function") { + // store the mathematical expression as string + std::string str_Ex_ext_function; + std::string str_Ey_ext_function; + std::string str_Ez_ext_function; + utils::parser::Store_parserString( + pp_species_name, "Ex_external_function(x,y,z,t)", + str_Ex_ext_function); + utils::parser::Store_parserString( + pp_species_name, "Ey_external_function(x,y,z,t)", + str_Ey_ext_function); + utils::parser::Store_parserString( + pp_species_name, "Ez_external_function(x,y,z,t)", + str_Ez_ext_function); + // Parser for E_external on the fluid + m_Ex_parser = std::make_unique( + utils::parser::makeParser(str_Ex_ext_function,{"x","y","z","t"})); + m_Ey_parser = std::make_unique( + utils::parser::makeParser(str_Ey_ext_function,{"x","y","z","t"})); + m_Ez_parser = std::make_unique( + utils::parser::makeParser(str_Ez_ext_function,{"x","y","z","t"})); + } +} + +void WarpXFluidContainer::AllocateLevelMFs(int lev, const BoxArray &ba, const DistributionMapping &dm) +{ + int ncomps = 1; + const amrex::IntVect nguards(AMREX_D_DECL(2, 2, 2)); + + // set human-readable tag for each MultiFab + auto const tag = [lev](std::string tagname) + { + tagname.append("[l=").append(std::to_string(lev)).append("]"); + return tagname; + }; + + WarpX::AllocInitMultiFab(N[lev], amrex::convert(ba, amrex::IntVect::TheNodeVector()), + dm, ncomps, nguards, lev, tag("fluid density"), 0.0_rt); + + WarpX::AllocInitMultiFab(NU[lev][0], amrex::convert(ba, amrex::IntVect::TheNodeVector()), + dm, ncomps, nguards, lev, tag("fluid momentum density [x]"), 0.0_rt); + WarpX::AllocInitMultiFab(NU[lev][1], amrex::convert(ba, amrex::IntVect::TheNodeVector()), + dm, ncomps, nguards, lev, tag("fluid momentum density [y]"), 0.0_rt); + WarpX::AllocInitMultiFab(NU[lev][2], amrex::convert(ba, amrex::IntVect::TheNodeVector()), + dm, ncomps, nguards, lev, tag("fluid momentum density [z]"), 0.0_rt); +} + +void WarpXFluidContainer::InitData(int lev, amrex::Box init_box, amrex::Real cur_time) +{ + WARPX_PROFILE("WarpXFluidContainer::InitData"); + + // Convert initialization box to nodal box + init_box.surroundingNodes(); + + // Create local copies of pointers for GPU kernels + InjectorDensity* inj_rho = d_inj_rho; + InjectorMomentum* inj_mom = d_inj_mom; + + // Extract grid geometry properties + WarpX &warpx = WarpX::GetInstance(); + const amrex::Geometry &geom = warpx.Geom(lev); + const auto dx = geom.CellSizeArray(); + const auto problo = geom.ProbLoArray(); + const amrex::Real clight = PhysConst::c; + const amrex::Real gamma_boost = WarpX::gamma_boost; + const amrex::Real beta_boost = WarpX::beta_boost; + + // Loop through cells and initialize their value +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + + amrex::Box tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + amrex::Array4 const &N_arr = N[lev]->array(mfi); + amrex::Array4 const &NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 const &NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 const &NUz_arr = NU[lev][2]->array(mfi); + + // Return the intersection of all cells and the ones we wish to update + amrex::Box init_box_intersection = init_box & tile_box; + + amrex::ParallelFor(init_box_intersection, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { +#if defined(WARPX_DIM_3D) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = problo[1] + j * dx[1]; + amrex::Real z = problo[2] + k * dx[2]; +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[1] + j * dx[1]; +#else + amrex::Real x = 0.0_rt; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[0] + i * dx[0]; +#endif + + // Lorentz transform z (from boosted to lab frame) + if (gamma_boost > 1._rt){ + z = gamma_boost*(z + beta_boost*clight*cur_time); + } + + amrex::Real n = inj_rho->getDensity(x, y, z); + amrex::XDim3 u = inj_mom->getBulkMomentum(x, y, z); + + // Give u the right dimensions of m/s + u.x = u.x * clight; + u.y = u.y * clight; + u.z = u.z * clight; + + // Check if n > 0 and if not, don't compute the boost + // Lorentz transform n, u (from lab to boosted frame) + if (n > 0.0){ + if (gamma_boost > 1._rt){ + amrex::Real gamma = sqrt(1.0 + (u.x*u.x + u.y*u.y + u.z*u.z)/(clight*clight)); + amrex::Real n_boosted = gamma_boost*n*( 1.0 - beta_boost*u.z/(gamma*clight) ); + amrex::Real uz_boosted = gamma_boost*(u.z - beta_boost*clight*gamma); + u.z = uz_boosted; + n = n_boosted; + } + } + + // Multiply by clight so u is back in SI units + N_arr(i, j, k) = n; + NUx_arr(i, j, k) = n * u.x; + NUy_arr(i, j, k) = n * u.y; + NUz_arr(i, j, k) = n * u.z; + + } + ); + } +} + + +void WarpXFluidContainer::Evolve( + int lev, + const amrex::MultiFab &Ex, const amrex::MultiFab &Ey, const amrex::MultiFab &Ez, + const amrex::MultiFab &Bx, const amrex::MultiFab &By, const amrex::MultiFab &Bz, + amrex::MultiFab* rho, amrex::MultiFab &jx, amrex::MultiFab &jy, amrex::MultiFab &jz, + amrex::Real cur_time, bool skip_deposition) +{ + + WARPX_PROFILE("WarpXFluidContainer::Evolve"); + + if (rho && ! skip_deposition && ! do_not_deposit) { + // Deposit charge before particle push, in component 0 of MultiFab rho. + DepositCharge(lev, *rho, 0); + } + + // Step the Lorentz Term + if(!do_not_gather){ + GatherAndPush(lev, Ex, Ey, Ez, Bx, By, Bz, cur_time); + } + + // Cylindrical centrifugal term + if(!do_not_push){ +#if defined(WARPX_DIM_RZ) + centrifugal_source_rz(lev); +#endif + + // Apply (non-periodic) BC on the fluids (needed for spatial derivative), + // and communicate N, NU at boundaries + ApplyBcFluidsAndComms(lev); + + // Step the Advective term + AdvectivePush_Muscl(lev); + } + + // Deposit rho to the simulation mesh + // Deposit charge (end of the step) + if (rho && ! skip_deposition && ! do_not_deposit) { + DepositCharge(lev, *rho, 1); + } + + // Deposit J to the simulation mesh + if (!skip_deposition && ! do_not_deposit) { + DepositCurrent(lev, jx, jy, jz); + } +} + +// Momentum source due to curvature +void WarpXFluidContainer::ApplyBcFluidsAndComms (int lev) +{ + WARPX_PROFILE("WarpXFluidContainer::ApplyBcFluidsAndComms"); + + WarpX &warpx = WarpX::GetInstance(); + const amrex::Geometry &geom = warpx.Geom(lev); + const amrex::Periodicity &period = geom.periodicity(); + const Array periodic_directions = geom.isPeriodic(); + amrex::Box domain = geom.Domain(); + // Convert to nodal box + domain.surroundingNodes(); + + // H&C push the momentum +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + + amrex::Box tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + + amrex::Array4 N_arr = N[lev]->array(mfi); + amrex::Array4 NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 NUz_arr = NU[lev][2]->array(mfi); + + //Grow the tilebox + tile_box.grow(1); + + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + + // If the cell is is first gaurd cell & the dimension is non + // periodic, then copy Q_{i+1} = Q_{i-1}. + // Don't check r-dir in Z: +#if defined(WARPX_DIM_3D) + + // Upper end (index 2) + if ( (periodic_directions[2] != 1) && (k==domain.bigEnd(2)+1) ){ + N_arr(i,j,k) = N_arr(i,j,k-2); + NUx_arr(i,j,k) = NUx_arr(i,j,k-2); + NUy_arr(i,j,k) = NUy_arr(i,j,k-2); + NUz_arr(i,j,k) = NUz_arr(i,j,k-2); + + // Lower end (index 2) + } else if ( (periodic_directions[2] != 1) && (k==domain.smallEnd(2)-1) ) { + N_arr(i,j,k) = N_arr(i,j,k+2); + NUx_arr(i,j,k) = NUx_arr(i,j,k+2); + NUy_arr(i,j,k) = NUy_arr(i,j,k+2); + NUz_arr(i,j,k) = NUz_arr(i,j,k+2); + } + +#elif ( defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_3D) ) + + // Upper end (index 1) + if ( (periodic_directions[1] != 1) && (j==domain.bigEnd(1)+1) ){ + N_arr(i,j,k) = N_arr(i,j-2,k); + NUx_arr(i,j,k) = NUx_arr(i,j-2,k); + NUy_arr(i,j,k) = NUy_arr(i,j-2,k); + NUz_arr(i,j,k) = NUz_arr(i,j-2,k); + + // Lower end (index 1`) + } else if ( (periodic_directions[1] != 1) && (j==domain.smallEnd(1)-1) ) { + N_arr(i,j,k) = N_arr(i,j+2,k); + NUx_arr(i,j,k) = NUx_arr(i,j+2,k); + NUy_arr(i,j,k) = NUy_arr(i,j+2,k); + NUz_arr(i,j,k) = NUz_arr(i,j+2,k); + + } + +#elif ( defined(WARPX_DIM_1D_Z) || defined(WARPX_DIM_XZ) || defined(WARPX_DIM_3D) ) + + // Upper end (index 0) + if ( (periodic_directions[0] != 1) && (i==domain.bigEnd(0)+1) ){ + N_arr(i,j,k) = N_arr(i-2,j,k); + NUx_arr(i,j,k) = NUx_arr(i-2,j,k); + NUy_arr(i,j,k) = NUy_arr(i-2,j,k); + NUz_arr(i,j,k) = NUz_arr(i-2,j,k); + + // Lower end (index 0) + } else if ( (periodic_directions[0] != 1) && (i==domain.smallEnd(0)-1) ) { + N_arr(i,j,k) = N_arr(i+2,j,k); + NUx_arr(i,j,k) = NUx_arr(i+2,j,k); + NUy_arr(i,j,k) = NUy_arr(i+2,j,k); + NUz_arr(i,j,k) = NUz_arr(i+2,j,k); + } + +#else + +#endif + } + ); + } + + // Fill guard cells + FillBoundary(*N[lev], N[lev]->nGrowVect(), WarpX::do_single_precision_comms, period); + FillBoundary(*NU[lev][0], NU[lev][0]->nGrowVect(), WarpX::do_single_precision_comms, period); + FillBoundary(*NU[lev][1], NU[lev][1]->nGrowVect(), WarpX::do_single_precision_comms, period); + FillBoundary(*NU[lev][2], NU[lev][2]->nGrowVect(), WarpX::do_single_precision_comms, period); +} + +// Muscl Advection Update +void WarpXFluidContainer::AdvectivePush_Muscl (int lev) +{ + WARPX_PROFILE("WarpXFluidContainer::AdvectivePush_Muscl"); + + // Grab the grid spacing + WarpX &warpx = WarpX::GetInstance(); + const Real dt = warpx.getdt(lev); + const amrex::Geometry &geom = warpx.Geom(lev); + const auto dx = geom.CellSizeArray(); + const amrex::Real clight = PhysConst::c; +#if defined(WARPX_DIM_3D) + amrex::Real dt_over_dx = (dt/dx[0]); + amrex::Real dt_over_dy = (dt/dx[1]); + amrex::Real dt_over_dz = (dt/dx[2]); + amrex::Real dt_over_dx_half = 0.5*(dt/dx[0]); + amrex::Real dt_over_dy_half = 0.5*(dt/dx[1]); + amrex::Real dt_over_dz_half = 0.5*(dt/dx[2]); +#elif defined(WARPX_DIM_XZ) + amrex::Real dt_over_dx_half = 0.5*(dt/dx[0]); + amrex::Real dt_over_dz_half = 0.5*(dt/dx[1]); + amrex::Real dt_over_dx = (dt/dx[0]); + amrex::Real dt_over_dz = (dt/dx[1]); +#elif defined(WARPX_DIM_RZ) + const auto problo = geom.ProbLoArray(); + amrex::Real dt_over_dx_half = 0.5*(dt/dx[0]); + amrex::Real dt_over_dz_half = 0.5*(dt/dx[1]); + amrex::Box const& domain = geom.Domain(); +#else + amrex::Real dt_over_dz = (dt/dx[0]); + amrex::Real dt_over_dz_half = 0.5*(dt/dx[0]); +#endif + + amrex::BoxArray ba = N[lev]->boxArray(); + + // Temporary Half-step values +#if defined(WARPX_DIM_3D) + amrex::MultiFab tmp_U_minus_x( amrex::convert(ba, IntVect(0,1,1)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_plus_x( amrex::convert(ba, IntVect(0,1,1)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_minus_y( amrex::convert(ba, IntVect(1,0,1)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_plus_y( amrex::convert(ba, IntVect(1,0,1)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_minus_z( amrex::convert(ba, IntVect(1,1,0)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_plus_z( amrex::convert(ba, IntVect(1,1,0)), N[lev]->DistributionMap(), 4, 1); +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::MultiFab tmp_U_minus_x( amrex::convert(ba, IntVect(0,1)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_plus_x( amrex::convert(ba, IntVect(0,1)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_minus_z( amrex::convert(ba, IntVect(1,0)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_plus_z( amrex::convert(ba, IntVect(1,0)), N[lev]->DistributionMap(), 4, 1); +#else + amrex::MultiFab tmp_U_minus_z( amrex::convert(ba, IntVect(0)), N[lev]->DistributionMap(), 4, 1); + amrex::MultiFab tmp_U_plus_z( amrex::convert(ba, IntVect(0)), N[lev]->DistributionMap(), 4, 1); +#endif + + // Fill edge values of N and U at the half timestep for MUSCL +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + + // Loop over a box with one extra gridpoint in the ghost region to avoid + // an extra MPI communication between the edge value computation loop and + // the flux calculation loop + amrex::Box tile_box = mfi.growntilebox(1); + + // Limit the grown box for RZ at r = 0, r_max +#if defined (WARPX_DIM_RZ) + const int idir = 0; + const int n_cell = -1; + tile_box.growLo(idir, n_cell); + tile_box.growHi(idir, n_cell); +#endif + + amrex::Array4 const &N_arr = N[lev]->array(mfi); + amrex::Array4 const &NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 const &NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 const &NUz_arr = NU[lev][2]->array(mfi); + + // Boxes are computed to avoid going out of bounds. + // Grow the entire domain + amrex::Box box = mfi.validbox(); + box.grow(1); +#if defined(WARPX_DIM_3D) + amrex::Box const box_x = amrex::convert( box, tmp_U_minus_x.ixType() ); + amrex::Box const box_y = amrex::convert( box, tmp_U_minus_y.ixType() ); + amrex::Box const box_z = amrex::convert( box, tmp_U_minus_z.ixType() ); +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Box const box_x = amrex::convert( box, tmp_U_minus_x.ixType() ); + amrex::Box const box_z = amrex::convert( box, tmp_U_minus_z.ixType() ); +#else + amrex::Box const box_z = amrex::convert( box, tmp_U_minus_z.ixType() ); +#endif + + //N and NU are always defined at the nodes, the tmp_Q_* are defined + //inbetween the nodes (i.e. on the staggered Yee grid) and store the + //values of N and U at these points. + //(i.e. the 4 components correspond to N + the 3 components of U) + // Extract the temporary arrays for edge values +#if defined(WARPX_DIM_3D) + amrex::Array4 U_minus_x = tmp_U_minus_x.array(mfi); + amrex::Array4 U_plus_x = tmp_U_plus_x.array(mfi); + amrex::Array4 U_minus_y = tmp_U_minus_y.array(mfi); + amrex::Array4 U_plus_y = tmp_U_plus_y.array(mfi); + amrex::Array4 U_minus_z = tmp_U_minus_z.array(mfi); + amrex::Array4 U_plus_z = tmp_U_plus_z.array(mfi); +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Array4 U_minus_x = tmp_U_minus_x.array(mfi); + amrex::Array4 U_plus_x = tmp_U_plus_x.array(mfi); + amrex::Array4 U_minus_z = tmp_U_minus_z.array(mfi); + amrex::Array4 U_plus_z = tmp_U_plus_z.array(mfi); +#else + amrex::Array4 U_minus_z = tmp_U_minus_z.array(mfi); + amrex::Array4 U_plus_z = tmp_U_plus_z.array(mfi); +#endif + + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + + // Density positivity check (Makes the algorithm safe from divide by zeros) + if( N_arr(i,j,k) > 0.0){ + + // - Grab local Uz Uy Ux gamma + // Isolate U from NU + amrex::Real Ux = (NUx_arr(i, j, k) / N_arr(i,j,k)); + amrex::Real Uy = (NUy_arr(i, j, k) / N_arr(i,j,k)); + amrex::Real Uz = (NUz_arr(i, j, k) / N_arr(i,j,k)); + + // Compute useful quantities for J + amrex::Real c_sq = clight*clight; + amrex::Real gamma = sqrt(1.0 + (Ux*Ux + Uy*Uy + Uz*Uz)/(c_sq) ); + amrex::Real inv_c2_gamma3 = 1./(c_sq*gamma*gamma*gamma); + + // J represents are 4x4 matrices that show up in the advection + // equations written as a function of U = {N, Ux, Uy, Uz}: + // \partial_t U + Jx \partial_x U + Jy \partial_y U + Jz \partial_z U = 0 +#if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + amrex::Real Vx = Ux/gamma; + // Compute the non-zero element of Jx + amrex::Real J00x = Vx; + amrex::Real J01x = N_arr(i,j,k)*(1/gamma)*(1-Vx*Vx/c_sq); + amrex::Real J02x = -N_arr(i,j,k)*Uy*Ux*inv_c2_gamma3; + amrex::Real J03x = -N_arr(i,j,k)*Uz*Ux*inv_c2_gamma3; + amrex::Real J11x = Vx; + amrex::Real J22x = Vx; + amrex::Real J33x = Vx; + + // Compute the cell slopes x + amrex::Real dU0x = ave( DownDx_N(N_arr,i,j,k), UpDx_N(N_arr,i,j,k) ); + amrex::Real dU1x = ave( DownDx_U(N_arr,NUx_arr,Ux,i,j,k), UpDx_U(N_arr,NUx_arr,Ux,i,j,k) ); + amrex::Real dU2x = ave( DownDx_U(N_arr,NUy_arr,Uy,i,j,k), UpDx_U(N_arr,NUy_arr,Uy,i,j,k) ); + amrex::Real dU3x = ave( DownDx_U(N_arr,NUz_arr,Uz,i,j,k), UpDx_U(N_arr,NUz_arr,Uz,i,j,k) ); + +#endif + +#if defined(WARPX_DIM_3D) + amrex::Real Vy = Uy/gamma; + // Compute the non-zero element of Jy + amrex::Real J00y = Vy; + amrex::Real J01y = -N_arr(i,j,k)*Ux*Uy*inv_c2_gamma3; + amrex::Real J02y = N_arr(i,j,k)*(1/gamma)*(1-Vy*Vy/c_sq); + amrex::Real J03y = -N_arr(i,j,k)*Uz*Uy*inv_c2_gamma3; + amrex::Real J11y = Vy; + amrex::Real J22y = Vy; + amrex::Real J33y = Vy; + + // Compute the cell slopes y + amrex::Real dU0y = ave( DownDy_N(N_arr,i,j,k), UpDy_N(N_arr,i,j,k) ); + amrex::Real dU1y = ave( DownDy_U(N_arr,NUx_arr,Ux,i,j,k), UpDy_U(N_arr,NUx_arr,Ux,i,j,k) ); + amrex::Real dU2y = ave( DownDy_U(N_arr,NUy_arr,Uy,i,j,k), UpDy_U(N_arr,NUy_arr,Uy,i,j,k) ); + amrex::Real dU3y = ave( DownDy_U(N_arr,NUz_arr,Uz,i,j,k), UpDy_U(N_arr,NUz_arr,Uz,i,j,k) ); + +#endif + amrex::Real Vz = Uz/gamma; + // Compute the non-zero element of Jz + amrex::Real J00z = Vz; + amrex::Real J01z = -N_arr(i,j,k)*Ux*Uz*inv_c2_gamma3; + amrex::Real J02z = -N_arr(i,j,k)*Uy*Uz*inv_c2_gamma3; + amrex::Real J03z = N_arr(i,j,k)*(1/gamma)*(1-Vz*Vz/c_sq); + amrex::Real J11z = Vz; + amrex::Real J22z = Vz; + amrex::Real J33z = Vz; + + // Compute the cell slopes z + amrex::Real dU0z = ave( DownDz_N(N_arr,i,j,k), UpDz_N(N_arr,i,j,k) ); + amrex::Real dU1z = ave( DownDz_U(N_arr,NUx_arr,Ux,i,j,k), UpDz_U(N_arr,NUx_arr,Ux,i,j,k) ); + amrex::Real dU2z = ave( DownDz_U(N_arr,NUy_arr,Uy,i,j,k), UpDz_U(N_arr,NUy_arr,Uy,i,j,k) ); + amrex::Real dU3z = ave( DownDz_U(N_arr,NUz_arr,Uz,i,j,k), UpDz_U(N_arr,NUz_arr,Uz,i,j,k) ); + + + // Select the specific implementation depending on dimensionality +#if defined(WARPX_DIM_3D) + + // Compute U ([ N, U]) at the halfsteps (U_tidle) using the slopes (dU) + amrex::Real JdU0x = J00x*dU0x + J01x*dU1x + J02x*dU2x + J03x*dU3x; + amrex::Real JdU1x = J11x*dU1x ; + amrex::Real JdU2x = J22x*dU2x ; + amrex::Real JdU3x = J33x*dU3x; + amrex::Real JdU0y = J00y*dU0y + J01y*dU1y + J02y*dU2y + J03y*dU3y; + amrex::Real JdU1y = J11y*dU1y; + amrex::Real JdU2y = J22y*dU2y; + amrex::Real JdU3y = J33y*dU3y; + amrex::Real JdU0z = J00z*dU0z + J01z*dU1z + J02z*dU2z + J03z*dU3z; + amrex::Real JdU1z = J11z*dU1z; + amrex::Real JdU2z = J22z*dU2z; + amrex::Real JdU3z = J33z*dU3z; + amrex::Real U_tilde0 = N_arr(i,j,k) - dt_over_dx_half*JdU0x - dt_over_dy_half*JdU0y - dt_over_dz_half*JdU0z; + amrex::Real U_tilde1 = Ux - dt_over_dx_half*JdU1x - dt_over_dy_half*JdU1y - dt_over_dz_half*JdU1z; + amrex::Real U_tilde2 = Uy - dt_over_dx_half*JdU2x - dt_over_dy_half*JdU2y - dt_over_dz_half*JdU2z; + amrex::Real U_tilde3 = Uz - dt_over_dx_half*JdU3x - dt_over_dy_half*JdU3y - dt_over_dz_half*JdU3z; + + + // Predict U at the cell edges (x) + compute_U_edges(U_minus_x, U_plus_x, i, j, k, box_x, U_tilde0, U_tilde1, U_tilde2, U_tilde3, dU0x, dU1x, dU2x, dU3x,0); + + // Positivity Limiter for density N, if N_edge < 0, + // then set the slope (dU) to to zero in that cell/direction + positivity_limiter (U_plus_x, U_minus_x, N_arr, i, j, k, box_x, Ux, Uy, Uz, 0); + + // Predict U at the cell edges (y) + compute_U_edges(U_minus_y, U_plus_y, i, j, k, box_y, U_tilde0, U_tilde1, U_tilde2, U_tilde3, dU0y, dU1y, dU2y, dU3y,1); + + // Positivity Limiter for density N, if N_edge < 0, + // then set the slope (dU) to to zero in that cell/direction + positivity_limiter (U_plus_y, U_minus_y, N_arr, i, j, k, box_y, Ux, Uy, Uz, 1); + + // Predict U at the cell edges (z) + compute_U_edges(U_minus_z, U_plus_z, i, j, k, box_z, U_tilde0, U_tilde1, U_tilde2, U_tilde3, dU0z, dU1z, dU2z, dU3z,2); + + // Positivity Limiter for density N, if N_edge < 0, + // then set the slope (dU) to to zero in that cell/direction + positivity_limiter (U_plus_z, U_minus_z, N_arr, i, j, k, box_z, Ux, Uy, Uz, 2); + +#elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + + // Have no RZ-intertial source for primative vars if in XZ + amrex::Real N_source = 0.0; + +#if defined(WARPX_DIM_RZ) + amrex::Real dr = dx[0]; + amrex::Real r = problo[0] + i * dr; + // Impose "none" boundaries + // Condition: dUx = 0 at r = 0 + if (i == domain.smallEnd(0)) { + // R|_{0+} -> L|_{0-} + // N -> N (N_arr(i-1,j,k) -> N_arr(i+1,j,k)) + // NUr -> -NUr (NUx_arr(i-1,j,k) -> -NUx_arr(i+1,j,k)) + // NUt -> -NUt (NUy_arr(i-1,j,k) -> -NUy_arr(i+1,j,k)) + // NUz -> -NUz (NUz_arr(i-1,j,k) -> NUz_arr(i+1,j,k)) + dU0x = ave( -UpDx_N(N_arr,i,j,k) , UpDx_N(N_arr,i,j,k) ); + // First term in the ave is: U_{x,y} + U_{x,y}_p, + // which can be writen as 2*U_{x,y} + UpDx_U(U_{x,y}) + dU1x = ave( 2.0*Ux + UpDx_U(N_arr,NUx_arr,Ux,i,j,k) , UpDx_U(N_arr,NUx_arr,Ux,i,j,k) ); + dU2x = ave( 2.0*Uy + UpDx_U(N_arr,NUy_arr,Uy,i,j,k) , UpDx_U(N_arr,NUy_arr,Uy,i,j,k) ); + dU3x = ave( -UpDx_U(N_arr,NUz_arr,Uz,i,j,k) , UpDx_U(N_arr,NUz_arr,Uz,i,j,k) ); + } else if (i == domain.bigEnd(0)+1) { + dU0x = ave( DownDx_N(N_arr,i,j,k) , 0.0 ); + dU1x = ave( DownDx_U(N_arr,NUx_arr,Ux,i,j,k) , 0.0 ); + dU2x = ave( DownDx_U(N_arr,NUy_arr,Uy,i,j,k) , 0.0 ); + dU3x = ave( DownDx_U(N_arr,NUz_arr,Uz,i,j,k) , 0.0 ); + } + + // RZ sources: + if (i != domain.smallEnd(0)) { + N_source = N_arr(i,j,k)*Vx/r; + } +#endif + + // Compute U ([ N, U]) at the halfsteps (U_tidle) using the slopes (dU) + amrex::Real JdU0x = J00x*dU0x + J01x*dU1x + J02x*dU2x + J03x*dU3x; + amrex::Real JdU1x = J11x*dU1x; + amrex::Real JdU2x = J22x*dU2x; + amrex::Real JdU3x = J33x*dU3x; + amrex::Real JdU0z = J00z*dU0z + J01z*dU1z + J02z*dU2z + J03z*dU3z; + amrex::Real JdU1z = J11z*dU1z; + amrex::Real JdU2z = J22z*dU2z; + amrex::Real JdU3z = J33z*dU3z; + amrex::Real U_tilde0 = N_arr(i,j,k) - dt_over_dx_half*JdU0x - dt_over_dz_half*JdU0z - (dt/2.0)*N_source; + amrex::Real U_tilde1 = Ux - dt_over_dx_half*JdU1x - dt_over_dz_half*JdU1z; + amrex::Real U_tilde2 = Uy - dt_over_dx_half*JdU2x - dt_over_dz_half*JdU2z; + amrex::Real U_tilde3 = Uz - dt_over_dx_half*JdU3x - dt_over_dz_half*JdU3z; + + // Predict U at the cell edges (x) + compute_U_edges(U_minus_x, U_plus_x, i, j, k, box_x, U_tilde0, U_tilde1, U_tilde2, U_tilde3, dU0x, dU1x, dU2x, dU3x,0); + + // Positivity Limiter for density N, if N_edge < 0, + // then set the slope (dU) to to zero in that cell/direction + positivity_limiter (U_plus_x, U_minus_x, N_arr, i, j, k, box_x, Ux, Uy, Uz, 0); + + // Predict U at the cell edges (z) + compute_U_edges(U_minus_z, U_plus_z, i, j, k, box_z, U_tilde0, U_tilde1, U_tilde2, U_tilde3, dU0z, dU1z, dU2z, dU3z,2); + + // Positivity Limiter for density N, if N_edge < 0, + // then set the slope (dU) to to zero in that cell/direction + positivity_limiter (U_plus_z, U_minus_z, N_arr, i, j, k, box_z, Ux, Uy, Uz, 2); + +#else + + // Compute U ([ N, U]) at the halfsteps (U_tidle) using the slopes (dU) + amrex::Real JdU0z = J00z*dU0z + J01z*dU1z + J02z*dU2z + J03z*dU3z; + amrex::Real JdU1z = J11z*dU1z; + amrex::Real JdU2z = J22z*dU2z; + amrex::Real JdU3z = J33z*dU3z; + amrex::Real U_tilde0 = N_arr(i,j,k) - dt_over_dz_half*JdU0z; + amrex::Real U_tilde1 = Ux - dt_over_dz_half*JdU1z; + amrex::Real U_tilde2 = Uy - dt_over_dz_half*JdU2z; + amrex::Real U_tilde3 = Uz - dt_over_dz_half*JdU3z; + + // Predict U at the cell edges (z) + compute_U_edges(U_minus_z, U_plus_z, i, j, k, box_z, U_tilde0, U_tilde1, U_tilde2, U_tilde3, dU0z, dU1z, dU2z, dU3z,2); + + // Positivity Limiter for density N, if N_edge < 0, + // then set the slope (dU) to to zero in that cell/direction + positivity_limiter (U_plus_z, U_minus_z, N_arr, i, j, k, box_z, Ux, Uy, Uz, 2); + +#endif + // If N<= 0 then set the edge values (U_minus/U_plus) to zero + } else { +#if defined(WARPX_DIM_3D) + set_U_edges_to_zero(U_minus_x, U_plus_x, i, j, k, box_x, 0); + set_U_edges_to_zero(U_minus_y, U_plus_y, i, j, k, box_y, 1); + set_U_edges_to_zero(U_minus_z, U_plus_z, i, j, k, box_z, 2); +#elif defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) + set_U_edges_to_zero(U_minus_x, U_plus_x, i, j, k, box_x, 0); + set_U_edges_to_zero(U_minus_z, U_plus_z, i, j, k, box_z, 2); +#else + set_U_edges_to_zero(U_minus_z, U_plus_z, i, j, k, box_z, 2); +#endif + } + } + ); + } + + // Given the values of `U_minus` and `U_plus`, compute fluxes inbetween nodes, and update N, NU accordingly +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + amrex::Box tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + amrex::Array4 N_arr = N[lev]->array(mfi); + amrex::Array4 NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 NUz_arr = NU[lev][2]->array(mfi); + +#if defined(WARPX_DIM_3D) + amrex::Array4 const &U_minus_x = tmp_U_minus_x.array(mfi); + amrex::Array4 const &U_plus_x = tmp_U_plus_x.array(mfi); + amrex::Array4 const &U_minus_y = tmp_U_minus_y.array(mfi); + amrex::Array4 const &U_plus_y = tmp_U_plus_y.array(mfi); + amrex::Array4 const &U_minus_z = tmp_U_minus_z.array(mfi); + amrex::Array4 const &U_plus_z = tmp_U_plus_z.array(mfi); +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Array4 const &U_minus_x = tmp_U_minus_x.array(mfi); + amrex::Array4 const &U_plus_x = tmp_U_plus_x.array(mfi); + amrex::Array4 const &U_minus_z = tmp_U_minus_z.array(mfi); + amrex::Array4 const &U_plus_z = tmp_U_plus_z.array(mfi); +#else + amrex::Array4 const &U_minus_z = tmp_U_minus_z.array(mfi); + amrex::Array4 const &U_plus_z = tmp_U_plus_z.array(mfi); +#endif + + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + + // Select the specific implementation depending on dimensionality +#if defined(WARPX_DIM_3D) + + // Update the conserved variables Q = [N, NU] from tn -> tn + dt + N_arr(i,j,k) = N_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,0,0) + - dt_over_dy*dF(U_minus_y,U_plus_y,i,j,k,clight,0,1) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,0,2); + NUx_arr(i,j,k) = NUx_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,1,0) + - dt_over_dy*dF(U_minus_y,U_plus_y,i,j,k,clight,1,1) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,1,2); + NUy_arr(i,j,k) = NUy_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,2,0) + - dt_over_dy*dF(U_minus_y,U_plus_y,i,j,k,clight,2,1) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,2,2); + NUz_arr(i,j,k) = NUz_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,3,0) + - dt_over_dy*dF(U_minus_y,U_plus_y,i,j,k,clight,3,1) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,3,2); + +#elif defined(WARPX_DIM_XZ) + + // Update the conserved variables Q = [N, NU] from tn -> tn + dt + N_arr(i,j,k) = N_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,0,0) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,0,2); + NUx_arr(i,j,k) = NUx_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,1,0) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,1,2); + NUy_arr(i,j,k) = NUy_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,2,0) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,2,2); + NUz_arr(i,j,k) = NUz_arr(i,j,k) - dt_over_dx*dF(U_minus_x,U_plus_x,i,j,k,clight,3,0) + - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,3,2); + +#elif defined(WARPX_DIM_RZ) + + // Compute the flux areas for RZ + // Cell-centered radius + amrex::Real dr = dx[0]; + amrex::Real dz = dx[1]; + amrex::Real r = problo[0] + i * dr; + amrex::Real Vij = 0.0; + amrex::Real S_Az = 0.0; + + // Volume element and z-facing surfaces + if (i == domain.smallEnd(0)) { + Vij = 2.0*MathConst::pi*(dr/2.0)*(dr/4.0)*dz; + S_Az = 2.0*MathConst::pi*(dr/4.0)*(dr/2.0); + } else if (i == domain.bigEnd(0)+1) { + Vij = 2.0*MathConst::pi*(r - dr/4.0)*(dr/2.0)*dz; + S_Az = 2.0*MathConst::pi*(r - dr/4.0)*(dr/2.0); + } else { + Vij = 2.0*MathConst::pi*r*dr*dz; + S_Az = 2.0*MathConst::pi*(r)*dr; + } + + // Radial Surfaces + amrex::Real S_Ar_plus = 2.0*MathConst::pi*(r + dr/2.0)*dz; + amrex::Real S_Ar_minus = 2.0*MathConst::pi*(r - dr/2.0)*dz; + if (i == domain.smallEnd(0)) + S_Ar_minus = 0.0; + if (i == domain.bigEnd(0)+1) + S_Ar_plus = 2.0*MathConst::pi*(r)*dz; + + // Impose "none" boundaries + // Condition: Vx(r) = 0 at boundaries + amrex::Real Vx_I_minus = V_calc(U_minus_x,i,j,k,0,clight); + amrex::Real Vx_L_plus = V_calc(U_plus_x,i-1,j,k,0,clight); + + // compute the fluxes: + // (note that _plus is shifted due to grid location) + amrex::Real Vx_L_minus = 0.0, Vx_I_plus = 0.0; + amrex::Real F0_minusx = 0.0, F1_minusx = 0.0, F2_minusx = 0.0, F3_minusx = 0.0; + amrex::Real F0_plusx = 0.0, F1_plusx = 0.0, F2_plusx = 0.0, F3_plusx = 0.0; + if (i != domain.smallEnd(0)) { + Vx_L_minus = V_calc(U_minus_x,i-1,j,k,0,clight); + F0_minusx = flux_N( U_minus_x, U_plus_x, i-1, j, k, Vx_L_minus, Vx_L_plus)*S_Ar_minus; + F1_minusx = flux_NUx(U_minus_x, U_plus_x, i-1, j, k, Vx_L_minus, Vx_L_plus)*S_Ar_minus; + F2_minusx = flux_NUy(U_minus_x, U_plus_x, i-1, j, k, Vx_L_minus, Vx_L_plus)*S_Ar_minus; + F3_minusx = flux_NUz(U_minus_x, U_plus_x, i-1, j, k, Vx_L_minus, Vx_L_plus)*S_Ar_minus; + } + if (i < domain.bigEnd(0)) { + Vx_I_plus = V_calc(U_plus_x,i,j,k,0,clight); + F0_plusx = flux_N( U_minus_x, U_plus_x, i , j, k, Vx_I_minus, Vx_I_plus)*S_Ar_plus; + F1_plusx = flux_NUx(U_minus_x, U_plus_x, i , j, k, Vx_I_minus, Vx_I_plus)*S_Ar_plus; + F2_plusx = flux_NUy(U_minus_x, U_plus_x, i , j, k, Vx_I_minus, Vx_I_plus)*S_Ar_plus; + F3_plusx = flux_NUz(U_minus_x, U_plus_x, i , j, k, Vx_I_minus, Vx_I_plus)*S_Ar_plus; + } + + // Update the conserved variables from tn -> tn + dt + N_arr(i,j,k) = N_arr(i,j,k) - (dt/Vij)*(F0_plusx - F0_minusx + dF(U_minus_z,U_plus_z,i,j,k,clight,0,2)*S_Az); + NUx_arr(i,j,k) = NUx_arr(i,j,k) - (dt/Vij)*(F1_plusx - F1_minusx + dF(U_minus_z,U_plus_z,i,j,k,clight,1,2)*S_Az); + NUy_arr(i,j,k) = NUy_arr(i,j,k) - (dt/Vij)*(F2_plusx - F2_minusx + dF(U_minus_z,U_plus_z,i,j,k,clight,2,2)*S_Az); + NUz_arr(i,j,k) = NUz_arr(i,j,k) - (dt/Vij)*(F3_plusx - F3_minusx + dF(U_minus_z,U_plus_z,i,j,k,clight,3,2)*S_Az); + +#else + + // Update the conserved variables Q = [N, NU] from tn -> tn + dt + N_arr(i,j,k) = N_arr(i,j,k) - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,0,2); + NUx_arr(i,j,k) = NUx_arr(i,j,k) - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,1,2); + NUy_arr(i,j,k) = NUy_arr(i,j,k) - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,2,2); + NUz_arr(i,j,k) = NUz_arr(i,j,k) - dt_over_dz*dF(U_minus_z,U_plus_z,i,j,k,clight,3,2); +#endif + } + ); + } +} + + +// Momentum source due to curvature +#if defined(WARPX_DIM_RZ) +void WarpXFluidContainer::centrifugal_source_rz (int lev) +{ + WARPX_PROFILE("WarpXFluidContainer::centrifugal_source_rz"); + + WarpX &warpx = WarpX::GetInstance(); + const Real dt = warpx.getdt(lev); + const amrex::Geometry &geom = warpx.Geom(lev); + const auto dx = geom.CellSizeArray(); + const auto problo = geom.ProbLoArray(); + const amrex::Real clight = PhysConst::c; + amrex::Box const& domain = geom.Domain(); + + // H&C push the momentum +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + + amrex::Box const &tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + + amrex::Array4 const &N_arr = N[lev]->array(mfi); + amrex::Array4 NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 const &NUz_arr = NU[lev][2]->array(mfi); + + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + + // Verify density is non-zero + if (N_arr(i,j,k)>0.0) { + + // Compute r + amrex::Real r = problo[0] + i * dx[0]; + + // Isolate U from NU + amrex::Real u_r = (NUx_arr(i, j, k) / (N_arr(i,j,k) * clight )); + amrex::Real u_theta = (NUy_arr(i, j, k) / (N_arr(i,j,k) * clight )); + amrex::Real u_z = (NUz_arr(i, j, k) / (N_arr(i,j,k) * clight )); + + // (SSP-RK3) Push the fluid momentum (R and Theta) + // F_r, F_theta are first order euler pushes of our rhs operator + if (i != domain.smallEnd(0)) { + amrex::Real u_r_1 = F_r(r,u_r,u_theta,u_z,dt); + amrex::Real u_theta_1 = F_theta(r,u_r,u_theta,u_z,dt); + amrex::Real u_r_2 = (0.75)*(u_r) + (0.25)*F_r(r,u_r_1,u_theta_1,u_z,dt); + amrex::Real u_theta_2 = (0.75)*(u_theta) + (0.25)*F_theta(r,u_r_1,u_theta_1,u_z,dt); + u_r = (1.0/3.0)*(u_r) + (2.0/3.0)*F_r(r,u_r_2,u_theta_2,u_z,dt); + u_theta = (1.0/3.0)*(u_theta) + (2.0/3.0)*F_theta(r,u_r_2,u_theta_2,u_z,dt); + + // Calculate NU, save NUr, NUtheta + NUx_arr(i,j,k) = N_arr(i,j,k)*u_r*clight; + NUy_arr(i,j,k) = N_arr(i,j,k)*u_theta*clight; + + // BC r = 0, u_theta = 0, and there is no extra source terms + } else { + NUx_arr(i,j,k) = 0.0; + NUy_arr(i,j,k) = 0.0; + } + } + } + ); + } +} +#endif + +// Momentum source from fields +void WarpXFluidContainer::GatherAndPush ( + int lev, + const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, + const amrex::MultiFab& Bx, const amrex::MultiFab& By, const amrex::MultiFab& Bz, + Real t) +{ + WARPX_PROFILE("WarpXFluidContainer::GatherAndPush"); + + WarpX &warpx = WarpX::GetInstance(); + const amrex::Real q = getCharge(); + const amrex::Real m = getMass(); + const Real dt = warpx.getdt(lev); + const amrex::Geometry &geom = warpx.Geom(lev); + const auto dx = geom.CellSizeArray(); + const auto problo = geom.ProbLoArray(); + const amrex::Real gamma_boost = WarpX::gamma_boost; + const amrex::Real beta_boost = WarpX::beta_boost; + //Check whether m_E_ext_s is "none" + bool external_e_fields; // Needs intializing + bool external_b_fields; // Needs intializing + + + // Prepare interpolation of current components to cell center + amrex::GpuArray Nodal_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray Ex_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray Ey_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray Ez_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray Bx_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray By_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray Bz_type = amrex::GpuArray{0, 0, 0}; + for (int i = 0; i < AMREX_SPACEDIM; ++i) + { + Nodal_type[i] = N[lev]->ixType()[i]; + Ex_type[i] = Ex.ixType()[i]; + Ey_type[i] = Ey.ixType()[i]; + Ez_type[i] = Ez.ixType()[i]; + Bx_type[i] = Bx.ixType()[i]; + By_type[i] = By.ixType()[i]; + Bz_type[i] = Bz.ixType()[i]; + } + + // External field parsers + external_e_fields = (m_E_ext_s == "parse_e_ext_function"); + external_b_fields = (m_B_ext_s == "parse_b_ext_function"); + amrex::ParserExecutor<4> Exfield_parser; + amrex::ParserExecutor<4> Eyfield_parser; + amrex::ParserExecutor<4> Ezfield_parser; + amrex::ParserExecutor<4> Bxfield_parser; + amrex::ParserExecutor<4> Byfield_parser; + amrex::ParserExecutor<4> Bzfield_parser; + if (external_e_fields){ + constexpr int num_arguments = 4; //x,y,z,t + Exfield_parser = m_Ex_parser->compile(); + Eyfield_parser = m_Ey_parser->compile(); + Ezfield_parser = m_Ez_parser->compile(); + } + + if (external_b_fields){ + constexpr int num_arguments = 4; //x,y,z,t + Bxfield_parser = m_Bx_parser->compile(); + Byfield_parser = m_By_parser->compile(); + Bzfield_parser = m_Bz_parser->compile(); + } + + + // H&C push the momentum +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + + amrex::Box const &tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + + amrex::Array4 const &N_arr = N[lev]->array(mfi); + amrex::Array4 NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 NUz_arr = NU[lev][2]->array(mfi); + + amrex::Array4 const& Ex_arr = Ex.array(mfi); + amrex::Array4 const& Ey_arr = Ey.array(mfi); + amrex::Array4 const& Ez_arr = Ez.array(mfi); + amrex::Array4 const& Bx_arr = Bx.array(mfi); + amrex::Array4 const& By_arr = By.array(mfi); + amrex::Array4 const& Bz_arr = Bz.array(mfi); + + // Here, we do not perform any coarsening. + amrex::GpuArray coarsening_ratio = {1, 1, 1}; + + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + + // Only run if density is positive + if (N_arr(i,j,k)>0.0) { + + // Interpolate fields from tmp to Nodal points + amrex::Real Ex_Nodal = ablastr::coarsen::sample::Interp(Ex_arr, + Ex_type, Nodal_type, coarsening_ratio, i, j, k, 0); + amrex::Real Ey_Nodal = ablastr::coarsen::sample::Interp(Ey_arr, + Ey_type, Nodal_type, coarsening_ratio, i, j, k, 0); + amrex::Real Ez_Nodal = ablastr::coarsen::sample::Interp(Ez_arr, + Ez_type, Nodal_type, coarsening_ratio, i, j, k, 0); + amrex::Real Bx_Nodal = ablastr::coarsen::sample::Interp(Bx_arr, + Bx_type, Nodal_type, coarsening_ratio, i, j, k, 0); + amrex::Real By_Nodal = ablastr::coarsen::sample::Interp(By_arr, + By_type, Nodal_type, coarsening_ratio, i, j, k, 0); + amrex::Real Bz_Nodal = ablastr::coarsen::sample::Interp(Bz_arr, + Bz_type, Nodal_type, coarsening_ratio, i, j, k, 0); + + if (gamma_boost > 1._rt) { // Lorentz transform fields due to moving frame + if ( ( external_b_fields ) || ( external_e_fields ) ){ + + // Lorentz transform z (from boosted to lab frame) + amrex::Real Ex_ext_boost, Ey_ext_boost, Ez_ext_boost; + amrex::Real Bx_ext_boost, By_ext_boost, Bz_ext_boost; + amrex::Real Ex_ext_lab, Ey_ext_lab, Ez_ext_lab; + amrex::Real Bx_ext_lab, By_ext_lab, Bz_ext_lab; + + // Grab the location +#if defined(WARPX_DIM_3D) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = problo[1] + j * dx[1]; + amrex::Real z = problo[2] + k * dx[2]; +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[1] + j * dx[1]; +#else + amrex::Real x = 0.0_rt; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[0] + i * dx[0]; +#endif + + // Get the lab frame E and B + // Transform (boosted to lab) + amrex::Real t_lab = gamma_boost*(t + beta_boost*z/PhysConst::c); + amrex::Real z_lab = gamma_boost*(z + beta_boost*PhysConst::c*t); + + // Grab the external fields in the lab frame: + if ( external_e_fields ) { + Ex_ext_lab = Exfield_parser(x, y, z_lab, t_lab); + Ey_ext_lab = Eyfield_parser(x, y, z_lab, t_lab); + Ez_ext_lab = Ezfield_parser(x, y, z_lab, t_lab); + }else{ + Ex_ext_lab = 0.0; + Ey_ext_lab = 0.0; + Ez_ext_lab = 0.0; + } + if ( external_b_fields ) { + Bx_ext_lab = Bxfield_parser(x, y, z_lab, t_lab); + By_ext_lab = Byfield_parser(x, y, z_lab, t_lab); + Bz_ext_lab = Bzfield_parser(x, y, z_lab, t_lab); + }else{ + Bx_ext_lab = 0.0; + By_ext_lab = 0.0; + Bz_ext_lab = 0.0; + } + + // Transform E & B (lab to boosted frame) + // (Require both to for the lorentz transform) + // RHS m_parser + Ez_ext_boost = Ez_ext_lab; + Bz_ext_boost = Bz_ext_lab; + Ex_ext_boost = gamma_boost*(Ex_ext_lab - beta_boost*PhysConst::c*By_ext_lab); + Ey_ext_boost = gamma_boost*(Ey_ext_lab + beta_boost*PhysConst::c*Bx_ext_lab); + Bx_ext_boost = gamma_boost*(Bx_ext_lab + beta_boost*Ey_ext_lab/PhysConst::c); + By_ext_boost = gamma_boost*(By_ext_lab - beta_boost*Ex_ext_lab/PhysConst::c); + + // Then add to Nodal quantities in the boosted frame: + Ex_Nodal += Ex_ext_boost; + Ey_Nodal += Ey_ext_boost; + Ez_Nodal += Ez_ext_boost; + Bx_Nodal += Bx_ext_boost; + By_Nodal += By_ext_boost; + Bz_Nodal += Bz_ext_boost; + } + } else { + + // Added external e fields: + if ( external_e_fields ){ +#if defined(WARPX_DIM_3D) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = problo[1] + j * dx[1]; + amrex::Real z = problo[2] + k * dx[2]; +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[1] + j * dx[1]; +#else + amrex::Real x = 0.0_rt; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[0] + i * dx[0]; +#endif + + Ex_Nodal += Exfield_parser(x, y, z, t); + Ey_Nodal += Eyfield_parser(x, y, z, t); + Ez_Nodal += Ezfield_parser(x, y, z, t); + } + + // Added external b fields: + if ( external_b_fields ){ +#if defined(WARPX_DIM_3D) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = problo[1] + j * dx[1]; + amrex::Real z = problo[2] + k * dx[2]; +#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + amrex::Real x = problo[0] + i * dx[0]; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[1] + j * dx[1]; +#else + amrex::Real x = 0.0_rt; + amrex::Real y = 0.0_rt; + amrex::Real z = problo[0] + i * dx[0]; +#endif + + Bx_Nodal += Bxfield_parser(x, y, z, t); + By_Nodal += Byfield_parser(x, y, z, t); + Bz_Nodal += Bzfield_parser(x, y, z, t); + } + } + + // Isolate U from NU + amrex::Real tmp_Ux = (NUx_arr(i, j, k) / N_arr(i,j,k)); + amrex::Real tmp_Uy = (NUy_arr(i, j, k) / N_arr(i,j,k)); + amrex::Real tmp_Uz = (NUz_arr(i, j, k) / N_arr(i,j,k)); + + // Enforce RZ boundary conditions +#if defined(WARPX_DIM_RZ) + if ( i == 0 ){ + Ex_Nodal = 0.0; + Ey_Nodal = 0.0; + By_Nodal = 0.0; + Bx_Nodal = 0.0; + } +#endif + + // Push the fluid momentum + UpdateMomentumHigueraCary(tmp_Ux, tmp_Uy, tmp_Uz, + Ex_Nodal, Ey_Nodal, Ez_Nodal, + Bx_Nodal, By_Nodal, Bz_Nodal, q, m, dt ); + + // Calculate NU + NUx_arr(i,j,k) = N_arr(i,j,k)*tmp_Ux; + NUy_arr(i,j,k) = N_arr(i,j,k)*tmp_Uy; + NUz_arr(i,j,k) = N_arr(i,j,k)*tmp_Uz; + } + } + ); + } +} + +void WarpXFluidContainer::DepositCharge (int lev, amrex::MultiFab &rho, int icomp) +{ + WARPX_PROFILE("WarpXFluidContainer::DepositCharge"); + + WarpX &warpx = WarpX::GetInstance(); + const amrex::Geometry &geom = warpx.Geom(lev); + const amrex::Periodicity &period = geom.periodicity(); + const amrex::Real q = getCharge(); + auto const &owner_mask_rho = amrex::OwnerMask(rho, period); + + // Assertion, make sure rho is at the same location as N + AMREX_ALWAYS_ASSERT(rho.ixType().nodeCentered()); + + // Loop over and deposit charge density +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + + amrex::Box const &tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + amrex::Array4 const &N_arr = N[lev]->array(mfi); + amrex::Array4 rho_arr = rho.array(mfi); + amrex::Array4 owner_mask_rho_arr = owner_mask_rho->array(mfi); + + // Deposit Rho + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + if ( owner_mask_rho_arr(i,j,k) ) rho_arr(i,j,k,icomp) += q*N_arr(i,j,k); + } + ); + } +} + + +void WarpXFluidContainer::DepositCurrent( + int lev, + amrex::MultiFab &jx, amrex::MultiFab &jy, amrex::MultiFab &jz) +{ + WARPX_PROFILE("WarpXFluidContainer::DepositCurrent"); + + // Temporary nodal currents + amrex::MultiFab tmp_jx_fluid(N[lev]->boxArray(), N[lev]->DistributionMap(), 1, 0); + amrex::MultiFab tmp_jy_fluid(N[lev]->boxArray(), N[lev]->DistributionMap(), 1, 0); + amrex::MultiFab tmp_jz_fluid(N[lev]->boxArray(), N[lev]->DistributionMap(), 1, 0); + + const amrex::Real inv_clight_sq = 1.0_prt / PhysConst::c / PhysConst::c; + const amrex::Real q = getCharge(); + + // Prepare interpolation of current components to cell center + amrex::GpuArray j_nodal_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray jx_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray jy_type = amrex::GpuArray{0, 0, 0}; + amrex::GpuArray jz_type = amrex::GpuArray{0, 0, 0}; + for (int i = 0; i < AMREX_SPACEDIM; ++i) + { + j_nodal_type[i] = tmp_jx_fluid.ixType()[i]; + jx_type[i] = jx.ixType()[i]; + jy_type[i] = jy.ixType()[i]; + jz_type[i] = jz.ixType()[i]; + } + + // We now need to create a mask to fix the double counting. + WarpX &warpx = WarpX::GetInstance(); + const amrex::Geometry &geom = warpx.Geom(lev); + const amrex::Periodicity &period = geom.periodicity(); + auto const &owner_mask_x = amrex::OwnerMask(jx, period); + auto const &owner_mask_y = amrex::OwnerMask(jy, period); + auto const &owner_mask_z = amrex::OwnerMask(jz, period); + + // Calculate j at the nodes +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + amrex::Box const &tile_box = mfi.tilebox(N[lev]->ixType().toIntVect()); + + amrex::Array4 const &N_arr = N[lev]->array(mfi); + amrex::Array4 const &NUx_arr = NU[lev][0]->array(mfi); + amrex::Array4 const &NUy_arr = NU[lev][1]->array(mfi); + amrex::Array4 const &NUz_arr = NU[lev][2]->array(mfi); + + amrex::Array4 tmp_jx_fluid_arr = tmp_jx_fluid.array(mfi); + amrex::Array4 tmp_jy_fluid_arr = tmp_jy_fluid.array(mfi); + amrex::Array4 tmp_jz_fluid_arr = tmp_jz_fluid.array(mfi); + + amrex::ParallelFor(tile_box, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + // Calculate J from fluid quantities + amrex::Real gamma = 1.0, Ux = 0.0, Uy = 0.0, Uz = 0.0; + if (N_arr(i, j, k)>0.0){ + Ux = NUx_arr(i, j, k)/N_arr(i, j, k); + Uy = NUy_arr(i, j, k)/N_arr(i, j, k); + Uz = NUz_arr(i, j, k)/N_arr(i, j, k); + gamma = std::sqrt(1.0 + ( Ux*Ux + Uy*Uy + Uz*Uz) * inv_clight_sq ) ; + } + tmp_jx_fluid_arr(i, j, k) = q * (NUx_arr(i, j, k) / gamma); + tmp_jy_fluid_arr(i, j, k) = q * (NUy_arr(i, j, k) / gamma); + tmp_jz_fluid_arr(i, j, k) = q * (NUz_arr(i, j, k) / gamma); + } + ); + } + + // Interpolate j from the nodes to the simulation mesh (typically Yee mesh) +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for (MFIter mfi(*N[lev], TilingIfNotGPU()); mfi.isValid(); ++mfi) + { + amrex::Box const &tile_box_x = mfi.tilebox(jx.ixType().toIntVect()); + amrex::Box const &tile_box_y = mfi.tilebox(jy.ixType().toIntVect()); + amrex::Box const &tile_box_z = mfi.tilebox(jz.ixType().toIntVect()); + + amrex::Array4 jx_arr = jx.array(mfi); + amrex::Array4 jy_arr = jy.array(mfi); + amrex::Array4 jz_arr = jz.array(mfi); + + amrex::Array4 tmp_jx_fluid_arr = tmp_jx_fluid.array(mfi); + amrex::Array4 tmp_jy_fluid_arr = tmp_jy_fluid.array(mfi); + amrex::Array4 tmp_jz_fluid_arr = tmp_jz_fluid.array(mfi); + + amrex::Array4 owner_mask_x_arr = owner_mask_x->array(mfi); + amrex::Array4 owner_mask_y_arr = owner_mask_y->array(mfi); + amrex::Array4 owner_mask_z_arr = owner_mask_z->array(mfi); + + // When using the `Interp` function, one needs to specify whether coarsening is desired. + // Here, we do not perform any coarsening. + amrex::GpuArray coarsening_ratio = {1, 1, 1}; + + + // Interpolate fluid current and deposit it + // ( mask double counting ) + amrex::ParallelFor( tile_box_x, tile_box_y, tile_box_z, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + amrex::Real jx_tmp = ablastr::coarsen::sample::Interp(tmp_jx_fluid_arr, + j_nodal_type, jx_type, coarsening_ratio, i, j, k, 0); + if ( owner_mask_x_arr(i,j,k) ) jx_arr(i, j, k) += jx_tmp; + }, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + amrex::Real jy_tmp = ablastr::coarsen::sample::Interp(tmp_jy_fluid_arr, + j_nodal_type, jy_type, coarsening_ratio, i, j, k, 0); + if ( owner_mask_y_arr(i,j,k) ) jy_arr(i, j, k) += jy_tmp; + }, + [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept + { + amrex::Real jz_tmp = ablastr::coarsen::sample::Interp(tmp_jz_fluid_arr, + j_nodal_type, jz_type, coarsening_ratio, i, j, k, 0); + if ( owner_mask_z_arr(i,j,k) ) jz_arr(i, j, k) += jz_tmp; + } + ); + } +} diff --git a/Source/Fluids/WarpXFluidContainer_fwd.H b/Source/Fluids/WarpXFluidContainer_fwd.H new file mode 100644 index 00000000000..77a6bc0c991 --- /dev/null +++ b/Source/Fluids/WarpXFluidContainer_fwd.H @@ -0,0 +1,12 @@ +/* Copyright 2023 Grant Johnson, Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ + +#ifndef WARPX_WarpXFluidContainer_fwd_H_ + +class WarpXFluidContainer; + +#endif /* WARPX_WarpXFluidContainer_fwd_H_ */ diff --git a/Source/Initialization/PlasmaInjector.H b/Source/Initialization/PlasmaInjector.H index 89ea4a00a78..ac64980b952 100644 --- a/Source/Initialization/PlasmaInjector.H +++ b/Source/Initialization/PlasmaInjector.H @@ -120,11 +120,7 @@ public: bool radially_weighted = true; - std::string str_density_function; std::string str_flux_function; - std::string str_momentum_function_ux; - std::string str_momentum_function_uy; - std::string str_momentum_function_uz; amrex::Real xmin, xmax; amrex::Real ymin, ymax; @@ -146,7 +142,6 @@ protected: PhysicalSpecies physical_species = PhysicalSpecies::unspecified; - amrex::Real density; amrex::Real flux; int species_id; @@ -185,9 +180,7 @@ protected: void setupNuniformPerCell (const amrex::ParmParse& pp_species_name); void setupExternalFile (const amrex::ParmParse& pp_species_name); - void parseDensity (const amrex::ParmParse& pp_species_name); void parseFlux (const amrex::ParmParse& pp_species_name); - void parseMomentum (const amrex::ParmParse& pp_species_name, const std::string& style); }; #endif diff --git a/Source/Initialization/PlasmaInjector.cpp b/Source/Initialization/PlasmaInjector.cpp index b880d4a46e7..56d29a8a1bb 100644 --- a/Source/Initialization/PlasmaInjector.cpp +++ b/Source/Initialization/PlasmaInjector.cpp @@ -14,8 +14,8 @@ #include "Initialization/InjectorDensity.H" #include "Initialization/InjectorMomentum.H" #include "Initialization/InjectorPosition.H" -#include "Particles/SpeciesPhysicalProperties.H" #include "Utils/Parser/ParserUtils.H" +#include "Utils/SpeciesUtils.H" #include "Utils/TextMsg.H" #include "Utils/WarpXConst.H" #include "WarpX.H" @@ -45,17 +45,6 @@ using namespace amrex::literals; -namespace { - void StringParseAbortMessage(const std::string& var, - const std::string& name) { - std::stringstream stringstream; - std::string string; - stringstream << var << " string '" << name << "' not recognized."; - string = stringstream.str(); - WARPX_ABORT_WITH_MESSAGE(string); - } -} - PlasmaInjector::PlasmaInjector (int ispecies, const std::string& name, const amrex::Geometry& geom): species_id{ispecies}, species_name{name} @@ -123,60 +112,14 @@ PlasmaInjector::PlasmaInjector (int ispecies, const std::string& name, utils::parser::queryWithParser(pp_species_name, "density_min", density_min); utils::parser::queryWithParser(pp_species_name, "density_max", density_max); - std::string physical_species_s; - const bool species_is_specified = pp_species_name.query("species_type", physical_species_s); - if (species_is_specified){ - const auto physical_species_from_string = species::from_string( physical_species_s ); - WARPX_ALWAYS_ASSERT_WITH_MESSAGE(physical_species_from_string, - physical_species_s + " does not exist!"); - physical_species = physical_species_from_string.value(); - charge = species::get_charge( physical_species ); - mass = species::get_mass( physical_species ); - } - - // Parse injection style std::string injection_style = "none"; + // Parse injection style pp_species_name.query("injection_style", injection_style); std::transform(injection_style.begin(), - injection_style.end(), - injection_style.begin(), - ::tolower); - - // parse charge and mass - const bool charge_is_specified = - utils::parser::queryWithParser(pp_species_name, "charge", charge); - const bool mass_is_specified = - utils::parser::queryWithParser(pp_species_name, "mass", mass); - - if ( charge_is_specified && species_is_specified ){ - ablastr::warn_manager::WMRecordWarning("Species", - "Both '" + species_name + ".charge' and " + - species_name + ".species_type' are specified.\n" + - species_name + ".charge' will take precedence.\n"); - - } - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( - charge_is_specified || - species_is_specified || - (injection_style == "external_file"), - "Need to specify at least one of species_type or charge for species '" + - species_name + "'." - ); - - if ( mass_is_specified && species_is_specified ){ - ablastr::warn_manager::WMRecordWarning("Species", - "Both '" + species_name + ".mass' and " + - species_name + ".species_type' are specified.\n" + - species_name + ".mass' will take precedence.\n"); - } - - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( - mass_is_specified || - species_is_specified || - (injection_style == "external_file"), - "Need to specify at least one of species_type or mass for species '" + - species_name + "'." - ); + injection_style.end(), + injection_style.begin(), + ::tolower); + SpeciesUtils::extractSpeciesProperties(species_name, injection_style, charge, mass, physical_species); num_particles_per_cell_each_dim.assign(3, 0); @@ -197,9 +140,27 @@ PlasmaInjector::PlasmaInjector (int ispecies, const std::string& name, } else if (injection_style == "external_file") { setupExternalFile(pp_species_name); } else if (injection_style != "none") { - StringParseAbortMessage("Injection style", injection_style); + SpeciesUtils::StringParseAbortMessage("Injection style", injection_style); } + if (h_inj_rho) { +#ifdef AMREX_USE_GPU + d_inj_rho = static_cast + (amrex::The_Arena()->alloc(sizeof(InjectorDensity))); + amrex::Gpu::htod_memcpy_async(d_inj_rho, h_inj_rho.get(), sizeof(InjectorDensity)); +#else + d_inj_rho = h_inj_rho.get(); +#endif + } + if (h_inj_mom) { +#ifdef AMREX_USE_GPU + d_inj_mom = static_cast + (amrex::The_Arena()->alloc(sizeof(InjectorMomentum))); + amrex::Gpu::htod_memcpy_async(d_inj_mom, h_inj_mom.get(), sizeof(InjectorMomentum)); +#else + d_inj_mom = h_inj_mom.get(); +#endif + } amrex::Gpu::synchronize(); } @@ -286,7 +247,8 @@ void PlasmaInjector::setupGaussianBeam (const amrex::ParmParse& pp_species_name) WARPX_ALWAYS_ASSERT_WITH_MESSAGE( valid_symmetries.count(symmetrization_order), "Error: Symmetrization only supported to orders 4 or 8 "); gaussian_beam = true; - parseMomentum(pp_species_name, "gaussian_beam"); + SpeciesUtils::parseMomentum(species_name, "gaussian_beam", h_inj_mom, + ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); #if defined(WARPX_DIM_XZ) WARPX_ALWAYS_ASSERT_WITH_MESSAGE( y_rms > 0._rt, "Error: Gaussian beam y_rms must be strictly greater than 0 in 2D " @@ -325,9 +287,9 @@ void PlasmaInjector::setupNRandomPerCell (const amrex::ParmParse& pp_species_nam #else d_inj_pos = h_inj_pos.get(); #endif - - parseDensity(pp_species_name); - parseMomentum(pp_species_name, "nrandompercell"); + SpeciesUtils::parseDensity(species_name, h_inj_rho, density_parser); + SpeciesUtils::parseMomentum(species_name, "nrandompercell", h_inj_mom, + ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); } void PlasmaInjector::setupNFluxPerCell (const amrex::ParmParse& pp_species_name) @@ -404,7 +366,8 @@ void PlasmaInjector::setupNFluxPerCell (const amrex::ParmParse& pp_species_name) #endif parseFlux(pp_species_name); - parseMomentum(pp_species_name, "nfluxpercell"); + SpeciesUtils::parseMomentum(species_name, "nfluxpercell", h_inj_mom, + ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel, flux_normal_axis, flux_direction); } void PlasmaInjector::setupNuniformPerCell (const amrex::ParmParse& pp_species_name) @@ -455,8 +418,9 @@ void PlasmaInjector::setupNuniformPerCell (const amrex::ParmParse& pp_species_na num_particles_per_cell = num_particles_per_cell_each_dim[0] * num_particles_per_cell_each_dim[1] * num_particles_per_cell_each_dim[2]; - parseDensity(pp_species_name); - parseMomentum(pp_species_name, "nuniformpercell"); + SpeciesUtils::parseDensity(species_name, h_inj_rho, density_parser); + SpeciesUtils::parseMomentum(species_name, "nuniformpercell", h_inj_mom, + ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); } void PlasmaInjector::setupExternalFile (const amrex::ParmParse& pp_species_name) @@ -563,46 +527,6 @@ void PlasmaInjector::setupExternalFile (const amrex::ParmParse& pp_species_name) #endif // WARPX_USE_OPENPMD } -// Depending on injection type at runtime, initialize inj_rho -// so that inj_rho->getDensity calls -// InjectorPosition[Constant or Predefined or etc.].getDensity. -void PlasmaInjector::parseDensity (const amrex::ParmParse& pp_species_name) -{ - // parse density information - std::string rho_prof_s; - pp_species_name.get("profile", rho_prof_s); - std::transform(rho_prof_s.begin(), rho_prof_s.end(), - rho_prof_s.begin(), ::tolower); - if (rho_prof_s == "constant") { - utils::parser::getWithParser(pp_species_name, "density", density); - // Construct InjectorDensity with InjectorDensityConstant. - h_inj_rho.reset(new InjectorDensity((InjectorDensityConstant*)nullptr, density)); - } else if (rho_prof_s == "predefined") { - // Construct InjectorDensity with InjectorDensityPredefined. - h_inj_rho.reset(new InjectorDensity((InjectorDensityPredefined*)nullptr,species_name)); - } else if (rho_prof_s == "parse_density_function") { - utils::parser::Store_parserString( - pp_species_name, "density_function(x,y,z)", str_density_function); - // Construct InjectorDensity with InjectorDensityParser. - density_parser = std::make_unique( - utils::parser::makeParser(str_density_function,{"x","y","z"})); - h_inj_rho.reset(new InjectorDensity((InjectorDensityParser*)nullptr, - density_parser->compile<3>())); - } else { - StringParseAbortMessage("Density profile type", rho_prof_s); - } - - if (h_inj_rho) { -#ifdef AMREX_USE_GPU - d_inj_rho = static_cast - (amrex::The_Arena()->alloc(sizeof(InjectorDensity))); - amrex::Gpu::htod_memcpy_async(d_inj_rho, h_inj_rho.get(), sizeof(InjectorDensity)); -#else - d_inj_rho = h_inj_rho.get(); -#endif - } -} - // Depending on injection type at runtime, initialize inj_flux // so that inj_flux->getFlux calls // InjectorFlux[Constant or Parser or etc.].getFlux. @@ -626,7 +550,7 @@ void PlasmaInjector::parseFlux (const amrex::ParmParse& pp_species_name) h_inj_flux.reset(new InjectorFlux((InjectorFluxParser*)nullptr, flux_parser->compile<4>())); } else { - StringParseAbortMessage("Flux profile type", flux_prof_s); + SpeciesUtils::StringParseAbortMessage("Flux profile type", flux_prof_s); } if (h_inj_flux) { #ifdef AMREX_USE_GPU @@ -640,138 +564,6 @@ void PlasmaInjector::parseFlux (const amrex::ParmParse& pp_species_name) } -// Depending on injection type at runtime, initialize inj_mom -// so that inj_mom->getMomentum calls -// InjectorMomentum[Constant or Gaussian or etc.].getMomentum. -void PlasmaInjector::parseMomentum (const amrex::ParmParse& pp_species_name, const std::string& style) -{ - using namespace amrex::literals; - - // parse momentum information - std::string mom_dist_s; - pp_species_name.get("momentum_distribution_type", mom_dist_s); - std::transform(mom_dist_s.begin(), - mom_dist_s.end(), - mom_dist_s.begin(), - ::tolower); - if (mom_dist_s == "at_rest") { - constexpr amrex::Real ux = 0._rt; - constexpr amrex::Real uy = 0._rt; - constexpr amrex::Real uz = 0._rt; - // Construct InjectorMomentum with InjectorMomentumConstant. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumConstant*)nullptr, ux, uy, uz)); - } else if (mom_dist_s == "constant") { - amrex::Real ux = 0._rt; - amrex::Real uy = 0._rt; - amrex::Real uz = 0._rt; - utils::parser::queryWithParser(pp_species_name, "ux", ux); - utils::parser::queryWithParser(pp_species_name, "uy", uy); - utils::parser::queryWithParser(pp_species_name, "uz", uz); - // Construct InjectorMomentum with InjectorMomentumConstant. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumConstant*)nullptr, ux, uy, uz)); - } else if (mom_dist_s == "gaussian") { - amrex::Real ux_m = 0._rt; - amrex::Real uy_m = 0._rt; - amrex::Real uz_m = 0._rt; - amrex::Real ux_th = 0._rt; - amrex::Real uy_th = 0._rt; - amrex::Real uz_th = 0._rt; - utils::parser::queryWithParser(pp_species_name, "ux_m", ux_m); - utils::parser::queryWithParser(pp_species_name, "uy_m", uy_m); - utils::parser::queryWithParser(pp_species_name, "uz_m", uz_m); - utils::parser::queryWithParser(pp_species_name, "ux_th", ux_th); - utils::parser::queryWithParser(pp_species_name, "uy_th", uy_th); - utils::parser::queryWithParser(pp_species_name, "uz_th", uz_th); - // Construct InjectorMomentum with InjectorMomentumGaussian. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumGaussian*)nullptr, - ux_m, uy_m, uz_m, ux_th, uy_th, uz_th)); - } else if (mom_dist_s == "gaussianflux") { - WARPX_ALWAYS_ASSERT_WITH_MESSAGE(style == "nfluxpercell", - "Error: gaussianflux can only be used with injection_style = NFluxPerCell"); - amrex::Real ux_m = 0._rt; - amrex::Real uy_m = 0._rt; - amrex::Real uz_m = 0._rt; - amrex::Real ux_th = 0._rt; - amrex::Real uy_th = 0._rt; - amrex::Real uz_th = 0._rt; - utils::parser::queryWithParser(pp_species_name, "ux_m", ux_m); - utils::parser::queryWithParser(pp_species_name, "uy_m", uy_m); - utils::parser::queryWithParser(pp_species_name, "uz_m", uz_m); - utils::parser::queryWithParser(pp_species_name, "ux_th", ux_th); - utils::parser::queryWithParser(pp_species_name, "uy_th", uy_th); - utils::parser::queryWithParser(pp_species_name, "uz_th", uz_th); - // Construct InjectorMomentum with InjectorMomentumGaussianFlux. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumGaussianFlux*)nullptr, - ux_m, uy_m, uz_m, ux_th, uy_th, uz_th, - flux_normal_axis, flux_direction)); - } else if (mom_dist_s == "uniform") { - amrex::Real ux_min = 0._rt; - amrex::Real uy_min = 0._rt; - amrex::Real uz_min = 0._rt; - amrex::Real ux_max = 0._rt; - amrex::Real uy_max = 0._rt; - amrex::Real uz_max = 0._rt; - utils::parser::queryWithParser(pp_species_name, "ux_min", ux_min); - utils::parser::queryWithParser(pp_species_name, "uy_min", uy_min); - utils::parser::queryWithParser(pp_species_name, "uz_min", uz_min); - utils::parser::queryWithParser(pp_species_name, "ux_max", ux_max); - utils::parser::queryWithParser(pp_species_name, "uy_max", uy_max); - utils::parser::queryWithParser(pp_species_name, "uz_max", uz_max); - // Construct InjectorMomentum with InjectorMomentumUniform. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumUniform*)nullptr, - ux_min, uy_min, uz_min, ux_max, uy_max, uz_max)); - } else if (mom_dist_s == "maxwell_boltzmann"){ - h_mom_temp = std::make_unique(pp_species_name); - const GetTemperature getTemp(*h_mom_temp); - h_mom_vel = std::make_unique(pp_species_name); - const GetVelocity getVel(*h_mom_vel); - // Construct InjectorMomentum with InjectorMomentumBoltzmann. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumBoltzmann*)nullptr, getTemp, getVel)); - } else if (mom_dist_s == "maxwell_juttner"){ - h_mom_temp = std::make_unique(pp_species_name); - const GetTemperature getTemp(*h_mom_temp); - h_mom_vel = std::make_unique(pp_species_name); - const GetVelocity getVel(*h_mom_vel); - // Construct InjectorMomentum with InjectorMomentumJuttner. - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumJuttner*)nullptr, getTemp, getVel)); - } else if (mom_dist_s == "radial_expansion") { - amrex::Real u_over_r = 0._rt; - utils::parser::queryWithParser(pp_species_name, "u_over_r", u_over_r); - // Construct InjectorMomentum with InjectorMomentumRadialExpansion. - h_inj_mom.reset(new InjectorMomentum - ((InjectorMomentumRadialExpansion*)nullptr, u_over_r)); - } else if (mom_dist_s == "parse_momentum_function") { - utils::parser::Store_parserString(pp_species_name, "momentum_function_ux(x,y,z)", - str_momentum_function_ux); - utils::parser::Store_parserString(pp_species_name, "momentum_function_uy(x,y,z)", - str_momentum_function_uy); - utils::parser::Store_parserString(pp_species_name, "momentum_function_uz(x,y,z)", - str_momentum_function_uz); - // Construct InjectorMomentum with InjectorMomentumParser. - ux_parser = std::make_unique( - utils::parser::makeParser(str_momentum_function_ux, {"x","y","z"})); - uy_parser = std::make_unique( - utils::parser::makeParser(str_momentum_function_uy, {"x","y","z"})); - uz_parser = std::make_unique( - utils::parser::makeParser(str_momentum_function_uz, {"x","y","z"})); - h_inj_mom.reset(new InjectorMomentum((InjectorMomentumParser*)nullptr, - ux_parser->compile<3>(), - uy_parser->compile<3>(), - uz_parser->compile<3>())); - } else { - StringParseAbortMessage("Momentum distribution type", mom_dist_s); - } - if (h_inj_mom) { -#ifdef AMREX_USE_GPU - d_inj_mom = static_cast - (amrex::The_Arena()->alloc(sizeof(InjectorMomentum))); - amrex::Gpu::htod_memcpy_async(d_inj_mom, h_inj_mom.get(), sizeof(InjectorMomentum)); -#else - d_inj_mom = h_inj_mom.get(); -#endif - } -} - amrex::XDim3 PlasmaInjector::getMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept diff --git a/Source/Make.WarpX b/Source/Make.WarpX index 3f1702a4214..d1c77482f75 100644 --- a/Source/Make.WarpX +++ b/Source/Make.WarpX @@ -77,6 +77,7 @@ include $(WARPX_HOME)/Source/Diagnostics/Make.package include $(WARPX_HOME)/Source/EmbeddedBoundary/Make.package include $(WARPX_HOME)/Source/FieldSolver/Make.package include $(WARPX_HOME)/Source/Filter/Make.package +include $(WARPX_HOME)/Source/Fluids/Make.package include $(WARPX_HOME)/Source/Initialization/Make.package include $(WARPX_HOME)/Source/Laser/Make.package include $(WARPX_HOME)/Source/Parallelization/Make.package diff --git a/Source/Particles/MultiParticleContainer.H b/Source/Particles/MultiParticleContainer.H index fecd39137d7..af826068806 100644 --- a/Source/Particles/MultiParticleContainer.H +++ b/Source/Particles/MultiParticleContainer.H @@ -79,11 +79,6 @@ public: WarpXParticleContainer& GetParticleContainerFromName (const std::string& name) const; -#ifdef WARPX_USE_OPENPMD - std::unique_ptr& GetUniqueContainer(int index) { - return allcontainers[index]; - } -#endif std::array meanParticleVelocity(int index) { return allcontainers[index]->meanParticleVelocity(); } diff --git a/Source/Particles/Pusher/UpdateMomentumHigueraCary.H b/Source/Particles/Pusher/UpdateMomentumHigueraCary.H index 359fc34d508..dc998caee34 100644 --- a/Source/Particles/Pusher/UpdateMomentumHigueraCary.H +++ b/Source/Particles/Pusher/UpdateMomentumHigueraCary.H @@ -17,49 +17,49 @@ * \brief Push the particle's positions over one timestep, * given the value of its momenta `ux`, `uy`, `uz` */ - +template AMREX_GPU_HOST_DEVICE AMREX_INLINE void UpdateMomentumHigueraCary( - amrex::ParticleReal& ux, amrex::ParticleReal& uy, amrex::ParticleReal& uz, - const amrex::ParticleReal Ex, const amrex::ParticleReal Ey, const amrex::ParticleReal Ez, - const amrex::ParticleReal Bx, const amrex::ParticleReal By, const amrex::ParticleReal Bz, - const amrex::ParticleReal q, const amrex::ParticleReal m, const amrex::Real dt ) + T& ux, T& uy, T& uz, + const T Ex, const T Ey, const T Ez, + const T Bx, const T By, const T Bz, + const T q, const T m, const amrex::Real dt ) { using namespace amrex::literals; // Constants - const amrex::ParticleReal qmt = 0.5_prt*q*dt/m; - constexpr amrex::ParticleReal invclight = 1._prt/PhysConst::c; - constexpr amrex::ParticleReal invclightsq = 1._prt/(PhysConst::c*PhysConst::c); + const T qmt = 0.5_prt*q*dt/m; + constexpr T invclight = 1._prt/PhysConst::c; + constexpr T invclightsq = 1._prt/(PhysConst::c*PhysConst::c); // Compute u_minus - const amrex::ParticleReal umx = ux + qmt*Ex; - const amrex::ParticleReal umy = uy + qmt*Ey; - const amrex::ParticleReal umz = uz + qmt*Ez; + const T umx = ux + qmt*Ex; + const T umy = uy + qmt*Ey; + const T umz = uz + qmt*Ez; // Compute gamma squared of u_minus - amrex::ParticleReal gamma = 1._prt + (umx*umx + umy*umy + umz*umz)*invclightsq; + T gamma = 1._prt + (umx*umx + umy*umy + umz*umz)*invclightsq; // Compute beta and betam squared - const amrex::ParticleReal betax = qmt*Bx; - const amrex::ParticleReal betay = qmt*By; - const amrex::ParticleReal betaz = qmt*Bz; - const amrex::ParticleReal betam = betax*betax + betay*betay + betaz*betaz; + const T betax = qmt*Bx; + const T betay = qmt*By; + const T betaz = qmt*Bz; + const T betam = betax*betax + betay*betay + betaz*betaz; // Compute sigma - const amrex::ParticleReal sigma = gamma - betam; + const T sigma = gamma - betam; // Get u* - const amrex::ParticleReal ust = (umx*betax + umy*betay + umz*betaz)*invclight; + const T ust = (umx*betax + umy*betay + umz*betaz)*invclight; // Get new gamma inversed gamma = 1._prt/std::sqrt(0.5_prt*(sigma + std::sqrt(sigma*sigma + 4._prt*(betam + ust*ust)) )); // Compute t - const amrex::ParticleReal tx = gamma*betax; - const amrex::ParticleReal ty = gamma*betay; - const amrex::ParticleReal tz = gamma*betaz; + const T tx = gamma*betax; + const T ty = gamma*betay; + const T tz = gamma*betaz; // Compute s - const amrex::ParticleReal s = 1._prt/(1._prt+(tx*tx + ty*ty + tz*tz)); + const T s = 1._prt/(1._prt+(tx*tx + ty*ty + tz*tz)); // Compute um dot t - const amrex::ParticleReal umt = umx*tx + umy*ty + umz*tz; + const T umt = umx*tx + umy*ty + umz*tz; // Compute u_plus - const amrex::ParticleReal upx = s*( umx + umt*tx + umy*tz - umz*ty ); - const amrex::ParticleReal upy = s*( umy + umt*ty + umz*tx - umx*tz ); - const amrex::ParticleReal upz = s*( umz + umt*tz + umx*ty - umy*tx ); + const T upx = s*( umx + umt*tx + umy*tz - umz*ty ); + const T upy = s*( umy + umt*ty + umz*tx - umx*tz ); + const T upz = s*( umz + umt*tz + umx*ty - umy*tx ); // Get new u ux = upx + qmt*Ex + upy*tz - upz*ty; uy = upy + qmt*Ey + upz*tx - upx*tz; diff --git a/Source/Python/WarpX.cpp b/Source/Python/WarpX.cpp index 3429f33f5b9..1a5b319577c 100644 --- a/Source/Python/WarpX.cpp +++ b/Source/Python/WarpX.cpp @@ -26,6 +26,8 @@ #include #include #include +#include +#include #include #include #include diff --git a/Source/Utils/CMakeLists.txt b/Source/Utils/CMakeLists.txt index 9ded83a0c29..77626c0681e 100644 --- a/Source/Utils/CMakeLists.txt +++ b/Source/Utils/CMakeLists.txt @@ -4,6 +4,7 @@ foreach(D IN LISTS WarpX_DIMS) PRIVATE Interpolate.cpp ParticleUtils.cpp + SpeciesUtils.cpp RelativeCellPosition.cpp WarpXAlgorithmSelection.cpp WarpXMovingWindow.cpp diff --git a/Source/Utils/Make.package b/Source/Utils/Make.package index eb318f8d23c..4b5888ef22b 100644 --- a/Source/Utils/Make.package +++ b/Source/Utils/Make.package @@ -8,6 +8,7 @@ CEXE_sources += Interpolate.cpp CEXE_sources += IntervalsParser.cpp CEXE_sources += RelativeCellPosition.cpp CEXE_sources += ParticleUtils.cpp +CEXE_sources += SpeciesUtils.cpp include $(WARPX_HOME)/Source/Utils/Algorithms/Make.package include $(WARPX_HOME)/Source/Utils/Logo/Make.package diff --git a/Source/Utils/SpeciesUtils.H b/Source/Utils/SpeciesUtils.H new file mode 100644 index 00000000000..b1a2618a002 --- /dev/null +++ b/Source/Utils/SpeciesUtils.H @@ -0,0 +1,39 @@ +/* Copyright 2023 RemiLehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#ifndef WARPX_SPECIES_UTILS_H_ +#define WARPX_SPECIES_UTILS_H_ + +#include +#include "Initialization/InjectorDensity.H" +#include "Initialization/InjectorMomentum.H" +#include "Particles/SpeciesPhysicalProperties.H" + +namespace SpeciesUtils { + + void StringParseAbortMessage(const std::string& var, + const std::string& name); + + void extractSpeciesProperties ( std::string const& species_name, + std::string const& injection_style, amrex::Real& charge, amrex::Real& mass, + PhysicalSpecies& physical_species); + + void parseDensity (std::string const& species_name, + std::unique_ptr& h_inj_rho, + std::unique_ptr& density_parser); + + void parseMomentum (std::string const& species_name, const std::string& style, + std::unique_ptr& h_inj_mom, + std::unique_ptr& ux_parser, + std::unique_ptr& uy_parser, + std::unique_ptr& uz_parser, + std::unique_ptr& h_mom_temp, + std::unique_ptr& h_mom_vel, + int flux_normal_axis=0, int flux_direction=0); + +} + +#endif // WARPX_SPECIES_UTILS_H_ diff --git a/Source/Utils/SpeciesUtils.cpp b/Source/Utils/SpeciesUtils.cpp new file mode 100644 index 00000000000..2ac5879f9d1 --- /dev/null +++ b/Source/Utils/SpeciesUtils.cpp @@ -0,0 +1,240 @@ +/* Copyright 2023 Remi Lehe + * + * This file is part of WarpX. + * + * License: BSD-3-Clause-LBNL + */ +#include "SpeciesUtils.H" +#include +#include "Utils/TextMsg.H" +#include "Utils/Parser/ParserUtils.H" + +namespace SpeciesUtils { + + void StringParseAbortMessage(const std::string& var, + const std::string& name) { + std::stringstream stringstream; + std::string string; + stringstream << var << " string '" << name << "' not recognized."; + string = stringstream.str(); + WARPX_ABORT_WITH_MESSAGE(string); + } + + void extractSpeciesProperties (std::string const& species_name, + std::string const& injection_style, amrex::Real& charge, amrex::Real& mass, + PhysicalSpecies& physical_species ) + { + const amrex::ParmParse pp_species_name(species_name); + std::string physical_species_s; + const bool species_is_specified = pp_species_name.query("species_type", physical_species_s); + if (species_is_specified){ + const auto physical_species_from_string = species::from_string( physical_species_s ); + WARPX_ALWAYS_ASSERT_WITH_MESSAGE(physical_species_from_string, + physical_species_s + " does not exist!"); + physical_species = physical_species_from_string.value(); + charge = species::get_charge( physical_species ); + mass = species::get_mass( physical_species ); + } + + // parse charge and mass + const bool charge_is_specified = + utils::parser::queryWithParser(pp_species_name, "charge", charge); + const bool mass_is_specified = + utils::parser::queryWithParser(pp_species_name, "mass", mass); + + if ( charge_is_specified && species_is_specified ){ + ablastr::warn_manager::WMRecordWarning("Species", + "Both '" + species_name + ".charge' and " + + species_name + ".species_type' are specified.\n" + + species_name + ".charge' will take precedence.\n"); + + } + + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + charge_is_specified || + species_is_specified || + (injection_style == "external_file"), + "Need to specify at least one of species_type or charge for species '" + + species_name + "'." + ); + + if ( mass_is_specified && species_is_specified ){ + ablastr::warn_manager::WMRecordWarning("Species", + "Both '" + species_name + ".mass' and " + + species_name + ".species_type' are specified.\n" + + species_name + ".mass' will take precedence.\n"); + } + } + + // Depending on injection type at runtime, initialize inj_rho + // so that inj_rho->getDensity calls + // InjectorPosition[Constant or Predefined or etc.].getDensity. + void parseDensity (std::string const& species_name, + std::unique_ptr& h_inj_rho, + std::unique_ptr& density_parser) + { + const amrex::ParmParse pp_species_name(species_name); + + // parse density information + std::string rho_prof_s; + pp_species_name.get("profile", rho_prof_s); + std::transform(rho_prof_s.begin(), rho_prof_s.end(), + rho_prof_s.begin(), ::tolower); + if (rho_prof_s == "constant") { + amrex::Real density; + utils::parser::getWithParser(pp_species_name, "density", density); + // Construct InjectorDensity with InjectorDensityConstant. + h_inj_rho.reset(new InjectorDensity((InjectorDensityConstant*)nullptr, density)); + } else if (rho_prof_s == "predefined") { + // Construct InjectorDensity with InjectorDensityPredefined. + h_inj_rho.reset(new InjectorDensity((InjectorDensityPredefined*)nullptr,species_name)); + } else if (rho_prof_s == "parse_density_function") { + std::string str_density_function; + utils::parser::Store_parserString( + pp_species_name, "density_function(x,y,z)", str_density_function); + // Construct InjectorDensity with InjectorDensityParser. + density_parser = std::make_unique( + utils::parser::makeParser(str_density_function,{"x","y","z"})); + h_inj_rho.reset(new InjectorDensity((InjectorDensityParser*)nullptr, + density_parser->compile<3>())); + } else { + StringParseAbortMessage("Density profile type", rho_prof_s); + } + } + + // Depending on injection type at runtime, initialize inj_mom + // so that inj_mom->getMomentum calls + // InjectorMomentum[Constant or Gaussian or etc.].getMomentum. + void parseMomentum (std::string const& species_name, const std::string& style, + std::unique_ptr& h_inj_mom, + std::unique_ptr& ux_parser, + std::unique_ptr& uy_parser, + std::unique_ptr& uz_parser, + std::unique_ptr& h_mom_temp, + std::unique_ptr& h_mom_vel, + int flux_normal_axis, int flux_direction) + { + using namespace amrex::literals; + + const amrex::ParmParse pp_species_name(species_name); + + // parse momentum information + std::string mom_dist_s; + pp_species_name.get("momentum_distribution_type", mom_dist_s); + std::transform(mom_dist_s.begin(), + mom_dist_s.end(), + mom_dist_s.begin(), + ::tolower); + if (mom_dist_s == "at_rest") { + constexpr amrex::Real ux = 0._rt; + constexpr amrex::Real uy = 0._rt; + constexpr amrex::Real uz = 0._rt; + // Construct InjectorMomentum with InjectorMomentumConstant. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumConstant*)nullptr, ux, uy, uz)); + } else if (mom_dist_s == "constant") { + amrex::Real ux = 0._rt; + amrex::Real uy = 0._rt; + amrex::Real uz = 0._rt; + utils::parser::queryWithParser(pp_species_name, "ux", ux); + utils::parser::queryWithParser(pp_species_name, "uy", uy); + utils::parser::queryWithParser(pp_species_name, "uz", uz); + // Construct InjectorMomentum with InjectorMomentumConstant. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumConstant*)nullptr, ux, uy, uz)); + } else if (mom_dist_s == "gaussian") { + amrex::Real ux_m = 0._rt; + amrex::Real uy_m = 0._rt; + amrex::Real uz_m = 0._rt; + amrex::Real ux_th = 0._rt; + amrex::Real uy_th = 0._rt; + amrex::Real uz_th = 0._rt; + utils::parser::queryWithParser(pp_species_name, "ux_m", ux_m); + utils::parser::queryWithParser(pp_species_name, "uy_m", uy_m); + utils::parser::queryWithParser(pp_species_name, "uz_m", uz_m); + utils::parser::queryWithParser(pp_species_name, "ux_th", ux_th); + utils::parser::queryWithParser(pp_species_name, "uy_th", uy_th); + utils::parser::queryWithParser(pp_species_name, "uz_th", uz_th); + // Construct InjectorMomentum with InjectorMomentumGaussian. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumGaussian*)nullptr, + ux_m, uy_m, uz_m, ux_th, uy_th, uz_th)); + } else if (mom_dist_s == "gaussianflux") { + WARPX_ALWAYS_ASSERT_WITH_MESSAGE(style == "nfluxpercell", + "Error: gaussianflux can only be used with injection_style = NFluxPerCell"); + amrex::Real ux_m = 0._rt; + amrex::Real uy_m = 0._rt; + amrex::Real uz_m = 0._rt; + amrex::Real ux_th = 0._rt; + amrex::Real uy_th = 0._rt; + amrex::Real uz_th = 0._rt; + utils::parser::queryWithParser(pp_species_name, "ux_m", ux_m); + utils::parser::queryWithParser(pp_species_name, "uy_m", uy_m); + utils::parser::queryWithParser(pp_species_name, "uz_m", uz_m); + utils::parser::queryWithParser(pp_species_name, "ux_th", ux_th); + utils::parser::queryWithParser(pp_species_name, "uy_th", uy_th); + utils::parser::queryWithParser(pp_species_name, "uz_th", uz_th); + // Construct InjectorMomentum with InjectorMomentumGaussianFlux. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumGaussianFlux*)nullptr, + ux_m, uy_m, uz_m, ux_th, uy_th, uz_th, + flux_normal_axis, flux_direction)); + } else if (mom_dist_s == "uniform") { + amrex::Real ux_min = 0._rt; + amrex::Real uy_min = 0._rt; + amrex::Real uz_min = 0._rt; + amrex::Real ux_max = 0._rt; + amrex::Real uy_max = 0._rt; + amrex::Real uz_max = 0._rt; + utils::parser::queryWithParser(pp_species_name, "ux_min", ux_min); + utils::parser::queryWithParser(pp_species_name, "uy_min", uy_min); + utils::parser::queryWithParser(pp_species_name, "uz_min", uz_min); + utils::parser::queryWithParser(pp_species_name, "ux_max", ux_max); + utils::parser::queryWithParser(pp_species_name, "uy_max", uy_max); + utils::parser::queryWithParser(pp_species_name, "uz_max", uz_max); + // Construct InjectorMomentum with InjectorMomentumUniform. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumUniform*)nullptr, + ux_min, uy_min, uz_min, ux_max, uy_max, uz_max)); + } else if (mom_dist_s == "maxwell_boltzmann"){ + h_mom_temp = std::make_unique(pp_species_name); + const GetTemperature getTemp(*h_mom_temp); + h_mom_vel = std::make_unique(pp_species_name); + const GetVelocity getVel(*h_mom_vel); + // Construct InjectorMomentum with InjectorMomentumBoltzmann. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumBoltzmann*)nullptr, getTemp, getVel)); + } else if (mom_dist_s == "maxwell_juttner"){ + h_mom_temp = std::make_unique(pp_species_name); + const GetTemperature getTemp(*h_mom_temp); + h_mom_vel = std::make_unique(pp_species_name); + const GetVelocity getVel(*h_mom_vel); + // Construct InjectorMomentum with InjectorMomentumJuttner. + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumJuttner*)nullptr, getTemp, getVel)); + } else if (mom_dist_s == "radial_expansion") { + amrex::Real u_over_r = 0._rt; + utils::parser::queryWithParser(pp_species_name, "u_over_r", u_over_r); + // Construct InjectorMomentum with InjectorMomentumRadialExpansion. + h_inj_mom.reset(new InjectorMomentum + ((InjectorMomentumRadialExpansion*)nullptr, u_over_r)); + } else if (mom_dist_s == "parse_momentum_function") { + std::string str_momentum_function_ux; + std::string str_momentum_function_uy; + std::string str_momentum_function_uz; + utils::parser::Store_parserString(pp_species_name, "momentum_function_ux(x,y,z)", + str_momentum_function_ux); + utils::parser::Store_parserString(pp_species_name, "momentum_function_uy(x,y,z)", + str_momentum_function_uy); + utils::parser::Store_parserString(pp_species_name, "momentum_function_uz(x,y,z)", + str_momentum_function_uz); + // Construct InjectorMomentum with InjectorMomentumParser. + ux_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_ux, {"x","y","z"})); + uy_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_uy, {"x","y","z"})); + uz_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_uz, {"x","y","z"})); + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumParser*)nullptr, + ux_parser->compile<3>(), + uy_parser->compile<3>(), + uz_parser->compile<3>())); + } else { + StringParseAbortMessage("Momentum distribution type", mom_dist_s); + } + } + +} diff --git a/Source/Utils/WarpXMovingWindow.cpp b/Source/Utils/WarpXMovingWindow.cpp index f968af81c13..62fabef2542 100644 --- a/Source/Utils/WarpXMovingWindow.cpp +++ b/Source/Utils/WarpXMovingWindow.cpp @@ -13,6 +13,8 @@ # include "BoundaryConditions/PML_RZ.H" #endif #include "Particles/MultiParticleContainer.H" +#include "Fluids/MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" #include "Utils/TextMsg.H" #include "Utils/WarpXConst.H" #include "Utils/WarpXProfilerWrapper.H" @@ -357,6 +359,18 @@ WarpX::MoveWindow (const int step, bool move_j) } } } + + // Shift values of N, NU for each fluid species + if (do_fluid_species) { + const int n_fluid_species = myfl->nSpecies(); + for (int i=0; iGetFluidContainer(i); + shiftMF( *fl.N[lev], geom[lev], num_shift, dir, lev, do_update_cost ); + shiftMF( *fl.NU[lev][0], geom[lev], num_shift, dir, lev, do_update_cost ); + shiftMF( *fl.NU[lev][1], geom[lev], num_shift, dir, lev, do_update_cost ); + shiftMF( *fl.NU[lev][2], geom[lev], num_shift, dir, lev, do_update_cost ); + } + } } // Loop over species (particles and lasers) @@ -410,6 +424,30 @@ WarpX::MoveWindow (const int step, bool move_j) } } + // Continuously inject fluid species in new cells (by default only on level 0) + const int lev = 0; + // Find box in which to initialize new fluid cells + amrex::Box injection_box = geom[lev].Domain(); + injection_box.surroundingNodes(); // get nodal box + // Restrict box in the direction of the moving window, to only include the new cells + if (moving_window_v > 0._rt) + { + injection_box.setSmall( dir, injection_box.bigEnd(dir) - num_shift_base + 1 ); + } + else if (moving_window_v < 0._rt) + { + injection_box.setBig( dir, injection_box.smallEnd(dir) + num_shift_base - 1 ); + } + // Loop over fluid species, and fill the values of the new cells + if (do_fluid_species) { + const int n_fluid_species = myfl->nSpecies(); + const amrex::Real cur_time = t_new[0]; + for (int i=0; iGetFluidContainer(i); + fl.InitData( lev, injection_box, cur_time ); + } + } + return num_shift_base; } diff --git a/Source/WarpX.H b/Source/WarpX.H index 7466bd92496..034c321ba14 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -23,6 +23,9 @@ #include "Particles/ParticleBoundaryBuffer_fwd.H" #include "Particles/MultiParticleContainer_fwd.H" #include "Particles/WarpXParticleContainer_fwd.H" +#include "Fluids/MultiFluidContainer_fwd.H" +#include "Fluids/WarpXFluidContainer_fwd.H" + #ifdef WARPX_USE_PSATD # ifdef WARPX_DIM_RZ # include "FieldSolver/SpectralSolver/SpectralSolverRZ_fwd.H" @@ -117,6 +120,7 @@ public: void Evolve (int numsteps = -1); MultiParticleContainer& GetPartContainer () { return *mypc; } + MultiFluidContainer& GetFluidContainer () { return *myfl; } MacroscopicProperties& GetMacroscopicProperties () { return *m_macroscopic_properties; } HybridPICModel& GetHybridPICModel () { return *m_hybrid_pic_model; } MultiDiagnostics& GetMultiDiags () {return *multi_diags;} @@ -503,6 +507,7 @@ public: const amrex::MultiFab& getBfield_avg_cp (int lev, int direction) {return *Bfield_avg_cp[lev][direction];} bool DoPML () const {return do_pml;} + bool DoFluidSpecies () const {return do_fluid_species;} #if (defined WARPX_DIM_RZ) && (defined WARPX_USE_PSATD) const PML_RZ* getPMLRZ() {return pml_rz[0].get();} @@ -1327,6 +1332,10 @@ private: std::unique_ptr mypc; std::unique_ptr multi_diags; + // Fluid container + bool do_fluid_species = 0; + std::unique_ptr myfl; + // // Fields: First array for level, second for direction // diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 6857ad85bcf..67b00ab8900 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -30,6 +30,8 @@ #include "FieldSolver/WarpX_FDTD.H" #include "Filter/NCIGodfreyFilter.H" #include "Particles/MultiParticleContainer.H" +#include "Fluids/MultiFluidContainer.H" +#include "Fluids/WarpXFluidContainer.H" #include "Particles/ParticleBoundaryBuffer.H" #include "AcceleratorLattice/AcceleratorLattice.H" #include "Utils/TextMsg.H" @@ -316,6 +318,11 @@ WarpX::WarpX () // Particle Boundary Buffer (i.e., scraped particles on boundary) m_particle_boundary_buffer = std::make_unique(); + // Fluid Container + if (do_fluid_species) { + myfl = std::make_unique(nlevs_max); + } + Efield_aux.resize(nlevs_max); Bfield_aux.resize(nlevs_max); @@ -1019,6 +1026,25 @@ WarpX::ReadParameters () "The number of azimuthal modes (n_rz_azimuthal_modes) must be at least 1"); #endif + // Check whether fluid species will be used + { + const ParmParse pp_fluids("fluids"); + std::vector fluid_species_names = {}; + pp_fluids.queryarr("species_names", fluid_species_names); + if ( fluid_species_names.empty() == false ) do_fluid_species = 1; + if (do_fluid_species) { + WARPX_ALWAYS_ASSERT_WITH_MESSAGE(max_level <= 1, + "Fluid species cannot currently be used with mesh refinement."); + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + electrostatic_solver_id != ElectrostaticSolverAlgo::Relativistic, + "Fluid species cannot currently be used with the relativistic electrostatic solver."); +#ifdef WARPX_DIM_RZ + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( n_rz_azimuthal_modes <= 1, + "Fluid species cannot be used with more than 1 azimuthal mode."); +#endif + } + } + // Set default parameters with hybrid grid (parsed later below) if (grid_type == GridType::Hybrid) { @@ -2222,6 +2248,14 @@ WarpX::AllocLevelMFs (int lev, const BoxArray& ba, const DistributionMapping& dm ); } + // Allocate extra multifabs needed for fluids + if (do_fluid_species) { + myfl->AllocateLevelMFs(lev, ba, dm); + auto & warpx = GetInstance(); + const amrex::Real cur_time = warpx.gett_new(lev); + myfl->InitData(lev, geom[lev].Domain(),cur_time); + } + if (fft_do_time_averaging) { AllocInitMultiFab(Bfield_avg_fp[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[x]"); From bd90d1ee2396570820fc41cd3e77b32638f13c97 Mon Sep 17 00:00:00 2001 From: Avigdor Veksler <124003120+aveksler1@users.noreply.github.com> Date: Fri, 29 Sep 2023 13:38:58 -0700 Subject: [PATCH 035/110] RZ m=0 mode support for particle field diagnostics (#4291) * added particle field functionality for RZ modes for openPMD format * removed print statements and fixed field naming to work with openpmd * remove debug statement * documentation and replaced assert with warning * forgot semicolon --- .../ParticleReductionFunctor.cpp | 2 +- Source/Diagnostics/Diagnostics.cpp | 13 +++++++--- Source/Diagnostics/FullDiagnostics.cpp | 25 +++++++++++++++++-- 3 files changed, 33 insertions(+), 7 deletions(-) diff --git a/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp b/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp index d3932472c3b..203872b1372 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp +++ b/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp @@ -78,7 +78,7 @@ ParticleReductionFunctor::operator() (amrex::MultiFab& mf_dst, const int dcomp, const amrex::ParticleReal x = p.pos(0); const amrex::Real lx = (x - plo[0]) * dxi[0]; ii = static_cast(amrex::Math::floor(lx)); -#if defined(WARPX_DIM_XZ) || defined(WARPX_DIM_3D) +#if defined(WARPX_DIM_XZ) || defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) const amrex::ParticleReal y = p.pos(1); const amrex::Real ly = (y - plo[1]) * dxi[1]; jj = static_cast(amrex::Math::floor(ly)); diff --git a/Source/Diagnostics/Diagnostics.cpp b/Source/Diagnostics/Diagnostics.cpp index 7eb574aa0ef..6e74fc2cfde 100644 --- a/Source/Diagnostics/Diagnostics.cpp +++ b/Source/Diagnostics/Diagnostics.cpp @@ -115,11 +115,15 @@ Diagnostics::BaseReadParameters () if (!pfield_varnames_specified){ m_pfield_varnames = {}; } + #ifdef WARPX_DIM_RZ - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( - m_pfield_varnames.empty(), - "Input error: cannot use particle_fields_to_plot with RZ" - ); + if (pfield_varnames_specified){ + ablastr::warn_manager::WMRecordWarning( + "Diagnostics", + "Particle field diagnostics will output the 0th mode only.", + ablastr::warn_manager::WarnPriority::low + ); + } #endif // Get parser strings for particle fields and generate map of parsers @@ -127,6 +131,7 @@ Diagnostics::BaseReadParameters () std::string filter_parser_str; const amrex::ParmParse pp_diag_pfield(m_diag_name + ".particle_fields"); for (const auto& var : m_pfield_varnames) { + bool do_average = true; pp_diag_pfield.query((var + ".do_average").c_str(), do_average); m_pfield_do_average.push_back(do_average); diff --git a/Source/Diagnostics/FullDiagnostics.cpp b/Source/Diagnostics/FullDiagnostics.cpp index 7abe923b316..d5109277de2 100644 --- a/Source/Diagnostics/FullDiagnostics.cpp +++ b/Source/Diagnostics/FullDiagnostics.cpp @@ -187,6 +187,7 @@ FullDiagnostics::InitializeFieldFunctorsRZopenPMD (int lev) const int ncomp = ncomp_multimodefab; // This function is called multiple times, for different values of `lev` // but the `varnames` need only be updated once. + const bool update_varnames = (lev==0); if (update_varnames) { m_varnames.clear(); @@ -194,9 +195,14 @@ FullDiagnostics::InitializeFieldFunctorsRZopenPMD (int lev) m_varnames.reserve(n_rz); } - // Reser field functors + // Add functors for average particle data for each species + const auto nvar = static_cast(m_varnames_fields.size()); + const auto nspec = static_cast(m_pfield_species.size()); + const auto ntot = static_cast(nvar + m_pfield_varnames.size() * nspec); + + // Reset field functors m_all_field_functors[lev].clear(); - m_all_field_functors[lev].resize(m_varnames_fields.size()); + m_all_field_functors[lev].resize(ntot); // Boolean flag for whether the current density should be deposited before // diagnostic output @@ -322,6 +328,20 @@ FullDiagnostics::InitializeFieldFunctorsRZopenPMD (int lev) "Error: " + m_varnames_fields[comp] + " is not a known field output type in RZ geometry"); } } + + // Generate field functors for every particle field diagnostic for every species in m_pfield_species. + // The names of the diagnostics are output in the `[varname]_[species]` format. + for (int pcomp=0; pcomp(nullptr, + lev, m_crse_ratio, m_pfield_strings[pcomp], m_pfield_species_index[ispec], m_pfield_do_average[pcomp], + m_pfield_dofilter[pcomp], m_pfield_filter_strings[pcomp]); + if (update_varnames) { + AddRZModesToOutputNames(std::string(m_pfield_varnames[pcomp]) + "_" + std::string(m_pfield_species[ispec]), ncomp); + } + } + } + // Sum the number of components in input vector m_all_field_functors // and check that it corresponds to the number of components in m_varnames // and m_mf_output @@ -329,6 +349,7 @@ FullDiagnostics::InitializeFieldFunctorsRZopenPMD (int lev) for (int jj=0; jjnComp(); } + AMREX_ALWAYS_ASSERT( ncomp_from_src == m_varnames.size() ); #else amrex::ignore_unused(lev); From 99159dfb47f955936adcb79ac1d3dae15cddcbd6 Mon Sep 17 00:00:00 2001 From: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Date: Fri, 29 Sep 2023 15:26:14 -0700 Subject: [PATCH 036/110] Fix clang-tidy error [readability-simplify-boolean-expr] (#4328) --- Source/WarpX.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 67b00ab8900..348b3125f57 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -1031,7 +1031,7 @@ WarpX::ReadParameters () const ParmParse pp_fluids("fluids"); std::vector fluid_species_names = {}; pp_fluids.queryarr("species_names", fluid_species_names); - if ( fluid_species_names.empty() == false ) do_fluid_species = 1; + if (!fluid_species_names.empty()) do_fluid_species = 1; if (do_fluid_species) { WARPX_ALWAYS_ASSERT_WITH_MESSAGE(max_level <= 1, "Fluid species cannot currently be used with mesh refinement."); From 4cc78e64641a6d6663febfa80576949c97727ec3 Mon Sep 17 00:00:00 2001 From: Grant Johnson <69021085+johnson452@users.noreply.github.com> Date: Sat, 30 Sep 2023 03:28:18 -0700 Subject: [PATCH 037/110] Clean-up docs for PR3991, fluids (#4330) --- Docs/source/theory/Fluid_Loop.png | Bin 300102 -> 0 bytes Docs/source/theory/cold_fluid_model.rst | 6 +++--- Docs/source/usage/parameters.rst | 4 +--- 3 files changed, 4 insertions(+), 6 deletions(-) delete mode 100644 Docs/source/theory/Fluid_Loop.png diff --git a/Docs/source/theory/Fluid_Loop.png b/Docs/source/theory/Fluid_Loop.png deleted file mode 100644 index 8d47cd606c5d8ee945d79fea180e6f9ec54a3f48..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 300102 zcmb4L1z416*M^}%L?uKJ5Jegkqyz~?x^w6hX+cuj5d=XArMsmWa*!B898pj@hCx!0 z?(YAcLD}8!``q3C+G}Bid7tMzC+>5fbDkksRau6Vn1&b&3yV}v_P#n67U3-{EPNh9 zeBdX=o>E4@H*9BhnR{5p?U!ePe*{?Q$URh2!eR$r6Jp_EU%(BRqu&{!xv2cIBM-TXpekqUsa{3)74f~H3(?Gx8O?WE}=htg|9`tiO z+@)>6HzG$_U1uyT-bVEQ*sKrqhOn?Cu;lLF)qH}zFih}{qHnlmHT?xeZCY-w2{ac; zi{RcWrVf=`fZsRPr!Y4QzD~n0t@MhiG!|L=jEnUnH`V>Sch8kR%|)7wjQTYjJj_W- zhP&_kcC24^|50&!`j&Ihd~&%Z$^#eooCFs3$(J&c4RDtsrC%I*g9$ z3xB-fKVQjG&cr21;)4jUJjFWs@)+VPFU*)?pxHXWPN(>2hqHoCIk9kKHH4hfPo3%X zZGSv)@`~>@R^+9GnFqHte~FWZ3tEp!3*GGvL|ELa;8(D)yVtyc85dqRO0^> z$m$9<^DVy4&ktCC?I3}NE37p}Jud0e8K&e}B6KB*$8t#9zJ!RBfi+Cq+g#aOBNYb1 z#r~;+|6EWa?JQp07u6t^Q0{BL&ZtF^yYV3Ko@C{yrB_F=egN@4iXOUCV|KHtNvwsn z?2Z39!e6`p)XJ}aN^p_fj8Bj0IO`+%OXoH!X%x`-+z+{^*(*LjJ{aC3I4~M=;6O12 zU8GD+EdPp2aQEDQ3H9sYKm8(C2GZ7Rt-qP}dg4szG!bHy{ZY=lh^xOpR|se;lppjb z)9K>j=bDI1>8Cdt-8SX17D*ClXmtS_Xg)SYoDHeGY?txT3UT};80OimBlYL!ejN^T zR*C1hfVvp1&ibDoJ%YSsRNQ_FClP7F6E}HoVQAVc1U~&iYD1UAG+&sA0@_v1+KYqv9_o`VT#& zsR6d^%(aYp%=pWeUl~iD2399c(1h3qn%%GmvwIie;1viUaf8U%T}{INIK{6APEO00 z5BMw1<`-V>Ut6VB<1J`WjejQfsdONxw_B5}Zq}h|WadB6Uu8q+hWt6q{$w=?EXY-~ z!f>gi&YwEyx2u1VuUG?uH;;+tPRoUp2ziW%QNTI-AI&N6$8Zt5QuLbV| z!L@pr1zxFhB0dB%^tCro+FyUM_5XT&{Q=@4r2RjoZGV$hLZ~GNgko){4MGh;>0ujid=@&Pl_y2fyJ@@%AC4**CHZmrd zR3-Pw1mX7ocpBdO-hrZp4Q8Rn}VSWz5Yzk~0;2G~;og@p5?8AEQc!Q9lF3G(+9q-Kx*|FRO~4Y7;5tqiNOs@>PDKp8Cu(%&b6zv@X<9pcf4 z>Iz}tpc2U&XX&ffD-HU$QOKGnVOi;xH1Pmo#=@_tk!;L?vLX8|mj3m$X;~n{UQ}?1 z2F1(d-kaL4ZvPtiu3wzVr&7^Q%GX)Q3Ot`*9`>)rt#0(TSye<_3PC8^#AUC6Ika?YU)y72tvvR^RM6SM_Gj=tKVuIPBlTX_Qo2tR7s3XCBb)cK*cNzXldN zkXP=MT!=0U{};_cbIPIL^P3zjKUtTxh+s_ANJtVQ8@6AvZ)qBswddFeZ3o#N8(_4n z`l|66yEkOQG(`7!bV4oq@Og$(z?I+SfnbpC@mQQr{QaN&BsX6IOeY!_benExrM{!cBds*cG=f7(-pzL`49jdUhPzp3aVQc5d_{UIEB zC}~k!apc&0|F5*-zhD|t2!gFOfkSi|erV-?qu9)I?!T*g7Ep;*_EWg@^j$>ZHf%ss zV4j0rZ}58lDR2Faa1t`-zo>uNy8>08YX8xTmWYx6`2&)#(FV~}H=(3kxQ^i5t{G%^uQwz3UD)%*KjrdL`gvB! z8cAW1~N1n6+Ko92>*69;zUwZ$)Jj{>IrcJ7A7~*W_uK^{8rl znWC3^JV8OHDzN*;xcJP-K$rh76A8*GP`d z00xyvX*@vHO7gV%*uh$pD`f=(esm!!VLatA=aEmGn)fz#2BNIzL=j;dsw#suu!Ko>KBpnk0 zl(cq~=Cnw;NVjG^)u2Y-R!RZ^%<6jGF~d_tfoE}qDr@M_>9~gGQnJ+V6n^5;Y3;ba z*+)lN)OWHJ_LK5>o0@B{S04IjhWRwdqaCF;vypplV<5zO`@Q3y!`$Pi8<&2a>%Zb3 z^9zXHMIl|fGvaXyLn*mV^Uv{}Q!+>Friv)o+nCYr?Esq3Z;qtsSqdKCXwxhaxwogs%E;S_XWq6|={s1lF2wqqZ z)Mi|LH5?Ehy1OS7UFya9+kVr@PC&w*VZ_#-xbq|~W@|UEb)zlK^>t>E zZmT}GBrOG692zAPCDIvkz%lJ29+9`%J-hUeOH!5$h}LCL-?^`_e;LZl!A3>AUeL&g zsBl|i?Zw(f11k#vtR?dW8iF(f5RUZ>3 zxY80pv@>HcGv$pGx{<2=?T38}OfAy?fDJkjvHX=hvz;ce%Ra%>vZ`Hk6b4FjUo>8! zoALtW5~(|`UIQ1)Itcv}wX&zsQQef`2xNF7Rb|QM*C9Y3G>6F;Qn;E*tjU>PPIM;wKL)~@+h%-`=E3*m3ay@rj>ghzghY|UZKX`Q+e-j z{3d3H-+!Uta;&roU~s%{uYc*prw$1XGK-mqJDRPx^A&^kE?p4)d!MqR!q&HEJTT#~ zMo%32Xixe|o7YStDVD&BHPE0QZfGiFm&V^72yK2f|D2o3wiG!jK2m_n2BIulV*7Me7iy8=wtLUjSQyJVfAlhw^=7atoVr4h#Kt< zQxg4N!5CQ6G%zuuTa`l0NEXf<`BE-lA~;zmN5YT)2yV9F%P``1Z zjQx5bgTZzC)MU-qe3+BfTok&0O}1tk&~)*>8ZXx0Yr5tFQUOjXULudGFD;h7Z;5Hi zhfuzgrhKz>JYM1R(mrbKAC6NYZQ`MoP)zNsLpkiLyt_q1eBwIaiCwxorP-yANeP$z zG7suQ6&gSNy}4}eHZrr;Wl{U8QVt(U?26i?mv5Xz>jFq&jxdrk^O!1NdlY>1-eQ@E z;P(KGwxpFFgoA%=1t)(7D7=hwtX-*vfSBk^0wr1_#aX3~l?a!`vku(8es0eFV+`@X z#A{_M$)O<=j{_pBEQv179;4(0N+Eesg*|u1BsZt8TPb0GU;XrEH?GZXx*%30^Jcc} zU2pRWV^CC;9nw&cKGQ%t?xPPk9ag7%<@Ub>_~T~@`X)xC$JY@nIKVnOQ#U%lDZZjd zpVbr6e&LB4&z{5mCw{NiYWd+5SbuLcd#ZQpTmcOCrK%baH9p5tS7p}A{fO*)~0 zm}#IKKb)sEO?&nCOeD<{NIyANn7T2y=0Q+>YYn4Ba5 zCvwpYl3_2Np4r)NEgw+uo??|mv(={(&ct5|OvBX|=Bba$TWNEJ+SCF$0XsIX2Td*?n+o1Q;r1Tl-T;kIjDn--?oGxA7etzS6Z zgQW27_%XQjrN3$JTKnV4NI)tj=1$(n>B;V}K(Ug~fgO&2cp5v$&3m@QV%eWbB{{rw z!q5z>!VuWTUN8#}<6(OgI)=kAIpXBY!{X8&VBxjTCjixmdlp=Wx@%g%dq&l}9diyV zN60taz+)w&lGkVLR>pbnoSp@OqWGj_r#c{A;v{LGa>Wx6X&0pU@YA*lhQXxW#uc`T zJ68MF$n#Aksw(zcDq1T*^t=Zh_qP|12YlLkvIx!Fab-#d${k>vPNqNzshDQqACwBH z)9T&NatQPe$z?VW^2MTO+VkGG^e|^QCQ8Z(C!wSM9$f{SaFMxGcH+u}B%Jxl0~E2u zEYdJ&=TYfvKDYA{X?OABK6IwB#F(DOQYcHKEi27(67N-98fic`q|WS@Flkjn8WOFw zq=je?Thr%Qek>!Vd*ba(L}%194)?pn*fgxmDFwk5eGnl3bf$rdQa+S)HdbF9U#!lv zJ6#{BV{}YVFOH*$#7G8qjERc6(@2fPU-!Lq0)BHIn5t_A#QS3qyZ6OYQ3FrC#~Lzz z#Tp~&VctJa;z*PP{h~pKH=>qObhHl7JOc=#ngdGsbhFoXIypISU%=FK@U~QZT}1G! zdw)b^(YYN}X9_zr$^Orzb)FfNpEQBKo)Zx;CpRdoc!_pq)t8`_JdN1f!UAd@1suM= z92pD54e^NDeKsK+cCP_E5|0rC7}p=1A8~Z`N-Ljl!o_F4Ar$}Vj}{zjF4FhCs6?BVPdx1g41D$_t8SxtX`*!M~gu!hG(Na9-0kekJW>s1@jZ;0r7e zKRYnTrV-_F!ITQXcw%dLI5Gr>uIQ}B$lX&lvP!3Q&NCZno71wkv%ghqz4}zbp6E;b zCld}{CIV(2rrHQJw?8QfRB1s1dFh?(q;+l?0_~J=i8ML%ggc8R#S~q6D=|9;eB8He ziDTjDFuJjtjWG$jlBc-Vsr%em%V1D#R&&a5<`WAbk&TW7`K3V5#kz8hTBKL*G*@Cu zvR?R^)9}H@W#>}0QpX{uVfT?10!qp+fX;AYPtsc^31a}U@}!MPBD}0qp?*EviS7#Y+RsQn!6<+(xjY&=fs${ zyp6<_ZO`B=UnGN9tf&ZE5Qf$KME^OJM&Bx>l+K%-46k{stB$1Q!U5zsi@r`kz!J=j zDWNLl$0+*rP-^SkQ^QLbGXtmtQ75`n7mRam@(m37T7*GUsV>~Kb@TXP{#D9a!*wS)Fb_5 z#Mb|f*MgTnigZo$DcWPBRGvm(YPD2o!#GMG`8(Ah<-|`W@a_sTW?t%(>pN*?AmbTX z7Ll-iHILZaXy1Ua6KDCZe;Hv>{1ZO{Zh_cY1XhfCL~3){Y7%?|@_-gF&yeh!xsC3g zK9z{a70|Agylg*IWd?P_q%QPNt8o{!$ZrVIjOuyRVjK^`PMB41^nQdfsczvBrRQhH zU$GCU2LPJGcH4&ikN)sSW+r#yDS$9aa{;mqYw~q24CZPYHyWk7ct9Q{rFrYOCZf*> zY?6&<$F*lpX=vMRoPXfKG0YPNWS63ICGLdeC+Pxn4q&dbKP+f{^+v%hzUD#8SA?4A zq^vbHJ%-WFbt)m%#iC`ab?hn8iBqFHk-X~04!nJ~ENM;I1#>D&3diDB)h1s@);|S# zK6k{O_}IQn`Mzc|SfEfYlTnlFC$8N|2wJy3XHF3-d+AHKys^1*eAHE@lLnvac}R+I z$YCPCUu##r_Fij9B*p zLmkkK=8>Y~ecS|ewp4P?zr+eG+1Uw#^9v8W`KA$lJQPS};)qe$CbMI;@Igvu&pwC4 zHOzvl4c)8bz=m`YFDp;!|L>wd*&K-;kNZBd29`NOHPe+ zx%RLT2)@6h^C-2k1CTuf^gtJ{^=Csvq7v86HIRXh;5hkOIKYBfBPHfIm|I=$B>bt~ zO?!l^Zp5EHnoiLFm3jtHZf_{ScjkU*cB!Ft1;K^bN5LN7^Netb6$m5{@ji)hRe#WO zzRG~`UL|DB?IgIOFN{%=N5AB@R_brjOKMC$XFB0U_yYz7%GZ zl3925G~fRTnJ}dqEBXdw@9YC~dqSdz&@S)wqaW0Z#!-TZQyx;uyizT^AtRHctvv!c zp*{COM~)6Ag?W<>DTt6icZ-1~m=3~{c4}REp59aiY#9VtsWn|b5u@lSDif_hCz2j9 zOt|sma(CTraJ;<3Ipyl~%YT0=5A$}kNX>+R_Dq$p3DPDJ(lyoQ6$9+c{4pS*Jz7G6b{sh7=}*iz zB@mo%@${ojo%EPzPtMT`X{?(2?h)q8omZENNfoD7MH}Jgwv6a;^tqhZchT6QbnRv@ zkGi&n<5&cy#8L%=26C&&FN=!F{q(#4&=mTgs@rI%*w^@kg${|g=Vzd=&$_(Y5}r~z z)vpAc&>I?dG}1NL7ggndT6O%C;g>XX=rC~gBff-QwUZBIDI#q{=EHpgXQSpockh?rNsaCCegQDC|tX>7Ox-Ksm?b(aS1?f)SWde{VDxu>}Q z1_R&$R(6muniQpnvLxGV-ygd@ z9s-PBGF2W67+wCVyNWlfK!E@)MmL$X{E0!>eaJ3(lT= z4b)-ipI#&RmeKT8_L7N6Hp81YGC%3*f2hTk1u%4th=mI^+R#t1_p4@QTUZ%1MyOAH z4;b(i$-6|FI})%#BYzu{XlCK+oR3a^=$I6-%JR9K34i}u=4ZVa%I#QW{!}!oD)IMA zq?r$IM1^6JBj3x$t}sZL#)$u^3anfpvHXp(6az|^X42&w_^)H;m_Q06satF17>PCZ z)m_PLpj+WQ-9EycXg9$r3CxiB!WzTHk7hzI-|qHW ztQ@Fg(|9{+1~r1$RgHMI>6<|C^d>?$if8Ogn(#-6ISM<4{4JS&s!eqOZN2#(nQ653 z0?Boxuc1&p`2|seL#OWe^@>TsErUqj&b#te4PxO+pII;y@3ea$@q-orDg>4Olb*Lq z98W%!w+pw4OeZ-L$8czlB|6lv!bUu8mlvx>xix$DJN%Bm+t;mr#A{OnJYr76WNTq? zgwFlj_^|i-*O5pSv00zZK5#ri*-XYu&v|G=z57Hk%a0|tcnSnU8G^ z#dw@~7@N@Rbi-P=YPmD>#fh*_Dr}zjXo2|V!XDj*4>>dp&kt2Wt2)H?p8y(nS0lev zIO=xnxicQ*oZ3f2&~i$a6ahamAi$`zlF20UQuq6>t*muowtDuzVC6D4P|T3ibicpb z8ppyF>4ITUt%RJVA1;i@F!)R!{t#?nd(x<``0(B8M_PEp@!@j2Ii2s$cnUo5y|%?n zZ(d|MXFMR99|*ShURh*uauL2gGBVSZxsGoF4MW;O;dSfnA|12>c5&{jh>KoBk816W z;#ZUhiVdCZpZ6e7sx?6C>|Bku%CAAPsX3;?LN3NcANGU%vgDc*gEW+QycPr z;IP{eSJgfL3VFOV4X10~Lx>kpp|GrOk_b7Ub_Xx)tCp^j@6ML$^hg2iVXx+LD~IEPxVhzyn3Li!6V^&AoH} zSBF1M4>ahTn5Vrggt>4iQBHW~@tF;-Dfl0C&O^$$w-E6O>Vx{QJ14}S{fY?(^D8}4x z@5p2MXn(nbqv68^g(rvm+Z_t#Ii8?VpY3FHc(e#RcHSlMpmnHw%qeOMBG)dHS3}>) zL%V+~S7+%rg0!DLNlf@9^8ij~@u!s9L-Oq^Bw3vTrndJ!#Q5#6@OTK(cc_$d`4I^C zHKT)*WQ4}TgFxN#hf^`p#|P^j{>*j$WMar#Kv2Cce^PpR{0rVx*GvkYBj@EZOgluk zzZ}>fTN!RC+_-6nEcrGsp25Z7v(eSXG<_t`Q8YR1T@D{#_cI-`uOb!LJ?(hrqg#&o zYtD`s*@=+;F;a9F6d$5FEc#MJMrm^PyMzoMM5i{zzTT=KVw9~uOEy;&yIp^(<1<`% zQW24LF?}eYX_)f(?aB48+-HsURl_mvn))?|tsKxV3gyXW#FNJz46ks`qihO7datwXg)8zbeR%u0h8j!BWFQ#{cF*@+%fV%@yzGn zXi78bON3MD8o!vn=tkthq-FecU3=HrB+{E+_1mqf&MmMt^R2A02sBBE(inSw(P-sk zIK!yZ3*Z%U2#6K%B2Yj0-eNJllB!|(ndMT$w>s(Md5V&z{=~=;q;4234EUu{f>6wecJk8haR`ki8{B^ z`0mZJEca@tY;spmzR{sfyu7|SVC)qxwr9n8L$;*UZFywa3jk5Lir6lbY?u%FT_ZHM zh36HjaOazWv>YFIkqK?eltV=+N!vr4!@ZEOo~^~{>1eP~u*FxA3T}vuWZYa^#V%^&5hQqC)YN3Z;s5@HSmc9*^gUbzogde{% z>N(HVVYF z$$Alf&-pOXFJ*_%^YMkmIw2BF)w4v}oEVh8B;D#!(9Sm0yt=*?xkgo>TqVQ~+GHCe z+Ya2jwsD95atm4Cx)un8AWCw4Pvp_iw$N~VRE_U@w4OShJCALie|KVZ;O*zDV&s)4|(P45oBWmPD_sX`rg%R#mjX$Lv@mVL>&? zQM2t$6uG6B9uZ@aNWD?Vg!_CySLG3VLE#&{D(VKk`g(N&>Q-vOM+W(?4UhFMUD2#@ zNsN96?p@>0>N9bQb06`rd@HBux6{m^zIuGL-LS7`%Cvds?O`F_tlsPEm5WZpnd)S+ z(#`-UxGNs^i>CneUYe@JqO|8I=PP{*dFu_6`3pzdkDr;Ls$~@zpu%9bEiFLnHfXKu zRq!r89%28Uq=%S9-4d`4Kt@GWg9rbV&{mofo+u$t|ZH4IRv(6h`GF4dw z76KzrzCSa32Qc`xxgwzWs243PlRGO{0Gg@9#~V5WsC#NE)>sVcGoNT=I?mwyh3Dw2 z;OIib_K4RJkdI&MS2i)<_HW20z^KTB^1*Gjc4<>Ogbd z0rlmh&U8s1IB{m~-u^=_tKXbErs}H?Kai{J#87cr zk}FkQ_EGTaB8?4CemU$uD%$^WgSyM7Ad>3naM|zTuu+B2PA% zK4Q&~gtOmkYiX9#*!_liWQJ1g$iZ)4e{_ilw&!oeF-b;$M`%^2I5pF0G}8>{YPtu| z=1603{v%o>GKo3UActvjJK`4d$GVwk>!d$czSvuJm(usWvv`>z@+b*4sQmh=L>}%F zRwMS$t*~Z~flot*a2Te3zxkpnV(pFz@XG;k6GV!* z5U5y#r6Kz$Tj=6YD{VGm9}s3J{s-HsEBQA_xZkBU0KT@5E~h5gxoGVx)LxA+4y_2we|L3ryL>l6}JnR@ss&Lr2Z46iw z|Dy$29W%$<-Bu*d`(J+xpz8V|iiVoO(CQ2jf&`s{6dE-)W$ zCwp%iL>Cilxj5LonWvG1FQqOHSZL$}!D3F5Q;2i0Cv7Izrb7#H3*w^johspbbJ&uX zaPZH3DxiJT!VE@oTz*rp^$2Q37^{SWfwcmRMb2OD*H)BecYxx4zF?RA9H1tXm8Ox3 zoWqnblubdMaDUKxhxn)#!hIKcpVX`Jh3Mof{91uGA9bvFB|F=J@+(p27yyE^ArFo! zf=1j;L)l)?NNq3DRLb3jy-~20tX!h3^dude zV;A80YE*qbAd6D@Xs1OSf3Uh^z}Qdp@l}7NZ;zkjoE5s_mJ;b3#_kdBv>98+e8Gg* z|55c{Eb>knZ4qQrg$RvnC;!c2pS~dVqK^@7NBEVdNyC~|IlxD2NjP8{Onf<5WQh6o zI_HsmuaTp-%{jw~%LR`HgoHtx*8uv>yoci*A86C|Y(B&xcx#OBz+T^r^q$Zt45*6w zdPk5UIOA&s4Bn0kx?V!o&}W3xLohC5BUWU{+3`O5X98#MtTf)!a^7v&>w}uLR03`C}iEi)L)vNd?_&V!r3Hh~n4Y z2HclRlYh1HT+n4FSsT&VaHS{@pwLNn&HP^OlQ{PJI&zMtYmcM&omotNDQ^&i?~}qJ zAax69bF(!SFJ@^m`)=2cUZ?C(aNg~k+l&@`dez+xWt|h}ym1+I1aQIeZ@$7&b~&;|a&?V^y5o^p^-}++v%JIY_$}Uf55vkd=R|xzM@C%-d%m znH^}!e!Hp@A5|w++VL&xTYv1{GEqqL)mhAk2M9mlEva|k`2M*%ifd!kgR(*8S6Avj z$P<||@Fs>~`$wX~(tGD*;XyOu-SHL%zb`6r1!g(yWa@apI8&4zf41O*Klye_895d>t7Oh=_ z9MvD`M+)De?cr_XByS_R@aChfhh9-*?d;pcx?%S!DXd6;ZQhj)_5hu?+8k8&V@Jjld$(sBsm?{*+Y zFL2N*p<4s)Ep)5PR1`HS8H>pJN>`w6{7}Ysndfa# zVW#oWQ)7;`*xt*I*&9t9Sum;v=#xi+RV4VUFL(3yM%|arFm3HsKHY-rQqUT!L0G8# zXy)44_Os~D$7#R3^C`%O|81S_)N7x8ZL#zU5Udq)i1&4hzJ33$#i)86c$E>s6B0wk z9SQTvZ>6Md*Lq&DmBI$&X%!5>8eM^Q^nCItSn>cq7}EiKKqU2%#_-Z(7u-2X;GRIj z%us*@bw0fgBIew2i}=xN_bdhOY9_Z-<-M39I9T*R9%ujKGP*g23ModVrIvlOA@Lkb z1#{O#ZrOxfGI4k>uhtnekMO{VIizFn!#A)0aO6-OdfxE6ElUs3a1?N=!;vn@qN^lSG=WBabYH2$#H4K;319JYFC%wG&u^k-N7v?RJk zOr@X9X1OI~jr z&@D^T=ltW^V%mAUBNhLU!|-oO6=2%Ojr=#?ii><{o}Lf@*bLSx)_Z+a;tO* zNB9aXdDN_VS(EXb^ppKTGuljVNx~J?Ix&@N(?xsJi>9uO-ejZ>#@dHVT{yTlXs#W) zTeT%0I@cw`KsxH~<0T<(kE*C^Hhzqt^sv~_7#}K|8c%cB+S~i#U{u|3vr5x3n{tJ)v|e;ZH=uv^ zd}oSB-oU_OMnK_Ynh3Ox&RN}uoTt%T@SFURl|1?ar+6*Sb7lt?W0O#w-H;=sS{ip# z;yHunE11`Pp3pCBKx-y^zS z2Crp`9FhcNl8p}XKlyzZE>-7K11QJLs1;5zWEH7kV1B%QS$kkj|$ue_% zwW;5AF7lEdq<>5c;LNEkaqHTL>#%1v=e~tUip4^GFV13vD!CdT;2#N%>OmNMcbbd= zH(6hTHwTiwtuA7N`M1di@)RgSHbD2q2^h1W9QSO-ESV#$?L7UA_8nCdmFP{ z78$g#`!&%oKcbtK}++Nm-G$%`!F*Zl+8K&)zX0orp;-fp&1^^m9T;|55L~;1N3Y&q)3)> zDmQ`LPg!V-=eq?U_+A;%V0@TAMAaZ~Y#fi@8+h-UOxQ-Oo=sky#STsN*fWv~;i`ah z!;K`>z5to+c9;0PtyspQ)ONw+@Q^LYA@fBvQ_2IF5^#%U--+Sr*2!e_1)*K`p~qG0 zrX?5q0~X|X`2&y_h1D$QRD*8XD~^3Ri${g;U=u$=Zm^%Ur1P)c)DEhQqr%Uzh`?Nm z>lDBi^ZbOfKS)q$4C1qBWj~>ih@Q)01uq|Dt>&xWerOw zB1x$qOpSdg0*qteRgEA4zdN@Le0QgUFL{>4#HGaKPECu7)(2ZS0JT7Ah2wj4rw-`w z1w}dyp-VzeF1B%?vGMBqMH*+-@e!}(&`sS@MU^`;ca!;`R~jl)6Cc&&4HW5D>1@LG zz5&C7QUU&^k3$a=?agMpaBJH$?m{5#9bq z&JjCBV_jgJjn20UNhMvut#p-weJ9Zu@9wObsn(A)uAR-0WQP4Of|4UvsblaoA9 zR1{IURFFfZP%D$IedXlF351u}h0}b9@8mN_5^ox}3eeEXCbP3*aP__KavkF zex2A~(OcEZg`o4SDn7OsUIhHnRW@WpGnmN&ecKV3R0=?5nyY|8Mte^PNvX8`vwM8# z08de}LIPtf(W3XAZ+yC>E_wi+L|dSxzL4&E6Z@)cxyk^}D0*thEFI7`1QRu(ziA@D zH&~s@_xe3MGR|-u=pU^-{F)FA&QgfwPSJk&NkIX%yV0Wp&q?uQkC~!zepf;#LSuLJ zzDtIzsx-&#C+@?p;F6)WYM>qJu?O>N^R*SISd%cTgzf0~%bEA))276HQn_OkVKrdf zty7M|Uqo8!4|&d{2i0A=^)MiDqaq`)YQV^AD*O6k1p?YV09HS4dkq6BKu@=ZlKoFW zL)0v&b6(xdx({ep&GS<{S9I>RWvSBw8}jSb%^cc%%M4j!qCB}rzzP8%L;k5OIfkJ~ z5xrx+bw^WEfuj9tr}A?ucM5NPkI`RKrd)CC{cS#b-e!p@SGA&ER-KZbn) z50w8{R`jI}-zS)kYDI;YLbw}n(q*K1q{DPqFu8V~?LhGeFrw>K7q58?OdPH&KP?Fw zM&Kser4LYj$S-_15t8DQI6(PkPxQ%(i9`MF4`;2mscg|)HvkHH>Dj{o)_}C%RTPoY zZqz1tpbvut9Jhw! zE5FC;aDXrqW+K~L)kx?nZeR9H=>hf6zm3A0{9v*QJ8nU zMa*YWJ~9-EA4<~F3we=LWT8ZIqJZR?lXU^xnae^qS>Mb$BTGz69oqE~=3%#)w@LRZ zq?uJx7DU&-+$vpw-I*K#I^4Ii8BDzwV${j&p9Bg zfB0t4uzkMZ%7b8%EJFfYsJ+9>FJy89_0hS#@T;>1z0Lel&>0U{tE@WGhQn*$W=OKz zgMC^we^K-;=3@=mxOyyOp6|M9sPZs3ik?c;Jhj!r_mXPL+Q^@*P)~D)cs_3cX{AB5 zAKb_k?y7YCZcGqAR3iFB-*oBGT)*bjmqQXKo6-+w6346~?eW=)v);Wk>$sh|#jEcX zTw3J12b)NAo}k^kToBiwTzoF;MG>oMaPP62l!y%ZkNaMxW$>LHeS}Sbp3t(u zl354*@C;p%m7hn2^CR&VV-DZ6os5aYkPZ$Qb8oYU?bmk+MPb9@3)8lat&NML-zOW2 z-)dgLTn}jEI_ILJSO?SRQf=|%-EFoWlS*73&JlXadIo=Ma36dDIudbkqnOn*_L^S!&Oe%)De zBS5)Ey8!d9`S?v;mbPo%+@qMF?}v!@yF-lx2K0dU8Px6*^tBI^OUx}Qi)X2{g+9el zrP!n(1Zhi{!PsU?m*LutYTZ+E(O>-zi1yhQ28>uoQoK;d+V3Rw57>Qts%6wuzGZLq-_P&uCbU>9p!|e-?b+>LuiYOc?%SHbSbAHl7h%g0py4XJ3>VfcO6;YRJ z+Yon0dOJ|?K2$%OZVC@?k9?zmKXnVMVFi`x++uzsY*=QM8__|j@2t2kBH2aXU3J|> zd#ZyXa&KilAaHSf(>g>T{%W4GdVI|Ls&by%u**j%s>I`qX7wQaitRm^&Hyz2R zXpVFT29Z!z8!Ai>*BE`Nt_tQJ;6U}f~SnNLaKWl;lR;O>LMak^Sl=QBM~$zf|DX zz_Z~V_03>c08)paxn|-mq5D^`jlC$Vt`9yy=ctc?&q~#7(8>&BHK~20;KOTSLRTLa z1YfNmrhT7|+79sqZc_xpMBNsUaaF^TfoCpV;6ml2GlgEC8asde)5EYkSJ%ACePjDK zF;^4inN3Eee}0x+l?dC5uqyAn1ex&KCmZ-vpYNi_1zfxKgE*6-Ymu!r;CAytewb_E z$PoKEOAg=MCod-+g~aqHnD{nmRKT_zlyIEqkONEI%=hm%z<6XWMpf^qJoZ2ono?5M zsdf4G+hqr+6X*|4ct|A%lqjDM9r6&-S>-|)Q>a0a8I?~)?9b?Sz>~{!D8I6{;btvr zax3I+uE$uMiISswB6Rz9Y)#;iEyuHy51PwgSpnwuPmKKzT@@5ySN)5=L{J%U+yd$W zfswqNlmzX*eIRp_gU=uV8;W{WO6hgtw0On{*aj)l1T?@^hp>2O@U>sDZ7V>_)XmWb;#}XxM^`ZvzZ! zjD+d3$K>l2!*VKM8oOy3JsJ40J!y6@guBZ`90%RYR4Cp8u$Lw`$d;8%O~m};t6=7- zEwC=)sH$!e=)mW~w&SQyQoa@XI;5V1j1qb@4w=9eP+hGcaO{b`c-*G_cu_;x(Bz({ zHutpIq2TciYCQG|Mz2+Y-mmKX_V*-I2ei#fD2W5vQTGGXpFgshF%F51&2I{D`=N!R z41tL`Iw1P-^^K0$5Gh6WSJFprL~j`*HZQD|F>_@-gUvlXj4pESm%r53x9(BJSe(D} z%KHn}))5uW{ zmMID9I0l-VcZ#k?#G!kOgE2S41MiV}v#2N^u{1FoJs`bk;0W37R=9x(F?Q|KG76^` zHJcp|0NT}D0_ni5FRB&T=!h@BYY;H({p5Azk$5YSfjIOws}InO$4%r)-Ab!*p58FN zjh;@%l#P>ulO3zgs)i=aTSAQdV;({umVo)*JyTkH0!(+zr7CFdadIg<(_VT(IX~La zu$}-T7Et?<6{#H3x7eBPA3H@!X}?&WIW{y0M7DaoXo}|8y*`IU6o8)HMlfEpK?h{e!iqpL5@~v&-I| zDc$#C_VBHo#f^qXESidPdUt2TIZ*k8_M~0~ka>#5P10<_+Pv2?6!uP%Z1+S{-Dw+? zggsnP1sXt@V6)4~C(W_5HxFeIn=}HvK_B(G#&mS%7}(q# z0oJ##)X@_fFM!ssxqW*q(a?UAqUWh=hQYAT3CTNT+l+DAFa3l$1z= zD5aE0cO%{1jdXW+OE;V~;QN02oc-m>1)q;Ty0pM; zh=01=YpVemZ&GSA#4;?5H4w^+-@E?f&DZR=Q`kvAGoo%uCY9w7s4fWuWzweAHupFCo%-N}vSMOM8+@Nt)vU;nVTBhiCB~mj~_vTnR z+T6ihtUUS15d=myfu&i$&I#Gzh7w5$o|v-d-cox{?vLtwEsRphaw0%9VSZ+$DL@3# zI-S=AlsF_=I&sH#J22pOP`5C-5l;68h$`jYTa;tTZ-OB(0C9BSwG6X#l#AsW=ALtZ z@an?r69*_5a%;~wKD=Hy28*~+12`}Rubd*(sK&_PqS4S$R9nsLa_eTWuYaQ+=Z|Qu z&OZW4?1SzxpaU!#IN+GfMwR*!)t$DytlnE}a{?1rXI-Z-V>ZJlC{^2rHqJ=ibywS# z1%hnU6rGz0Gp-lgDwirE$^7+cKWM7L%G0Sshq6rGe4Wk_bU+LOJE;b+|eRZs^ zedJFa^GwsOY3G4zu7GLN@5uQSiL;yNYS8REx&b0XlefuuE_h-ixg_chsQ_QqAb+S% zz6kIcx!><4iZ1p6G_*3lw@nm%xZy83ldQ(o`adn7K}R_U^M;l8Ldo@YF?{%!eLyo> zNyiG=N;Vl(+<_Hw@=*)WLCvUB_)b*z7CSzH?y-Z(-_G1g9+a6QiQ&KXM$+vUh zlkc)7;9fnUzLwYv^h2eRRqT4r{;TW5a5vujv+7Bl3G1B@v(LpW$&Q?ss=Y6lr(iZ| zxXUS$;%8~{G)^4t^V&rcS|47{&&a!dQEEnQ+Y>sH4t3|Rb*ErFbxer2cbm4T+TKSZ z-&$86sjWqy#z5s-Bi2Us#gts3TyOX^Q*fC4b8@ul`tmcPlVip@;8E}>xdn@0?;U;M zN^HIPQT*I<77%k)Y5);teDuMm67DvCwiGY z&vshVwmy9rZCdP<5kw{m*_bs`7@6#A4|pP+CswXnfRC}-1`drRr&!*aS|4*G;2CWCy(B#;S zaIZvB-id8aR{xNZQMhthosu*+YQOD;pX@JX7=)!QRz$Xj=LEfjuEnSg*X`ot(Cpfe zv&rT+$^^1oZPwu+RuSFetiW=Ba!ewQ1!*PE8(>UPi(ffo&ZCJxHS9WSaVw?tmi18*^Wo2}7(hcb(@<0-zpN++6$ zD?8kl20u1c{iGy^D^W)3p-u;>4Q`Vbq%odOyE^NfD87>O$mE9&b_YR^MFiMz-Inh=7Vc;e2wGLz8uey;tjyWdSaitWL2k{ zfo{ho=XrnOvs`uKMkyX`ZN~a9u+jV&#|W0ttkf5^`A7}%1|*7ua1~u&m1$m=^HWE1 ziN@glkW8z*e4?QNR{asScL^`v`LmAr#CpyV(3eeFSNO^u7gd7k1Enn9=2$^8Ah4}o z0B~D!OO$*^!55wa-6J2G`Re`SIS9?#VLcwO_!QsxI{iY2Q=!z$AAUH{Bcyg7lDVkm zS;FFZ6Ps+=8@7Tr^CDPp>^y;kSEBfoLnMRQha+LH_ZNv)_TgF@ffs=OzI zJ$b2euZw*jw!k_O`nqII4(Bdk(Fi3U=O0P8jQf3^K-%{eO3%sBy4!zHFHN5lB8ULX&f)$1=V$>S9F*4`o( z@WR3Y)Mu6OHW`F^6G6yrau!OIlX4 zul>nlPY*fwnlY~c|D8^0x;DoWwSs+2$8)H5Tpy_hhP<>=u($x~*-QLW_9px=pK3c> zEi}EwfAgA9IqHhQWxs>6mS;?cCf(rpN+DUaC14a`RQXVY3?N*mAsU}qYA~Lk5KmDI zZhRRCtj3!3M0b0N>32#P!k2B|QPhjH8oOQM6u3_>%_Xwnq7Ywi1sMhb3xUCpryd zNofKv2D)3|1H4emf}=5#>}kpidU*L<1x@CeOa&>2NNg z9^lIEgEm_=--Y%AKy(Z(+@MjJ=^&BWj72=i8nfGLc%REGYw$kJd_KS_-mI8A@ zH*6VVVg@5ttTj;UzX{s&9=m1GAkwH%R%3{fYHH8h1Q*91M*Sp{Q;w3+EnkT;y#xbG zigmPKCV-A6jH>l*G-$l^qoAiuuB18-`F6}WELJpwKCPKP;&Qo|0VllynlZ2u@N|tq z{0}zlPppI5IC#y!S0sv~L+JmG0d~!wT z==tPrMZCsk!?+W8Y*QbaiZATxrj^i6K{jJ198hw~$HG=pGf7q9KDRQKPzTuXRfsGX z^-1@sG)gbEKJB9~lUC+p+E#l2JvPjptZ}lx0Y#qKdK3@BiIwMsI!t=w>L&a#KXyAG2r6EN7e9 zs%+O6b}Kh?UBf1# z&>^SQg76I(-nU;7SQK_pME2Iy#Il=UW=z;XPHNy9 z7O0fmX$1Nrsgvy8h@qJIZa2NJU#SK-_2dKI%$cK@z|Ib4#*K7#Y7 zfK~rzQM0AHv4a*tuJII?d`?C%_QU0@f6_Tq9fWCojR%M&G{}WlNc7T7+od&_Gz}`t zI}Rh$rB*og6fl0IAWKL@i5Ce*b)JoIaamr()SMKZaPCIV-ke{B6`iQ{p1So+y4Lhg zY-B@Uv1xqK!O#Xo=yOMLG>!YRg>@+&=;>dt1ZQ@5kNNR9gu7wtIl=s9ov3S3F- zG}};-6=t~UWkR`9XD!KIJL3Rh4QcB@HJGeP@ypz$w+Ns{PUkK z+FLa)&b}*I%^ysY`mJt~G~(yV<(Bt3ejQe4Xu|~+ENoIo(MpU~Zk~FMXmuLbE5j@B zc#*yZ22|wA1V_5;^cES^o*5G9B@rFM2DUE2DU*GXV zs#mH`(blN{#&@IDq%g@rEFRlMrTBeeC{p!hZ`hu4;pq>%Zn@IWVh-xZ$TC9Dlvsq^ zJIFTni;bCZz>`($HmpB&Snn_nrGi=?fmaiMj9;wgWzZXp&@`Q-!* zUKL#Ksl=EL+>6nHJoEz7Qq*o3g{>dmcB6XSlC;_CCWzRsSJYwD#YLwd1pfDz-aaKG z=!$So(&l#qZs8sGX{Mnc$hLflbW0<%MJX~rY(ECKxWO@=V0eFKivQC_PBjY3GxVNUGh}4EgU3Z zrJhjSpMG+TIe9kiYL{;mU;k*`I$@18Nk}9R4F%)sN;krovfnIi5fv{y+|7yc7>h^N z7FEFovrfg}4IYIOZG!*9``6A^;i-1YD=mcTJIYa zZr93jIm;KnF6E~Gaee(LLM-;4Tke#B%QTYRwDpk-^@yu^12T62dQs2xq!~oPC7RleZmC$ka=mca=9fu zT^v4f9QW~`NOh+I%&xfy@XY^h(cSrv5z)GWTviSdz||yVqvbzj{?IIJ<%~4a$+e0-mFqIN>yll?G=F5EnyYhQE>P1;@mQeHS27_*Y54+h$ zms}0(-<9ULtjdwZ3{3-TkFlT8|F>@g*PtH@4u`6ktT(ioej|HmzF>~E*qf)JgNcRl zs$!^r)~ZP>O(GS!{esud|Iwet8A1Lcm9YTdUzDyW`0f5375EJPo1L8=#2D|?Z= zLW2prqJn2mQ9A$^s2bJw@W0;WJ&z?2iGm1TN(w=%&CdbDnle_(D?+M!)6nK!k&iGG zHeZT>V=eD^p>iZsfE3v4MO3OUEig`6E1t&qJL=bctFwE<3L4Mr&{}Z3N?tjKD42XFQJ&2!(Fr}mvt{5<`_|yh4vSF0J z1jxA!GJ0)NG$+-o`TVp;xxZhUP^Va4gs<3#C3HP5J-i5eC!%>yVWS$7AdM~d2yOD@ zxzA|M>$#x>K@#X{QvO~I_~b3*ZsvXx^Z#pOVZo%1D+_}8rPiv;U{r2qoX|#Htm%Ru-b0J`8<8F@cN;4*>&hsEqd#Tx2*%m_g)9%|d zR{Ka|ozAYHu3mGBR?+DH74YtM5fv0{5FgRsRb&3JX3-@2hT^1a?r9s2r>NMIiv8$G zZGKAFD75Ml0xqVmQ5i(q??UDmE`KW#H=JlTx-4vW=pt+|7szh$W^9|L&RG;R-j6@8 z8mtqsbM44SGA^B)3WWV$?seMcSNe!5ea9;)S-5vJn&r{vj+ZLJ<_ZyOm@Ou81j)3p z3F?0hh{UAO!XYT(eT5co1(Br`Ey8GTUh?Jry4C6lRE*&+!~>CCbRHpFZ3Uj^^{R3Y z5FCB9D9)4D%#-Yd%?s6H!p9zcIyLk&&++Cl9M~O7Ps-8r>W;^g{}8cOl(hSe4MXPV zo^HN@Yag2y=60wG<9u!GR3LVZLb2fIyzJ>(GR$i(odKDaysf+^!*gE3{d)3VK0O%2 zyIPfB?d&mMe6h0$#FMIoqNYVsG5mi$ok8d@0_%y)1JwSvTb92c6NFVdn)C71Uf8Ez zIFb*jZL9<%{$>Bm36rR%Xr^WFUDr4+W)a2*gNz{^O$eL912C7>8c-M72nlXNBQ_UjG(x*|+z{->8$$Yx?X=sVlRL4HWqLd4HdA3XQ4Sv!@8Eu%BcUq(p-dwoW1G z^xwA)F0^Yz^9QJIQ!ybub(PA zBChqL1g4cqn3|UP*$0(;nda1(3fU5YsW~S+_;IBBO5jlLr4h9mpSUK<9933Q1Ke)9 z!#;jWO^EO_IiafTD)we)ZT4LSJ(;H-v#E(#y#6NnRd6iC|9y?n+Brcuw629kzKzHI z=*Iey{Y-m%rRfjl;WSnyqr-i2JNPFHC{{aL8j?nvk*Qrb>=)yc>QYs-v&L6`H4uXO ztklo8U2H(yUSU$bXMK$_ezI>-;M^~bHsM`m@vzLHNWU%O<_^U_=LjRh)Fj)t#T)7F zAj#*$vYAil^IrZDIsg(|>@5Nu4%q!*_iHmi$?P$k%D9ORc8z2)`GL!*`{iu&Wu5)g zfmzBI{{+<_KIq`Zm*|i|H|Y2Eo|yE~sN_cLRZUX2uqE2C`wv^++G(dUWA*-EllO;{ z-P@^!oVjk}5jYHwd43B9M&V$|_pE5q^e_uM`K5)V=kz)sUy*oniyt!e+kqD$)}`C~ zZcIqa7NDrJj!es>qm6X)yr)}#o?ky8@A7X~mXLsutz9=$4aV0iKuyNvkI&9_v0WDL z0EXB|?;j&$!<-4uEqr^)7RzPFl%rO{1W{Rksh8#cQY*>)rBa}KXaiLJthCCx(wiTE zH^lqLJcsgw>2l8o0l(Jo!KPj&yW0WO#8exlp#QQ!9?jc*AQm1FNxlE{!!@A0!<40&aM~FL%W|2Usq z&+j2=HErnmw>Y{92H^)>ap}U(u#eQL7Ri(uxO=IB}1h{Y2HeMM!W z+a!v0@buzPf@yPOtHgTwcbb4FTz|p(Ik6k2O3C|B2Oz}vheJ5?g<~JeKwoj2y;A!y z36G2;zkk4DIrGYm6S z_jtVKm%B3`s{8KHXKs!d7j2YSbZtgYR*q+9y4@c_(Vs@srj0^z5a)6bwiWAVOUhw4 zwL80fqsYE!`@p$y?P*daWAu09WBBgA{Ma?w$L7?dm62t_wzU~OR}U0W)c*XiY|iK^ zjth{vsVtJV4KP;ecaGT>w#%KcH&9r&>gbWlEZ5IeDMg=FRlI3Tse^yd;CH?J z(ZiTacyy7epH>7FBo*Z4tjq?KN(&niaOTH@lepygDl`a$giC5&DUU9)8t9Qk-h?`< z*~bd8jqv)sCCOIrBltrp#((3#AL8ixcnGe@K9BqA(6g_nA zbpmG2l`T}tklx~K-uu~tSt;P+owXKQ&d!vz19;kSr3>+-4rcjxD08JimSqJcDPGVX zn7~DQ6FE{jnzeeN;Aam{7Mtk7p%R$|@Eog{Rt#_f+W>Pd=e?FSJfJAmxGaEUGnQ`x zLvvl*>eNvE(cvt|G}UTP*H) z00$^n`452d1s5g2*B7pZz^WNt^a0`Z#q5nJvrx0M^FJ}^KNbFG1dW#uZ5M~7SZVLU z{LQ)i7Kj=zrk!}whtuv0Bf$P<@o4up{Ggxh+q^#2fDQ|T)@}*&CQaH`Jji47thAeL zQg{c9e0@^V>}B|wIE16vEq5|WZxAZYeryvDB#Q;MxBBC2ZZ{+EA3N%|5!%}hAHA1j z^CkrvfBo$DYrR0qK|4@tJ}K`)cFgU1dD?eH_C}tGK^RcttHrycSl{4<+r(fX2TPB} z8HA`R8_f!g5^~x4h{GZG@^<Z7^-T3;f1$Hx4|32>8Y z7msIobRXdhSwbBt<@F3*om6LCI@FJQ6r)qcmxdW5+(r36?Dt)SrsW!!cMh}s3ICQ* z9=2$GU2LvNjSCZg5AA-SXq_~E3JV0|7NHO8P3RDo$rXn z)SB~}j?f(de%sF>sp`Hh#BdF3U)6jak2An?k@@-v3*+0*DdjAsZiw{00x1743LG>w zfb7|@;|KW)FupjkcABt_gDh+MyHEkm%Aa9o- zoGR(bRZO{8=-rBYbDpSpJo}B*l@wgLAlCOnl8^pu9LMZ0@oQ0M{?mB@;XGZH8yrBs z@ho{9`w%ApC|jId#bUPIfH=8-HDmf zM+;AQtQPm&mdwr<>kQ^e;PEzksO}V>}q%G(ZTNMbLJ_MR*!Szpi8EKjnncjc;sN^oHS(o`X zt8#1#_^qf$b^~#qrpxHBH>0XzF|Ady9sZoqCN({ zPu$AkWWHS%kH+WuT}&D$rdeT)<$qVH;(7judBGx}kBfo?VJPqF-S4jeyB!Ab;Vv=L zA4%KDa5$egzD)ssJcd9*Meh_OP{jO6B1Bth^eQUBxwH!AU^`nr9 z;>dQ>CIWkyUfl%Qb+Zzf8bm`<)1q~NqfE5-KQsi%E$v;CSDe9&+l`C=qrF3{Dl5Rm z5{1Fj7<>hip=PNxj^ojWfh_0p&+;_#j1FXcHoBfxlz;X^ix4Pf7gVJTp@>4P4K-;m zX&X5HuEPv0hhbwhlG5#oqC$hIl*qv!=+Psmd7G&ZJdw`Utl+R{m89QO0Jd#_(HW%B zUJHKM@f$4Dw(4@yH+rTYSoMReM$Q0r>;gFR;#Aaz_Uc5f%LPZ(0mWWB{E>;fSm71- zF5RF=Q7wH?g^=td%d{um#{*vA*{rLm|xfW%1*B{v35 zJ(bw6E{-V<&K3bzp;SEr{4%_vCmaI>{<{Lx5$VUs1>dX)s z-TGFuRg0v7YQR6`FpxG=6&omjZ3atBM>(ptMcM1YW--R$WNR{^IVu^gRLYQVAOLO_ zi4&!vwBI6eNAk2Sy-=U+=&n9JoTrJP5CS5+Zr+P6%+U!ztTUkkm+*sVltn{EKv&%cO#W-S;O{MtTg{BU=6)&R*=UKXC{ zQ-W3DXyv(AH5`gDg{Qu7V9E!{O7>>+%}iCYM)4ZQ8OAtP(-CRM+IQ!jCrpmjqPoQ2 z0ywAJJip8*@X7fySoZ*7g9XV5r20+CuxG^*JpM^3ZkA5W0-({T5{y0*MODf&~gbv ze(uIEi`+VpLUK!of+U@^aEVw*J&B&a3xTZFux#@I`AUpgZEWN0jLmEejYq995`|Rh z80hWTLH-llVmnMHlKKFgMMOI{3vjif2NUrRsMgAIJIt!(;(4c&4fMEd*K};V3}isL zAu{^YZX3v5;lp_b;ZU+Ao^l3}u_3MiOCQ9Yp^bmH=JNbP^1G1L;S$Eh;y0(=?U zbr|R;S501f67kO9mk+k@6MrK$gYK7 z>}d9N+vmReGp%hCwy#a7d{au@gD0R|sn`OynB2L-vSx)3d~K2k_1}-?n#X-w@a0fv zV#i}x2yMLSV(w+h{3SkFYTm7BKOHJLEN@pmLNabg2nuT>ZC^3{PuMg}ixVZL-N2w_ zKI$!v09UT)YX%mxvA(xbp16PWH)y~DK|j(`mevL3eSUgG#R6Md2G56&|4t$K!w@gu zt>=M>|KIK}$PVWG_y(jU!qHubRyv*VvKxON2>*(3`COefsvHNilwU5|H@j!$H!#IstT)X|^6lndTfxjL({h;vC#&EZ~WerwDcY~!FAT%de01du| z1rL*uC&9kLKtXO4;u1)&hcQ|UaM!?aPa~2c2J5IC17+sV574tgnd4K1q*;`IvTQ6D z`H8?&u49Q13K37YpDh6AeXo?Qf}W$4EQ*qxxxEbtPHDh6GEF9$&11F~om4a;lvcUb zE{D$vs7L&ZO!yoLz8aSF#l!RL3$Nx~#Q^m}e|=Q>BkGw?gyT#=HvL3B8Sid40Ng+t zgf?d_!d5tFbNqnn zXJ@Lyify%k5%pcjp74m-UpL-U?YFZt&Uu+`( zIbavGzsHAJ{hL)aK+zWp+p&gDHqi=vj@}vQ@0&1b@&AvgYOh(J#q) zYgMi9?h_Q9K2l1+)Z_XosFB2SBFNZo{{f7?rq}F+nQ=b(7FtYKmV{?DmpY$W+koOQ8Z1Fhb6D3`xI@sm z@%3}o0Z2h(fW!TB82gEg9el)&^_%cSSauoQ}(-`Bd!H%Y7@6iogF&KcN2RYW5}) z2nRWGJ;6hY0xXqRJz>B7Eiz2)1eAn3adS#?U@D;fBH^~?XuE}w)mZ$1!!=M z9D!)$Pc#_Baprn;E)uB!VZzS5?&jJlSuF9BFI>nNlinI5N@dt|mHkO3QN-{f*v~AW z8Ts&#X$u&R0u@S}FVRuH?fUBjpmvOa0=La{6r1_`;_7%~Vj;J<%<;g8?<^(_AzgrjiC$RemWj#s!on+9i`Y8c=^Pe3CMgzC$ zNZuW`y-VKxTWioQNL@14S#5j}}W=g5B%ASQ07RPmVGjB47m zVp2%=bz<~CxsK(w52&5qc_qouZhf@vEVHa#=X&*H|Cbi>Gv2ALc;JcyiMQE6>zdIS z_tI-1x2Y?Vx!e8z1C7D{?}Sj4?j&M1K1M;Imw1Ar)Tc*amUN#BPZ*kzRr-zB2J&4U zzG2z)KHxyUB3iRTeX41zprL*qI|f>pZb0-|`~Av#%BoGZZLLMID2)JQFi}q}MHZZe z;z8MOhQTUzt~CyTS=ycdu4$natx5v@p-O}AG?+qy73#w zFJGC(iOmkQw$~|+8sMK1lp-oufR4N3Gj4GZ^MfxITAN;Fh6)H=0rkTkEeWcO)k`rp zJ$p>6{@td8hvJ@)g6qR~M}6;H<%JVjTF}nbq=6#7)?6AS-a#WI(GN=EuXu%BwkJxy z!4-3Kw&WfeBk`6;*sGm zoZHFrXJRnO5un5L{BpM{I84lc$iZO*g&OAWvVt*J_+o4|9+J`!Z`n((6>oxJqCd z-~cDJCTg{?N5u6wwiE0G3;frzF0PxI_aCvoAIcAq349I7A|ruCOjeAc{u+?ybndWx zGN0I7soXyGQgP|lXRuysA2aVrf6>l=VMFvXDYSIl5WhDYgB-=87FY`KNJ>Zj5@2RK z?Ep!18GmXJaFdeTG>pCAUIJ=TVEFjjNp3Nd!@j;UX4O{Vw@$CWhv}Y2BPG9~SEII& zA=noD#PQIkk;`iMXY_RqPSad39O~V!iKs_mpVAr0`4(7Z+Lz$LIsXN7fQc!u_{_wF zBN2Vl&{nX&TwP}VrV;671V~XWXyUQ zF}iOj23g8^AIDia^gzdNHGa@CSC-3F3%Z?pWLze)%KLe3oNEdgt{%vaik^n_r=UYf zZj=2I8Ly1?m1wpPjF-WiUnBki30BqbaTpSoc49^wUp59HTr?(@Cd05Q3HK8yl36>A zd0p##M%A0Z50}I2dilv{c0oo|yBAbrb4JVe6gpqmMYyev3- z_->pB0nar23RouEjSuyK(6AW{6Bk-&e5gK}KmrCBO~vMuBQe0Bv40B-8qKUGe*9eXz|yYUB@)eOF~}0qwA1R!*t)IO9JQ3i(qo?@0PH8iad045Nft z6oc;>ue_a*S6 z`{#h;)Q^8ej6{{$VdG6Jx<;V@CXeNvXl znctuNliah>`&J7_57f-hUx@*@A5oZ>#gkOeK)e5!4SZ)DDVVD0N4dj3DUXxUthAL- zF9Y9PlI3#NMsE#9gXu$UppE#nv%D-tMQg}qQ0CByg3y5_q%lT%hDbOHd& z^MjRco6ta9S>R4v?r5zqwgmM0-zpY)mrwo63nywb&EmwcUuqF3v&CCnj+B5x3HR9w zXmOWyDOA}6k7&iBgvo+h-`;?MHF7qe{fJDTFya_u zr8k!<6zBMMrX^L1r0inwR_zSTD>b$^G?rC@2H0d=z}dlqIp^LV`!XB7Y;ST_Aq=fl zia-BZafz_d-{sOXvvg2`LHpWxNZxQp{t^e~{_X9wPiai*`MuK6_g zfy&B(4eQwap!v6!<*;ca!!`;5N8c;J?zfT|0S)Xq+aN_bA^JPiT)e@H5Pt^b}@lI0mY=&L(JpR(22l@JaG1$bDF`c9L#)68&3Fdozo%!MI99} z!~0Ow(MA|!;}chneVNt#&#bW(t*gEa0*=90!mmCy8zaVk=b9D zL&b6zh0#tgUK|NO2FIBd1&>9XCEYep6&_@#W3!=U{dqthtw8=e7{ z6q}I(4XS^D-TKd%DfIM?7~25}Lm=D>1hY4&lZrDp)%i+5F8|tjr7N=k9ttN$Li0y3 zj*7y!9D0*>*#b(X0GU5|KlS7)!D_63dPrJqF+H&bkQ0yYY^|D!YETRTQ`-k{x>)u= z^J?tn4T60H&>xc;9$&^V8A#6A^V0%pYgTT3Qcmdvjjc}b({Nz8(y9oCnl>8m{JC<$ zJcTs3yp&ojzj?g;-v+cCYR)w_JItV?v=ntO64$y0<`DHZ7oj1D)t8#;ImLOK#=gEg zULIr!Ypu1T=$N4OF~-Ol`{rn#|K4ApJy5SEWg?E~MbPO;710aG>%S%H_-ZHtrf#nm z_{pz;n86IXD+b``6sTJL$x4D757r8_T%l7Jnx7sz8IL>5&c|#Fm>G^mAQepX2ROoXv3M6F%ti)VS`mBX9P7Z zb)`wn+Mvu^9*TMjU;uAh0k8v0%wsBe9vVURni^qd?zOa!bU}%rA6NkhDI5E=ZHTf} zxXkJ@`3v>ZLninMFmZmy11AtD2z(+x>YV8#r8sGZtb=Vd%MSYNsw_Xla|BXH7V)ID zJu*jW2@_#Y>NMHd4X2@wLW9oi?w2b0^8-k({=mUM^?L|S2Hv5>i|AsJMM`brTPxQfVsM0oW>eu z&>&|kq&=%4_oO=oP3aK96B5W4l@n>^113O59>7x!57+uBBD=LF$9TAr8MJ__d>TOZ z6o#`g1n}4IcUMfUYbh(kV>bcZb8-LuhwnY?;$VnsRK4Mif`jnB4!nnB1%yIYXaK%$(Kx#4D z7|n|}V;#1Zpe=%Yfek-@CN6%6A{oVLvl7 zM56|~D9UYSKpdM{DmIT=+H}VGD^o~NLHx9mw3+OSR;DEVZ=)pc)L+Z7nqt+7l?-&90!100Ug=98=wkQm5|x`OtMz zDyn4=Zy6kIe}k8YXm)SfE7XN(CA9?^=X<_a35ZHv1hu6Nt35F0uP|fu>@mGO-7A>< zAy+3&O#YBf`xTID`cfo|<=JkIbQ{REG5~m?2yD7?ny5tKI=%lC>JZIrL0iud;w2MD z!hS%a!(*=jS*IBA9U#skuK(3l-gbIe50-r%J3#W-Yn)C# zj$@*Lm-ifW#U=0=35?TVOj^%uUOscXv`VLTgls4contj)b^rEx{KS+{+k*I!>y{!} z3@y$1jkZhwf{yOrfLTqhgTz$K(|hny5E=P4c03BNZ~Tg+>1jD$nA#bZO@iTz17x{U zjb?omBZJ4lT{uA4=I|O|^?SwK{;rlYoA`&|Z8>4%#U>#|pQcJ`P58{l7>dB)4v>S+ zBLD@c?JSlZAijQKRz#?YkZ*g>7Wj}A&(EWA+u*rnQvi#UOY&?HVCKiUfxyA8$U z=vqAe0&x0_Cqa7i8*u}XJ=p^ti?;Q~MCAcGi&H&l6Bf?TZyr37+;^S+=9+e|BC%1B|6rgn^(BiQo<8UvNHAppFAFKsZSga<;yD-c zl;8!EPi4EhY(hvVr=g0%B|ievWS)cu-wVk1BpP5z)7csoOt~`v=#T+T)`D39{?*yw zZmBYZV8o&%a5FH0SRJPHJ4H%4YE~y5Q&2N?6q~8|>qoydMQX=!P!|pr(p>@b&*lX! zxJW{>wAhXHT+GyY#_CK2sa4Q}J8S}B^h7aifJDoikH_guS*1NsQmMP1`S zU6da`pek{gl{LnEjyNxHf+#k(!o)sMHGuNf$7R8R) z0pq^s7;9)_aj>*1lfETJ7eCpTNkJLyBK#TudEt1hqLcY#slT|p_bvyy@$=U?YOJ$t zGvmP+t&5a#-mK!)QvjCQs=eVe$cP6(%n5*Ao7$^Bm%D z34ca>jzKj~qh0cTxE<7I>H}#9f^MzgvUPd+4FNv1+5Oh9F;|2Jk7O7O>by7rp6#F3&=6+sbZNxRwvA7%!23FV;N=!ZpGI?csy zotuXGeqiL0>r#90NUj-vjh49^utpy_Uh+TlNog+&=*ZG6F(GaM#*tXTq!bJ1LF{`E zCMQ8bNNO!Cst>$ht?X(nSg3&ANt%cGx&NTasJpkVc;RLxP{yYutM{^zg$M{UseD-| z0vSqV$tS>=8#1ufAw+5R^(9b8SDESH+Eeqq4_<75X4-Q^EvEg&Lfg|Q2EqY-^fM4N z`NrL`ui_|s#2y4SI=jU#E`bx@&HzfxrFa^M6UO3ks-YXX<9~ctPK+Va6-npU{)?mD z^VqsY2zw8<0)EWg28RS`w7w@70Jxx!+ysw&dALV>;MY<~5A;I80olH5umW{yF%=V~iw>b))ImbP^-THsGZdQIMZfmiQ1wB<*)1RCFcq z{x@7@MK}8D0B>K~;xkbgy)H24oLk;6mT(V-BJra05C7&@XGpf9@sD^mB;~MEzHiG)^KfQmMK*I80HnsXcXM&H)Q!@PnU) z)q<0m%3{A89)FN-=E>?R_*g2e`Y%0{}u{Iv@}%Xk3f zL0LaP+7x^6ghiYr`w|cvMQ57X!623e4EGtVxA<*^CTZ_IH;Y!24Z!pXumbOv5@m0& zv#AALrNS`lfa#UlRCz(L9?B1s%DK+pRtWvOxw(nfBS$|-iMs%VxfwMHiZ2T?8!;Af zG2mN_03ZII-zR>pl3um2nNGcwF7w=_zeIj#(yU0PxEJIDTy=a`^VM&fb*6aIMSvUg zgT&;64?()_Qbsri{!?&VWXz>ZVx^z` z{0nXI6WhU*t`=-_#yJKK{RMVZ5Vip3PrvJfPO8BM`apgw43#pASWYnJ=>~IlY<2;j z%-fk%k=290*@J)lIp_(-)hrF6d_s&5>sn%~JNc?WvFRUu6h9u~SX>({a|$|(%pI`L z0RytGH5LKu`=7S}3l%R6KDz}AKxnHaE+a?hya2N^<&>i977d5)Kpc`eO3WuQ+!g>| zh;=4m3sOrS_p^xHZeQpx9^6-F@O(kVh@v?NdsimZ%OrrJE)C$KRsmor842*6w%e}W z?^``!M|VLj0Tjv0ePgX!aaTZP!|3=O{x8qUoqwDHx-Ts^L#WXw#)LI%KrZbLk^cUb z?KVDv@B9G7BTpyd7I(t(|1R?Ye*%C7Rsgo=Ybr520+|cP_=7B(@Cm^gkQCFv7}OBN zXR8ok;eKrZ_$i2Q5(U3c(wSx7SstVdu`7GZ=cT-HV*)jL_%|YsHX3VV$ITLdYh%h z3fY)JUh2O7NOu_?LNx| zUw7Yz27{IO_eXK&`;+*wrNtvAL~z}y{QRiN(ku3!GOt}VA6@-S^IM6WQZ4o{W>wgB zD_BETAe1Xj`D$C7l~K!;R_Sj{+V7JQU^H*2NgN;l?V2LWGRbsW5(Gw@4yM_10N?(6 zc(>cNu*Vq0ZtoiOYM?2LfSg*P7qpQl`v^Z@0i_!YuxV#upZeel^}oLV4_$8=mF2#D z4ND402?7EV1_;uPNGV+cV$huuk}4&oNC`-X2qKNrf;0+Bw{%LUh~9L)>$+j@bI$)6 z@0T;iJ~-g|)rz_1nkx*pi17&j19WOsLCI_m>iYtSAS12)w;+mk(3NWkY9+$7pYj+z zxdpH^rtmg@oQ9J%8bJuVP*ROglVRIp>MyBkwP}!gI*56%(g&AxVfma_z#-#q(gJ^X zV5&h-hK9DH$+V0>_(k<+lu}gcL|2-koMZgMMnBj_E=W_Wka-%Q#JW{woH%YYpK`uX zZXCb7(<_19A_Dg!&d|T;aMkb-xjK%@lR3G0ynxJcL*@K6WREJ%mQWX>Y#+6U<4=S- zV{JnR97A|5Hoe=zMKr4)P>BYjJ6Kc*I+*nc5YB+V|Gx86*l!TprGlR|dh_H@`=rOn zmO`)ktiN|n(hc2`Fv}tzLJ{NkC^e0ez=db2RDt3wB3h`c-A6&+89dnh&!zkQ_&e+| zFjq>|7ecfcx>|Y1MQ09+yXqrA^xuxAo~R zQXy}x%d7a{xgo9u?{R-AqcwI1q^y?AFxj-%K&Pw;7N%mb9T=1g zR84BtZZ5sGUS52_GO^6S#iZt~Pr85dhQCQeGHa}@GKV~SMhU#OVnRQA_cL!Ul%w+f zXhOpAyHEMCNzVt2Gbbn(Jipr-zrrbI{rd}F>f-*dhic!QQYC`uFQbiYlc3LB3^BYA zB>~+N7mFgZ94Y;{10nl+tHnK}e0S5Uji)j0eVDjNHC;C_qr!PG6U@>fkDr6Ng#z_2 zNrb>c2qb)b2j@k-9O?g$r2#ztMQOI0E-D_O$ofoODUi?jbiX905tS3HGJ#uW_ULz( zKdT@1l6k?ZX78@vd6r=XlyLWi^j|GhN{745UT{q7q-3JfNPd7gyPHoIy_^qnSDW}T zJc*b84l1E`K6W~s)FZ=%upy|hx{K5@bM9~KPPA1~U&8`0lnf8}7Id=r!lM6EeUZEd z7*IHYz;T18_bW;1H>G7Fy*vJuwimYlgwM26QXmq*9`lXPwmBd%_>3Hrj@eid{L4wP!)Ro2CSo z*am{gm$sd2MGC=MXr%50w1DkE2S$MKhMa0to*@*YLw?Ni5VN$Mbc9(^U5d4j4MZxD+YGx4qRc)mo%zEFlXZ4FHT2dcnp(q5y zf8wyZ=PsN}UjGOq7rOIh7|1_f8di1owxm^;xES{(jPNns`y}K5M*iMNGD+b${-giT zreKpz#sx}BpF@A7&p9ZRNZBN^usX{`i(Vvs?GQpPCRg%t!L*xkC+u9-g}W?gp0!`= zmXMY>lh$@N^=i;A4)0Te;@cr%!KLJV>@e7Kl1ZBwIc?y;m(CZg?j$Wbbc_G%6#jQB zrT~8@ed%|cjmnc_jF<=5k~67{^(u$rTi*Bel-}6JNnBy+_-lhxG}(LkB7Ya7e1@XW zX6hqcijr-CNt3h%`cyv|p@8qE>AL&Wl|K>ck0dTcCH1BAPjFvK_x^MA|7Y**p&|B5 zJN6m$cSXJRw&hF9=I{KQi3<5!{X&4ZE>hZ0aA89;NVg@aXQvbcD~{<9OU8?cg{uGK zO+ul7r9#R_HhHKJkB0?VX!{b0joQiWG?FM3t36mU{D&y}pPPsN^OCZ71@Td5KOjvf zk|}NEnw;2&zu%=a5@6fX@7?{TYl!_UYJHr7dv2`3fG?p`*~tlA@Yvy z|L6Xs`AK#${O;{OZF{`Pp(`o0b*r8kHNk@t>+43GoS1IX^;D^)-Y>pik8`A0EcTMD z)o3`9l%yVo6y$bA_gdFYXm_L)Dap~zC)K-Fq3j&EfZe?%!}nTUDuF1f?Tt#c;e9_ z=T|38-T!A#?BL~njyFVGB_9;&wtm9(ubHf3+NW4=V4Jtx1ucsWNAD@@#p0WiwXuO- z8t}34Oa*29N2=Z8sjt}etRzF6A~#3d%{ExV9>}bb^_|;|kMW?%$8}wpbN={TVfjhL z%Pic>!ZKm>!uL=%-C}Qu8(qlDLl4-OyIE?3Ygk02QF7{6j_|i}sF#{H%3mGt9St5b z8w6=bfW%0EbF9Efs^lf$;R(d_#y^>vBbUKZMRMZd7Ws%dVk%!K z8LHQHZP>2cv|3@%Eqz}(jbUbQ5n318gYuFQxFKnkmUQ=iwLD59bt^cJT%~Mg`l$e$ zlrf`UhYlZm$vGEXKIibgOzFsjGOZB_In)((v;8!dhzV>PYa4gF`K=G<=@54%dighu zH;R`DUa7@ggFLNYy)#j$vn?TAp(7#^(y(~wj{SV?0JtIMe`cCFE0fZT!7cq@m_d$QGqVV9TMcO;s@FwQk`oDK3NDF!ST%}0=3NHnFU?X65(woc zwOJomXPFbo$69WNy-T$Spl25`;McojIqlM+gGKASKM=UoDU(z5>F$2Ar7C=VEK5W( z7(;^*ryCL$^;w9Co?-PtEsBLqj>uV_cSWNLtZ-3GHTf924c zB7t~2aLneeZNfiG&`|BWqWlvXBw`M8<9-ro5d7<7jt|lwFRD|mlK%vV>vHZ}_T>IS z>8`=RFgJBsO~_DD33Gq(@v_cBANwoW2037vtNH8hx-Vj-3I%8Ig@3s|TI$HBU-GFp zZp!7|DQ@lEYZb8lQJ6Cu)_r|>c7rSUv;3lz`4ZvHixP!QQVR?>yO z+}P_-YHUZ;+gf6u94*=c7EfTa0aG}L*{AQN0R2keJQ94<>Qlkcv;WVRe?AJkCPmlH zery}BW73*^?`1D8oxk_#Zo@r$>`_;{>CEs|L;qjvW0k*98|AK?Z2rn*T{cFYfp24* zZwr=HS9^q#_r1S(C;ArcqbX7AAfx9CuLDE*M-5nQEg$1vIl>zZjq95KBwltFZByhg z`ye!4w+;TAcdHVIG?{zFI6eyMIIG95ecPEH3g zs%qe&6pE|cG`;J4qHwJ{84>*!-?t1OF((;;NQE=q2h#PX?Voj87HJNy1`l-VFe@9YsXrp<;2%a*B3=6I)~%%5DfTAR2ye1Xdh_0j9#$9u`DnM*vs*xZUO)o_V7tli53nJ>Q(95{BzDBTTQTt1|ipkLCTgQJB@5r9_OwiN2ic&7F}Q~#6+j!i<7 z08s0iHVC>TkOfUK8;)GS-=eCTn$uV*&SfIMK*Oyt#=S8TN1T-qq2smHUAAq@+dS70 zdml48{gY8@H2cSgbI)64w$2{h{juq(KB1{jWcgYD$Nl@G<^`5g)p)G1xbWT}N*%T% zF_pW;_nzYLTE4?c#gbN~9AkVKdFC56s@Oa?jOzf~rt!wm+kr&}e01MF%Vq4E{Xa?r z%3FgbB#`6d#Ex|q*bMPXJP14homtLu;)-oZ!90WCxb$n)h+p_W0f-~wjCIvARncbd zrFRC)R%B9xkwigXKJ(_Si@zDDExXL;Dz)A!+b(u)%9aybXvZ|(jj6mL%RIq+s_xrV ztp*?u? ztxp@c;Vt|vO;2YX`3sK!xXw9k)|sE}b+ZNO2xH}B%(0enlf z2|)u2HQXn=cyAcGevnVqN~DCef5Z^TA0Y_HueBVU2s~q3g`p8vVbSbW8Ni(QdgfyG zU!xM7qoWIpYhNSHFn6o^=JR3~oV1e%j^A=xeBJi?v>wNjI-+ z_)R@fEWtf^%cI%y;cR=x52A&Yty2=eP-nZ|n{Sr#XUa*B+O5BIIS@)Q9P9T7%ATdZ-g`q@Aoj?%bMXyf%xxs=JBQ^1NKHJe z#?(7Nz}y)MNiEmSk+l z_7$Y=8KrAK2HmKykRDN0Jw4_AeZujdjgc zZ}4qd>AZ77MJ6@+#)j2F1-cywC{xTeFjzPdvbpa=HbVOGbY!h-<(aF=+uJ<`9_ufx zl?%z5OUQHUucCMP-_Q3^CeMmnB(^*z9<+%1G*e=hmKAaI*_*wVyt>%By!GOt@jz`u z&ST0|)?sP>kcGK<{M2=?F(Qp(l$ozDwGC}?`dSj@8*E?w%NLf|bz7Dj(`w8K*CV+r zPrVS-uPy(+V$t$qbz)xg`pCS9Lnx_O#~t}Fp{HY0jnQ`kd=PyL@m(iA%D`MdI z?PK&gyu6a~MnLVms&nLliG&5gW@dUQy_Wh=rX5vbEcr(3w65j!M!fl(>vURNC0~A~ zmKyr^J5)PHp26IG(>MPy*<)$Gju?Hhbnu{0^gA*=ZVqBNJ`fCOqDEukKJV{7H)~B_<+LOf77m5-;IZlm3G;)&TdpDK8%a;LdeQ zY}{q>QV>k^HQ<&4&UlEqtE8h8){O&(`~3yZj?`{GY?96_i9ZJ(wSUv&b`CiK8m739negf-5ZfQ)@FGpVq6ps-f@$-g zUti1t-oy(&luF1-85&4bmivF2?E8kssf{ z#J%$mQJ*l}ykW;klA(4^DvOr;jHiz1SP^&17ZFOYcdrI`qeDK&DC9oVQbAb|G;_d$ zVhsI9^N6ZpL9VwaLG_;-W^)J7njvy0P_{IKnIO0tg;OySxdL{hgCR5mEl9OO7V9*S z+Y`7Rba1KPHnuYP3PPsj&nII!Zn@vipxcr!OjBeWo8UUIyYydIixe~s_hWQy*jhZ$ zG5%Em6%;I}i#WzSo^=GMHZ@Lq z565Iv3_s=2Xbynx(0x;DhvyUJ>()5#IgsS%bZSX-8%iDL+M;-(2dG{^mNy25o_1fx zuq&U90g}EZzF5vrWQ4%jcTZ;UuJ;KE%q<5C-dTYvj4nedZS7^|65=m}f{yX)>3e(8 z$}ypBprZZ{B8Wt;;nbKqx0?8R#Xp`v@=@OOJSo{t)pxcZP*&1Q{0?Nfqlw+$Y9AZ? zuoSDk?ss%n-h149>}nh?8OKXE6k#d*o}Dy6$$HG4Fdh6@Xh z96WS5WX*XLyr*#0p=x!$;VmVmzK=?Wzko0^2;rvO95Hu?EXgp(>d7NBtHO%yrZz*2 zD5F?l>K~NpC3hZ&3|C0ci#{#TD!e3)TM8YX=0JYIXBsQZeE`IZ_0+|qjkv&qDXOQF zACi^1>mea&gNf7AAM_oQly5Lae`cZZarUZ3ZIksE-c`C~_8*7rBg$OIL+)B7Oh|-) zPp{7wq!*RimY$>b_e~77+g~234Ac(rA>`SPG^P?yrfKswjj8kZo}5lu{nqS4ziCUVoCL#Mu{;h9eD^;Ni7YvkTnTd}=U zpT_;)PWg}jf3>Y3x@oovD@?`n?PIA&1T*c}5@*f_K4WOsr#O`z^=kriNr?D&=G3@& z%ANh?*!34+cGur`?vFi=;p8VJXePq&%v-9pcxHTt_wDjd@;_Pts9bb~6n^<T=S&z2{c}JuPiU!9Y$?DEDT#xrjBhzS+i+yZb@28zIB#UXHpZ;N!9VEZU8X|G%Lp zA9h5CueELwY3GonW=?4&Q!Z^gGDM`fdKBu8`bs#e-;~Zqqx#ki_q1qo9J#l<_9j0 z<4`prQeYe^xF}Ao2P0lz(QBN&4;fc$U25|MA|xUy2C0-z!APsdVvG=lvrP?KxzLZ9 z0Wr10E5EP*;|Z<+%6G2kA*4C<>6SD37XA6!eY^$_UCood)_P;46GXu z_VqAZCU3v0Bh22cOkc{I8X1{V?RN>hn`M~IKR%`L05OFk>Q8#PUtqQ@{QPlT}X?kEzS{0p;8AmWT`Yz^qz$2lsWSG-B;F{~SfpwiO`r%yoyfqZ|N z`Bf|OjFfccC+n+*>uf^4)a>583q@k88`nQ{njAae3)uUi#)B}5`9k_Wm)t8FvuB4p z77u((i&!jY5rk#fo(9-M7A?`ePruqjSepEl(xyB0g}O|U6r%>(Jr(ikxc%25cuVdg z?nykR1Zh?~h4~Y&%9{0Y`1aQ$vyL88PQ2%g_67mGNoj4?snJqVDy8z5v+eU2$hr-r z%P2XN{utGnZe#@xW0-yY+!=9;rVCiEE=F&XdyTkC)F*C!DAm-8plhJ##C)KC?P*6Lw2;2tl%9q{G?_{U@CbotFhk~v zgM)s*ePPK6$$Tz1ziOHQ(f6u-ILHURF7TM+XjJ~^F~3J1GmAdWabAgmB#mRStszg<^Q_;Y=yt~Cx6JTfggWni^1fHI{i$XOhgTy%Rnyf(Z|Acy zko+f1Not9*Ul*!4Dyj`I3XfQKqrR;Oz?OTAB$XvOHK|LjMb9)53w8qj@$->isY{0i8AOi%*-4!|DbVo{XxTa zkC88%u>exQuMF86g~{E}=u_G#qw8ST1LPFAq+#3&_57BY|9NbuFtrfyJk^N~iWUQCD={wb%In*?DJK z>0Ly#@>>U~gh0PY=sjJ%Qf(9+VNa)C40&GcOvmgS*LT>r+n#K9bRSO`U%VU zIUcf#sFeiv{&aw<-~;zT9<#UJ;u5y`7|VaDj*SWl=E$S9+jAw*)ryEe_GYPH4x{twj4f`V@fAjaz08vFPnw5^*QRDNi#IzKjl`VOLd$=CC$ zbv%{p={EOGz`;9W{9X9AvaW~>-fwvuEA6V&9U$ygK+#mRR>!}UDe-rAt~OTp%6IzI zo3t}}X>CraZv4-2FR_#Mogx!kGNc71M;c)=+4~CbG-SrSev;(rSxytZRrkQ36;-^8 zMV1kG&17~&o8P9cedCUx-WTqka}n4Wp3jz?ekh^Jb0vb0-Op1%K5Y7p0qiQh4(rzw z09C~ZCCgqqG)A_VcpGvls2Fsgj|LTqoN%N$-5fE6%mE)nA9 zIKT*_3Dv>~5Jph57%&6>1=LAd?qkjdyBN0}*AV|Ccpp;CdA8z(p?w-TcE93#PZ@@G z>FJBZ9?~rfuc^LWqp_U4T49wpbk3RRYg7N71oqVhF`NM(QrXyNu?}DE3JIjn6?`ibG#Cvkl~@ zHxhnz8~<1vyRF}G&eo0aWofIwYtShrEtTZN-F+0f=dVlXKRPLu0=3Z$=(L>+y!tkP zjmLJF|2=J^+y#f{T8J_TP@vi&DOGB*jCn3ae7%8?PNbxPvT_@t<)H=xfIvL754kx{ zG0@}!pl8Zc@fSH4fALqiCni7uj#>yBx9)!dUCD{aJhBp-p&n2R_|g`~Q(py^%^aZS zw=UWxmPR0>L#I)A$%_Dm9c5&aGm)=jSN`n>G?~)~PhYqsrz~uRrojC3D$&sdvVa~{ zNg?)uz=v+0Tw<&TdOfM9eM~&J8~YoM@;}hOG8>F)FZ{*nM9oGs9sWyIYylN)p8qz3!b{I-{aL(i~SSl2JQtAf6) z^OYw+{aw^0heF#UK5CYzO08Z#Y+gymKkuk7P_wzeyq$DIQ~#@6P*NQY3?*JMCKnqj05*f0)X98ozYmvsf-0vVuSLfNohREx zl8o_d51`lU5{dY>q4QmkVtV390=*8s>>+W-y5}Guij8_=I=|ND#sM;$YfOj%0GUmo zo|L>xyNI@4@)6!OylJfY2Mt`-eg>{9r1DoWJc;sNF8oaG1sQqTw08qCHv*SvS0;ky zZq>ZLJz!cmy>oFdfqC{ZMYS$!Al$__Xwf>nqPa45AtdFi!Y;(_fxSA+{+=?YgZ#zj zE*BCVl2;o7F-TjyH7U;K@=xu(ZOZkkO^K%tMx)-!l9h{&~ zHRfC|JHB9d+44&g20P9gVcif5fQ#D!34hIf!qeKWr0@`;t%Dc+e1hyw9Qgn}&ePZ$jEoU_7ew8qE0bd&VvZESU(w~UxU1S`hEPL$K} zP95|wTa0f%H5VgiF+ce6SM96}<>gDpB4@GjQ}M20Ub^N`v$bm0zqlw|ZZT)jsj=dG z&&6fqky}~p>XRJQL9N1{#}N#X`-wB$i)>cTpG=9w3p*R>%pVP#>PW7|x=o-IpKa;n zd$P!#!oY61GYL>&ag24>>6T^%KZ4sZ(2>t`*O{$}v9Jc_h9Pp88Tw*-$-~ac??Xje zEG54Hq@Vm*X>$3M(D5%=MHl3Ims z(%+Re{PO0YU^NGI^3oX2?z63gbp0DK!n;tFNdz(j6XiM$%x#o}kAna4iJ|IYIYOz;&QdS)o(_AhsZlzPaYq0w>VYtk6=WfL9srTc8z9H z#ZkfBGS#0?vVR<#Unr`dU&D3$<3{JkM5J$jCa1g5y6zS3x(!QtwQ2RUd4*7e0h8HJ zBQE-iNm>01>WIM368S3=j?)&asLN&BB#G-&fq#Y;h zkP5x|>D{V*j{wR9DZ>4p606=1)D*x$NdM{JdFpFQF=gS>_hKi%Uqiz;KunL3G;751PT81l{3}5m-g^9-)ZieAm@F!?S zN_4lcTpQo~cU`lwgYxdKn@Wdq+4V)m#+)x031_1DyG)ed5$k6(_9atreQf?Fsb({+ z!rxaY5usVPoJuM@Y7ym@+_x1C+W@~awJsyk?je6>gy4t`-}~zivO%}Q98!^D$Gp&+ zwL53=oIIJhc-~zevC2&P!&k@7AmisN8%A%7%qWe^(-RPU1HY&I+^aP_?_1BWVS@iChh3Dh< z1Sc1afMb%f4P;8zZjrF1Pamy$9r3|@%EBT0dZLfBe4MrWf1)-sc`)(PuG;q~T#ZM% z)AWkuV+-P;8QzVqkP6lYn}$PaXzq_$i*bP-`L%Ub?6|dx`ixP$ zv#M3y+OOtqJ*e)CIJw^`P1uMzb7p#Mv;X$`#~7MgFr5$6oN)%RPxLiE`|C&B_yg?_T@MteU_ z#c}FYJh!DUe;lu0PeDP^zZKZg)LUex^xhycHg#RZO7D^t2nF2#RAB((0ds&(2q6sw ze6wyWgF*vcRUxa?)55&xHrKW!0blQDVfp!bx(5zgSq^(zsvndXui<82~>L& z_q}`f%)dUzC;kd?f%b&cu9z!`$q)J}0f)sDkymj-lp|(aV{dGxAK!eREadP$J$b&4 z+|*gt-wAIvFNJ9gxc52GyPUg>e)4J^ur$=JMAG!G;+JoQc04@rNQeg~Zm-m5R! z5^lS&`qmyDti5Z@RboVQ?x1<&?cRb7$>|IB-WHx!sX&sNgH~@r==Xk9mBms!H%-%T zrpupV1nitVS?F@%^}{9Njbv7FY@7NG0c9dIN;K~8=H@BO9e*fe1tvu|9nEDaWW6kW zJ}+lIRDTQ4Q{Sn{o{5taJHDHbeTYn*@~WFkKPB1B&QIHw))&2&!8oWLb#6`)+r7X% zd)4iSddMkEam>QPjN!X+UrL#ia$YT3auxLz`+x;ARR>jXac)E-^}QUC{lZ9D{`;yf z9GWI%Dl*XGl}FT^uep2HB`Ne6`ZY&h-FsY!RPCW2-Pkg!l7W&e>8mGqOL2M!^m8DeJSw|FJ+uqL!4IMV{gPUFx9F?{^Nd; zJ>~JyoAodjcikepVQ?FHAWiAsxP03O%BwN0e!Wlhu~9V(TlCAGh^@E6v*ecg@V&vt z_5LT(-iwIA*g5+dAP2p*b#O>V&v3)Ps?T6pDPwY7n5Gl9HzcTjDil40alx}S?6J;} z6p7f(n9q-4`XQ727r~m(I#UBc{7?U3 zaddc~w-OCn>#Wj@YLZ@Zw-2keu>2Ij6i8CyESVK@Rh--{avveH`Fu|i;l9k!YsXf+MWt@7 zCTmNhI{#lAj)w!4qiK;PX#Z|Ul-OirPi-R|`6Dxx_7FCigG@kaeM|iVr2BEGrDtp# zbonr!td3QdJxT;XiUwcX-j=*1fBClA-7KY-*i1<(m!eUiUin$@2uY{fG_YU}5$Fp_h7X$}hHUyWSfh-Lc+GI&a8gmr3PCD3(nXSe8Dw`nZ`v_ z%(mZRW#z9u9z&*mylShN%<5&Lggo0)u3YB}NC+YP564o`fY7ykuY=qn2r%10n`G2K z{Ek+azy~EM8yH~w*7N?|yKzfP4?B0C%KYKwTK_YXzx%AkhFLZ?LtYt1Zxro--60Ep z`U;6qyd|$-(?}biBZLGB*65?{^YhQ_!)79DA={&eL--MoN}`JpLjeD92Y=z9YgIp+ zjO_}^0M(Tf-_9Zq)7ykWLeJxL9*Ls>m57zv3tY`B8DKIlTbo4AUTlL1kF4@ZRm|+= z5b+hFNWt-^CL?~%TJ)Dupnc8NCkPW<-p1c}k~hEjVq<{VuJ+)uoJ2*fPWk(H=PZFQ zd^`0fZV|t4-wYVRC)Nq~<8gVLc+Vpgh5P*`)sQo1fS0h^pqiuUU*=}9g^>t1RoZ-U z&U{#FbldQis2whMvT%a7`$WLH`!*^8CA+LNdOj)L2OXh#OGW{79ne$xMH79*{Gvw^ z5hNQa!ZIz7BHA)l#JpM1p%#lP#({P9I7xNHxLgv$lfMsD{&j?kk`^?dK_Ksv0CU-w zg{VygZu*!5!q;=4Gf@<6Ra^`H^&L*F z*6Y?AVLI-ce-B`=22T=JOStg~#6^%H6osGq;;-bD&(Y{m)1E9y8|aU>YOyXbAlI2u zYm$gH?zL`Qfv5IYYkhQ_|M?7i#8mwiiHoi!Gc?Wl3eOaaIMpX54D$IIW$xFliDzx8 zR)_s#;ZFIy!C$}{hJNai-F$=En(IK0u{DdgyTWj}Tte@yWNbX+?0rVsFNE+ec_9N& zF-buL1Cj2<_oE+EKoc4f-pPQVzjSGvkZ6%Jj2jA!jUj|?r>ZkRT@q>3Men{oFCI zi!S{JBt~R+5j1kNybe|lABZhztm@blzUVf)KH9U?m%D6k*fo%b!s*v{x1C6;Z)-#3 zT_Q(s=f{G;_U>kqZuS?piiW_Ey*xhtG$=`dzHzIF=r&n(dvti4V^`hv>iN@XtCL#CJ){}0 z8UDNCU6r`FQDW@zMnnbItvjZ=F*qw#Xcmr3ATI3KN_AkPG=6uAW%3kn&TQg8au5Dd zS_3M|eD0ez6G>GW^G87A{&>2iLfU%{?C@3V3F5f)phW0~($#(I!Qx9Ov4;szC!$i*U7qr9Kot!b9O zyCC8>qc!;>n=J0x=5u=Yo9NXw^~#U)2~|Iy_kZ>btK5=gJ&vKvsVu|yr0lVe-_jk* zxU23}SB9)ZTZ+?v-EY>%Ah{!R;*Zj>QAc)eM}HOFCVuQ)#XmOHP5A6UkU{9`vJAXb zyhtl@1^^Z>F5K#7aEDkK#5}H-{_V>CGe#dUOq2fkEWcRoi(^}STrDvBvfqp1#_gv zB3W;btr%DEDO0{Au3p6qLbpQjkAb{*t?*d&F|M!s!BinCSueXCCc3{L%a)RxdX#7a zoZ+~KA3sAzXMk=^JE*T&vsqvrujLlFWi>pP&_Lu6No6M|dYahcZc^P9z4gb$NFbG8>| z+ws0emhh4X&x|^+}vI1d$XgU)xR$)@G@9!)G zbR%sFMmRx6bByTqTH!UACVMHAcDig?$mz|_(nlmQoY^TR%0Jm09E|FTMbA#Q#e+L* ziT?y1l8l*{yWV`)7p3}an2BToTQW9`H8R(=gu?FPaNpWMR$YCRr=PPQHDGJVNC=C+ zy+b4s799lw@+KyUwxxxbAU=4>!S(E_A{LEv4A#LCRg*n<-bB7mQNCC2Ax7Gyxoshf zLnJN?VoJ!km*P@FX$aovu1-;Yp0Sk1I6ebcn0kAWr0D`Wf?~Awd1==9u=%5y+h^(j zK08wgDd(e|UmI2%oOqXj50Ws=t__rU&WP6l21w;W_)XM1T`4vJ*dD_U1a>61=w97n z>GBSQ5SQbHSXJT@|KaZM*r*yw5fI6f3g?`a=T}N90F^XBIqfuRYOM zSc@Fmv929Biw0J~{YpV-n3b2;Q9$*WtHk@ZSFPK|!$$5=`sX4v>1w0~B9~t1i6;8T zSATJuKX(mLKYrheot6;6P-V>kOSa7!(_Gd0^w7)~lcZxBTMW!PZ>@7NAGW7DJ+lkXinlW;iax~sunQz}d3xV^SGYftfBWk^$RV2)Cs;vyDima_7@?1Qy+zLTL zEnyfhRu==6#cV5}YWCS7!?qB8TZD{sdC)jk?=k)gTQ4p=E8ZI`1%BP>IJ_!uzd+z$ zJ~#0+gGU#ld1P;+I>${0}ym=!qm1f!85|4nOd@jMd z4M;07>pWv3@LF@_Jo-tKHcb0JXd$AZ=DVmxJ3Z}zO_u>Z3q%y{FVb=b8D)8?QfAkx zyQY^c;#fZSr8^OB-bAA*jC#O)=!rIwKwOoN2mU_GX&TNQ=P)ed@IPLPZ&%e-0V_l{ z?c233b=-qe;*HGdL$LeGv~a!^U{CX3puSk~ z=TpCSD*MfkI@0*Q5z_ECX}wO5fOx7gt5xT7I?~su#97{;9MAjhNl_QYc4JoteLQt4-iEkB6fJ0Tnaa$qf{RM+OmU|F z=JHP*FPSn3zGF^EwP2iT<52$JbNO=#aNx-eQvNZF#XDzjW`i1So(kL%6Sg(bw6{%s zg~Imry+`Ip4b*%=K8RgMd=P@Z6L6hIS{%#Q0ojRbX;FNC6Gwj$WOXwU(W9yyzH1~- zMD92<4J4z0?Y>va-cE-Ap-nsHYre{$NdB5h+1gHlXAkl5o3p`x>e+Ht=b)G5_zn$` zeby2Y8GK*tlD5y@Jn&z32LBBQR{vh&gPy%h$1xD%NfPTIkp3y$FS5LSP%&jmuStr; zIg=D^U5Gd|Xr=biO~b^F5AI2=R0uN$Cbp<+zXSy^as(7rD_cV(BK4sFQL%bgpWmwI zO7z^j5#07RQUmi_vCf}&%;)p7wC@Auv8%B)?Hw-f-Fx@qAtJ1< z?PXW0FpPyG@wJFI;|8RnBI#&nf650FuO!Muf5Az&nqkTF^>IDHN0Obg)$q|{6&XX| zw$H`r*zo{u!}?mKS@kM}8aALA^*Z`l>-w(4-t%Xr439$@){fXhIcd8=&1zsvjiJK8 z;TuJsOF6p@3>QH7;@U3}=D2OG@@|n!)VEhD5D;bMEh3P3b1T^jfYj#jVmhWQt15#! zn_$|CvfO*9Hx5+6tN?ScC#4m^Ts4CPmcrmUH zouCoTLZ@Zf%%<2{e<<&9xa-IYHAzmFMOx$%Fpgl-Os!G0rI0-1jPY59br|jpVE%A79Tu3L{LRbZ$fNLW7srOrHHaE{MY#)*V?)E#2fy#Fat$S zs{&^i{|H9aZU4$kzOPDr>i3vWO9&1$`0y84AR`3;11Y6Ln4R}YBR_J>(Sl|@NE23e z8{!gaEgodlRKz|h zs`JZjgnfN|%&sw^&_Ub}EYQAggVHnRoo{V9I9`ihmXeWqL175lfraMFQBW;v2kEGf z-u_5EMKA==+RlV`KVdF!5#-*l+pUG$KAb_kj!Qt^YIRN67-pRAjTF-3mwJH!xQkM{ z6=+1HslvMh>xSl^VtTP<<0LGmJ(c#=LbekZDSbawTW+ROh$Z26e5uvm62 zV^r_L#_~YtQ+8lIKQm+iMO}VxB%|j+r2q7i5f6y3jhT|i!v(}ybMr1kw6IfNbxoHi zI^Wmg6x_$8K0H|SLZ(}Nd#st1Gz+$<1d|4?fJN>NXixsQybXv^S#)O6**O|w+VJbQH&3s8{L(RV9tuBg)L9cNc|_# z(b4D!R=*H&mP8B{ejS{M?d9h$_|q^C9Ibt$FJ@7vCR24&#?{K8mi$mgfD0 z9qdk!xwP;Z2qR`{k!l;DPHbP&JFBZ*iqhHXC_IC6^0W{En>PhL=wwuq7zfpx6|3}Y zX!`uCfggKfgntI3jgMa{!L^hvq3SSKrjHi)ak=PP&>L}ytpN0oC|aW5fs|3@v=~r-1C)956pxh}t!x4_|gX{Xo}2=FhMGJJ_MO2rzL_?@H%@ zM2lV8hYQU`d=MI0WtA=p{()?R`WA>(+l5}AJ3o;pfM*Eon0|qE5U=d})JHAmY>vA+ z59*LPY$SV^+5|?SI9c}#(f<{$>5O=?1b%A>LZQgu-r@eWpED)jD0Sj+iJp2yt$YGWCvNjEc4fB_1+!}N9 zDq21ihfd^Jy{mMzT>yZDWE`Nr!a7O+rYbME|%(~AEcp~1|zCy0G2 zvl<6h&tB|<9xZD5<3&NVeiLVu-X7dtEOppo^aEcLLFf>tdwM}ne-Z#Vmy%_e1AUvJ z;0Cd#F^+Y9_2VqQo>M+hjN!RE;!Lv`L1w3k z1JY?j?}_T}I(Sj;?}zCKaEcK!U+jJ3mJ*DZjU58c+(AOxT_Kp0Bj^4JT_=Hr9|ywA z)Zd)kxM;WNjQzdkwu#5%AJ=iperIPM*Q|ZA@D>i1ER(-19Nb=W|JS6Xq+nK<$~)#{ zXDe_jTIn#1jsrmBv$u`I#}4B0gfMBq?bK?&I%MrSJZ2YhzT)zY!sY0P0o;`;+3^n; z{Wv}U005Q%1sMMH!>LzL61PJN@W8X7=PIhn19kW}_FwVe#Wdl&6IJ5yq5C;u98Hlg zKa5cxtea(e45JK@8biF>M$unOi=iSjblt;g;1#eLuD7rp^EpyTG1=Yj{)2FYb6^}q zE#=WC^~L_ly+ZQ)KaK$wsko-2AE-rLWqOiE!KfqxWVi>jtJ@xw6usu)(E)r>qF~LA zln7xQGXCfmGMf~QrRo5`!o|k3>j3`M;1Ze8?iP?$nwQVve8uKl0V#cCmV$EO{O|Tg z$cUW-0~^u?{KN+bmU$!7_?|5MA@c1E1G4t4+df82ed-`fDdwHwg_ztyHdYD^`qU%rVT5_7bta`NUcxyWkEltr!a3U| z1W@xaM7HB?X7rN4O6eOvi+OyCk2+<7(Ay>OFnkpSS(8Y`C`9!ae568U&zoqu-C%fem&u|KZjoDIbF#$Rk!3nif&%Ldl%6DKjk?#s&^I! z%~lUR%>CH*o%WFp(prE}0iAPDK|us;%U69A@TBie7Sz9Sh&$TyI*R>c4;tV) zpS&&byWKhgV;!E8>1SLc63v;6zKv^3Y<{KuNtj+(yLcg-x`RfL||a^o(0Z_ zGHQ?t9k&r+CO0v@ja=EcR$N`EXpJRjlF+Pdu01w3(y}M3>{NU)K7VY1n6CUrrsuf+ zo!fbDYAaPN`>(HYY%KT5&Cm-k~Tw(}hV&;9YdB(g; zixzx;LK_KL(MyC3S#jvh`dl{+(hqA~yy}<(_}amE-Flj;jAH)<9)4jHuu4Kmc06SH z{*^L7oyhPvOQ0TDv(<4>&1ZWeGqC#0nK$#lxRmR1URl?yI$_m>061k7Wr;Skq%J-; zdF!yQi>TP*rMNV0=4I?*l9;I>@e z$a|FH%HiPPNlT}Is0%vM0gO7CElY*%{s-<|)(ybYzXxO1BBVxHKGt1KWX^Z_QckY! z>0tmSi@)H~e(s;m;eOylxW2v=f@gH6(4ZcvrBCy7eV+8KVZDN&>X=|iKD93-D()Z)&>*4EmXQ0zVXO8C?g|p7e~vT#=7|9 zScAY1-d4XrY0@HT2vnO;CT7&zv@e7wW;+$p5{Vio`tFOo`S8HHzM{W{5!G`gqB={z zVnt59~!e=qYd9s5$vWBa8!62ar~$>y3glt$f~@put9gk#wk!*iphOY4!r_sj#u`h&zxxW{?t zMeL;)mQv)$&o=0$VIbijhX0SP?~bSP{r~3}nVFH5J+k-8I70SFW@gAH6++fYR<`Ud zWJETR)xweND49hWq0H>xb)Q3TeSY76Jl+qz-S>50*K0nn=j-*7!*b>FY}gsT7J_!U ztpLR<*k-Q~)#cv!yC77*g%Cn&>E5_B8hSQLCH-j#lal#EFYQsh1DdO_N?V752XOxa zt<=*YhHz*Vk|V=dFsY!n*uChzjJ1S@ftEUMg*JIMI zx*-0DUx3Upb{6ytC%6kH?xq5m_)9lf0Kx$enidGbf%UJTskJhj!`H5?Dx91a)XsEx zlB@%z4{;H2O)oj4>VQr42K9H>Es~Bcbetk)%h9}l7f(oItk1Fr>UIWI9O=UlF;tGVE<6ZGDCd==f9of zIbPhtfySy0dgp{m&&yBlu*zq#gDdkblQ6^8$rgs2l%RnJ0^)2z(_!jYAQav{TIo?yirc?^Tej!{}$wm z*|Iu+&+0nZ#(h{vXxj zL#!6&JNrqSnl4`WFVu1YfUe4_JOFe}eUDgB!k36>A1u^})_#|hx^Y9V^c5&TjB@u| zll>@1-r4SqJ$TO8xx5ZFWC>zpnNkCKQ{B@#`NiV=9zbJu^>hJ|=L)uURVsvIeFh_T zQB&sv4mSJx_5tYfc0P(5_|*>+81ppfxSSU%_+wXXLY-=OW_z2h(vbg^kCuztTYp1B z7r;okvU&-u)5v$~c%2dL5Ai9Wwy`euUkWv-p{Y%#X;U3QrtF@#Y+YBOcmS#BtXeLR z$?w1+2tsotpoC#5rj*1$tUnLx{0lDcW1QQbGlSb1CLS0xV-^b2KZAG}m~|T$e?x!) z0W&ym=8nlzIcD_P2YyIa|E5OeMT>Zu(IR34v|qGk$zAtb zI_MBsfMPxL;j>U1H+xs5fjA~j#-}>Tz9;ntNFJP^<{;wZ?us>l6qpX;NtySXR;=zcpNJg!9{d)>BT->D*(Xa7d|M^0z4uVy^d@?9eu}<;bJZOb~ufcR?KO( z!MS%4l){iKnL$Y4T`>LaJcE%;i4Lfs{g&aUjDeuXD+AZMBYnAVpNmHaYzMFqRb z1ITb~#(?ExQ9I(c zO_#*Fc8DYg7Aw3C25i{P{sN4{K9`owk>~!%6dqK`*AEaS-gII+RQWN|BGWp7a7ZYN3&E z%Z%4S`esM_`Hsv7+gGsVOiKXslx*ZVzXsHvtc&zIkC~$ReTaatMQq!jZM$_=pyoueMd%ix49Y1CbEjG8c(N!W>o{Ln4j)xv-kbHnspWrxut4C9Q@D1QSt!)4y(@gpG&v+YIz`C z9=|E=yiaS}s7;L+*^96o+>4(~L8nd1T)zP6I0Hg!=dr%O^#CkxA6((LUly=#H4KAk zqYc)5!dW}aQUGm7g4PRijj7A@1mnQ4Ll=JEK?))Ua==iuoBLj`-GWQ+a9_(l^LFUK zXW4`yfQ4t9*}6`73~50C)Bue*u(w90FG1EhRHVbGBKW2|iN|#r?dFd0j8u@%b3MHm zS%qR%*>dNSq{PIHkLKq`7wlUgRqh|(4#(Le#1Mo@PhZbp{+8rOAK+g0)UAS+Lgbq* zPYUK)-->~~2qRw&?gtMTnzss3@IDBIcky>XI6Zb{5Zqdbc+BgXrhN(JKZwipa3nCd zV@aB&>LIoB3e%5K_+o{bg<}J#)pnpx{YpxzswMzCRmW>BI;SME>Sn;rj>oX`D?mK@ zcKw2YABf}VLH^VP_yF%LNBlu=4c7^03v}L&vdOwGAPT4aa;d``C9lF{{ju6vygMSu zXjy4#^=`|@`^6{3YAMR%49~pRM9FXT?RIo&Df9e0u;|OCanl;}q%}IF40ojliz?&` zv{x0kV%K(m9-nqCHkc^q#77t>Qwnb6$6m;)YP%J^S7a+HZ!oqoAZuEx55~=TM}(^yApzW7fMLF_6N=N8Y3*R7E#z{HBYVrk#O*9 zV|?-A51&bF-RUZEB^<02HDfHxKbK$a5^lLQfA9u`EJa>!j%|j+s3=Fta2mv;qR64G zTVPnW9f;oX#lC}3Yh-}`Bh5JvuQTQ&Oj2Y$alCpjmccXyRLtaw(Il!2Mh}@S)w%tA|>}P z-@(ceN2f=&3hh*H<8m-?x34Z~lHwzXSE6M!=+vVpB{mLck~Uf?WR6~pPNK6FfWHLe zd8i>~JwTad+M`F0X01n;p>gKj01RE`TIWd>yzziF_X%jG{vBf4PPq-wU7d!yVS(YM z_vott&FVWepD2-&3M4oe0o7m=cwo2vT*`lJBaN2m~iG81hWOc{=BF+ys2EXaVgo^%WwH?-3eIr7UR&90xbiTxOvKJk7jjJQv{M$hyV8&{bc0Lz^O7vYhHBbirY(MR z??x&p;jLeWD6~@;c&0YuConjO(Zh2ANHUNMF6nMzSt7R;VX&ea-mf zTnd=t?vN_VxaZ|}$f`cNHR={}WQ0X4>YOfHu=VVyjn?Jr#cy>9I=YIWv}!K|7(L?m z8h`MwhJll(M6`qJ#rgAdfP4laVKZF!c+LYdo=KH;NVyXTbVQmomAR38sR(usJ+dBu z(*|&R$BTD;O(8`P%x8LH69gu_zR2fooCf-(&Mypt)w_vT{}ZerxDd!PbTwD0Y)}{J zUG?qm3+_5di$I#eSn=q%af3A&677pbar6kR`-5<@JQdoW(ly7)^uBGCHW(f0O{h>n z5ZBeHoF{g2$9HhxV_5Kiy_#RCqjBJV{HH8b{W{G@+igdlnF=>=7UuUnn+E8Fr+#76 zTP0p95s>M&<}*TB2@an-p`;Wj1cG&L0H(aO-BEuQKy9NKUUI-hX1Gesa`0dNCP$@5 z=p3mF@6W*ipK!1(nnr~;t2-^aaKB#XC@{Y1%*oY19;q=!S8m{qiVfFN%roYNX?Dbt zU4~4-qY9hn53K7tBm4HJKh}FKM5Ja${?t#|ILzI~2Q*RYmOeBg^O2bqNY?FM!R-C+ za+5?(JfP7tx3=z1y!ahZUF#he*T|tLq&zD6HNM~GMV=DxyiJZtfb7XG!2c&y`AJub0RoH1cq;f?It;k{s^>+I)z z0|)kq(7?s0b-dXF%i#e!0K7} zwE4oi6Y`^+|B=QbhidlhP$=p~27!%ehvO2naM_Y7zBVpT$tpeKHQ^ON_ng$>DlZY4~b;g3)P^$w8@ zYsvlV)uVSPMMbHgR>WliQNX!{wC82O;pLW{eejV+1ti^fN8A5lS zUuolDYl)pdP+H|dD>pfraitaFnkZ(hZB!ncdwqWQKMojz(5hN&qR^;$ z3T#&>bqIF0-*WL5frBEHBkLx^XuFtdgt`)OFQ^5>JUW7{HE4&Y$@+7Po`{IDVT;cD zzul=)cSYYfb6rP(Y2wly&k+O)3*h=n+zk`2$r8Y!!Cnm?0TrX~JD7x5m$cAvh0d|3 z?tVIGM+qu7AKhm!G@KZXG7BIE7%?Be4)X zhq!A)dts9*$G|@aAW)v^q$xLR9CWA*GHL>`><2I{s<||pUnD*ie2yJ7O{Oz=!JDpu z9k2E|m^J&GqrO5~0}c|Gftv&!v5dE#*|18$+%z;nT?oG#dEy|k-5-#j)0@A<<^84Yp|Gv7ydF|AJwBf<$(c`+i4qX8qoDf3}( zhi(su4;!xby|TB9NV=8I_f2c(5u_x#*I{n2h>+sEKeji^>8HdK|9{~m{}2*N%crr|ORKM4SK&?p6XQe|_v+}zw{pJ_m` zCFL!>ZBM{r9$A8js-QOT(?0NQb@fNNlDzu1Q8dsW@ z5?)%N7((bnMRXB}hX3q3e;(~YLeBNK@jAhS`Bx_UZMd=u_tlX1)hw0=LLuftZUIhN z*!DTQ$|bek9_#6`)>MMcak+9~yQ|Uu!`93kPoH!X{Hh$SLLvLQwXhEv5)vR>ieh!Q z0<}d?0TXVd=gh9)E;QW>bYpmSsSm_elR(+zpw5n-H-A&3Dy9SWmOp^sIk)b7&+x8l z1V6O^@qe-Yn}GDa6&F!S7OKd1k!2MSy>EA6A9Mgo1*uIsD8m|@*ChGv0r+X4-GLNm zD{_<`1(wef5h5>wo%upe0cg1;117Q$5=`l`B4(g21u#=Ey4j${*K;}0r~&ZsI>qRR zFWz-QX=pKFGY^)G<=TWC)`a5X^CX#mGzJM*BS>^*Qrw^T$t6UO zd_{C!1+Hio4%Gy~)$q+c29%-s@-bEqpz)^!a8(&FNEBSY^~zfwUAhq^mwMrn;gbn+ zJuR*10eYRLJ^<$0^~xF3*OV1rfF^tfqGO1G86YElq0%A9Yejz)x7&86HV?udxMVRaXlOQ=+ujQkv7gEPF)y~K zjxx2T~)@pW)f;|ucZd#zwZ zfF=s#%(mPd+6DTLx*WBq2JE>KoRFaE`FN_Ko)2j z7!gDP>gZevZGoNjuxO+W zUVcopOT!hNmwL%wVMWqVpNRSZ%8(%K2^0w$w2`^g@dbner~!{)OdYDJjNueCvkFv= zU#rMI(7D~z2a2<9f@VFeHK3mC%hv5L^MJD<_T~-fN$}0k6-OCjNzAWNgS4Q*nU3Ht zD|F#l)n;AX9~0oGJ& zXBOx?nFf-z&0p7380SIl#%HTwV!!R%&&=wak_P7ZYTLlf?o_}bP6K0kdmwNWVdK6O zTm6;VxYi*aYyxnUt~lcJgB-N`3H0!5`}W;5$G!r7L{fG{t*t2}qI5`-sh|2O2)a^* zJHd$coVA@H1q1%h&?+W#2LoB5vDFk<9|vxzR=Iqh+d?Ukt_3Ws^!<__MW11x|8Gef zs*h~R-4Euj6uGK$*Gk+VM1?Y-EbB(qcl`u_$MlLW=uyzh2W%PY^QSaB zF)_&B?PY?rm^LnYr^^V3Rb#1fH%dh$9pH*JL0_pJTV(10`m%tHlftUpIx0EQPNARlz;}`Q=Wvyn-3lF7`79x_^KeeH_g!BetEDiT3 z7VnQac5?RXhTWYk?3|nuHM8O_f0-GlR>>3;1aXn+&K%d{0TR_`nrsq>#<1b369F%V z1pJZHTb%P{dqn4?FDZIrUU|ce#wO6<(q9kGbsg%sw^eR4jd}+Tr7>Tp+?BOGOdCms zMbB~%)IQU~+(d{3d4dmGZhGOqJaam5M4cq+M*&1PO?O9gGj+FMZISJV2}L-n`Wh{` zVo;)kctZ6n(9UbCGw)V#2=we{f+M}Q!CI)fm{nu%VK_?>k$T@#BrimF`MPRQ=9kqEB}6UH8)tK zq0|BqN;ZgaT9fZi>Qwg*PbV#EyrW2{Lune9HYT_P+Q^ednzzgL=O=uo8E@o5^T7W; zkxLkJJ7_AMz^T=}5IHLD2v+*baXE1q#%5nBlA>QnX1;&t|9|s3Kdij^Ct~J1CT@{V|4x;4;2` zsFO15|H?V)1>I&4GT(+F=@0!gqKwdtTXvG7e@R;JsC0Ju_*s-@U;`o^#FpXVPFD>- zOf{r>HBZ9=o8#C0hT*<5o0d;}Or1nYRAO+@-fD-~ocm%XD|Gt`Ua;?bR}}U9HBw1a z?e&?UYp%?j`Rupo`ePu|&g2Vod(WwKD{+8i{!58LuL{;m z@62^Gn~uW%&r#&i-<<=(rZ@0}5540v*w%wTDUmwZqPN&LD>Noqav4z4HiONr+4h>D z89PRES_Qv3BJ;8d!CRtu!karCcL)E|(L8KBgD|Hn&^|F;0}mEHXB!AD8WVdQ$57+< z?S!MgmZs+({yt)cYvP{l&CtI*Vb@(aAH$ymhcZ%i3R!;;!MMT4F>+M@%BF!Z0J zo}q5o{_URi4ntFuZC@INBH&9x2#cWaSyZi%E-s8`jy^VV|4w^j{bS#(&~khDrBa|_ zfPw%5^+SL6hEk#_Og=JrZV*f?#^$9~(bC^KE7T{j^GXud^nO~U`Jf-m43N*B%;>{LD!=<6`%mM8!_BFd7HE6)G>OqkR1@;}D>8&#+ zu{wEkUQ8P?KxB_i1^glQpDfACw{R5|ayM{5T7R&eDuQAOb2YaM)4WC`Oc9({7byue zSd+tCM&&#NoYQMot?t1yUEm?`Y@7{}hW+2-ee?9_`UXgNnS?an{7~?jhzMyvX~?}Wq2nz9n=^YYXbQ1j@}Y_p zZWRrZKr;gL^Jk5$s_5NFkoj0=pKZ_Yv0GC9yKa2%-rni0h+6t+gxe`Q8vZco%W6CM z{(G<;-uKR9!`OMC?P#sbLz>y#21?7DfrOFE`c|Y z%z^#}=qpVqe`RI8`WD+{A|qpddVh^E53)vW3`1X^jmifbBn6eJo?R9Gfr3s*4G;#x zl@x4KG}b{y>KR6C4aZM~*JC@;Gw@yb!Sw_>LC`w@7}Sl}rbs9C!wo5gzv_YzKG#3x zIRT@27;Hk*y^%g$OV(8bw$}&uhdX+4Z1`_&_so!^Y)LO&SNKC#)mP>X)o6}oW*iId z#3Fqiq89PPMt%}mU=rsTHvUu~`@1O^Fm|Tt&6m$dS3B6sWd?T2p@lIDW!!W9 za6B$XXeeeLjtqXsLbhD4Ej+c;$=GE&***sYBTKQ(Ceq>)|H%VW+&#>SsoQh+tnF@g z?Y(@3NycRgG_iZD{Ks3=IkRxN*F-IdBA;P(j2JFDvoX>h3kHG26pok@|9$wZl-$#3Y@dt0Ugw=Zv!QP#z=a1CC@zyTC1w0ug zllDz}8^3n*PTHYr;nsIXe~$wb`;iJ0lWNM*d#XNDq$(MhOKxY!m2@v*9p$09*b(+~ zwru;K_s_fQ?K1tuMUtqyo$pUZnRasoZD-6%POrOYJ{-I9t1vuQ6vujG)oJC&(;Vvz z*|#k*dZ*s=ggdwG&F%;B@7tH$_4^`jGY7#32!UvA++yZLA)~raVBhANz2f$7hOb{z z8*AonGzhO3MagnkQP6!sMJ>6ocA$++doba;=e?%i7^(!Hr3Z!ntQi8%==h-E&#^`0 z6avFG)z@BDY+kA~lAQzp1djo#(M85Pfx5V3io-#q4R&fLbUvJSHga5#pIuf&3jwJG zlU;`t?RtdG88vkrx2G39;WhyEe-Kn>SUFWRv=RHNsfS2jB>$I}5Q?^qGF6xom&h7KwQi^8$C z(JCwvtj6%uTcp66wJ>=h+BXxCzIG8U8Uhs=42nIh(D)0A?z?Yk_R( z%Pnc`J6khUkadN$GuGJ3Ng&PcHue|lpapu3m6XJ+KcBr-du|?EsNjjRDJaT&3|X-U z-vy6%+9Wr4V>@%!f_XPuEyJ0eB)M5y>3%x2TW2wo)7|6#P`~KwwlcPUofCZ>TV0DO zKQ~nG&9om{(@_BkcW$JH)3m>q{+X^9AW8PaSxj0jx%(ace9F3ZdK!O5k)Vsw)ob>CqLxCB}&c0$;t8&1hxA4-mix>I@w`dbY0~GEum|{6w@Q0@iGohU>vW-Ke%hHL|Ap(I+eM9_ z(p;7mJx~jEa9X;WqA1uO&a?&#Od^vL@EM9@ZBZFB=iS+kTXuwM%g&oPU1p)yER4Q9 zIyw@BJD3Bz@QR36=r<=BD41_geb^ov@bzlMe})|RH>%>2rP!Nni(KA-r1%bM2YS23 zGKvbVv+@Wky`At`tbJPcVXgSlPSr!MyeO#8Mos4-jOLRVB_AOK49_Dp92*=_t*mLT{=$Z-;^ zviv|`^gkSyr(y~aCS4{R=GnlIlsnOH1164^2`-;|Nco5Jjd`^L z?P0eQE~%j^4Gv*zR+9r(SEGfbSrr;Ni2yNmdLKLup-M>FMY_-s_Ucp=5EK*{^u9CH z?E2LclE*u)U(_@kHVHh6XAh|z@5RijDApGNx13lK^y5uj0IiNe2dLoa{`29-F!WFd zLhs3!o}YTUNS7sI2Tna%w*!){j$3j?s1Q_l4K{_OS>ZNG$uhq1)g?nCLxoMAv(Ytt z3z&YyopbIPSqgk|<=jSOeRoq9nx4I<4iQ0d=)xbsUCKNKNeQ9g?zHg74%xyMNVdilrutHVJ%=sLB=ho<0V2d5a^sq8ujp@??>!sw z%`GnX$x8X)__HZD&vz2XF&Fro^Niqf~W_KliSK`N&C3TNPd(dsp)m6hw5sSu-n;&mai1{%3M2aFhYWlyBQYDd ztNv1l-ICd-JZ`TaWYX%E@W2aZ14pnL&=+lUruB;^&hzNV7yX=Z3ZskyUqx)@QO{N8 z2mK1-+Ym&k&Os=H3V!e+b|sIEI{Y(!Pn^gFexGYdQ?;||#03bs#GyG(QGk>}cu!(Q zd+(h^0gVfS&e}`tpIriD1=Wix>E~2df zQv(cJcy=UZ^0Y8B1%fz!wlAF(Ei9z$h^d=`eX7~)K1RrBwljs|gYca3y2tx5vo{k! z8M>|eh+0Ac7=Id(9O=34ec|3wB$M7BgPfKSkn$qPE6#WhaNQ{^;nze5| zSN}7wAdrxUUf7wIkN#Rx1I#SP88xJ|;u{WTToYzjUsXBnHHidH4WK^o1wwj`B1ye! zxmTk(SoH;<-9%5nnqpSM zpHe=@hF2E>+%Lz@YC>QlJ%=k(w1*u~)t6zP5#rzhiLl*+@~I!Y2hXRrvr88JY($G+ ziG8@hBJWsg{_IhyZr>~CE1^%M$#793qU!{`Z|qX{f6f@PWPp*??s+1qw-KGQ?~)Tn z{i38Te(u#(3~MLKXjZQVYfo7eD`9}nk@MR%0hUYK4FNe}>f1-7&EG#)egd3@IEB7f z0tTtC)&xriEATzh@@Q)s1P^tI9ArEtpLj~jxh&jRH-J$`y_57J!u+P_w%Vj<$W^|i zdI{AF+&kNpTlp73II5BxK{UsLJc=6^4Y{F}QeUKTA%E|!Q;+eEUP%RmT1v)z=||-R z4-;8%vJuT(JI(n5CBelm~0t+0baJ$XvD3&z58AAksmVDMh==W19!5LK#4XSZD z%ZQ|4{=a9>$2YEaj%3yX)1n$^uAmNvnZ7M!mrUq;B5X+46K~dws;mTl<{VQT^RrA3bOrRW z#*$G2`02qd0>$GMv%mtoByrVO_;-RSJBAg; z-FSRx)}KSNd-rmn#V0-Gq!MwE;f(3oUkZ294aOCk|FoZ+6nFRW-LgcHlK9L_?(#F5 z$iib-GL9YWOEtEd#PJEBkHHahhwwkR8jrFcqpOB(iL>4myZ*37LDNEx-KTQaF_3B< z_&FA)%Csm27J?qKgljv2udE#dLuLUCpZqh!kFlO@<@415O^;G-UgXhIrH1-j%_MB$ z;dzX?eQxDG|B0xFUOVzMF(~t>TMBF49vQQGxu_?5uNMXTo>-S|4W=ad>&YoZ_9xcO z4Ys5t7{9v?B+}v6Q)}|I{tTrwJ13{EFUnOu_R4Fr{ zTivY~A&V1ZT3JW*%@BFNKr%CF@Q@5iJ1nRH+3hvXezmiO3Phc1ZcI{1qSt9HxppG; zz0}>7_Aodat8m2}x<9%Q&tRQ*>LmN6X4H3=@KZ(o-NR!mXWYJSk^k~e8$NCUs-et@ zIj775uUFnq7P3OWo&}6SkFqPC<HH25zsT9|r5DfqRDTW6Tc@%)4sHg9 z>hXuc4^eq0Vi?0^_v_u*WS!yABGa-|;)1 z!ll4=QHDS}^;=v>6}?lBE?b$?6!sDZKAqKZ6JIvWwfQBNvINp1!zs{X=;W?nKS&-JN(tMw7{lb4;OczwKc=Jj|Y9u98Q~Nn%6L zEv8_hantU-wdtRy+!6pX0P20n_Z=Jk%dVGjYN@Z$2_%}|x_0q@*|SY?@8pYa*;8Y8 zn^@hK+96~oi5q7k4yV4E^PVj*siT!<)mpYpXru?1mNIEo--e;mD}7~blqRa(z7bze zHUq))5{g{rIDji|Z26Y?kDRiB1UZfuUBeA_!&W2rrAI-|%A~}Hvz^EQAf$S0i=um0 z64>hIx!IYy0kMvx!DTgZhJ&j@#7wD)Ba>2}km(cfy${PS#9|};HI(gEn@PW7b>LX< zwb&%lysCEdAUN(lYV}j!49{(bprnfJj2g5ggGyf6xm?=eN~82MIUIWj8f&Zxff_X= zUWyy;bmP}Gil2WGD!XMX3IsE%Fj&msk)Z$snqjE@m0orRCtMkKTiKZe5Lwi z5_%LmYn<+r_EW;_jsS?Jr_;vy#5n_x?+5e0*|n?k*@LlXF(5}nPUns@c)q$8yu!JqbyK=; zc2B9TZ(}d|BpfHB`h{oRKaJaxa=L%-=>-Gp#haT$8ECP_ja-ncJW$;7S6_2WFu9^Z zstS5rj#gN7`pPbOX02>L43J;gJ|`lC2SZ&1g@wmL z1dAjNnu_(`4*dPE#1OcK)O(;jJW`Ltns$*JDeR2AK73#|o^a%j7Fy)^W-V@}BR8K< z3GDjkPsyYBOdzWRP8~+`Xg1n9_5a@DAov5gqK|*=Y%H7oTAb0gIhW6iS${T)PfU0P z5Af$Vi$-VMk-p}3zOGB#VF5Yw+k_{VypL2txWnRGbDsOO6FH8NA)`Lctb1`zK+S?r zYACGjcYA0{p}++ULfFM0fHt+TDQVl?IfKAN^_UEV`DxHUdaa zwMnBpSjaSvgSXAN;xYmzNA1n};FbE=>FVDO6G#%uFo~Abdqv9=&t6dvPzC6N5jx8R zdk>E%0Gd>Ai~bF|u;rB0pp$RxK~e|KJu{9S9%LH70@+kx+D4&>m>KU*1qkfD!BOOm zMpy{=PQ0zi(`;MZEE}z}m?{t^50BpbQ{YBJ<6a8Aq=cZW?~fC{U3jBiz+uYbmtLP! z!S-Kaf(|JeZn?9471^fNyK&>r*TX6Gz+L2@H*EIbhPDQYOH^?-4n~*Y6=z!b$txaf zS}p<61*%aXOsVSKC00+QlXDIF4E5_wGxp6QsR|G1)puv5q>iI_w8AlIVJ4@kQp8Lg z%Q@2^jL+6XMP2Ul9Y*9Pn?28h=n(~5Y{Gsz5WXcnjDkVRes<0Jlzj>0ef1S~lN2eW z;dMTS7WN%Sq>j1;Z<@XxY{UghIy1d)PLy=^keZ0JG`VF^qa+(BFM{al){o|O`aeFa zqIoEW;NML)W`zr4r2yC)C7+2+LE1xs;kE~&{`FXN!)_(E#BSb3Yxfg6H7vN&^fO`7 z>#t)H?jlau#S1YbKW=h=1=JG$&NV zQ_Qol4z`DQPpVt|_@kc+9sV8O%VT&LxfZMzDx8ojQ2k^t_3QTzp*p8g6-s~mFC+#=R-eP~e z##5O`zfzI#QhkNb1nd0Bj>(hSn8vU_kbD!HzDvo zG&3R&s85*_d!_b|!Y)FWPi`G(P8rx5Ul{Et_n_Gpwxy!n47Fa*!9uP>PEbxkJDzLO z5E!r7t(n{Svo+e5*sQ;KIo;L#GRvrA1U)>3$lG8okd2}p7$bLB746bkiASwnBmNnD z=>8R+2#iJe+RHbePCgS-q}+e&QNaHuhfUk9;PEt6cGT9NG`nvq&F=kg@?biy4)~at=fxgox8$)Z7Zm5V27zjs2d!%a4*Y@oaZZYl9t|oj zJk(}t4BDW&yo%q#7B~5LN4{_JWy#2WN9`*!T8iu1(yLxYUYK!Hk8m~;@5B4lkx?Y;p zJ9yL1%ID?XFAo?N`PhTcjf}=b6Yx@1_Fqd;FOa&vKxxh!G;_gRZZPB_{3^Dv8MgCi zQ0T}L{maZKX<{eq`bE<_OwLYLq6i=yvk1h(+El_P zzeK3QKKKrcH~lP&)Y7^U>%62#h^W=r<@$FQ%i5Sz8$jz5w{1g7M3nR-f zTAPUa3k_TEw^q3ND^X_%zrK+3@!hW0x}n>ER};6)VA?V7mH2SGuGi{VbTc^t-!Fe}bu99uktn z&YkNgw7E@ul=j^Ldq_)Cw+jmSxY0{-phLQ^0Ldla~GChDtUa-L=rur0T#@qkuiYL=KmG&z} zl|eX!W~%&DU6q%q$zfuwxkFl?=81mm1~;lwu`okje7TCi4$TceG4 znN~D)Y^rplVF$HWk<+0cJp2!OZx+an;gJ;RW71u{k<5SV9>2h%jj;QT*5{8&i+wO# zjFa7(@Lw)9h>dVBi%qFXLwYcr3SJ|&^>D!K^m~d;z<&tr;o&u zF|Jg+WxmMAH*ywegWaO9`;Pv)D&BAld@RxqTV(j)uFe?%JnUjuW2Xrb>u$>4tVK5i zJ@(H|#^5Aq1J)d*kKG8 z_qkr!>2)Poo(qK<_so5mihtnQFpHO( zLrkG(Je(rxWg@51Tof5#F1sjVhE7{c&|DDmPZ+V+P88_)V^ROpB7|t|1_UgeW~Jci zR{%-*xHsM}e6qT~ZWeqP0Tkva_?4cHauA__c$7C|3Bx$;8jlqP4A922#d}nVoXTU9K@fB#aK07``O z)YDG?!O5$|IQwb3NbiY{KNiW{u{{mNgU2Q+1vzr4zgExtzy7k0Esi}d50dt}Va74k zh11DH@rIi)SVe|+MO63xJ+iPM39YX&F4d@$qX)6bu~qr+xVd@TyNJ#Oz=}~*n*JUd zS3P`GS&D(&e3i|O$7?6iC<{j0K3K-Llu_+H$XUg()T84Tc-yQ-G$Q4Xjx?j|mxQ_9 zb?JH|Rn6~9**Hx-tJf{dmnKASg~;&_bOztdJDl#ns{KUpFbuv);NfrAls=EoD*VE% z>-1rPn=mF2W7EWypvO&|NF7%g+*&j=Ra2uRlJNFL`nnf06Wq=Ys;}{grO)TR!g^Cj z!skOl{+$0W}-hi^?lE*niyh zdaNo@UT%cEduXqoOPs)m| z&oa}Id;a{BYu{k`NiTpNQl8oYfq8FYorQ1L#`cSBW12}XU4BEaidPK0NHfuYN&@~1 zF$Oxxn?|l*s!dJ%7I~z@5oYb7OHyxE6D-fhgfZ^-Bx3DRV69`q56;;p_QW z>Z#VU%1tEH3qnu6RlStm9rd)dzOR+l^Q7@1F*48LRuZlZFQv)vk}lyacI%xpyTux# z`QO^qqDW@_(P+zy2T)W{XZg z)=oKX34r$)w5YXdipb3N^Q4c%GYeOHmi?&|L1?;Y^ZSTz;t2w;MSSpPlGaH+`CK5pwMiy$m<`CX0ik(dM)Xj%s;=) z?Z(-FSu`zh{M4Q`Y_a-cF=x(Yx=v1gNx8K^f#{1zH2J09S>3FP#p}JMZg2I5VK@DSD7R`q~8ln!-{IrZ*wYTgEa%yKqmj7njSg7QNhjvMJ|dFr`(5Fjll-;`>SA1I`;vNBwH}u&)dqcf_ZsI+C%SeZVH(sBN?jvmGZe%>zLwZ)?-W9 zoo4V0qEDYg|1U^l_BAjtPH#r>JxqZE4+{Hvh99n*lSfth)@AAJY|DCUe8#0+iW z&-f`sy>HxSxP2p7wdl5P2zIi6KiRn85m|iWkn{rvTZ7hUlX6bEr>}?~-z?OOS!Ngb zVM_WV$q*R7!Ck$-;S$RD>7E70(VeHsaC%)q@DLnfY*E39hjJ2=JSrFX84s4Ee+&&f zYuf+1Y{uxhAUE!FK240J@=j}4bC_slUJL7s8#6Uy?vYo!Sbf&6SicSZNINcQdPsMP z<%lan#_30*DDZ#<2xjzk^@)2a7J{O~pZNKW|Gr%g=6hS6WT~>&;I*P~)u?9DRqvldEKygq5Aez|pD7TN1dxqaI)rZVRZNHKY}6jj7ZV$G zZpCpfo++;Pz09QODr;GL%&R!w>@K$cU-HH`U*GuWhLA~D2j3uI^9!Sgm7`r2oB=V0%L**8N|6guFS{wwoYKrh;NXn~&eJJkcmhHP_Oy{Lx zARcqY;7a=t8SQs@7UBEVxZ?BGvmeL5%ph%i2gRzD4k^A^%6(|T=nZE984D$Q3@S7F z3$r>w)WR{}R)=Gr^6_5lRssj-os}A15GhURpa1HQ6BH|s`L8&fOkz5A zdGL4~Fh(z(ibmj06{S%ujOV+DrlQafL;vP$Z%xxR1b#Tf2a;$BZ{K#&@wu2nAti1A zo~}<)|IgMTe93_`$j;W=tt{7A&suq7;O@ZQ0#e2I*OYCe8Pyk+ z$>;!Un+s}8iZ*hq8>iZ^fMgOnnGWC9g?~>5+;xBf4V8hMfu--KiNdM~5e~3Oi=TK- ze`YON-cLq&#S}f(2_JtNe94UWeix;`g71K~OJ=?>u5fs5=hwk8N!ks-(m%Cr#gNTi z+-@wi|3}X-U_N-DH>~_dZT`As(lt_dLT}nw!Bu8rmlx_Pu74VOFIV;Zn+c9SmJnTH zF+Y5IjYCQ1qOq#KD@B*CWglnDzhWi~H3m})^Z%b$5OacUvizvMbk3d}#!hN1Wf@HS zoN{NHJpP^Z9mU#>vpFrha!A z7{jS~=>Wm6MLg;wZDxUJB$?+~wvPWSv0zKi^JKWim}EDjEN(Ko+p&12b~~S zIp`zfJLW?j@6#hb5dKyvUnKcw-{865fymAXEx-MfzKX|COpKUgjPUy-Va`usvgMX2 zUeSM*_K^MAZ04uzHN8X%r=9a4EZF&+zx>Z0zz<>LHxt?z?Ej5;iJ9YG#C;-_)moUH zxM=Faq^pitd82$K`}vH>tw z=*DbrJ8IR$qACjiTQ!0f%I!LG;OXf8t|$7DsNVkIe6v?ecri7(XG^*MroLk%wg{-< zxHkIXktYSsoyxA{?eM7k%C>**i9XG{|CGOTe}w$^ z)M(*Ae{j7M7FhmyZCqqfbeC1WkYul8@ufVm+?gq-Uj|`VCU}XKR_}hWFq*Ng9Wp8# z`q361X4r;gf~d|S=8k(EgCdh+oh)lDG*R-e@g(JH!x+*`oXVvANBt3{RH{*}(@2__ zD;;z$P54jK-J9{cO5@<~;qmcBkx@D9)iXItzEM3RaTTgHG8NZ!%vQAGH>(-UwD$0A zi=WyD*Z-RcBvDw|{{N|o?njuT0GD{zuN7(GU~`NSo=!IGQr&jbi)+X_meqZ2+uxCC zbn{4I*=cJNmbU0+f%s6*E;Flzt)~a$Rory5HANmxiRfuCCxS4X3{92$X zc#Y<1hC^!PBiepJ^H&YrbkgUT7^^pB=T5*+E0egL z>9tjm7Vm!_{(BRI`X+sdD*ti+XTSKrZ+H!haAda@Mh--68Jk^a*zrs%d_`pM9Hw3J zOnV@0m-3fkgx>~9e42eYZ=k?kM%lFTeELidlI|&M*J|nYS1T0klTTTw%n|15w>1>r$WBbJ)Ca->!Ais<%SdNM0b z7^|)n;-iN0WSeSHj--a|TxQSmuUSVEcKMj)i?Mccl{UA2IPg)8Q^wx$`+8nSw26i6 zQ-AU&uYCcIr9lQ}uM>8ceIX5s)NfF3mSb(h!{=xH<`O3}8LnlGS zNV%S&!W374sL>#j)|Dq^b!J-#l?}iABzJsVw7SA0Ry;#V?DXf8r+$YO)+qkMnrNkh zinGJ|YOirUXL)1P%Cta==RY3^G9_w<(gXf}tpE9S0^z*!J0%7$p=!*hDXP95)>Y@4 z0&d;OP{DS(lG~WmwR@yvyKlY+2bks4FACO^nq;n+W@Mr+oeDbo)lCP!$l zl-MPNDF+VUDQi~32+mGok9!4U9LuNfjsM5q0kD+=ARC)oQG<;CEF@P6<~t)a8BR^@ z=@Gihv+$1Hy3iQAo3+v-x$8?&q~D$Bkspi>?D_{9>FW~aX9wAzYwSF>!46xKc<`_G zf%pls*M&Kli~n;EUhD)s=OK%&prkKoJ#cFe~+mNu%hz#a}1^bMfjO?H8F`Dyp`X7d69Zkj|z5<;+J8hpLbf! zuzPw}ci$c7P#pfKOg3^NEB7SfrKp(kG}q@XGcqum*BrC|YC175SUQ&cjt9d(b_ULr z4g#2ci&z<txJcr;#FKOC3Pp|Tx!03%G(@&A!^m0?wGTla{hfYL}xh;*0q1|*~# zX`~yaI~4?^Bn9b~?%1TDlyr9p(jh6aH{aTxd#|4J{ZZjTczM@abB;OYSaWFT$>2!l zn{w?{w}&W*Vh3V7uLef5a~$ z-*9kXqvdRn|4B2IpI(s};-aqNF-mGnDDvjN{rmqeYXEu&R`Z`X0)K_a9-7P!EZ)Dk+d25k#IpU$K(7D!y_a=TbQgq|C34t z)%@G9!vV^I27gkgV99#qm-m|oS3w}wq*^G%$|H@X-ZZA=dHil_kb&VY(cgsMe;+O| z1Rrdtm2@Y;e@0=dnCJi{-XB%z=g&nqobDoNGIH#^eNk=a73M@ocI%Tt#{S=T;(u=) zo+d<4gWt)#`R}UyC)_on&nNbGkVil!?&OjsP!CvP(Cx^iTsB+dvFoG}{OiZ6;3Lp% z*Vq%%*@79G2T0RNnvpyS*0zg|Ki;#$%6Sy7(p zzn+E;$7+g8J1J8?{7GgrQW@i;OJxQ}vUjzOGL~=X-H-p7DX4~%kt%KW>Kq?;c+}R# zy8WguG$J9;+J}=&VKRF3h|lKX9SR@ZV)Z^(OHgh^kJmSLfc7|-uGl>CSZQ`z7he7u z))%J*uS?p&$fW#d%(+BPDb9NePzZYn&RcDHsMRm~%3tpeqKN)5B>QX@-`<{Bc~>F_ zg};W*=_le)sp#KIwFxo~Sxr&X%|p>w7m2S{)M`V~`Frxxe-{^QhE0M(D~@3H0O~>R zKk5-kEb=N|Tm#5&*l|1+D_DB;6&(cNl8ws4TbLF_Bfvyn|09)i@=ENK$X00lsmsYwofzI2O z6|q05QT?x6=07kjP@0fIheL8u9ILk-3d>^y5thPy;{WRkTG8)rO?Aa!|mnW%zLasPhY00|9YOb*e%xD`JuLX+I#~jqtddx)XMBit*a0-bhW3=NB%pO4)uVclbh_d%-q9b= zSn4EF`!=z7k9D4zm`cP;ytw(6H&|OGYJzBNu*i!jv z{bH$ChmYspSs7ffEQg!CxaHa#b{2j!_6lBEBjJsG2M5u<~`|V{0bUnC3NdTV>ID zJeS{-KaYoF2`HFV@38LJfGJECeWH9H<-m+$X`5gREzKM&ETl&ChJ25TWBN^cts?cN zCfI3NqO*DShioz9ge-A*xX!;3RSg8*fwgB1$F&Z`BI;27TP*&oTa`(Kl5R^_*^VkR z3Y9#D!*>~?N>FhQ_mgUCQw2vQEDwWv&YBk3ZKt{oX}NZL)}Gsw4!@9g3hv&<9}Ykz zyXzmZZA-!`K@|jPGHzA7X^mSsD9xF;d8fRVJdOKFmiI3CzmlmJVb4zYvvLC#U1~~+ z=el7y7$m=s053y~!?b;@8!+mzLc>FPek+YFJ41<>lg+#dJ+}6fj74u)?VH`}B3b8~ zwLmxiqbX1riuGbq7R!UzGdMy7b9Ai@pQMTjhBUPYsUh<`JM3bK9CAXrTZcuY!%1(@ zS(2yn9^Sg+IWGWzSvF&;| zNtbB0;m48cwnLQ)Uf+MH_q0BKi-u1t{=-cqVGwykbWfr?T6L0*bTKNAFwuO6+d2yH z$IPx|4QXm@#;_{<`+5gF2bP)jp6Vo+}*a2PK7C@BPIm3BU0-IO>1^v*A*W?Ct!$S z9d1V=Wm(4`kFn+niBc2FXQ?GKXBoJR?D;w-G}(FMudUVZR~dF>;z2C%@0a*6Mvp@h zI(?YGQZvC5yC(Ly)4TT%jwzE5h09?2#|hhomUa0+)X261(&7S_M7P0o7PE)AnK!on z{g|x}t9!P1Gfy2K?~4i>#IkQj8SrEVKA#-J&}1;WP5Q`o0!^#X>!VcWW2E=+eLCAW zq~pfsI&%DZPr4VQZlHeJmz#HDr&^g;OAjB+RPOic!(Q7M0o*U#9QR8o@V^}WSD+W} z8}hF)ZW^c&J<4#&Izx@k-^k)Gq^E)J?J0;&-pv^a*8xuM^7A9xT9(Y4MUw12G4-U| zJTIQ7(VUq#8oo%)S|6BjI{&Kc&}Q(Y?!vLI)9!BgZ9tT2*4>^l0unQe$A-4hLLXi= zd?Y%&H(cRCZQE-DtY^ts#uYHQ>&*|it5CO}wp`-MlI2du0`t*o7ELiR>iP6p&>$`; zrQ6=a-#Py8Q+tL&gTf71uq$OAk>6XC0YYT}xt+sJ5TI*Nu-)dFZ(SkY+^#hv>sD@i z_z7Qk65lI!4?~A(%r!ogH7|9($dia8Rv=PT*=!-hLNwp>K#KJ1VGaYDlY%EQ*t4dk zTM0+w(yZw5;#3PuttOiGx3KtVBzpt{+YRvLLj*FA%h4pTI1NGGf;ZSD7I`IU9Y%OI1+sM6+z3-n2e zztwgGs_jT5ESc2F_tU5u)STxQ^G0b**9@>9-%rf?fyraFVl9$y+Vz_7=!x9}5qil{ zNC+(Nsl$k*E*JUP=Q2^faJ!p%;<=eAyOgkCKs^rbhB`eF055bUT6ZYAUzOOEN<}Ns z1etdYziatm%_&kKwKB9Pdjr-;v6CJZOzZRpHGYOi_}}u_0TX}D>FCV1w! zq`ww^FZ~;q_D!tVBAY!3_Npjk&ZicY`DCh_ut_+B%b+Xx$f-o}02Z0te3e`4Y!jBA zrS7%egPBfLYiF^|mdfv^p=V=wR*K@)QwEqL+N&&bRYVQ`^WEf2@1mV7A7myb<3Q|V z$`_GZ1P)qU0Uc-4URlu+V3Ld! zE#~>%I?CihdON5>I>p2|0af&pXD7)MM|a%%%mMA^FV79c60#;aw=sUQq;ku*j%a4h zCbpn%)PS!dh~1vwAh&#d4Yb_BmH>DC5(?$=9cSACL~ST zm=8g{l(1Wv@KX~$x6d1-4t3EzLG%WBWy8^vmc+k9VV*9pN)`+%kf*o=NO#}7`;-O9YG)MO?FnD|u7Fb^L7 zleC0nCL*VB@$o$p9Ax~jK>QbsXhLRb;Qz3?!s0}#2$3Rvixa*#uGHLLwjFf5GLW|jyuBoNlU157S-3Fj75;LWU85- z*dXH%P2MV9S?qSHdZ1i3-Zp^t>i<6sGtz(n@3=3qu9DOXjz2J$E8{TdM;&j2vq&X# zg2Vk-*Sx3uQ^C@&?(VjPp>oX^({j1oojiPRxM4_TQmXaONZ<{A68^I9RD*{ymg*G1bGLct>P<^HdwoT_JLfCDX$=d2Dn6*R!0XJ$ZbJ14rGAR{o5U zFZ(*OS(81ZV=vbGgO_pTrDI-Q9#NWa@u>f8Jl0*0d$T(#E~IT{HdC2D(2lT&Wr^PP z=NuDPm$7$rypyg)asHD9$^3y#KBjPAV)kGpl71XV(IWje zY+B0hR%%GiC38S^H-A-ubA`PY>M&n&3!ChcTZg^)FLlNEgw#G#!_1#BGBkPr z$etvMRj&907#!&&tRbkb@JjX%3~FV3OB*6txu0TXJgK|Y#4mNZ^SRRd1nHAS7GFKq z6fUSavNn3g4V(+YN>Wl&rb;smedcI+%OC3n*CY!8Kw|tdJSUzU<)k-pLLj{p2s%~X zK;pi<)PZ!8ely6!ns`c||1clLX^-w+ASo{37ak(r7xZh3lySJ69LwK25Bi|Kc2rH# z;5V21fq4qw3?l6HfV(zpWJkNZ`FesC-*(LxEVRV>dilXz&ljD$b|;s4qZxsbhW<>f;KY-I0RzGk`h|7FmixP@SKY4nJ<&BX0%+f4U6MYH z_+6us>OpRk$>ELIVPx z^FbZxu|2e{eEeYQO{;^$XAkVP9V7Nr8RB~nbg*t-6pc7<=fQo%WWpzIq|3#YhW>)1 zYMeP5xA>rLTwOA2-Q@VMhD5VWTB#zm_R)?}hyp%ybvGTDbEAi;^9 z(`;4ov#v2#N`^VjelHHyuq@KOS8Yn&cHbYX8Qk$ml8YE@@QV}eM0+JTo$G#Ek4p09 z#aqeMY0v2G*R^Tt$`PkHGv;-&^{qu0c{7)FqBn3phc~0vtygn;K2?U(t z#=WUDLpH)A;nANGJV(pkUv1s9mG2bl70+x!28eh1bNX80^9w_c;xdOZ(*G4lfi?um zuF^8a7Ijj#Y4nx^q{BEV7N0^`F((F-mj6*;qBrd0Is0XVd=y3gvWbpX2Iw3)aMk%F z5NpzViU9xiITNrgtzEA7dI`hPqfSs<)+z1HPX#q z0sFn1%uKh~Y&OG7EJPxMblxe^t(60#njbN1m)SkLE%@t#Dpj^Plh_C(#4(h`gq0@M zke%zr&~RBi7PvFFRgiqh-yx0FsC%2V?%cANIh^dH^QcE)g<_3LMFi>6U5M?aqgr2dS1(nF*y+1XiUqANSus4lEf@WFa;g2$l&A#Q_g6X z0_-oOaryRa7R_2Xs+4g1vflve=^Pz^A10<@fBM(~htXbLXlN>KT!u}B?qd~JpN9hI z@dl6^!+Pty%l?X}hbCeK_+2Vz8v%EIQ{yz~)8`sjtpL!%Gg0-TZC^M97B#jX3s3}p zBu1FRt$`u>@7{A;y0K>X!y9XAYGi>)hS_p^AeY@V$IOdPZ1Zjk_xM;^87f=QR^h(5i2|v_D_33WrFc2m&W;;^#(H zLW_SAr5eQnC_F#ayU!lN9GEZ3G*bPw*>=DvitNvoykHRG50i@_e>iQDf!ljw;6El% zJBLKZQnAjo_ga!rfdu-#B~%t^ z`zlLNtn{Tnqv1Kwd z;bGZW?0vf@sIi<`N_Xew&4Mw(Z-)haQFl0VSVnDhcO!wn`+LKi>98ZdlDoE0=-sO) zNjQayfv0|KT)S4kC-yQA^P_{o^1`|!Vl#lb3)@#L$WYCue8Ay%5+y6mlE@@ ztpXnw-1TD9X*)00hXhn_s4v3dFlg?zCcj1`(bpHclBdaoG%5o~rfd_qq>Fg(&y8)E zO$AG1==p*0sMV`&xNkhoj#|a*7A$o>zqU)>2%qveFa>nic_7dUDH*LxR{C5<<6yjub8}xl4V2_pyoB})S>TnA&Z+Vvk zQ-IoHc7%m5Vk*4B+(eDq{&~GG8}JOIebQctMUBUW2uw{x!m& z_||ua1aSrV-ml!TDBLB$?@gq8aFa`{yg2MHyO{KKxpEUJ+BS^nta*z)@yq~nJc+ZUWOlDA3WCGVG%7r@-FJi<*Zvt*4~U4}n;zk%VHfRj&!-<=?; zr33Th6IL5O5_p`9$gG3;*9pK*o+Hm!#QQ8B;XC$j3&{*bm^EN=oVZsvrv#kATjGzS zA4_bEKBSHbZtp;Qz!ELy(5CL5D|uJo30v#W)in!Ek-FO2USLzTG?5QD;li#fZ{QMA z<+3%(mbxaFKlE;mQ}~xOX*@K*J!3aN5X&FqTw2CtTEjAJv*o3ScGHLTjP10xd!grA z+8n{K)I(HQ=$g{yz22nGNV)=*TdoFKt>;z(p+vew8(<{3Zr1~ByVO`kquP56Q_s9*j*VIu)wn* zA;1IhskgWH7f6WNh;3P8z7K52NZH{LC2$(-C%|W05)M}U^x5{II-@mGlNCr zaNt_c2=oa6krQt-pTb*zd?Axm2BW6!XvyqzCBx%>0k&;J2)kkAJBvy&RMxCPhZpHR znZ6P;B;4N>CKrm;?^RkeUD_q4XkV~WI7w`TpEqYEILa;5wdbcGk~%NO31zXk&6I7# zhx3aM8w>1Ln(EHh@-Xzx^GLpOgZwSi)iRzv5?ZeDp)q;o-vBWi11z_2Ulv$u)z##W z_#Vdt{N})+DHgbikOcZo?jXI{(s3m&%L&I3mk&N7MH!|>w!d&`hB zZPS1N-lL`+mncW0(H{Pyx>#AJQ@CNPPb0!xOf)TO!p4_P5}WQ<%FU#@)si`^Eo0W8 ziHTt}{J^iIM^LP{<@$3};N6e58mRsm(mP)Uz;aT;6(%DdwewI)<>XQ!hcj?kvBzTH z;IMH#S~zJd=FNO==0PujP9Q~9wj`9_Tfzh_+Wx12AM-HG9fw81<4R4aRX2f0C;si0Xg*t!pQ6DqoUTOYvnp|63a3CeBc?Ki! zH}DH5`FnD=E5k5b+oYkdz<4z!=|kxILBWyu4cC$mv^Q@jw-^^iv~%Z3X?AYJ(o9U^ zRCqNLU*<*)9(s*?ViuR|O)Qe9+GKBkzhC6WGaCxgLFQ{#G3&LZ$ymZKarh85<=ko^ zm_reSRvih^3QSf&r!Wn~(o@7V|NG|a-^CSV0aJKNu!wnj%7GWpTKRhETr7yh=YEM+ zTYKY1Iy11PyT^Mnw*;@7ey2PO+*{W_VV;J98hfLf=ehSeTI<3t{(y>3ef&MopR_dg|oQTU9& zkjQm!yN?W5UV<_H6}Cc+CX#z%=kv~0RwLP6RUSW9%Uq$}2LAeq09|NFM^mN=OO><& zfT3^bcd_1e1dOV~C~U{9rfblMjpjLpof!M~ZQh!ipKebafxOU*$=*x_%#z}P6B=E} z$-zW%k{xufZgh5*iEmx}e1GP^#|ME=fm;r)vrd)S(^byK^g()^wnmT=B91_w=pnT& zp+-1aAWToHz?K!#-5ccExw&e)i8~&uCv#DYLqTWJZ&0!83v%hb6f*0Hae>Mw`Y*T^ zwY7lU<6u(0Vk4X2;eU1IJ7wVB8-#*+t+RUP>?d%!!5=DZ83K+N6J1xMSoPB;m;%6W z*sML^7NhxEH7Br<_dM)CTR)vLQf^k-s}=Mt8XriYa+hSY@dCU-xhWU z^P>=j^*cyERjlIalERof{*T0h^wEd@V{Ig3rrv$FMMBdORA{$uPVjoiSYn>95mzfc ztoHmvkD6JAVrQeJ$5PmlA=t`|#sx#$Y?8w8-t#IQZ)+BL>fcR6t)dechEMNxhKJ4O zfBo%&{3J)qA<@SnqZ@$NlI!-tZ%_ZNQ7g(&!#({b?Z5|ou+Zdsyy$-e>uMC72t1!^ zgHMdLhtzB!{DBNkP|i+PbIV^H+SG~i z@bkN0SXBhO_tGVKn0|TtRIkkwVadJ{jwE1FCQhBMBob3sVrIl8BKo5XKk%J=8!%Q^ zemz^FULD`M`i56y6}IP?64j1#G-IA*&~(=mWUQ74H$S@#ms))#yXi7^RysqyNeM<= zmDre-MK5yQffKSP>F~Fi9ZnsKCK+tILx+ZykP=NZ?{b189lR3#+i#l@fH@uYp}D<( zli$TL@8gHhr25<#MfKE}jBf`PBAtoM+_cP0+OwIdO<-=4Z!VlPdOtk5o^gJB{I#$= zz)Nf&e{Z?)qd%8zVc@o>cT1bq<^v>N_QS=DPyTa4@X4X!Hi{OTP1?|Wt=NEt=ilLj zBU;CfB)_gI|L2%P43RRFop0vk+k~##`nb7*je<`<5;m-cGk?e-e@h{x<#!nTAU=1& zt)OiIBaM;5YctE*eJGV*8Mq_HH8{_?#?&rIqrAvNn2;qbBv{g7OfM|+FSx{ga2b92 zZqMUJ$(lCp5xsXKWet~t$#+I7GMy)cw` zuVI-0-wRg{R2~)C2}ZKwh(kGNdvFR4%tA)Ov-a~mX2L}3on|J-7v9g=O&)hY zvR(Q~Z=&ecE_dVgQ9ph}o9?$SbFcS;!BMhEu@wwH%#Pww`PNzP!WaB5uH6EC?> zf{QEsh2oV^dv!&hqUqOafUY4>HQsv=_SIWM+Oz#&it4AY&gcK9ip`3_X+|{SJ)b`kBj{X}r#x zs(r|~5HZPhzxpe1G!o`~(!98^C;a7sF^(`q#x({cA;U@iF%g++Mb78qSVRi3dnS1a zz&rOc$H$V%e5O~>?hZQC*l{dq`Z!4;)xPfiHT=9S38boLF7dY;fmzGvqoCd>&MOy2 zy$2<27N$eubr4j43V=C_{JG5+LcX%|#B|)Lshc@LSy!0c2g~!rLwIke_=A1%I4Fc= zFwH8gajuJsik`ag;390k*PD+AyN5dTVg6tvfC>1ndSv>&`5G@#p8t#S$*bI@(bqFwfiklc=F+0 zCji?46{{0v%>PGwC6azQ#c6<7B-JRy`q7D{pqeIzzCY}Akhz`%UaM+c%SM2bZDvEHbq4?4eV+6L|fm|Nb z6{vqsPs&c>`JO?QL{2r7*;hkSvfUGfLi46#ktMoOx%GW3;5)BJ$}U4eu1NmA3i8%5 z!p=-t#^>-v!g|WDZ3+M~vJx6|y@SznJDr9XS^OS{%PJ&dS2II+S8?5ql&7n{a_`rt z5fz!FeamiDf1j%Bq#zRvUWaP4EZ@^@NDM!ZYl*fC$8E7XP<+qz`D(JhbzQi7@ak&m zCcSdW>-myr8LEsY@Uz$5)K7IY2mf$N&$(rg_052!>|8D@gASaw`huLHSF`40<_(KG zXR}MdWl6F-nv9< z_h}g3pS@yIeX36y_<4Id$&>GxY9{zE)fL2cOV&4)q$ z+?B1LiS&Aa9X(#(J&Wec?c$D-E{auP)!i^nG0!I$thgRSJ_=9Nb9g#Gu_wl8ASADP zHeRH@4xHMFT5x72UL__HjYOn#yrL7%nIlA$<9=HF$0-m=RQ(B^`*07iB$Fm^iT{tU($dXi%*LkuxX-Mzw zE|L8*Z<_rKZrBVq`%QVpZlx}5hhC)h0Y*%PFR+!gydpMg{O5LM2b1^H>ZoVVX?FNR z5{by}Rgn;tb_a6I?_t?&Z7OLWO_ZHM#J=xia$Cnz{tOf4MC!BAm#+TrnMP z+nZP>W2O(|43KLhayvA_s~&i+#ddvGgfeU?^C`+SK|ceIQ20CvcN-_jadIhU@?Y)r{Ftbz00|RS{gf2wD zBH>N%7p^CM+F4qv_9BRR+4qKMe1EGa7(yV4MUU>k&${0eugVoDc5^W_ShcrmBEa@# z`*43ys66`dD!L^!=)D_%<-!Gpg6d#~u&0v=LCA9Bq$Am9<1s~k?0q@?C%#9$^yVYk zQenP)H`f6EQwWw#oA1`U{4wHe#h5HbhZ-s(t(TTEGTeKF5-WSc?uV&v772Lq?z zd<5FY;p=m<=>9)DA4t$5Pq&&?#ggbVj_rV&T@(-V(bGqmkE{Or3v;+H-lZvZ*nzC2 zU7EBvakK?gYb>DL^L|n=hpy=jOwY|vs_^y2<@nU(ns6U6{g}bqzK`G(`2lm!rAGg) z=~*y|nmRv4hxP6RP35?1nbWtj|IPwtTO?4ZEj8Wj`D+-E18ZrEY;oADjoDYQ6vQ1C$_`d-WrPTy)wG zSUfuSy@m*SjLl~&&dS!>dUj_WJzuvz8cj%LTnc&u09Lg^XzqghXF}~)XZCHdvm=nT zULSFrC-0@$mc8GK`EaJ$dF2hf8wX?6?ZydH;h&JOx$?Rxy;*9!cc&{&)C+Ru0R_-H z?ZVOcNc*t8;pI0(i$N#C>DM|`#)|s=h%kD!U6UEkkTdZtK%2w9c0#M8q-;;lr_M!l z@DAAY2?0}&6h{@yMwMQNzNw7RXqhZ8h9@>~`_^6yoj87a4AZ+%4cqzHzdB(bcsX~bg23x6|z4BO%3O`X4&UhD)TE$)|-@_w;q#wsa0_Lnk(Q>iF0#&j?Mg<$C2 zyWbR-jb1x!;c@ydK?l}T49a4qW%$*>y@f5u$eIXuP^IeW%Qym5 zasV!k!_QTbsCPfrMz-Q85Ov|^?Js-6_e_%PiRsxJ*YNxa!%oMPgcPsUjbGi}{OSQa zo%+nn)%k_Y@xy+6*K6OTAp~&Fk%?CFHgpQzp*e}6mNe^UtyiFYcQ$gGQt{V2z-kNc z3B2Ba3_BSEtOPqtO>)_B{j|h}-vB2sFE3rbeviwFKoU*y3d~-;XZmV9qa?%sqTi?r zm1|mn4<^nbh4BU>JeRA@_CP0s_~WB(wbKXnKQO&$%rgVH?COVz=y&&DoX4d&y$nGS>corDR zR~fdmo$%t8aQD^CH0_r}J2`MhZ%q_y=i*9bPa;xo?HFOQ$t|QgyK2Qr2VwBaP1L24 z$G|>#@RiA9F33?7d`d-tZ$YVU%?nW!jxH|lmvB)=3mwDWmj!&_oqQ%4B?kAnd2XqnyCj(Xt- z?peIyd;9(daHfzCUOAnWo5W*x~|O`Bt+&&bz0- zzDxUei}LS2We9G?L67sCdXw!^-szIg9vGolQK6~`bi8h}QX5XW)?@hs>{QYuIvukc zO)7$jCJV2P#^iVCV69&{biYYwXuu7^+3)vrC3L;0z-50JlEUwpZkGCp=7n5ezM{2p z=VXC=ndq|^`FQGcM1AG?Es|eCO2*R=(^v5oQ)JBZFO5nVxLMO5?g=0$U|@VIkdd_i z3h;s(U*CTthdu1uX}?@cVKN!c6j$4gC9%^80iMo$T~-~U?~(l8(KjqIzxGKK231>c zHk6Wo{)Hropo`U1ovr!V?#vNLCbTs>xp7MGI)c$ckC;zq@NCc#G5Z+lGMX|KZ-q4M z>GG0@BBJ4Hy>E2vTk-+_kh)g*Z*Gl$QYKVg=OzA2< z%B9}Mk4apC?RY;{#^kDp9zmS^5}y9=4~8dHhdGASQZLz%KI2zp^(+C%rxNHjmGp!} zSH#7H5+IcTVCEgkL)ZbJ@8^(;d-)8+a(j2}pjMP1ePXBM&$GGBAd*&)CV{y($ab-! zzL$^tnQHEpMGO-mR{~F8p`ByPCPRF_J&Alq&g+ehYKO5G8j?(3cGzd4Z9M$v2soIv zN@V#Me^}*slLM&4=()wGU!GGA>ROW=%C(eTUf5ATdxT=AaUkEBRX=dwO}M}C@bOVt z!h`h-hLe`n?NHx36!my(`PQq2gOGges`CZFRgtr>RT_kWqFAyA46L6eeiP=aGs7_+ zd+IV7c56k4UZOh|S8AC6rgs|dr+i}>A1shB*)Cl@?2_a_5doY*#$zDL6cd(`u(6|U z&JO8&3Hwe(7L<5E3Eqj>B?)e%{i7)UE6@e#;Bw=r+p>}PUO}1d8`l%7zNgqWxe@XX zu{q5i?sgEYBW0&K|A(McQ>ur&MO4IeL$xnEvH9O9S}rT|b=`=gQLGL1*|sE`&sp7K z%Nfa$=?4HLP5bH{z3(@&t&K9antN`3928DH)H>`E#Fr`PlO(QJ$)-`bP}y|bCV<-4 zZH=yU`>eI~UrMb-oHEjHhbW?{lxToJaM}q1L7*C+lw93OA9HyRCj0aN7?`xZ5uC>#SYo43pz`Rxek;3LrcdzHAt{Dch4-CEccsdwi8eve&)RFMob!_oMDmB}z$2Sp$~Fai-C(M$n>h%ZRqU51z|`AJ+9|>+TFe;&Vy-g|1iI&X4>#b=>^(cB%`tOwat+I2-TT&y2u~+0lf02zuH!% z>Q{TX?&LzJt;dApX&c#)9GbTBt9l=$nw9$fSp5tdNA;mm5l$V~U`xk`QhG6zJ4@1| zaD^MxEnGZMa65+&3LTJtWt(-&@@)W?)e`oVm|yVem9W;)xb@4!5dBlWY2)DKEW#y3 z=WLqi4&0^v7Mg|>Q1KLv9ypR=Q{R7M+Ht!h;)kEuP18>83t$~ASMRWJ9hz4nb~>(p z)cF9ek`9`W@ji3+`RtyVq{P9_=9JxLt?3bw)+B|{_M5J4DP2eGJ*EpC1di_oz%g|+ zlukbJq4X^Q@GZ-Yx7RSY=j+w|#O-w^k)GsN7kxY%O%BJ{w4$)TOw0Wi6}=&Nds#+lbgOKY z&=#NYx1OC!#T5y5%Ith3bTohF=6{85acEm%$cw~h)XZ$({Nqd6ZuWQ3d}rZ#*rbW$ zTIi2y2K;+frDxDa0nQQ_1)_ZD!VCl3Tzp2 zAG|)JUXZeawx=-Wc`H77&-YNhF>lr#v@iPR8=cj|qn&OEufM``&^UtGFF3W_Z9U0j zmm-C&U2J6BGsvvx;=Y)bf;sx`=w-4a4#`VtE^Y-)itNU}QV`4Fg%xqo#y~3Z?D;yozd-y(UjtwQC1?2d zf^0p`mIKQoO5Exj7I?v!WQ!R$PQm*X&-bdKee&<`Yz0T69s=0=DkW))#^uO3wW{MZ zchR#Un?SpC*Cn#z~O*#xkCzjXl0wv=k*Chi0^BzC|uL%clpAB7}I#8 z5s9RPL)(DwsAF>^C(ZSO+hBv}TXk2yjVCydFims)dI%e)Uj;5>zEWBWC*VOpj$6uq zQK|q4*hUWAdurP9_aBiikPdSt%XA+$)KK>=TKO`&P~ZjSV$6nkG*rK#cLH{k8iDQ7 z{&-uJ4h__tjqOnN%l|)`Cnf^&eJ<9SG0YT~Asf;Z;eDa?{J75OMF&dQn4Yd~7n{}W z2~wF0kYK&4LId3;z#rH%h$a76x`Au!Z^$$n!}p;FeW27OW#Rfpv6K}&W<~(rIDc$(WE^U$J`@&$p_Ah))p58=66JwYfG*EE$KC-eJG-)^h+v<1=3IHEQt1XA( zo8^A((IN8}CvmO5SK*Ogap)TJ0lA^Qo9)Xnkish;CGb6Y_zl*XrweG~qux`V0Uqma zR)T0kneQ^_hz(p~uTKiZ2+4}NM!}P@A*h6PU~~VJg>T^%%3w(w{L%m*60R4^?Oi`q z;o4zCi41$I_d#%=4$L~D#WtN59A7>4XC3u@#*q5<9R_($4tH~z|M3u$4)&?~A@^*; zkZTwmt=jKWNg1cmH}E^0l*PTW2QzDgKt`7RBGT=!yLRcR*Z-QJzxpH^N}#p?GZURr z1NRr8Lg%_T67iWxH@$qLpI#FiPA8ZXXO4mGAzpSdyO(WHhgkt6hzk0fJ0A<3Kkt8v z;*xr8@NnXRue#E~RJoDrej&Tt3=nqs$qB`g zb8VyGzTld~@~na?59HKoDa&eOa}E-f=k1vMK#Xf(8PWQ>O1w)89f}+Q7bW?dl3}s#`WPPIREuh;nT~ozk(S*f?Q26L6Z)%xZLA2MtNrNmVX8 zy5-|Xl*3_u_2)%m;QB5UB;jV@JEi?CyhaV|SQ_3m6QJoNMfQ4w z-8sQt%Xm3N0X60OBId1T&{t)y?yH|iyuDzw+duj$rRY`BBb)C<*;(khxYLLCXcT>e zg;IHX_VLOUf6@}fKmUCt2vJ+SVQ3dCq>*^}%oBy>ul9u9Y~67);>_~}ccY7wF=ch7 zUYC{8AV6=aLjn!jmp}t;J>7f%Wf~Je4u#xH83MkSE(in$(+N8bzAMNOpJXe|*r&vA zcb%7HNZN$IFkuM2n$@>Mhl8F)pR(T({?u~=PYY8RdOpxvJxpGohXcZ&vhM+(T{7qr zuw`8ve*rL${HvTDG@r17f={BGKEr2KxM2K!1TlLeXwHe`k#fuN<4N=dn$H)GUtNVW zm!Y3l0p(iNkm%K`qhJ!g+8}1x_WZ*ak+Vnbx9`2mPpaVJj@+z7NC>=FWyCI3D=xd- z;&BZTuX{+9>~%+m-&u8rz}2qUqoq?P|GKp~r9t5J4DQm#tzPZ)OK)me0(r3MS#}&|Af?wZ^28aehi$v4E{6`Y9xx=GY=xqm|y$X zlmaS#FapBqcy*Ht!w%FPgDG~&=Xwr}pBslVJQ1};0&ep+|KifRcR=Mk?Qhk02|Z{{ ztxB7Jd~<#LIltks0O*|_eUGSDB}?~*d)}1AyJ8mhBqF$ZfqRT!C{FVj=Jvn38Bo=a z+=~)9dv*cN^aog_F)pV5G!WLV=chVs3x3hmno=^j*mLyXSyCv`;}9o!fcfE9M%$;v0S{jCJ`lOFm}*LBH2EA}X0P{N*t5zG&`l<^_mB&SWd$mpkP-D56L-NF$7CPE zzx6d5hMm{97{PKdA70sZplnIsmZda}Cc^o4M{NcWAJ?I-rO^DGiZu0ng)PA^Z6+D^ zK=qksc4xYe06mtac>3TR95qeRoB8}7aGEsG(l^OTOD<{|1PZP_{gY1<#Pm}7%;%Q! zT^279S*|g_JZ8u*|EjQi3^E1I)gJ?^P|WGiN@RP>Z8tE-LatP9 zI6wgL%M1N(D_`>yenEbsugh{XY`gi;1`)?a;Y#X7`@TD5c`G2c#}ug-xEZXGiTX6y zRdk>p9d_Oe{YW9~@d7Z;F9HA7kNBpbYtXb4bx@Xut0kxDb_0i*gBmUI?Cea1d3XO5 zEV56lQIqB zdyv9M>m8T6#`!?vH(iCrHvbfJT9e&y;YV<;)nCml7#9!jiT*UM`dQWuLWS&?$bP@1 z>K2))XQtRB+!k-A8MyhXx-3%dCpDvN^FWiX>=N7!3?wDIPa|md_EnCiOa}8PSB12Y zru)F6oVQCUyj1Afjc7?3bdj(?65Q(G`^*@w20|hY5+^LoK1cpZ{v3KhYx`pI$U`I-T?h5 z@fGd7+#1bW1TvKEy^7Ae*7m0oZ3TgN>5DpxuIaYxfcsO{lde#%Td1~FJ^&4GOp&Fs zs)+ae{_yK33vfPPaPnC#2&Gty$qx~fJRK^$_k$0Y|N4owfXbU$vMD8}@46KMO1 zBv>TeOP4tgCC*3%LyW)@m2b9n1}$ORd8h1ddELZmFCh&s=~&ijuH`+=`%^y_1ex@w ze%xk;|5XKZ$ljpt$BbnL5Lr34Wa}$b2ZY)^Jy<^5rGd7pRXqk=bJ$ z1q8HRJ01eqXnBiqj_6^gdzTE8azWXl%g4z0N+07rBA zD#81+hT}l^ltB`TDz3oYTgV@u8DEeOd3zKxXRg$Cjmu74sXpnKqguXBOE3;$0({G_ zG5LRUv2vKmZHfHg4v%zZ3bGx;8PKsuEN-zONWyuZR8k1LS$Nbem1UZ1;{TELmSI^& zTeq;JAl=>FA>C5a4bqKtNq2)Z(jp+;-Q6Lgba#W&-FnvJe)sjg=Y0S4QsTMqwbmSS z%rVAV@!HRdq|t-oVQy1YJ+E~1lfMc!J#bofbZPthd>Vk)Nu-A=aGN$dDS7`x3Re(&uN&T6`yW!6= z`XvubnDEzOVwaeLFH`+WV})oy;ZZ2kiJ!^2D)C)uSlZUMoa#23nTR$3Xsu1vsan*k z4Lf&6jsAQv&3N&}us?jJTJn_=2M6$pATDX-8GQcDy!)sZqB2kv@i8N~8pbDU=?Ysl z5ni9BNaLKWoXYj1;$%)fgG^f~Js0|0f1m}YTe=5$!| zg3PAc`7eE`&*PeQ?{jl>3aA05F4N1Vc*sfd{Sjz*?djSw zQ@>uC|DGw0d}sqUk$mY`xgWP(bGo)+QCWa37rNg`GVMimch-xjLSw--RvjNGmcRX5mQ5N%{+=grY|6Ciw|lq^1ZRxvHU@p)bhv+?;! zI}LGk2owZ-sHY_zz5l~X_py`L)fH#**L%!2LV!DOg4DUlTWS8~Px1z!k4Xl~L%lKn zePW#&<2UoAxYX~1WCc>9nSt3rt3rFSaa`-D@ea_J)WMubep_;2AX@cwWGU)${9%u~ z&oa7x+k7sr=LgRx9m|5I`J~sTl@D1N8k|t4Jil1{c4?kHM!zM?wN+Eq^^a1nJ`|bP ztGYp7Hb@B`dhDl_RaxNJCBSf9BqLP$F_zQK`?}m8Qnruz-Gfz)HcrW-! z6u*;Y-mTzWZ(9S~ZJ>lFUgW{*9mZUQ5#V0(rk%}+HEF#1fXs^ZEs0z53V_-p6;*GM z#7AksW07YiB?i}OW>N^IOz9>oEq}+?KKGoe_ys)qs+zBRU|=jV>O@_j$YO>)SgUvQELkg;Ia;p=94|GL z1;~_B+sb_rk5pOIO`r_Bx&JzDMf&=7O(*-=t83k*gY>n_Ue#&q>-GK5J}3Mn5|{GO z6y&g>V(3t|@IGerQh3(ZWhBxXwlzh#Qe&74A$m8QL|nvu@F?B;c1*7q*!OA@y=rVR zqDakHLynWmLIcr1CcO|RoqcATY#Z(%KsxX}WNEA3P|@8Mrg$qZhJXO|@7v4LvisgS zE!Q8;RqEM~50BxlsZw~pL>})wLuR3b1>c3<9hPVO{kXf!k;CHJPj^=*9-qc9EJMhH zCpPjC;NQ`q!8fCp-T+KQ{BQzPVGKS8C94E{@q6|3NCCIs*tVxEO6wL(;FDw^CKr{r zh8qYsVkgts_2`P zYCV|;$8Gm5cRw-N)6>NCJ`m=tQA-kmTIofu9vQ%oJ&!OZ;e!bj&O?dn2na#MbT1^r zHKhazb;JZs>vV2%@n7%U6(;Q={(De?=-&{6z&sDbkg>%AG>px{%q?!Jx2G-Js4Tw* z@r)EaF)05%FPY>(?%0fapOai-(>|uy1-)AbA8)r0$!xH3W{$8%b?>0@OFyFbrkfN5 zf=8E=j!v)lwS;1Q%GS7jBN194sPpy8$Ql7|OG6!aW*`NEC;lgq~b%uoNWDswds?7{3Jr9owAgBk=`NyxfT!9+}EjLb?E zLHeJMjt*@I4cN~M#=8U1925W#gUti`Ab{9Pj$QXfbOidFz>=avnnnC?QEk!IFyYiv zgv__=e$NEqvwP^c8zsfs;Z1Je^jSVieIC%;zp@DRRJWHeG z6L-6!=J#62nj$BG1yf11K02K@1h(^4g?NUZL10oLUa8ky1R%8}>&@Rkr++Dn6}c#o znG;hyJ!{{vx*g8NvQw}oF2e$qn4mXk)PI4)>D;%;aXlLgg$+>9SwPavW$_(*{oECZ zAonEo3NavCAxHu1tPA0X>#^F`|Ey>5ITEm7aanW=8eyuaw{ZqeLNG%0j15G*E_Pb9 z>{>2IyCDDZ1#HFVnG9YDF2b|#Dn)f*XRcv*#-I>tT_A1lq)qJ_%zN;pKXzyF|1kf# zDcD#?2X=uW7QK3835SpD^M$@6@2e8J;o#$b4&$GrY@MaM^Z|%J_KG~3zZh}#SPixQ&kq`fayZ@8O{8uPYaNT}@N1N;-A^C3TRoFiJ$pYw zF~5x`Rn+dFNPL8)RY(^DEqMIgg#M*_V68XraWh$NvUdezr6~Z8q!&C2fSc?JNRrgjSPh6%?EZaW zBnG_{v1evS_O5%W#EjTyVz~;}krY2@i&jLob4hI`&fjIz>wgYm1LiNPTlJB?Oa?*Hy=6%Gy&}k z7&uS)j-5n#D{#!|9-q}r6v$#JqDh3mGxQ1rlr8)NU=A8CpH^eXgf12O`ohUUH4ozW z0oSL+cS>q3iB2v0=Z?zlcxl$^hZB&< zzt0paf4$O3H!f`v#YmJ&(SO~hvP?JpZ<)Db%KS9Cb!}4l4%Yqu9KlmcIjZ^6x+I>9 z6P-RDgrfdJ&wtua&b>j~dIdZf6VJEn$}GAKIRJainSBe?#POt{bU5Q`$b)PWCFnWi zrTBW{N{eMbY}H>E8M@eB@-J9}47by^dgU)JntVPHp_;*CfR`496$H(Ly0%TL9>1hU zk!2@9@t;>&hS;;wWBzTT)x|QO0_OKRcv%QZB`*78h2STzFBGIOy_>4D7>(H{F0`g> zRs6zp1rX9Sfc`Qj;mF!tLFkx8-smUL@OWp8pXAQB!#qgXPtZq@gP>sG7uMJ|pr|I% zi@*>!_jWzN$o5S2Er}GuX@Y!kO(jtHrp*<&cIwH)8CBLoK8_edLAuzT+9>ur`oaI6 zy6DiYZ#Mu{M%AHz2c|i~v0r!{NjG1E6Rk|Ao_4m%VA+4<#e5tv(Oea{q1FULiPoKT z-Wm$V=ciTvJ4Z~Q3$2m&uOF6I8YwWe+M=*|p?zE;Xlvd`7~M?dbPqB)baG!sJa%S2 zc84n;JMghEf`@gHB32nBnSw^WlqWe|YZb#CHiUOYv+B)f^vHM&IkV^xJHXg#$X!b3 zJoXIjosl0m(8sK654E`+S^@DcXLg7h3P+F!68Lil$xOI-p^MOdLM#hcf z2IuXe3=tivvOC&k)DbpFc|f2d&nDyGRWTI0DM+Oz6W>t&Ts4ElfW_}KHdZ=ybHhNgjH$53jFP((_8rult_j0gW8>6Ic z^XV~Zoaik(*d~7rQm-eQPNDq@2S308TG7#k?0A8V992!Z_!pfd732rYums5_(OSdR zVbGAc{NTt~D>wjtBU$^yjox16vnr@HBH~_X1ce3ZtV|LfJN(bcNI-e?o`{pn?wjaZ_ISQ$ zQ-R8z-q#0#FsM1C!rm&yN?9@C==buHx~$-MRzhFcOzB-t{-7B7JNx(Sh^jLcd(1m! zmVLY<58UQBsUtLTMdbK&twgXu6WCzOYAnMuj&e=5qt|fpCJN`Yy$g)-pMU)i#0r{0 zzXP1cu?mKbwL!akVad>oqO&CWYH$+HlxZ?G9xT)nnr=Z#DuhjT7qPOuI+(n=+m5%D zMOTAP>hd{7DK;#{6Ey^Ug|RtR%uI8cc?yCu@!8rYZbyi~-A2f2*|$gbyb-t0oEER1 z!Neo`{g9c{&Ita81e-33*%52VvJg{)9Z08O@J-bDFX%g`Qfd8QApi>OvGVz0)|Ct> zfxeW+Kl8*hB8alxF%M*tseLc>pJ;y-_`$0kWFCqJ`66 z|I#N+ek@DZTN~4zuC?%3Oq=oA5;e=I4^dhf6LJco5b-e6`%LX)ZeD{q`vjO~hyxUm zf4tnhtY=BKDX^Kl1$<+LMT%dBmR>3oK+23%H0~t8pbimtp%F--V+ZrqWXGVXu5B1O z;=5)x>{vjhY4g6eS?ch54e=B?7d;L-p22@tXe zA!s5wh4N`jUw22-mMIiWW`b)lAna^~kBkpUKA0x)SizX)7i5p=<;&!;&&Z!@Iv5Aw z>3s*T71I(zZ~6+*nl(PDVk7)~0BB|!pxq>@SW`2ROp}P3&9gWD6~cU?Zr)!5d_4so)KQ0L(S|xG7dDG^ zG`)8f;9Y3cQu@D-MhFt3?O{WSu2_7!oqg8vLiyW?g8`|c|4{gU)_ae{i}j?{^)I#K z>&2IM`(k5US^WM5wa+D47SEUQ`cn-+m=z!~zBU-Qcc@fSA*teFf$g(BlR(inmHXM!TL=!m3`DP(0dk54K zx|(Gg9CL0%1eTHvl)tzl&jf&uGuGa{C5=IS2=YXan`&!At^o_9L;Rti7cF9{hPC6Y zui)VVxIOjko=naQIvs_T9v5nQ+wS+i+!t$|Iab#A=dCJ_gp;J(OhiLl`rOdl$1cm-adKmEE@ z#Q@n4-G~U3|MyD)WeOFF$DA*!_`}V-?#zxr_SPqU(pdQs71s?1k)PcqFSmE)M-Djs zK>FJFR{@%W?kPnIl{$nT!G2mkV`o{Y#-h)zT-P~x2D*zlkQYuZ)q~9$MEBbY$UHxN zA7uOkvGt|nW@D<+K%f^;K2D5_{;j}S><$PRf922e<`oXDp->RXHza#B%!(P{q!l@b zZ6*<|0i*XKFpo_FUP1QXtL+{tYRAM{*7fIztFF_Ep(%H&uxr#C7$9;1MYWdg0uS}LhV~6y4OHAQ1|>bp z7+4X?O}v<%)IskeJwxfb5h*ll ztll2cr0JXbm^xi#G_~y61)7WY6QePM)d(RlUPm&->%DZO#_xxiJM`>Vh8N7YSsz7S z(;}#D86Gaw<~x?jvvnpL`s*M@43#fS};#78MnAYy*Tpf&jt)l$SemkST% z3>XL@-$~3A7I?g6J9G_3VB ze+tw|Wp;X*=DYJPoQt52HQ!4`NLy0ke!Mh9OX_n}+r%*dT$vI8cAr2G#gQC}Ag{hH zK~gw~CwM(%_U%RUMMqucn%}L#mgWN&;69gUJ9NpXe9s5*$^_hr=afUI0x@8aE&6Dq zRq0#KqWlGs-&Q2TdZ^%vtA;jm{1AJQGzH}7gxq!u3G9)_Jmz8x5rBDE%AWE2U0Q5M zMHY(4jwr6%0VZ)p>KP|Mb5h%H)}q|AVYMOl^dzgK_H%wU!9KzZ@kDdK9zZWT zy$xKmmqRM-(rw_&B86imInAA3`2pEhrolQ(Iu$Gma`;NH3I`ZUAcNyI1_+^FnmSV6 z+zs%l!BJ=>tWo1P1KyCHf;i^k+^6gFcZ@7_8Ez;)>f$zd2Ic>K&iy;|&{KL8`lj-m zbT4SlxKillsDtD^X-4p#@-EIXr+k{LmQJ5R|K6PQQc+d; z+kaRGLuiO^jYEDRv?}QrXmWD&9475$Ia=cpt2Mqg*)JoM=O`u}RO&Y7fohks5bMoY z_KJmyz|elFfdUmr3`DWiXrZ+y#1q)`eB7z1AFF#O{jO4XmWPLdkulH6|6XeJC66k^ zhF$kg@h>0*G88_%%PNM$`l5+)`TotI;R zHw>tvZ-}RYH!unSFVU3Si}Oz3gv&;^)-+5va~~gG{4KFFEZ}cgiz0$hViF*tk91T) zd_i`*K4DZC;%2^ZQHQMh_(L1Q<`%Ze01S!*7HB{RvRtTDLpr(tUDpJH)LgrYDpq1k z9!vmCrAh6Pu$wsJCgS?R3I+S#iR|xYyr(8J6rNt2HZ-Efp{z#ayP@ZcuI9_akCns& z>Agr&abT~@ga%qCx&m!@WWyE_DFHi1)hn%gGfnpIvRdo}eCFTKIV)w{X;;eMRcG6A zf4ZdWM@li)Y>;JVh?4z@K_xv=qMBa<8va`Ibn3T87^&o{pZQ&AQSjM&>f6ZXUnW(t z3vnO{p+ThMTtGN7Ze6j1hNUj*%p!WX0n@j`*zwx9u35kJgS#piR44M}{^luhj?vuU z7w0w?g90%RzANx7MLg2yGNyzu+PQzhk-0ptVgcib4#Y`ulj%@Hj3&X6+kgzR9$E2i zNOERW1CVR@3(;5W%nhLvofsp}eKCa8i`~)Tx?c2`LvIn_@}ri?UrECvOm2r}IZgGE z7umkMnJ7f)>$pP{m+0A!Tuy+pcK%ul4j4}3AJFlyPU=F2Qmvozg;rqj4=KgT^&pLe z-cF$*;&dFKNNz@P>1|iZ}Hy$K%W^0fN#;^ruNk;%b9-Qi1Dz*3tA}-{<0`*N*t_ipzN;B z;5u8Vj(wBVLXVETi8#PQBH(()ZSFLsS)$Urcx`t$Q@YL1L(((Re@p^nyFW zsG1eWH+tIhl8A2OH@~MXiG9@ z2tyLbD{=3gMPDs)?1n=Whr}PFL-9ZFU1C>b0a@-n9olUAqb4Nm>8lp>2~OWh)Ql{{ zfXv&N>%nNeWeQAo>msE(fSUUa!-N25N0=2Zk8(Cr@^NEs? zt^2p3*Dg-}@;yj0_)?_s*)&4=Y&2R{Gbvrmv3(lS+`9H5(hbDGX1ZvX`T=;!7ty76 zYhzD03ApeqgYCKMulQ7F7lD7upz~uu)%XZHLz}OC=6)gYC_*FXrlwm1k)$N3Gv{SA z?DPuuF15g-jz_!lzXZkGc3c~KrpwQ#Qs}KSFjMfi*B%4O zKh~u`raGa!v9|N|?7L;lV4>BValIzHNjU`0zzNJ~aqv!ZI07t&BEr6QXM2Qz^A1^i zbxh>RZ<6?KGv6eqHH4B99wp8%E7-l#p`0|*DR*Q#u}$J$6dTH4Jb;V6Z4_BZk% zcB|s*ZO1)1#1((u;A6U){K7HgAd-&@V6dH%;SAMQ$BkI8O6jl@XHdpt<8|8Os%2Qz)ay zSQm{8ZXvr(&Uhq|rl}ZC{SSnjck&;M!s!Y}cJdH$KhhR-cmi!^nwms7o{{gTG2t|n zRFn_n!nYF}y&));Q{@-iv&G6}oMEOadLf!iXL32;0sE;8=vU3zrE|cPRR6M>tB^fj zYMj1`GX)bb;8;cDkElHcxQxKMk*BJAg-bO$Q>N~8~-7B03>Kb163@5s>c;X$DkA4mKS3RbOCqOu?iDH^VXjj zg4eXU7t~4UGBi-Wlyna__UdNL zDI|{#H)5r_yEWi!jJ-hHRYrgU6d$FW9Ils^BGFd6yK63m2V946ZABZ}7x3b@MTu1Z z%9C^tmvbr2D;13rl^|5nC#N#)T2ryIJLOJbz;M@wRd`Vj9gm?(3q9*w%6GGk;|5Kf zG^I^(jH{rOrwn==A`Pg3!Oh@fQ&oc<|Jq-~PWa_;t|dnJ+&Hu4u$iTp!_Gpz?5_xx(q5($10A zBL|bi@Lyw3BQdRFLUTZ0E79p{N}508s-G*ll}Qa>YN0WnD$zXtTl`@F8#VaeUxd~` zYv0Gp)`F$CIQ+4&%eWU>8PO@%AJ||fKx}{{NSnIMl)o|tZkwB^%XR-tq5Y#RCH`IP zw$1RP#lj3v2x&9Z9q?CXm3vq8sHR`MA34X#Gwcp{=Ni3(kr^N@ONMAbSL!}sfr@I_ zF?4Cups{K}tdWa#M|+XK=|v|PkgSC|lLC4Cs1-W;Ku`56bs}dLFXX>nU^9C*hz-G5;^`!IwdYeD!&Vgi4X$S~ zmtd4(`;c>pnqSn{Cm!s9oV4-K7dotdB?U0SxbJve2;>&gycM){U z<2g4Ow5iL@j>>N+*c5gP^JD3;NTr5iiQXb?asVj&8+6cqWGR5c@r;*jy#XF6>W+X9 zNGB32Nu{a(9(C#wA0%nhJ<@mvi-!6qU|#Y%v0^cGcfdnNyIO$i&j8IsB8U&Nj-R$) zZnCz=4nzND3BC21;%jwwg8{-0Mi9t|ng$U%GC1JLlmjGVbMJ>=6c5bPVAgd1x07hV9o$;P*w~> zOr;zzgzc5(p-3==o?~80iWbsmY6@~#&nd(RLqEgp!5c|4ndm8hH|MRqdy8?q?{k9Z z`?+cSG+X8n(4h#_FXAN+hrN=pCq$l_>-2gluHtS(iT~A$AFzg=n{eBR-H3Cm9_y>9 z&&Hheo}gt;yh#sSxMDXCbPi?A5^&2y#G*x~4XdX)zffDY&Pa|ME{C}&{{SLm9@~8W z0!#RjhptUFIGUG+7|TGl!crER(#E{g^INYiqt?4j4? zo^{griGbpx>XHK|dmx1bHVmP9mVeuSR=(0mOcJ1DsdrWTUAx~RvP`u&n|TAi%IaVu z_vTTn`!KdOmB_K<6RmQ#9Z^}Ym7*j9_Dkr+$5a*K1B+ZVC&5O*tjln3%VVtsMIr>* zVFgClgALEc6KQRu3<}R+LNBbeD+5&^n@ha#4gM;K4{C6)^zfI6<%hqGa%#q>hBU=7 z%O%iMQ66iE0e=w=A3@3EwW;O*j4FBq6m}s!O`GzbJW|>6iv?_MNKmmav@%UKpOqNq zlS_s$1mi7~5hUqU(vrG{u>P(tyx2%PSiGR5fxuGu{$ygbZ5k;IO@)Umh(`W*pbMvf_j;WOUnH zOH)yk?5ex}v`sRFrt-zU5;Z(C$Bb1N!zs}iBl08u}lSa zha$97N~r7(I8fJa(d=hLmlluS=DGEoV%Ea12D3F8QzrUDvr~Ek-?=>%yc^C7Q>eb* z1oPi83Qi~UBPOqNli>9Rk|9*jYFW=SfJ-590>4XoxcGF+G#DPhL~ENf8G(hXA8|tY zsiq5bOvha%vu1;EU4fSBat|&HUyceLC^l$&m=u}n2>Jt71$?^y@Y}7CW1Mrb=6#gQ zA))h)Gm=3y4uP*eohzZ)c`wjW+s4fMg|-qC%vLR6H?rPvzpt^%;g0={cU5$%rWKD( z4Jfr*>Ok!yBfsmEIbMYZI4_CgmDc*yVO2=94}}?g2VS7Oe}&yUI8xnpz@7PsYYV51 zY=hpD0qM45A#MsF!I(q6#I%Zcb5;+`KN)DT>?$N(Ytt z_dkQ{toIYGsc3!2fwXVAyhhgvY~iB>GHv1buzZ9`=u<@eb#M&RRm^zEC)=VqQnP0} zAyt_q^nMbw7kV7IU~$-H`RojRLYu8BY}11-ZlHqaXoZ2Fx=Hd{d|4kzbB8z%W8#%X zpOb+LQGgI=#Pbs?(ZA_IUxc&Avy#v*oJCSTO~paT8a-%fSvKoXF#*J67QIx6J1K+H zBI-07W}FN`7hCl=J!wka39^qt)P%ya3kQ4LuK`njBE8NKY(x;jII9i;FgQ@ z7~83WNN+ho!wyf5RsYB1n_;ffGK-JupXk|&2^HkZlakU8hIZa#bIgibO^~nG5Fro* z$|`7j-y9Jw&V!PqwgcCHENDT{w;tQ{&CB}|mokbUx6KWFxMAxeF2hL>$^9QGSQ zivomDE!{Z1HXT6Ne)R%hphBlUysGUtBi$G0X!cin&C*|$?*KNmO&qG&(9MIpzpv&@U^Er~MAvhuayIyag0@2~qH{O76il^0q07Ydcy(<3 zIvS|kGi3eq1#a}_SL-~dHRL5R0N7#xHBVH8rz2*m%QjKthEB-{fv6iO3rt#ms@atT2K-qY4C z;+Wo`jVqRrJ8GkGJ0}E{vinxL=;4gC4zON9ibkE@jT~II6K-hC>>K8FzR}B`FyQjB zXz%fiwabXGGiO%iHu315OO|XZe~R}!4l4&-Z<@fXg3u$`1|ltF5t`*@P$fJ8JKsLJ zf?b56?*&jKc2Zyl5L~yv%0G>S z*3t42(}7AoizRZrP@y)(J5-GRKcM|)YO1%h=_9%fNs!N-)U`<|6YC)j+{?E=y)O%( z04y&E74zeHPp`5?gr`DMrU;{^)`bmJhmey|3NM{r(*gvj!k&hjY*`uyPL!T|;`;$V zy50PqjbA#pKM0S4k)Ut66uS6E)s&|G7Npo2Tk_H@n|oCnw5Ngy7~x75Rk}jOQ{?lt zZ;oo1IDUX}()Ck;t^C3B{-eZaJJFD`;AKBe-3F^ScLP+B4H&nta7kNH2vqg~)B6Ub zhrnz;r+hUIz5}^W(@Jk!Z_?2tC}Q=Dom>}8B3d;W7L`*SorxSHW3X<+*jMY%N-(hd zhDRLRPD)>~1b^5d0At?YN^Z6G$xVZpqjBipOd#}cKi+V)wNb!x<0Lc|N5)wM4;idc z-udp2`yMfgf;+50?gQ|Tg>GC?k1QaRt?1zTY3F- zpe`ZSGhsUvLofq~+X>E0SciDjIHlFH>ymxN;g9|MsX*}bbxio3QTmZmQuQO4(3Z`) zKJk_7a(ig$TRyZVM_ES0YhsI&8zEvRI5es>;;Dvd-7isDdpKay|NQA4WQ5-|_H%D= zwanUiKLHuMt{t)<1Fs-C#r}dUsiQ8Lf8t7G+A|;kAX1k6Y%C=$3d+`~#N7-%bT!f$ zp7wB>boyF1%th%Yy(JgbJpNk1nhsjFP%aIjzfo~!&qR{<-;+@nS)nm(G$fc1_I z@PC*Yr0|&M-+;U^Z;+=0#GLvX%nho$R~1TqLr&U{(kJ@+*_XBO_hO>$yzUYeEGC7q z;{I%*WLxzze5o7B7nx;hb7#I(9E6;`qETFJ(+j@LT7)_A6p0sqT^;^|g z7|-q?BPHNJ>z|(v5qz!XTaMZI)kfr&0~f=@Uo1tda*Nj?{#Lh#sUOspN-f2CqX55y zr|%F6AY`(X+6?p(xul#MwfLJ%8j+^3xqFC$N^t92Tx&f{pKD_}e*5_2vq_zoK%wEC z!RAM;KgZoJBX|TW&)k*(6$X{(q$s z(8-DwINm$S>`b}?(plHm7E?e*(Wmyu5&9au%*_}0|f=98ISclZb}LBEi;|N zkIRrxzOQ@*ZuY&OZlh`8TQ7}4Q;7(V64&DiCAyQZ`z&~$S@}!nFw><8s2{GLTHLSv z_3-msC5*^d;7q^Zy!8STEADrG+6*tsxAi|Mh^DDs+l}D(PoYvIUrcNawGDrdEI|{DhuRvxPm}1D{s+3 zpu~{^ORsLfVZGohohtFmkqg}#W2nun1V(KWGAS^Rmw%!%AwDnP+tGiR#Oe6r;LcU2r` z;e1EAcooE6h#N%c+f)rP5SU@QL7 zI(tmp;MDpT7&X0=K!{g7`lw-nCsq&Rt>mhsMWRC4Fnw2G`B#OTV(OVNKDWCI1ia9= zK>jUu>3fRJ9cM?Apx<#w5jZbuwBNxQFo~mH=#d9hn7X3cpjbO&km^gHl9SCE;r`{ZNN9e=TD%Cp3POP4obO_ z5vg@1x(o)CVS{+1@(j{;BXgS?7ue#-=FsU@AWUvK1|iRzKZum|&5h4ria%ZC=%$W^*haKkbqQv<6n zXQQ+@TW$8{m18mnwP%FGtG9ttA|UsS5u#@pI?{fE8OI0Mj?UyflB%dK(`Dg-AXoBN zaXaz!AKrCeGqq*AKTns+i=lOY3=a+5c=DvP(&;%a7Pd!n2}pJDfcbN zqyQo-9}k@ydnivT`T#V5TJ=70&VS3F`|BlVTVI=Ckl`AY1$zyFD8o(D@j1E~vwWQuR(3wEMForBDiFd4+PG z50%}&-K34C&-D{6{y7 zPh(%}2+EYpfU*h9AAe^`Hr&uU9>dM_dPnQx5Xa~MmM1_zXvOY==Q(gqfd3VlMUdPoo z?#(1d#4XKoEl+B)(AV5y6`{S-3k6)XXv%=k43_uQB;>X!C~MdT02rN?$Z|ZpGAo40 zx5FbC3TdIA>Lspp7Z`&VR83W@s}@C+aN@J^#jbzc+K9gNo7xoKp+1KTo2gjs7%BI$!fShm~`0rJ)nN03e^uv!|XsBC}6|yLx2x<8bMIc zeU<)**df=GA7ZVx#FiXlq@#Kbjt2c9$kD)kLxhBCxgz1e#+Simy+!@nf#W?p*?UgT zi_!9~@)aPSGjJZnaRn0wdGLtl=JfG&fS zfZ5Qm%HS^;iLw<7(3x=+v%W1Df+U?UwU~}-MaOng?%v7|7|{#wj-_+-gY=3oLIV2) zb9EN--KIj`afACKiVd6NKM7n#)G2CEPpUEUK{+_agoht;dfb_o<{do zb`NkI`qRxi;dvB|kHG$QlW0RdGCQ%L3+iVn9bT7}fLi zthmmNjTAxbr!8CmvHKSL;x=37#?mC$Vza)iL>oo*mx4RO*-pt|Gy=2Xx7_DHdwOg| z+at_Kr(AijB0N6sx}}Bqb%eYRfsYmjhWYc*4h!#hm3#6xka*DOCEs;dRj)?d`LQi2 zrlAiDIPKiPARbK@=||w~>mB`W!}gy4DIBf|6pV|Xdil~Q(RBMPJZBRZ?Knb+1PkHp zYaJi^x1`>jPzpcV@u4pP->K<&K{uUI$4mS)6MMI>`xAw@L>=08=%zF-EE8j3AV*u( zHW~MOwp`D69G-lr17bd1W#xU@acc&H6)jARf0?7tgI}}2H)QIFeNrKNO4mPc;3X2@ zOm7BBaO54UbjF821g%UR-&6Eyqgg&tv~X>k^WtMm1b=euETFj7JC0iCNFz=>`dipm z&DJ`{;SJ`v=k?8MJ79!)suPndIRO*cTud4eiccW?emk z`4j&)9b-kW6D)FAJo6fjJvODLaAd_eIG4}W9yok6RPf{6oXw~-T z86|jd+UKMmE6&+PeX|%tI9t7gOxl}qn(I!8;hbkd<*eGG`JF(OPWcBI9d&-qe2L0!{lV#fgOw z8}S=ozJVQ>fIpv3N`&=ddTVA;@sm#21wYnXMt$C6;M*Xd-KNN4e8m2vMgD#n$V}`+ z>E#==qG@F?DQccSd3eF5xxFjW(~~mm;r5g)gZNbJ+yUj3`Wg1L>FflD^~d=b?ECh6tAHl;-| z7@fHG!r@H-7VXIE?~xJ@i0c#p*AT9h_4s*mh}}gz}_KLBoWYt(NKW5Vz*o4rXlQz65);3+V%Ft#8q)k(5fX zLl;{Q@}h@qR z5YP8Lr?C}C?f!l7q{sbOyFvDX0IW0$oqPbMbFBIj3$vHYTQS}EcY8tC3>6Cfuv%?a zdI^K2?Iiw>LbBU>IPKnbEW!6^agIJ&9~S>3$(49#7jZ8Bmjnd4E!MVkMWej&J=|w2 zcYmBsSyO5UHyCJDnm+GDsshA6R%@KQpUG=)B0HK?QR{ntQZpGg7{Cs*d|HJ!w{n!q zux3+{A15Gp2+U`~TZ|DXEX?jOmc3gX-=LSQ*#agTDs@8`Wq0vWh&oRxl-jxehqV1Q<=y&0 zgDM|UBC}qD?xS{ukCyQM$dY(1gLD42J-@xb{QXXjXAi`;rL<=-crDX;f)6TV-f|8e zT=OcWw6wbrbBGso_od^XMT-umDsV`sv@;tvpr^CiJO)pl2e*AYzL8IVW$39)G^2Fe3b_Gc8r$%| z+@!HVP;tUvgKY>hyEy(Fg7@YwHCFR4%uCb;KIX7uMgNzc`(!gbx2;Q!wtAqxRF}wj zcdUo-B73xsiLC8zuf~3-`fR)DfK9ZceOB~BS}E&CIEte(3GH-@ zBq$<|{Cf;gwM5=?2Z;^ZI20xjXY);lPN zkl#?(`;3`m2a0|)Nm?{9660U%SmjVqtGCEKe>0RXIo%E9BVA$(^+AK>nGHfkAA()& zpe(sW0w1%=tJ=QzncJ%PPq(s5#IPFzv3!_16mdbIjK;=7nsMXlIxY8wx+9f9l!H@ep++YxiX z6!*S&qQ6@BjrWr_m`-NZ9%;l+);Pm7tDVcxR;g-#nLfAs7!hQo8{?7W!_!oEJBq0j zPu2qFZ&xwAx6~bEuYD1bF>iV{7_f}ER*hUkF1GY(heDR%Tf{;ize#3e{q{rbXTG9P znDQELKUCUl-C-SW`1;#2TK6r_Q1B8wzZT$^Vb(7u2Qy^nt%s1wJhXDY=c0e34=U8x zn?1qj=M2RvV!c-HE-teFaeipL#JD$<5`^8)?f3$|FG334f7L&|shX zWgaE7IwX*pwB<9ti!rlm`+OdQ`;{=P3@-{29X}RZMis+G_X0pZ9`_TTNsK*k9zF?#t4yG9i5^i_4 z6R6&d2=8<9lV(Oi@}_{D^X8&~;CVpU>pMg7f}sa6emK@Li9IeB-Y8q_cs4tIf7krE zWq;L;_Co)|KH8GKKAZDc~G~8n715g27_=(xhFjcE93DDa(|HF7wFu>DB)w;tUn*!UWbL z-_8$$%g4Myd%UXculL7<6|&FcnU2*#9#VRhR;BKZ+zOCB&-N9NitF8l2^>rpkB{}C z_&DW;p;ge=zS60D1lh7puiIf0-<7H~j1j zn+*BNXU3XhHsiPw>ZRhmlpbqqZu1(y=x}l4iu)6bkbSziaLiJO5}=#_m%~D7-R-D@WP0x&TidDGxVf*toXP=0WWR^}H&FJJZE?N=u75H{_0SA9*uUR9ooXqwky_eyobEU$)==}~B zoooRM?ZQmTgE2Y}!$SExw#q-?syBv)F>=E*)ECtl4$Z%cMp{5f`KUVD)R!1UViC9B zGqnbAy#`5)XXYDtWqCCv!U4nEsqBJPdf`XucbW0m_D=B4w(jX-u5>g5%q3?XX z?`QqabIyAIa;b~hSIo>c^NEbMj@>Vd2^WtV0X$V0EE+t_d2D&}NPHu#GLL_nn!wxQRK%=ShtCrubRxA8!(I{Bl)vNAm`c;}f%PN)R zzC`QA$I?FTQf5VyHcek@=}*^N%uc(xuSqCYBreLg1$WjK1m4Y&j~gjQA~ z9L|Z?k5>fL)F8i&hzdQrMkSpuQbeCkURcX!$%aZGW6|DNyqa5EUg1IFzFW-&1>r#` zxqyUR5FI4d8wmA#-vw_MZJS$r*ttNX?d>CdaJkV*38!0JWDp*VIvW`etWrh`6r#RD z|1fU78X;4+u0FPidub8$WBXivjI3m{$>DFPNaK49Ttk{73D4JGo^Fn~q#Z4t>B2af zarH*1A9Sdg0tobEYPLVLT?lG;bt2R_Ox9m>yCc^)!65aUq&LV9{rPD^KkBkc`>TP5 zQ0u6UdE!@`G{Q%!aX3&(0h*3NFKJ}T5^eR`T{9M$Wu&J$B^)p%O|WDwj(5>Cb4={U zXXT{4m$FlI3OY=#?qh|d?%mNUe$YewA6zg>yXP zi#Y!8v&8SGYnVq1Z=yrnYVv8`3pd&gcX%e?%GlU_da}P-eF;ITwq)O`5R_pNfPt6atdB=O6pTX3@Ze>Maw*Cst4Rq!}5k!B79oe(j5JgX(AmdZmGK)!c zqHu#08M0*w>S-hVt;u((Tdc>AkdNr>r<_lR6Sbesu#Du5rDxE#@Q-Yw5`_ifKYe!<%PQ9VzcBPN-lA6wI-x!SYCE3)}teKkD% zuvB>d##oY84h@9(hOZvjrT_AR-jK1@Q_#l4`lyDn{mR-YDpnUrI#$P(8s>vS&dtJ3 zB^nCIehEaN8k;IVB)QHlyI;huR!pEV=DErD-vkpFaWfU1)CyA0@yO+x!Tf^Xy4`FO zK_MhCGuKByUHrK?Gi3W@dFuNZ%;k(IlZc-8n{;8Bv;}-KZV2!c>Ftc}2(8t7vtr1^ z7v*NMi@>BeUSqswIv@EDPu?a^=bw z<=?-K<{OrMQ-78+L<8wpXJ_4TY?y~7xFh|D>gwhHMfQmfj}Jec5;hoQ&UF*P zdPk7WQBv_MY-)!xSQWO-b;BJ39IJrYcF%$!%g$F zX6=wDJDFY-y5AaO@D^N=>Q|ay?S6cOZ_v-%SIOOrwGk<-XlGYgbB)?oDsqI-sLZ}* zMHeshx08d6rSCiO8DB~qG-$SMzmL7U)wTQnM-F=Px_Zs4GP0Lz%jp>^R={B8igY(^lrIvBMSk5%sA(JIJoK z3OZD=AB!rWj*txb$1#sEOy=|ldfXQOD%lHz2R6Ti4kPXETAV6sUveXM?{<6|qfd}A zpsG$yZPZoJMH2l3m<9KJ%e~;pQ}UVF-aTh{G3qxfo2s#JVhPBL{iMCE%s-hb@F7$# zWw^vKA1U=XCW^>V+^qWdaNL}_k_*M!e*slKhatZ8@cQI7X2kwa{u)V!)&9+Cl= z^g|?BsKrjpU)qGf!8ya0=h;XV`M;UMoDDLCc{HQx0nF800hoQQ7J%8^fv>BxsH?_> zD~dG7G**%N_MQ7hg&*5<7Nb@8qbsK1K+nz98`YM8C`)elq<#wo5utRO+V)f0`0;5V z0`2AWZESL zwt3b5Vm2)B@~GVNH(-CW4)Rd7i5}Ij*MN@SDC=83FRibdDx9}P?LU+CwHY=*v;LiK zeoC;!B`4)MIe`CM`}lxf8iy9gl?Fxbv$jKoNk(qp&0PvjI3yc5RDnSH46_?N(6^+h z0yGeLKY5~P z`AWR4-M5SU4FR`sa_8T|j7~&z(f;@IU)Xm7@9-5j^j)3y($L7C;_60)IMZbW)TCh@Boya8HQ^lO?&}jt(s& zLTSVS%<9^_xew*sXGwGwthf_Tfw++9(IB*6eiOQ;m6cOBYTNCxxur|vpPeJR#3XS2 z0yNOcx;j-Ts2`ULuh*caja?vXl!qQ|yOTupS7zVhya8z@$I8p2vdyL=?3F6pHO!}G z06mk%;n2xa28j$&TH}m;sn+V6$FSlXT@nv(qnnF%xU96v=IY-tbOUvvR-s2z!gY#9kLND!+i%AavqvP7+{cW&bnYEuD)8>SglD|d zPeY8k#u(W5Wx`^t6669mOoze_@pc0X6}_?LPK(zNTISD}+T-2yC;J`$y*b3F2!tR? ztQVk{#bGVfg)@BK;I*+T)Q3u(8~Spe^Lwyccnmnokt#PM+L+zbsWydynPx2Q?9vg4 z-t^1T?4;5-WuF)9*k5w~jY$6Jsx5te?+1Kw(oa=YB7WuD29yV90LF1Q46QXc!r@xf zQOr$N*@w3inT%z|3IIC9W1HeW+60+VOM27E0>r$Jn05QuHr1%kS?5|?0B9WU?7(<+ ztVo4GXUCLHT6NBeUBtUKIC11qd1mW}RHjRmk?I(KJ4g92>gnMXxGKezBpKU~kR}y7 zqhm6B@RiHsp-7XyTl37}?c(9#2q3FwEtL2k)}|!xVeHVBt|_;B>Nwyr^F((4x->UD zCY%4(chJpa%&T_zv1dX|`Rx=xMJ8qxcJHfdii)=kC2YOXC>0{#91Ok|&i#dF=cat* z19Q_aCgu;lgJ#U9X3CHF%*ocMwh5N*g^&C@L7UtTp)SM@ID%=Gmv|g|-g|7tKINJD zkjK$Z;SfvDw{qvdpQsq9#9zFka(t)Tua|>KOFywlsblQbE~(m3Xp7j9O5S7YJB6CI z;fVo9+QY#W8dOSu*!K7wMJ7knWsvI07`e~FaUBB`dqiT5;tZ`*#Rm8HtCj)tRR;X0 z5M2imiH&G(5tfcYdKnoLI^gf;)!wlEjC&g>;Svv5VU+R&A-!we0TIw8yQ1~N3?aM7 zFDjUmM5_OxsN&V^ggAwBo`rCJ^#1QE1^peEZvwJ5E{k z6SGe*Ulbb~_)yxDp@8(%9w}lJa`*}J95@#JZ_O15Wfu^xTk+(O7)@kptWI1@Th!o1 zfA%{u_!OQ^GuwV)C#4eSX~onD+zP(*Yq~uBX~Mp-B5Ub44E`VQbb*C`p7461(>`Le)yw8)(mhhY^ zNTd3K$p$O^T|?P0u63&7j_Q7`)yjAG=3Xy>sHs8*eh{|Q)!mWo+H?Efrv`|Z>p zicAcs3A34)t1JPN`@jW_RVEKs!gxHL@=ID%@V9qmo5CA!Nx7Acy{JO>fNbUl5T#5e zL5WX!FKueIn^5Sx5V`3$EfQZ_`=7d||4ADWr|ck*c5`>e4mDi=u>p$DXtQmD*N`&C z>v|rsYqh2>D}el|Okr!Zs&h{Kuv@5=E)ao(U4!~PmkFvu^zO>ze~vo-uxkfp zVF~XDw^V43Gz;svKNhgcQca{)+0{q&Tu1@IJs~Sm?cQz&mE^y{D0_xDAm8abZQ-~0 z`ssCFkzt&&gz{NLBB|B?IJDWAM$~iRl@$RdnV9K#fJ2VB_a`%xQ9yL6z|8I5I@A*g zCGexPKL){AQ<$>*m~AZv zG{Uk-7;xqV37vv&gq9F#Q1E8P$UnxT>pi%VwJPB{{{3jR!(2*7ni(JB%RSDB{de(0 zO@=2pNAVwQXe(M)&HlSnf|RyVNHV|`#OH_-{#7&-j(HAUwOKp>!>Dy=lZ-x6%`F$z z2>_UDdTZb$)nr8)$n6@K7~H4i-%1MTE2wPeB>5*nNEQ+XSx}g*U0Tp(=%TAJE9tuHB5Lp^*)#hx2c?cfo`@P8uMHftJ>m2%siUU54o6Wy7Ss#-Ai5YkLex# z3UB@bfse#kVTC+{+8e|DCzggMG{8^zn+Af8Io)F9KVfM;WY8;e*CV>cc}3x~GdIo6 z#7>RV+{r&$Hx94y7IM_~TZ2SuCZ~SSu61Gf+wEnMh07HN|97cLQTrUi?9@gU13*N- z&L`xl_%;_Wj~+@=xc8&*65H1O(sn(onocIDn)0JOKZd`(N(WEa_SCrFr9P}ojmU2D zjF0`rT94ChtT-NkT2~pI(O*t`Uznt$?O>oQ{|h`&0| z@i~d?bZlP(jP0 zoZ!)aDpTnY&%+Jk)#=5RBo~4m%*rhQW4_9}JXUYlzi~$M#h}O7I@QHu>YwQ1qm6pO zWq@@=BGbe4G?GW*>|LKM&bExhM<-Iy5CN+8krjD1%{vkK~kCH#RA33@G0t}|T*&No@s6Qi02d}-t z-rWIAT)k07`Tft+b|A!1?>Q~Pe`mzqf^)AdKTH#h#{-ERW=)@o9o?<*-5~1b%y5L| z=V*Ab{V(b_>QqLySrqXB+*S&`!XwDP+<_lG6Mhxj z(s9?$RGiDrhaCXrtRDp(gQ5cA@M<n^ zHb44k8qv=C%2VG3FD@@ZL&)#Ie(kx9dU`JRz~K`S(4;6{qRnfeznn4G@3Neibkwr4 z?b3cy5!C4Dz}Gvgry+<{M7o$UXCmdL#{ZI9)Ky*iVH4Kk`!j9{*RQF(vWp$d3qBTd zZ_*CcrI8&Fjo`7GF1kb2@JRJkGDy#a`>#kU=Kri;B5olKI;nuw`JiROVqpyX-ksX} zhpgjM#~=kV(j7z4;A_VGKhU`49#?-C(y^4LjSU!9rUsXH1v!9DRi2?vAz8957$wi} zgz5F&xv)5mNw$Nw32#1t_lywx{q;`#x5n|DjsB(dG3gDeAdjl{1oo4Kk8uxIi!Iom1M|YdM_P zH_jpkjXM5zEPQkp-8|CHwE_U1h02wZ}hCwvx@T$vKZ!-Fyxldo6{ z(}@1mN$t0gP)t2VB!WC)rg%hnmsHDf%5AEe8Ab-5e4Ov{3}{89a9>>R8a~&vGMIul z>~y8;JMFM)iZIDO)?Czd^V;%lfnNvG_v9%4MjWo&DPlB(uD8{CQQQ8_?{r*%Mxr(6+ z9xsz_Pp@Ii`Oj`)wepxp%Zs7A>eTf)J-gHcZ$}FX|!bc2@Yy8An=f zFM!FKWFuaA%}S_I%u2YAaWyj5FE`ClZLhcat=or^cWxYmgktz(K66rD5S}ovG5G*8 ze>J0aC@YHSC=j%*q`!G?`|Z**6j>f3UszFP&ud&R4}dpZz(8p8a|FmhSKYzv0$bc? zZ&}mi+H+9^9U8x2Vl|TzBc!eG1nUtHs#B<{EB#HZ733qv`DTPOn3Q8%S!ldk!H{Mh z-V$T`;pb$`7iXHW$2 z-3gcshvKna7oGsB9xMHz(pEcKgQw3`{Z z!0SwxJdHQ$f`M)Wt{5vC#6B1fI6)5#oS^wtmsn-|G9)K)uNo8_D&NJ?_i&v*Q*bQ? zZ1B_(Q^GKm){bS|AgzA;4nmEsk$F#)w-EMdFjVdXQg#)8HTpZAiu)+30jw1kefiea z$mk4pbcy2-0m`c-?N9%LNz%#!@g0|z*E=`Qej3^^zxkHD0iBQmEKF+Ar!>pYD#$59 zC=k%|f6Cl{gC>_M)}==4hvM0M!!E1hr`D=$^jJaNuS39f@fl#v8S3g^FKNaDC*ruBc4wdOpEV6mPp%d269vrqF56vHUdydHt#PYmo7{^+xC* ziSzYSL|gFlpkbgeWvDoM>wX1a({n(?OaLfgm3#DdXDoSPyROWh@_o+nxoMW2I;v=W zqOHm~`(><6xYUiRNeo^f8S&=xpd3G=-q+dnAbZK1XP)E^%=y{zb;D%8Df`=+7(%`> zSHaL9=?p)9@Y2T&DM1>7zdJ8n4xT<=xBVY2fRe`7vFwyDTqpOX{r7rHU-yYqhOUkb zflRE+t?yF5#*v@&t)>fyr?1_!*KWxg{m2p7_{^D7ZA;d%Ozmu=U!M9WjH?+hD_tjp zp;u>6>+U9GjnIdyfFOP?liEcl-m^vxwV7@c_F+Gkd*X%7^}XI0Q-RRicyDN>^DFii zjpITey?pNKWx6AEkU=QMclg=91{c%8TH;#WAJfd=rjDQU^bT8Ab~PNc-vXx0rnEn@ z&3jre8RHda)k5yC6S0LW?50m8-7X{!Zhd@wuvUu2tOY6sl33e=F(Bo|{39j~!Y+G6 z=fdfUtFHNI;KRo)yE2a?c9V2QLnlmB*zNo@8UMRYCa;L4ct?%8!|}ebl45nSV>gzB zP3iz6HGTO$v{}^N57*3%Z&7lT6*by{$KSQErbt-=vkbl@Lx#K0t1qoYO1HV>%c9LP zjFSu>r<4LNk}?SR_}h`wZAz9-9QGrfyjjbYJ#HPX*u_u`w`4O0|A8`HoK~&ezr5e} zColUkm-Y8GdV6G~Ad(7cEuNa-863Q?%@7Dq9C8D{*04~{#c zWno7WbCX=t5@q|-i`#)?XCh8RLC<}G+EDB~{;?*VM9+MqIq*S})E8?0>*UyX-+maf z0pv~U(;BgLh6^?Ke$UzM2tOXQgd@b4Nv}`Xg;p7L>lK&^UOQP>WLhvzN4r1Z;2PE(~sB0_ao=?AiG{O?yyEr7xUd}f*yS3pQYM{aX#caN)Mfc%E8zYS zS%qG*is;QQ9HCwYR+-hd3p0&g9&LSro1ac#LxcJfL-JcT19!^cC7b6sle6Ix)wW=8 zMDn?le~UYE4#$o`R!#v;emF+TZ8YXlMj3lD%a4GLcbu98Jg>|xD?H0jTz5VGu^uph zKA?L*;*Psy4KB)fCt)s8EQ(y%(It3f61^WEK@(K(H8O&vyQiGbU&ntkQ$i>3@pQHXvV$5bWz7@ue2JVCD0q4qU!^JWO~o>9&F8?g zP&GrWTUk;L&Nse?rt0wbdycc=omSkCl2$qk8PR0kkW`wMM;sneq|_mR8bW+p2~V)o znrz}Enfql_(-H9P+H4%wWK@qh@rw{nYcH=mA5C1w0+x?!anH|Yp!yhGuYx|o*5<{A z1m%w$ceaukWS4gFwu%ZOrY-p z+4SH_Q1~8sp4QiMc^?2o-{dvm6k@&H@_L?TYmGe5OTEWhbn98&h2=J;!3kN$u~Oqa zt%ExD6bI%sI!$f!s(W#!Hk;p`t4vbT8Hyn(`WYHUvk*=?5~g`4O?}vz=xu!aeT?$h2_@MkAh(yJ`HSAy7$8O zWO?UnK3i2G5%0gYMH=my_cEooo1S;`*IDH=a);PlS4TadLu0fPcWOmjfHIRZNYeOK zIsPN3vy0_3EIU6ky6apANji2Vb@cDXo(Fsz&IQj+es`_5kunCSihIgdcUX8RJio3C zvceAk!F&2ez|T1<_F(!e{zcJ6E#nC#Yscap!!ly78(6=8MSY#CRQ}_66fjrRc)Bm> zp=vVUm->lS*qN(k(mF%{D-!J!=Q!)YfA`Om=T>bv>}kI6V^6hg;j#DwxBIP#kDAEhIlZZSo<@wXi0crG*7t&qb8+NN^NsZ}zH#1h1kK`xDhXh( z^n{^h-g>F}ZDb8FnHTjI0rykDZB#k}3E4aJyYj$C7;(P85(SDPicG2)s^>vRsTla2 zhCsXkl8$#R;YE8qM}VG1*y|K%*KFtd>7pGoFxmw>Z?{@32AsvQKJfWx<_0w$nkcBo+H933(bPl$0A7RIt-f%;MUB5`+z#g{0%Qj*dZ zcauZqRP4c$#{N;tw_1MN3HfZ!=B<9*b~l`jBbqFuCkV+B@_|V*W!Fe@j{*VIeMI!Y zg&N?+PzseEfS1h0=yY%>^G4e{W=(L(d-qesP`E}piyUw-N6z)Ya8 z4kBWpbAF$r0sTaSl%EWfpn85eU`rk34VdXeTz_8e_M7wv;i6H{7QH=taTTk19J+X* z!J!d_d!g~90gQf?2eXW`j2q~2XmII#FbL@`jlB9TBoTM@vll^v4YUAVp5heN?B$79 z=0LF~26}i7`|wT}xi9!i9RXH-YoRSP?#4P}1h0s>pykBI8YTT>fX9#KNcn!vzWyqn zeYW0>45U`WZNK^e_Qo(%L}ZR8id7m6l}M6>$X;(nY6jM-VVFLIZq=&^j9WFN$k`4< zos#D5vQP6P-Xmw&$06v0LZBbK#?-Dn`%g6d;=HUV+ z_z_Qu%b%zPe-5axw=UT?cS|y+W!3jVzZ2kkqO@ChSobrV^bV~JnY19XVdQ!!(EG{s zQujf+IVv6`8^{_& z_*Zc*yJ=MzmXs%ZebrCW*nL1@tyH859bvq_3*HJ}aj7Lco1Y+XB~V=X0FokH_n6NO zO}F~<&Vff;BPYogF5dY&)r;RANN+QZhhrb52JI6TaRPkgH<^BJ=Ogj^_ol?39$ARL za+KsA`ZKS20l`Tkg~K7EO&~=ztd&W*?dgxqh&txSbi z+3>Gq)<+Fn6Oxll?GqMJDwF&7J>(2(qwlEIuPl(vJJFX+6S_9(uoFc8FhytJEq?P7 zl$glzkjo9>@5q?X);g0pVAqO=HC|zhxR6D9*7$L@zN5lJK`(2~WXPZqz3V$YWIFdZ z(_}JZ?BHbN&_f^CrHK0ZkG<`mNWT~^FpdPyJk^6Mzk@=+q_)9t7tjcQegZ5dTqrXk z`6*&*^_cP=oNhAHwM*cZOopTrsP6iKhdoFcA*>#(c3K_~llDGn5a03$M z5x8++b4ixKr5zc1uF$Lq4GNyT#^>VTwPLQQtBlD7vO&*DJ5Nm7UHtDaNSjiNn$->N z-!iq1+L>j6CHzUe7^Fve$Gj`aaQLoO@UIo(2FSbyc-1N|UfVpDJ)*oL+ytDySLG@o zx(Jy^{usR&T6;W4)7QnW|1{IDz0ck*=|@faqZs+$BvS)A-j4_S&rLQ-{1MW)4^I;H zxX#aA$AOPB{lYl%r3|kMMMOWZazrGauA{5pgjNCEAn)kvsc|*fZQ9;{iKo1Qd)Kb| zQhlW<&Wn3ZQ_@k_X+Rva<}t26BY4WZ3KlU>ZzJ(ED{M$hq~go_8OVA6Tb*6_Yi2Ay ziMJ#Bg-Jh=s=4OJ`X}KA`U6m0_-OVGDzc}D3YL8o#zNyxO+Q0lyPal7V>OAAnd$H^ zpc7Ef#BrUr0 zspTkicsll-+B_2-KJF4bfoD}^B^`c4*Ozmu8qskis!}lVd{qbj(`>KRuPMkAb8%o% znq)1A{zr+HkUPu+g#Dwlu!w~oYW?uX9I3fL& z;Jj20*}D>^M%eEMBjEqhs7Uz;;_|`(arei>WH@X^%A4vvtTo>0+ zi414bH^Z9na+>JKvR~HAz*AvGzA&6A84rbsZf$_tR>*ouI_mX#2AL^B_VDp_1Jng9 zpUEVOtuQGVwU5KWd_g3$E8lUsIw|e@TwCCUuRh_#6gew(0(Z{x!;ULzvKZEY-?SLZ zy<2v0xD>pEsEMNY67+F%I@>V&@O`-@W+AXU9N=Rn(KX{Bt=P?y@VcZpV*Kv}8okb1 zU_xi?piY~+v{g-gD z^GI$dOlrjir?N%5tbQ$ydUzVA{{%VK|5<7htmwD;EL56r`Q)$`yEnlYNFQ(+#-KoJ zN2shuDXTZ5qyb_(EJfe7##&sn%cJFAwOI@t{q}y!mbfx;YVPo zIh#$cNRoGpRTvR-v0yc_TX2lndOVYv{$ZK7@z5O$Q=wo4{q)T)B^|RU`EjlK8M5-f z(YE$9DSH{zhfjcHu>r)~n@YMqDP7i=uiqcRMDy1;4wCAvdXwmW20vyK=x*A{zhDF` zqpl?eUIARB5W}?L)*nMHBQ)y8v(MLhggG|6)(6f-vg}YDH^xix}Svj8j;irF+?<|uU;U&1LFXX@DN5MM{W?Y-2+;{AR&o;&JS*+ zxLf#8X~lFRiq?XSP?%+E`j;bILHq!k;a{t6co=^+bMNu{=Y>GFAfl{q!>OAXO$QW z9r66f8!EEzDwUhp6|CULRAQn|r30ncq#6RmU0;eBz&~L{L^6d_9Kh)6Dw9|cY9AFt zsSmOkhsF;rZmzwk_IWQzt;HbjZoINmosL_zeio?`bX>%gGwfm&7FvA;dXrWqT=h>c z12~ybwhXP6?lF8m=UcNX`B0ST_{cX9G>Uz1KE`MXX&gJQTR}lrWOm9i1Kt*7VXdP;TasH-HMO@Ye zo6OkS_9TDL8dgL%^P;xFJ7Yo$zubJ(bVmty^M5roFesD& zk94bd43jTx^2aBiSxg>$>dbo^`O#TAjt>#?GU+9DtAh+DFriQX z`zB1rL?zhNB??QpZ<%*#BSRS+<1X2=($+(pS~q}|Fpk`) zR2%Q>hrZlE)g!gP=MNh>e{a%K16JrGJYKw8Z+4iUWu?GJl|*A){`wO@Y=M>Fr%~eA zI>qxhDh9Juw~8!lJXMOq74Idn47T=hGkHu*t*RWA?b5#5bAv{G#&|Y$A4Yha0Xsz7NpuWDJ8>6@${@$;k zqviPfu^*EFVb#jiD8W#Vtsf#c@*QrFPy_CnRyx3wNooL>37_qNXczK1I21Z#4|R8C zc|5Nm4r9uG1RuC9ozJ`ogt^77o8LK(RazSW{C! z)iNl9X(y9dPdZ0?K)aeUbK!Bg$5DImch5l;LHK2%J(*0S`J+32`rqIFKYz%7U}AEN zEovdRSJz;v<~Ao(JQqi3PDN`RSU$2=Yh6vw1z-L8E{2Tg*U#l%IkU$6R{Z~2fk6^^ zvv_!APY%TGk|Gx4g-M3s-DP$Ls^zM+ps->(-Azgz@4&Nfp6>k-_+Ldr705UcV!$gG_^r-qoIf71!!Qwg-|4!ig_<5AJyJ+tSWid5}U=!i9}8| zlJbUEh7I*5MdHRf4JDd#J*JN^;y(QSjw?hFumft~FLnb{@~oV&>a(8Qua$b9SX`WJ z#L#d9h}yNex=$hLwBE3E#nP^11aFWK!%bn11%HL|yC9itc-7?~qfS;MC;`YJ<)DXL zCt=5OCEllTv6aDZL_t-UfRaxi)RRp>X2NxYPDTSHw6>RT;SJy*Ba&8NyPh-oJQINx z?VE{?yV6?ViDU{cL9CSufft+H&mgJ%Q>Og;$Nv5Bo|p{RLgU1&Ukys`WLTPv5$7uK zCRfU71od0K+-#;Z;dqS?mmQO&ye`rF*Ty}AtU={}?~bjSE$|QAp-UTqNWs5=PThU9GhqK2@=}|_~3}>9;}b5 zeFJO`>^y$d?*-)A$A^ zzZ>ukwJL6$KkvK#n#E-Ljx>yJ5;x{wwjr2^b7jPjoxt>A16-7dzK%+ny}H%%P-ozu zngTq4AXA*Or{etgae!d1To5}mSbjU#iZG1})`^J{c6A5S!YA^yBj~oGTHiCl`osNX&YQ zJ{$W)_Z)QRx;A?sTXl6vJ>OJ<16Mgt5caeu5c_!O!wN-6yt8DTvp4^k)o_j>$N zEvPV9nwh=GV*vjp?}?>KR#<4UHMU2lSswq>OhZ$lz>j26Ms$o=L?Q~<~7WB4=dr;fSEJ#5C+-nFibq=i`&PzM08&TC#Ld2=orYFX|14ZF$M|p{KdK{H0g&rxs^lf}_lMUS+$)rbq}Xa5 zgB&k`l>eG0Bj-uX{YoGM1jGCz1;_4eYnrJ2Uo&Gn)`OCG~EZz$s?6g39T@=NYzXuj^&2A zgN1Bo?)n#I&it7bh2oQG_%hpz=bRv1ST%QG-`GTOeGPWVCOxKAStM&fhD7q{-6jj=MSsK01d9^ z4>C7Za{;qDFV6yw-#6}?Q&nTo*ugX&hYzu#xdtp5!k0F}!TXMqY~27!+PK++KV7}e#B?CJCpH6N9e9jokmg-jMAEWHVVZgK! zQYN2~;QL;RbMp!QcJP#Uo97+XXYNf~cyM&=h~bFX`J`X__pS7wSQN1@Ul1#jNzjn} zSZ($RMWp-^u{_37S=>mq9i>i&hSRgR_Q8g!NUUZ>(Pb)hYm+ahk!HhJ7bp`qgRAqW zw4E?tlG;M>YY8(^Ju&lNsB;+}5J418$3fl7e>qW7wX}g9&ejb@)}VmyHVo5mJ+Gh+ zY{5!`@xXiOZ{ADqFbZWCOIRjJCj*FeUcqza0Qxh$+FO!w7#`Z7dHMpC%9)9K--VLig33(+=tBLS%axWrTCEi zT&vjidCRMVHSkHmGtO*}8L1urXD&kSmp#_NFAKue^1dIETn##5NbmAH|5~T{yzU0fTI>k=}(U>zs-8~ zvmwa+XY%$)xJ9(WpK)}aV70Rvo0{?j38-K7Y*bE@1mt&&5Id-y`Wk;!UW0zA2J#Mqf*p%u>O!s;1yycmFdt9dny zo8k}0>MSatCND6H*0MJT#yzFydJd=Aobx&sPx3h=;iJJ;p^&LstK@&5E*hrl(&!AX zs46ZUt}v|Q5vDn{74xN9zKr8rltalZ$eo%ZkuQ&5sHa~36PuM`wN-lm_TO(pPJr4C zhpE~T^%tg8d5Np(n-qRuM-<-WN>37&g!_=%S<}9kJMr`fd-RHT60_QZ91}MLHZ>=+ zG1_Sm{#aKd1Z!pg^^YnMx4H(QwNnHmg|8tRi|>6~5ivqgqOyGU@^I#p+tu{v->f6J zLP3$)Wb6?*c73{UMXZR$wL}kfajg+WkljS;|J4|IdKS@J zFQn`%3e~y)z1752nA)jf5NTESBhKN{wtDhzfDu@TTjxe-GPoOulSR9At_XJhMs zyq_K2zj6W~if6Wx9Iy6z*rd@oFkv!<_goA3$69ZdF4EeOUWsQ?Mw#y1#dDGNiM)!r zg58Cca&YMtprgj2nU8uMVT>V&g*2}tUE;N)>#(DtS_*k`4?Fhkn}Hdjw$be7RG9Yo1;3`%bA=Elm20`=v81AOHO3gQj2}(tH`zCdQ!CyQ$j= zv1+dpWE$PU*iuv3QGV%@a)&`hDTWkQ%Cq*RNSw7Z zOa@_Pr}DtOMo-B zK3#&hVC(bu6PCU@O=Q6zRH!JfpX(L*=SdCa(+OHk{GR3LiX^q$--TRf3TJmri5p2? zSK@Gt^&3f-iInGNuAZ1EZ4fsWHR7iI_bZ7aAie3z9)0rds{kzH^vLS9h-AtPp{8KI z@UjR`jm8byD`7Goq#U2>*a4YR9CEx~Mn)CbOQIJ2rDodu4Q>*t8ofWc#TRIt^lvQl zq)bJ6OoGK(<5$FEhS^h7t^}Z8rZ33R7amn05;>|Q84lAh{3LvdRL%uIkXc;G9m4PP zk#OM+-{oJq67bUYY4?Fa#;wC?mks!x@U%YXSwM=kq|$kar^VC-4TEu*vRZ0=_)Q?t z+?gmY?);V-JD@~;k68%1bnwo5tKUOB%5Q;6cbu0cEz@o$k7|>+^=T1Qo;JCY`H=Ph zTZKjBFzYL23d#WyN3^-bRw0~tDlRM~eq?O*Df7c@ISpnNAL!LtiUzQg0pSp);tG;ipiwDJ@$ zdyc&h*6uW0g$7p2L$BX&0poGi^#sz`%GJBu*Xuv1R)UmNq1&a|NpqK6c{ALX81?tR zuk67^59u&fY`Ry-Mix8EswoJ#YE{-cuj=<#c#;W z<3M6Y9V}rm3;aE4G9*w^1zS(%G~)9Plg#ow&+(rycLz1@!a?})eh0ue|Eg={~Whs{Rku|Gch7FthX*y}?o7nbwL7H>6o7D%RcMGEkjPC8F?GXXRyL30z zC(g%I)xt2r$(Z3&9$;1m&-}m5n0#wqen6APnW=RVvw82FMxP*L^E$|tZKstvJ5=}S zrLLD7Gog(yU}e$1d+N1+F+mJExmXd}#PWBy?O4RN)S_b-xk3{RxrcXB19{lL(Xh6# zAw5q+Wn03>O`SYiHNp0Ef1`4XDu!KzKPdhuUbL2-(J z*WITSe%{(g>*4s$I9Y&rh&kK7j-HK@ueKgv$ga**! z4|PQ1pS=t_gT~0|-HJ2BiR*B5x+}7WF$}%ms*raH;?GBJ-F>7c!((EybucsHV?h7GAP*&2G~|UOY9}nq3>fAB?8J0HI_$=WNFS9FLnLJXSi1@3n(}RHmOFZ$1 zm_d(Bh`BC-PF%+6ab^Btk{NI~_AC1ws+)PYn=4Y&KUSp4KaT~C= z=C?V9v^PGt<8@M6WW;K3y$zW6^SVA({%{7S@8-VMLZb8<{glyJfWDBO+F-bIdgB0a zpfzhZ3!~Q(ph%hFqBSc~+TNw=<*)w+?{CHFzeytx>AJ|g@X)o0p|UYvb4byVFhF$m zuhlxnZ$Q8NiL8l<$p#`ARXPj6adLa~^-XIiK6?$Lf|2{|U_uTeyX;y)=M zVHKe{In+$?;${y2QwR^KB)gjuI~K)0ss?-kLZ$JVaM$_J%_$5f?v+L*bS6qV2WPQ{ zkC|A>oZ2VkD^$|{{Py8Pl6s5cq8P0wV}|pRqO&^Lr9lofIX{ZGGPV1%hC2g$1M zZEo&_{&`0X{h_&GM(eURZXD!^f6wqZ6=uQ}L95m=jDiuzSQsB%lwkg$(&RmN{&wj- zzfBa0BY2dk9sqjfsu`B)TR6RTw}cHC*c-7k){|*G8+*HIpbU6gVWxcy6u>%I_myt% zg1i*n%iP5qa;LjWF)eIX2<+C19m$~js>s482sppWLZxeFpc`6DyOjkqqHHl|IBrW> zv}i~?0a2Ky5SU+3d}1q%GJSxe(u3ieNERSmuQc3T)l?F^*Ui%Dueij)|A6Wd+vDXh z5RFQq_*}al>5EvY%?5`;1h?w@3H20(V%lP!!d((JCSBt17X)cEu?XL+09sA*dq- zl01B#6nYE1w<@7OIC<7^eL>E<(fD%EIMOpV?4Q>~+DF~|KiTvBe{H`04P(P`m;BqM zRddzt=wS`xM`Cvt#(faxuVFTTO0L-F zjx!73EJfRCB)w*g54N~Teu#9L?Bk~H&~13W`>270GvPe@Q;hqE(CHwm#~?!poHp{DxG`#r0D2coz+lE^ zl$fRra*@Z?rwWpZ*c8$gQkVlP_X8-7TJ5ljMH9&U7@*8~mrH7u$y^<@XMX^$XBVHL zFgf4j(#U-Rgki>UF1%<|&2Nxr&XAN5X&^1UOx8tK;m+@hUs#^Rz7z-=2T=$COl$l#&z;!%+CO)yd|ag3o2A@V(<2RFQNghVXvBSqVKe=;EsTE z(X*h+fia3Jinz5Ed05W=Y|1lz099u}9Eq9KH=COh=$Svku?Ot~c$tN9FDz=Z8zE)X zaCFZZ7$P#NX@b7k~>Fa^(UsSpV+I1>!bZI9{qn*opn^zUE8f`HocMVO_yv!1td2qNGm0sA_z!# zhjgdXp|pS?jdUX*N=cV=NQ1;#+xI)~`OaUCXAH-1Jo;m;b7|9t`t9x1Q{Uv~#>xHXy*)wHZaX%csmd?b!BWmL4D2xip4wqIODT+YpbT*xs z16#E+!9^cu_X(maYlogl9v4N!mXFp?89zVz_>`Ua(xjpx35)F|E|i%LK$`vHeUN+kf+ul=n-1~F4StU2Zz{Lw>eo}B3tZUFI_!DSg6;-ErU%A!coyvvWN~!lXGWWpJ$lBImyh8s7sVprM7r*cz=?;1( zhk|Z&*m<~DzlildAVOnv#p9I+xF9CLEJf2#^adXsm2L=~EJ9=xaC`nKo&iT_wdjqt zrDQ#1lTPI4TO8Bw)}gup%5Kj;7{RBSbRyLtHk6xyZMz7zy__Mj#GG$1wDox& z`xEDJx?r8LW!UBeic7Mf?$&gIn`I($lW)ejq^c?FX@1y`=g5ZO8$b(W$I9^7;A>=h zmI}%BuXfoQ2S#fEg4EF6!hF%_^Z~i`AQ&%ML^;VB6!S3zb$hL{uY`ldT;9RoU7L(a z^aNv-@z(~AA#1dCuEG@eBY+O74{~R0H-juiF!fK6CAytfnMOgcgV&igtu<1zCm5EV z2X<;3i=9ecnk$c7G*bM&?1G&qxL8LZ4tS?^zii5C-{T*c0M9Z0T4+%yQc6Vv|5020 z0uP96%rW(Z0l!rtKq(ZvfNG>r()_pn0+6in{6xHa?F=0{H@to7#5{s!Y3FT$e6D z(pLv5zG|>5j6(=e(OlV_z;lUfz6?j~V+gU3K#3Z-i-Hi2jOsNF!^E$j5};N+1EFcp zu3fNeC@qd11nGeZq~l{O@EC)xQF)%g(=uh7d{@qJDGPO1DWQ`o0M#>AF=L1n~;DTe?V5*wwdA%hu*{2MT*_`(Gp~_lsz6+ zC{If8iq}PLT4lHd(v$Gi4WwFpdAiLcgM$8o0YvgtE2MW#JZ=I_z7T0|w|}XSSlXdV zP-Pe#%81)Tu6V}|&ha4M=5Sg3sqNY*=*)jQg@4>2WKXQG96|OvxZ97>gZ^t^UV$6y zH5HzB15H*Fh(zP1N&+}&mJx_)Q8I#F? ztJ#uCkpHX?Ar9t_4u5iDVRLK>^-TdGQ3JngMnr~mwFuSBHaCpG!sY2ffn9YExxF!k z5sJ%zm7&+XZf3vA3bSGL-40$29iCSr@W)Y zQja?T`QRGOTn9l(=T6byp<09h^7UZO+ zEZ?UjE4|qq+hTaA`v@z|yb(HrhaqCKh0j&sJm|aGZeL|Dhk;6m+K}Q09viM-xXvPA zT_#AKD}W4@NdQ(G^2YNcFDC?x$C{IkO#%GJK< zxA~Y2(CJhx`yz9Yo-?p@__h6g7kG=+KJeDjH4yejvL2;EFFsI_ucA?h?9vp(9GCce z6HaZ&8pWHgddx;?#Cc10x3VMNmlqeSi|$s*WahKRz|_T|?akxzgx#WbHA$KMOHT+3 zv64-_5AKv+;d8(1#(!R$~|R=L6W9jGii+C?&n|HJFtW^aSoUvkZ#5hGSrQ ze+Du)kEGvrf5FVgK_J2t!9KQuu@&q|Y*Zo}X?&Enj3{l-q2vX~i;rLntbF?#jlL{D z!eG2byl((`ZG0y5tK$bRhx^m}4}q>;3LcL0uZBV?jEa#JxSMiF;cGnt@$JYqwU)~m zGTo1_D`%brTyg`hLkCt^XCTTk59amicP{Db@>WP>!gF1%SU!9YjV`9gLpB1`Qwh#W|!_} z`56zw?-9KRI8^ox3k;6d?hvA~FU;OnqJ?-Lo}pn8pB zu<6(iLFxC;4DgBnNrSEvTXa0wDXO35|CiwNwML)Zzqjr#6k+_6m-0NAqROTFH~%pA z4ltc)gEHQ9_kMdRGhr~W%@3?q@%Q^oV#+kC4;Y}22aue7x-gK!+yvGaYa7be<}M{3iI!c=!r6<2tCi4=LvT z=Y3dg4{4t$e=g@7b-UP^_mB`gh0`I(FpclCHp$wU_;Qih_cz zpJaVc(41f0gR5g%p%c^4My{`o4e28{3=%s>G_oXC^0VQNJ8FqFzVEAC`?FHOM%xMq z??Ss{jvq(f$-uLj_O=$={9nCz_--Gmtl2+)bq6Hu?c8r(z6<=e%z`!+t)@8Ph093p zFkugB-Q?;-SjAp;0a0v~&A7yqXzj^Q$X=&Cfq_TV`fS7z_fpf@ZZ*Mo zFN{1L>mDv0L{suJX~ou)oVE}%cB8&+TB2$}M%)1yh5fTkeTo8>{r9b3d9&-1BkQga zpX;G_Mb~y|H%#&AZ%j zcj*n80&nCjLkHUf(T$5R0e$NjFmgrA!J#!2x6M=Y>tyLnt7qkFWZ+sglMDEZRqg9vg`Uw!|MR}~Quilahyxu3 zzTbDQRfJzlW*jKZY#$;j#5S|S8h6C@h{*5iB2boZRZdd%0xp+N6`LO0cKIpO2IJ+M zV^qz+C`+eIzH{39SUxOPdnEMNy@`l|i$;4tWvp4YxySk(i$oFXFxh$a16}LBnX;ZPs=1XY~iq zvwwq6H8ZfgnYBrOuww|NQD{}a=A}@My|e^VYM{&7??;nEv2%B#x~IFmG=JC3<{QDa zMCZhR@M+s0oFIl-*@VGaz`hl@NV?-EvGbWqYX2;zE-{K-|ETsf{b$h_;jWE?$=e9g zc-pZNdI=xU5h91z1t)s#L_%(j-MH@5Z+Kt7SB6MDd9o1B7L)|(*>Y+pJ>5O4GjTcF znHn`rQMJF#0vyIB1}LfMu)@xI2f*1oGYZDBzr*KRHSd4Z?0?rq#NSH?Z}byv|AqN3 zl$lta1M}P~;OA<+=>9Db78n*9?nV;JM6E!l;lphd((M%=?xer%Nqqh_Df# zWE3^!>m9oQ9n$Hj*V4=P=i>Cm{Am;L7NimPR#|pEX+U-+VI2%2Lbl)SqMV~su>2Hl z!Xzu}b4GXWy?pcaR?TxS)eeVIJ;(wpwIwl89*B6H zsBEYPc~cs=PG|)_BsE^h;yW9ets_mFcW4V$6fYm?<#IIS_$V$)bp>9yJBl%@K9iQ3 z`_I+q%22f1@5u()M%f+R`*gWHXh@;_ew$I+gvJjh-km_ACieXm&CwfBdB!PDnRb|Y z2=+2fz!YlV@mnLnEV8F$QU(1pUVT0T_xZ-R3@e@@U}fQ)Z8`JOpH9b<@*u4wyc0~4 zvW$H3Efpsy!cKbbwZ23SfxZW^7Z7z-p$mux zhU=y5X0RDW{Tsl%eLOy0m&x5&=ypMqGP<)eF*Fl(wy}DbY!z&)gpX>UnAb1=2^d^Dl|X{=5LJk*MmaN^K?8$%7fGYu|v)v&Vm|D>gtEt-CG*_ zr9_Gp!tBBm<5Yr{q^hMgi)UDTGCAQZYcCtG{G(R)aZZ9;;+-FtXyTG#Jf%Wyp*b~{ zXc6#8cSMi0AUYqFb@>kQs2bEpO2jm%_tC!peIgK{H#enR&dq>Pg_4vJyg6I{=JMe) zJ*0duhBj9pW*41eSQ6SHPb}3Qgc6SCg-z6FJz~o_r96UI zqxhyfWHTW?LXvzY$zEcQFlJXW@a|fDW?Ix{5K>$OSzJ13*~C`+#hk zmXlwR>Bi|UzgPTBpVL(^=FjrBj`K)1$#yg z@d4fdLbj5vY@+L>I-!!>xXlz2`C%C`Gv2KFQ0`|TjwFu67m}TFX(fe_rPQB)Ul`cd zc&c?*JRiqFV~s(X>T7QnSnLeVh*u}Mx*ON+yZq|%EC)zqg<&YYohP>%Kc}C0JyL{& zpUFe!0LN)0Y{}!6P-HVVGp6hBPTD_Nkq?d0;_ZalX&q|<3>wsK-@MnG{eh3f=4fqT z8;F7mS;pRbU&wmUpOXqQepq$mAZ*FU@sUf)2>b!ona>`gP>*e417^&GR^;3njffg> zpSOr~)vbZTL=}>cZ{Z}E4y=`EflLl`oDv!N15E)7{sB9EI4TQH+X4GPPt|&;NIm9> zUHf!i{0Th{sryZM%oZzwZA30q_&1@E|Bb6F(kO6#Pi3VHyuyvXNBYC{pIW_s5V{;b zoWaG?LFn0|4lMfuPU`)5a`asIzO~7bf?0B_^MrxOofzchq21Z_%ueBGc&C%@D;_%9{%?5b7U|D_$_y`%>`vDc(CDKNFA}BK& zyD#>d^t}7Y0i2EEF_3u-ZZc!FC26(+b(5w^nRM-!&Uf5BP2yU!Z&}ks%mT1KfCv4| zvxQENu82>8mbG?J@B33XW~y)=SOc&w zP_y>h1&OoEq!s{Cw&tV7vqj`G%@b}i^o3Xs_m6F^`5;a|M|vi#l3;EtIac~@;n|HS z?Ir5Q_g-)|bDG^;C2>PBIa1vbR-!1xfh4oyO7@>()~ZBH?fNkUC_+sfkCVb^3TWGc zychawV)9c`Z!LNy*s>aS;`eXTV(CH~;PlYUR^havyXQvppUIGRD51;t1l%K2r~)Z> zY0#sszK$9vO=*ZjH(ES~6q7O+DX-Qt1skc3miXz<=kkdzi>C5Y>v_de-x_it1X;y* zL|Cri>ekJN9qj{(-(zb#!GnlQN{&!mw4S#pPY(4foo_4F7N(wR0l-noe{x&bumHzr zxKbtk=2oTB(QnKdaYM0Hw=9sC_~+nuaA%cOIUqX>a7Yyx-;jmY6K?+V=a;gB%ey#x zO1!K>9-D~UtubYho)I-E0_e>qCTR07F+?J-3Lp1y>5%$$q_XDr4Z0at<)?y^yl0|v z2Uy2NzkrrK!-~&)?@Xh3>Kr0i<>iPY!9+*J>9*+W{ss8QX}k>ofQa`)WJfSt6IjX$r}GPZx~Z2QsGFe4JVB5K*FK&ic<@j&@zWew z$JlD~eX~pYbLl6zM1^X>Zs<1W_~emkXNWoxjuAcdwQ#lJYnN~exB+EV*u#dZH+}%blbp(HCV;=+uky^H->z)^q#zQNc41 z7RAqtVvTHj`$8MoZROj*TRO7Swee!fE8qmr$KlPh2F@=4*kv)F@uY62`<(bHexY@B zUTNAZ3J2+YUuQqf_rC(7Q89qvX!(No_r7o7S5t7VYl6Jz1hN(*SotIr(JsHw-Tjx? zb){$~#7`~;URnx0RLBDJ1X!U_2ZE*Ag5kyb0q-+PeN0|cVRq!B;2m>7Xk7igTZR(t zQCzjzcM7rNEVp0Q?{t`0k?+PA$VklRhUm~Y7}BH;7FeSKEy*k8ot?R`dNV)Z>95v? zcYP7vcxO6mSAVLkXm|WVr*u4=uF9JqWs9Pv#g2RAX3MEHj%;?bc4r)vsv26-1{d;! ziWnJ5hb08&D7{9II&o8g`WToF46{UfZ^Bb{n*!vrFx1L_PGXkLc6AbMZmrUfW(cVP z1<+gG2_ODZv^cto0CckR4~DE1CaId8hI$C{RYeB686wOgqrUi8O&Sw!%wU`@-zCBi zQC5jIa&xBQ_@&x$giuir9?=e<+T)Hs-k`;-B!Goka@B3{5Hbcb|7hn9v=d+-o^;ve z=yv_{%cMpKd`2xzmfVmvT-3KM>ZU)n!WD_zh=KtA{3?M zw0Px!=uWTh1>nSPzwM00eQw0j{K4pdjSrSO+(rU|QgfHLH-BeUu}^drD~WtQ0g)0; z9F(z%Z}Ab`{xh95NV8#!mw-LE{5p9L05--v(50)L)O$^J<-?8OSdu2|rdsOCO|Ksx zuA!m7hXiFE@VP{#xHav9BQTGo4Jk+OvnodtQSu>p2fgr}P&c-ib%I;wU}Dijg%bF* z=84)A-S2y09PHqGXsD~VnRvP^(nnTaz8aDd-uLv8A)jYJ#12d6JOf7<^8*y;%+-~! zI7(1|5xNP?%QJU6ingD{-(b_ZsK4UB5FULQ9`T<~0Z~1Z@qaG}VT|;Fz(_#5_ zI%Sf6lgAt(%{Eb-`EUk|hI0|I`ROZqqai8!2H=*Pmr?Eoeuqg>vZfo5Th~o%tCJwr zu6YuwgU4(J()rIioKyWz;};PgS~Ub;XTsx+ZozNY6SwlTzSf7b@)L`L_UtLVC$8(l!Vzn+2Pl`_&m_h>Q&hY8L(O1acmYg|u_W8bP{CNGsgV1iG(w0vdC!r<6klr=S2jMbp!g zJi)b0uGx3V3)!6-A2a$j1>O6y(^T;im>{YDjQC<0T;Y^ae%)Z4%oz-Dd>zfxxpPmU z6toPeWF^so&jRuUWf2a(!iJ{RSW$d6cgp&{^oPAC1uau=r)J^*AA%|dy_^t=R^fXmsKzp z*>!%DUFHdyfh>n~7#i{^>H#wkI1S$lp+ftY*cisaX6a{RF`dV_w0|crC4R6FkLuDN z8$eaEIobTaTiS$N4P5vgrUgr|E>vt5YyqQG zS34&(PeV0B(2lgz5axrX{ikbn;=!mGiNcO#ZpUTB8kxQK_{_WQTYUcWAFb?WwlvH4cO{URCU3)Pn3VDF#bIr|L?LW-l*I?ST;+M2upgJAr$AJ~}oiM8~ z5c(Q+%tXqvf6NkeNOrL=)5sKrKtXLuJ3V)WNof-n--{Y6N?wmHs!QaE&sCSjjpsn~ zl_rwe+VGfCsRO=YW+XeN(I{$Z;Kxo;=PM7fPEisIgLnv7+)2S`QKL0hz157XB0WEV zzn!KHD7F57V!VVP*XB?h4(CUk8}uYiEPN1_GSCRc4>4O$HU{C#wuLHR{E%&4=R&p5 z0^;S@#m?#7A^a ze1gYz3tZX`ZCp+{%s3|FvT=8>Uo+1svL}GmShg>`b56DE=D{5Ktig4VFtYEV6u>%5 zz$QnDv0F6}%KT33&76fF@F|i}d}PNh0teHYt74y#B-Yk-&e2BNjR45(Z7WrDs>wo& z-5?G(>;cfpb5IutJgRv3(6A3vQtJz*Fg)1(7wNEMiVY@8i_qdRH6%gwZ-L?feib&! zx5IQ7q!=}<4_3Wew3SxKWPSfvw~i(Q#;BeLAV%e4clQ0r{6|iIjQSZ`0a98 z|Ndjj(?>Y##=74UkSptWe+(qRCXMA+ks5=0`L`MrqgM}J1Z<=*>r3T|!1#QxPTJ?N zf4@Fd?l=a}A(c{ejB$BzJcUc6+?bCB@9x6yTr=;!u~@w2I9&5#L31GLR#`LQgOo|F zK<2u-dg|fdKpR1AqS0uac?;%I`@cP2TrvC9@)A3t&-b+)Dv}fm>%ziYK

po43|W8{h?aWEV9*Pn+go|Rg*J|-{n9JwZ*Un=a0_HB zQC3)utDiw|I%ecDo!@*;%TI9XCr6hvRScQ?#U1i6K+th+NxZp0*|^0n@@`}QQZP2f zTGDzIs&M81bd~}=WrpCHMeFPY-jG$dmc}aj+pDp> z_Q*O`rPPaweF<&fo_8keSiH-L6lFpA7-i8=0qL|VZ#r;X2zV!6pSZB+r+efM9=E8x z6Oapp<|{orpftKIbR)fLV)-)ln!WBn$BC~LtPi5ezplgA)L9alpsXSM@afl=vv;x{ zkH=)h0)JvNF6CVeQI;-nc*HV`&`x3PygU(4t9< za!D!~oflBWjU0*oKqt{N-?|ISlJ`*OJB5fjly1q%8b5NDoGYo?`S~iY$><3;P=;# zIhFUwq{nc2P(;5G20=38cDL|&TgIp>o1mk;&(*{~xDa7dDqu}IVqP|mF&*0GhgmQG z3in!admI5H#h~Cq<0oBJ{@U(LV5S}XJ?MeV=M<>k5T%Fg(lHtlDJ$WhuA4bWHWk*}BqDew!MleY9ku9C}`KCMMMu2=) zJYkKx?-ANE87N<~f>Mp;G=ea>Ia$pBTSnCu0*=FD{@{r zUun;xFhnI@T6lLU5LVJGAG)3RZF}A?vb7!eFQ*8JYCd8T_nP=08UUsbF{zG6nijjC zAQbTbR3|2btHy*0W`RXQ@|^{Nf}S*2x(ytH3o!#!MI7btJ3_MI;m#R*X$BHI)uA?zjx|>ZBD8w0n`41lAr+;i0^#_4Oe2UrfPpY8PsNrp z161v(2|{}ZYVL|_Q}k<0)6h9!V%*ee{0r;d1zOG9g^vNRCy;Lx;jAbr9nH=!B%pG$ zdAqcXR7f4vU_P0mK^dIVj~$ zc}4pPSAaW-J%aX}3TrG0GJV~!xTJ{wDbMax(0BO;9Ks}*93lPUT$Ye~ZH9__94D@d zHkEA^(;NHCRd>(5{2s&gF~~KB4*%)|v0!YL-ilXmzab?y)fzlGR}Uj@rs^>b0!f!3 zxs+nV!f5EyEjydt0=d2|aq}V;#pPVHxGDo~h(7d}OGFZdXM^Guh>T2}ju(Oi`s#ea z5SVyN^09jUS(#)4Bzdp++r?w29;{^u@-&1~i^Lxu?#Ycu`^Eh@Fuk%29d4f|0f8LF z-A6II5y@bQqt9c|z!y)rNN_<2bR=o^8L!YSjC*Rv;UXGBU%eaD<1XsZH5SfV*JffW zcx>h8ya$X^cQ;T_XT2Mt2e#sKa)j@rn3%7mwH}vh%f)l3+sVC}UzUwLY{|nMQUs5O z|EAmjT`(v>fs6kOYR9WesmnB6B19qKKIspCW5ulW4ovJ2ijG(rnz~(#G8(}l=uMd( zD-lgxd*-qI0Ug}<;e>P|4_!_rg-~$JP_$d0;G|J3)30=Qy%%~zPZK7*yAY)0w{O(B z=8=giUMUW9-@G@qUX{(eO-Zsk2MmK(r({@Pl5n*tgNAeQ2D^MsXln0+8{&>GlRc&| zX_K}Z59XAvN%U0Sw96FdBDeF!stQOHxy)Xare zXPVe4Xnjv1Jq$epGaEbJ`FZG`KNGNVv?Lny5rbU$K{2rSj#s*!cU^gTnZ={tfs;Rg z`rriYH~4Z_2^;S2bUTPfWi`?5 z75q$^uR03(c>|Wg-eDzskERS(+fq5j>&=hb$ACu%-na>PWeeY6ycZvI7iMDfM`Sz| zFe1=m$_EQS$fg#vj1tQ{;FLl7NdE)6`JZ1VsKAh+xlxuATIU=bz#(YEH)4DWs;@zi z%GbV~!&Ts07TB0g$#!$~gwz>>#91kTgKp`307BSRy2XUJ0MciZz}?Cpcm>LUsa*RS zzapKUi{OA+%w1NcWEIyrY;#NXl!`Fs;i> z3QRlonN;!IKHM%a{L@TPM?lFLhkra_%qGD9*wq{bcZi|U;UgY3bcec5f}$h@Kd{(Y zFHMW0%&>kk>9jy{Ih@L0$wK0kF=WF56dSSZ*CQKdoDV^Op(8WOtf8!zx1`Yg31g*_ z$k`RY;>KOFWE(I8V@jbU5g8Om2pDa>fM%ANewe$HZFc~!BW@Zg!n{<61zyLicSc^H zR727unJDs&hgoDBxsS;U9I@G1FV_w|=(`!Q)5 zldRpZM>KNZE}^>n#s`(~rt`-zhpdfbCY_>V6aJjY>!X$62>UC@Bb$}|hqew-zXhbr zJ+~V7z%(^dV(zVL@gPc|84|Emltgv1x;xpQR#%)j6@}cB{`BZ!K(D=x8u1$ zSf@KY6eMtgJF%p0-c8@oPP})GpIuS=sq^JyLy4W0?J3vv?%uTz@RXF4NwO2KU&ZFh zp%dnQ9i=ffAg0pP$@3lWB_N!Vm zw%@2>l%TAk{bTn8K8KC>oC;Kd&Y#H)jZ6vB7Fr)*jt^{dgtGIFJKEWScp@6(VEQ_p(5>L=oP)qgITsh#3V8wtoesl(1W0GIaItgoU%doH zD3J;H^dc(;B)qGj(euTq1BBr-$TNBj%-Jz+pWq|>85X8pP)i<1AUA}zM=Y|&btoDy ziBL193VpuT@VJsyW0&qO^RoUU8T^-Og_N*0{!yJ8@IP30gJWRyQ)yjA%jYlSbt}4Y z?lwaQTYb?~1yJ*;x=z|HpbuE-Oat>$P5@Tavryl+ zp&DrkjcK7QUe{*Z!g2K9M5DOU^(1En|I?|`7X*g++Ra;Kb1l~}6ImSX%`Cvj+wF$$ zmkGne3bw~a3shr=<6YD}!QR$B9L+v-38;_e6$(S=HGx%Fho!lA(&^n~4Fr>SYD;>H zz*FBF@L;euf!yVDT*WOwZgALRE&w`0X@hqSkVN4LU@Oy*#0&s~cstfB!w*Jl->q-| z@LfOxBKfEam74ZLe^fIsPW3DTW&8A~=&%3a5HFaJb0H3&@~X!l7?+7_5D7hIVr0Yh z(zkf_cp<@HkzgUU6EfJ8@weeXX&QGFw`qev1df0Nb1bvX)X& zMM;oOD{euy-DLNOPY%FrnR4#2NEkZFbt;`_kL^V{obOh>;2y#25Fk{jo_>b=kEFTr=`}~a8GKhL}Kni{&(T( zxWE_x%rw~RvV7=um>u5vZxB^YbEHoC;3$qm`R-9qt@V#+DI*pfeb*b1ENIsKWiZKf zEtPC?I=wqn6Jy~R0jGthFTDXvHlIDSE&~3TQ;)&hXUO)ysWQ|j3y3CRyR$S3_dDRe zGi6V7l$`yw!b{`m1WGl^5cj^b7gl@i0>Dhno|o|AD3*4TBR$_rykP(;)+%U%myo30 zmk<+}#xfBCxHe{S(x!7_I58UoQeF)92_Y+-yOS1ML^f4jffH^n$IrJxB)56_ElqTX zb@?j-@0Sle7m#_hNm5^@0i}=)$S5BB)~^I$R;d>?^llXxzmZ&DPpg2bD`1qDac&fD zq_8^b+n$1Wf zuQRbE=2`=)3{97}qJtDr6H(ZaVANtfjiN=APp!zdzP)*TOU6Q~eH_EH1@AYZhpbF; zApv{y{RwC_c6A>JUrRF0o+^3%|2Ocz|AHdlU$ZbC4CML(v;M5fTx|HoB-jX^!|}lpX)z;wi*Dv zjaKFlQpW~F7>7=#NmYtkyrZBheQWPyT{>;gl5TQ9W~a)tU>5TXlheWOTvm3l+4)L? zg^&yzaDh@*-8%pP=t;Gr*seL|qI;}!9i_bD^0hj$KTEkxFyio#=A&;KI#Zd67%3!njeLett&zWQ4-U_~v*Fe#6@-DJ0mSMHn$2zV~oRww?!-_BM= z^7YQMyY4t+U!?+PLkrw*cl-;5rNxo|;G1CNRW8@AA?Dm6|FA!`y6&hkrD&$~`QX4~ zv^j4&?v?_D5day{-2B0;N$H2oC*0Tn}Xe6g>&A@xYQWo_0_ zR6L#_uN;TNN_WJrH-*I{OXoxklP(+KQdPk4|5|ke zuJq4E{KU}_FS=QJmpHK!fCY4EWm~0P5FlO{l|$i>MkpJsY12-|tCrhlLfineOAvp2 z>ob^BceqwPKx&KMB^#}q!sBmyt7e;@Z~eFQ%usi9hye^e4k%^wLy5oM_VB5)K5Yml zy8x>#F&0=MCKd(%yXX|-1){R8szj)u*VKUV9x?*@mWW~BYw~bsb$8=<21RN>zBsU1 zKSUUnUy1@>AUL<9LRAbMVrsE;l@HcpL znnxHT0s=E$DR(;L6D=`gKw9MI0D)-NM#zJxqqVZ~q`bUb^B-$7v}|b2QwCL#2nNX; z19Yi#{|CP$0DTbDvb7}By=wLk;vx|}@-%G0vM_t|@vcaW#)2C{N{yps z53fEC$sON3kZaZnq=d3wiBfL?gM~L4d-uJo;<*s_%m@pgE~F*kt*3^kUC1oxpP;r3 zV=NTwwKmiRSJf4TW1zmcJXsj1c`~m4t|NVJ_ysK4dXxQf)!Sohsm{xx z6e16|cE;?iZ@$CwWs)i)mhz?K!q_ItE#FRrIoIa*=S>(?Zv&@qrc`bIOkRX~K}DK@ z6Hb~KGhI)x&~`*;B3n0|iEL7@tB1$jWIvf12YF%;`# zKtY)q(K>S_x!z}dHiUtCl2lgv^}%V;);mrd#@Qn3Gw;2DNul=h=PJ=HB2{FZ))>GI zPkUx4y-eve2|xL^iHdg}E3#eOe|6ZF3CRrf;lN%LcEs>yZ8bU}2rq`d^X&$Faw8^l zQQ7#a->iuZ!4JK%Wak{l4q~`31b+;*{{iJ-$TzKF-9Cc~WG*TNDZO=8msXQ%3LM&G62^|Z1WhW~p*!;z8;($bt zjvI0ij;BDkTJ8*emrmB75oZA~7gJa)xH@ zyzYbS%iIW@F272;>3rWUKRj8L$lPmN<*&(oS(=Fz?JRs#Qy4gpD17w;Lfj=QKw+cd znj`l;nZo4-PncaF@k#5+HE9r!s&<&zcPdZs^L@Ja?jV#VPydtX$m3q#7^?i^2K&Z% z+XWI>+SQ+r>gK3P1I-4&RxY@;CgG`PGDIcpV3I3A`%i&5qFr91X@6Z*-wG~*R+j2I zT2qSk#B-1YXVpL@ZWsM=7dv_W;2n&JnLjuC-uCQRjfQ4tY|Trj3S197whC%Iy+b;= zg}6-6XCwlk;njb6R2MFv-3=W|>i!SB_`hGmA#<+#=Id?T$K)8o7)x9aBYuL0cE@Y2 z?hRQn?~qiF+sh3ymDC!oPC7q;rXVGNb5a+Z{gCBEk~hg4SX%$d>~iQ&Y(OjL+Wx(}%4vz$6=+!th>9!s~ms3y!jbKT>GwbIiONQVM0e4o)lhnCf^{z}98^w>c~ zZGpk%k@s%sTKlQYE5Z=<8E^rStxzKggdG)p4&9fwN}qBFj_C8CPo*hpaVR@(1PFq5 zik|m9L|HKs((~PF3Q^5cs0ywU^ngR%Lc~al0VKBnJ&n*Ox1f!BTk_lfs%Jh90p|693{FPx^Or|0o|Yc^4QulKaRAvC(kl7q4(D9F z(*Z``VEow{d+^(->p=H{?Oo2%D#za-L>{q2mx{8Mp(eqjlrNZhjMw+b2s@pX?C?03 zdGYe@L(QZMEr=16Oz`_u3)~m<{h9Zj2Vf;YAmhV)vVT0Y)Yw&Oa0xIdrsm0Rc}?VOJweT?Q| z#Z(w_YkxLuq*+4hx6mxYb&atwyv#^^)1CPwO=YaUGhLrHFVv;_YT0ox@nI0+9>&_`@9&8l8GUlKp0X&5Y!}&# zs1bh`v;5oSY?`o-ojmaHg3b;m%PEY!4OzVX{6cY`Qkxp{r2T4-gwnj@h)p zSvGwJxFSs?Dd}q`7+HqhR+x`rIza<(@O@B!05_seEAZagZ%~u0t}QI|XjanDy==?6 z^1MmcM(4a9Css~%3U=B`USID(`3R>-ZNG=#GkRLwp2Wk)~!5lb8Knt^{~J?NT2me z;y0o)p#GkMfwtNkW4;&tzR>DHkEWArP`XlfBa;$JwJrs&A@uT7U+(>dy{4skf))AX zXa>oJP5tK=_rD;^#$JB$4er@{kjvl-GQpg$^l{#n zHbR>}Ff%-mWIM==?0k5oRec8BNgi^H@R(XkhAyU{h9Eft$|ocxevtSu4);GV=2Fe2 z$QFhKJER41dn`rmHj7kQDZFaqIT?a(nII*O7RcwDR|@(ecco&E`j}#fK^IBRe*7|60p{j`t>k@tFUKWwOR)G4}F_aT-be*9&k zcTi_LmD~Bj;v!$yj^s8wm_vvRd+5#0_QsWXAC1s?ZRYJi5>k!Nhq`tW#``R5Nu@Pp zcf}t7xRcJI$1=s766sfFa;p9uwKKeE#E(J$R^H*Qdn_CdFFg;e-wb_NcGbnhjKkNu zw*HQb_8nsK3_}w7YfN&im9JCOIN5<-sNiU+Wlub<{@$t%=eX!4ffs46iR*?iZd7~gG3`wGut+zb|-tA%|$WqR=6gHhjLn=rDR|W~85KHAY!2(ijGs_xTxKB&* zw8vHdJ<))TQbkYG?1*XS)SCs1QNV)kmo(W05bKJOgrEs#7F~l;RQcbK9^fA^FKkD= z%A7E+J4AtOS&}GU0~cIimaIIzvey%wmQY6$_Cq#eRzX_8c67vC2~b%g#?8aFZ@Y05 zRQ!Y$3wGxYpTEcnLCc`7yr`52j2W%87{Ol(I*omxES;q6oLfpj64X(ps8Y@Vn#xa5 zsG?r!*uOmb*h)Ez119rj*B&gzuF@(S&tT)^9KUpz~HfsfPRI z`E3By*_AlZC;+0nsveaXI6v7AS+0SFyD8hWU||5VcnR+5w{PWT{Bh`M;|VP{l#rMx zq5FEen~y^NO5fou)O2DnJ|+H*pZ&rvsH$zV5(pa+|A(-*j*7DVqJ{xS7!ajV5a}+Z zLBgRCq&ua%qypJ`Fv(Mg5%!QW`9x-tEr2STYURq*MWC_92vl8olb?Joi5{r5QBQ#-&isZ&LNc zvWzWX^!!BGmuZUlgVQm`pvNh94;q#wq7` z&wlhnpE{X;!OvyuXok*1#fpa77iC}gH(mh%ZP2px)twk%!O4|f#`b9gaD-4{v^4T1g!fv>gKt?RZ-Ov%RS-iQI@aC z`tNO+{Mos{##pYb^@yCyYfqI$%eeihsb!6pMLC<>meTb4$i9PTH1}V6di&mPbzrC| z8@>64GncBYnI5d7aImQny6$wbx`IJsV0AwJG(n29TH-XnFQ^d}EY<(WXNjsn+Uf6< z59o-{-Aq{5FE;Sk;|2-p5V~E|t*zP*u;{GNq=`+TzyZ&zd-@gk;zsq`JROq1E`*x{ zvaPOsAn7*NuY6r{_Lm3F(W%Q!!@U>S^gf&>YmC5_O`26k(JZ{#e?uUk)}mYT$BRhv zd*Y{^qsKvM~SXL?Q}(?g5id2msMB3bm*)GK9G5yJw`cRqqhk@GsyK~atmw@ zEXN(2Ui+wbUYhy|;~uJ8RWq{diI}+?3HI!%kBpzxtKr@hp{DyV)S8v}^LWVCbIgvy7CdYTyi<~L!bg7u9+0~104ss<%nkpF$c0ZLN`IsfmkWT zU>~fy2-lbMdr461(H{iu5b78AUI^)ERz@$O%&zRdjPzwCk*3E(ny0pIYerEDmw^jp zZ^gx(!5ZH}{XR{_YsE_upEn@)>dD|#x0M{PtR^)@Oko*GFdLh3I)~qqDkqYm3%@&* z;@3-AKcw35P_ICi-0rN$iiHWysVv-vSBMkWj5+SVj9pYCzq&-PA)vv!24)5$job(HWusd5?J72 zBH)E}ChUWuIG|32E;xs`J27>|W4Gj?gDxGMZ8t12Pn!j7FO#QSNHe|Zg4 z>Je4X1nHZqihW;FlP<<)%7J-+G%D9^CCIJVQ*!yI8lKR*$GQju-#SuTMxKN#$>qNo zuqN*}@W1iSQ72;Mi9J5TyWA)Ou49MiJN5Kw3B>4k}Qcxh8i?6RI*cWqbLTp;BZ|?oMKRhUImNyo0T5s*K1e z4$bk%Ty=l-^&fAK)}`}yjP5sLG;7@p{|AtS=4CsOPGTta1cIJOZ!_Hc^u1{* zJ?h$}`$Y?xOl=ulC!yx{Z?G< zde6g0HJW~BuPKbKx$-IOW1`jnfIw?tcK;@~^j>eO8P zc85)e|9%>OzswX@*cL>G#o%M#fB864=E+)n`t|~Uy*z{gEL@PUsPaK}KzZG+psVu0 zR59&uZ~M!DApK-|nyl3yPp=V2Y*1s4riOL)5ld>6+E&R2c5qJpn>`PgGcVWtNmtUZ z#1xY(8!VMT(Okh_#XvzpVNuF7^Z;bRjzHHWZFC?Yon+5vnn$mIY3vjDQ+=s6n58OVdDSllG%mPRIlymB5xtr*{MTe;#43uVg8*}iE*Sl zG`bP+!`X>FepqJR`sLRXC@d5*x7SA@2I>6eji3a2e`*>qq#QfM3JJ)JVj$qUC0{{; zb7pR`ME9cW)tIQs=`*es1L8gwoZYRW3o>H86k+O1dcnUAp3(&$-8IP&a-z&4FDTjl z1B$Q{m^!HYl__%~YrR2~ja&B(c2PKU+$qp9bzYs4m*I|9=F-^**Db=zg(McT=jIA% z&_2}#0~%Sj5Gd$7$RWOQJC_2n6(m}()j`gPOcBB9y`$^?`~KT&zE?u;xO;-F${J_O zOlrK*{m*!#H^p(H;_ie0bPHeUp^p1ThvjPm9~o)&|7^kKqdWQQDhxaM6REyBU)6Xg zy}=ZW@0oaEpX1W%df~rb0LWMQZzn%A{D~8ppSp%pI3~D5%*0G{U(%f|&==NlgsXSt zFPj#DkKK0La_P7=F9{i6=;^JqW&HbA+Q&B9U`$_jqva9)28KVmm=XL7!?cfsUM7tu zhL{p59_+L>Z39WQX5qWb@C$CF&+S z*?rA5*dQqR{*KgS2(;L(d+mEBOS#7Y_Q(5ExhRxaTZW`(Q0d9eO(Iv*Z&Te@Sa@!p zPSWh7$k#9NiDg`u3sEy=w>{g67XEVPPRf5yW^LijIK2VvO}D%jXq;vI4AfA1WF!bL zM9Z)P5Gc~H;OZ9NCrcRr!F6909OZd`w;HBj z(+U9Zm!tX*a%cqJC5gdIv5!WUqac$p*stpfGeUOu`W5mm!1n>JuqU~j2Of|4GxR9l zq;Z(rQ!E{YeD8S&jAikSd+y6gify&n^+GiR<19D-=4R2^^yY0A$LCJK-!;F5q_dL` zNpKI5dwJnn!4&cQ5-}JS&^Uft*a#FvZ{7f+!{|U|tR|nTeo<^!zcj6}o-C#(! z)*5ElAYX~?{7mRp5RpjLe-v`r?){C$g2#0W@{w5*b&bIXTlu8`FP>G`(Q31^_OM>R zm+a9YQOENTG5maK8H5X(aNZPpi3v11ocJiu*;i$Hu;pTe1Lh%?+q;H&zR1=)g* zM|I0T`RPgIN$BZ^eD`DFKOO_jimTwI%iu{^9Of8<_4>_4Vyc>(F_y9AP>k-*`s#fQ zkjZ>OfCh{Iik}B-&$SGGEO73_;(&fnBvEq_apT|fIn)T_Zh}#T-p8uPH^`{zJup#P zAt--j@Nul0n(4yf9XaBta4f|eUSqzs`^t>c^f=glq0-0-a7}EzL|O#i!qnAHeJ*tzJ>aA16mclKRQ5fHmwjxsgA6 zY+f6xMIsyo$|S{6S`^acTPsAr3a2_l;`T%O3K-d=&Z8$o3e9EZNuwgM~)U~|7O%gF+&kXh}c3Fm6VKPQpiC`_Hp*-RO*P0vyd%_5&TE} ze63U?p52_vs28RtBk}p&!|Nw=(I@MM$&*iI`p?bE2J^Eck>uCSW__A$R}i0Y&IqQ* z`@uG*m=S<8{&YpC4xbFZI7=QVRMQPMx?QH4T(VXv2Tc;5T|TJyLJ&s@;TU}0u0~EY zi<_}ulfu!kMbfD>Ec4+6CTHKB-V@s7vrey$MAp!u5X4y6#8~JsHgj)*B-?Q}B|n06 zddw5c1$P1_b9G3i*ns}rK1mh}rwQ?4yEf+3nyG;@6amk-yQcN|dz2mIx%i2!zVd(i5*Q3v(TXk@_EP^Kh{5zxV-x z2@zUCoHi#xx2B(65Q$M-Mb9*@_~c}MlBGc~-p1Pebj`HN&9_72W*|bRr)JTKiU*Af zAv0&eW#?x;mVfgdg@yS02ZwC;f(V&ounlf1a+DqmhkhXwE6b%hK*JBN`4-m;6cTq9 z9>vXv5cLz%zGSKs@@{%FT&!|Amazwuc?{fbF=RlJw!h5~_vLK_g_Zf;K%6EJvSS~N zKSGu446e*Nx73M00_|IR<<)!6P1Nkfnn=XV7U66D9vD8Y!}(4!T0TtS(%*7^Uv>AD znMZLI(Zt})=;m&F4x=_~zYNF|_fVfF$HG{`So8H{|lYlXPux#l!DZJ=!S2x2UuxUb%L_bdf$~6T3^e7L6g}ri%o`_^ZeBvkyI3xvk@vC-*4GXM~@w4@FE7eiq;Sw>W*B?DrYd4kNWmtUJN?=JR?Rq9{g`>1x(BoK}Xo%vnG zAI&`iQgI_|ZgW8;5?8R-!-VU z40kd7A%C~o)zlwmSv*%`n<(hPy8At0dCfrLIS~b}{SoJHjfFxaubPNHpeht+s!g4!5{$Vx{m=PKsLxgfytvj8 zifTZbKZsPL{_8n2x5E&By~`+b9MB(-gS3j*9sSS~W{%kd* z{U&Xhkh;VS;-y+8tWEKT%8vK9*DZ`@AAp-67Ua6TH>VUB0o>xz>P>TZqQbuH9o@WV&jxz?%5Q1qTde z3vIrwm5rETnY~7MDOD znn!^C?}rMzgOYqf-ggB#WcL3xA;pkcJwO zJ~;q2-lnLY)S@^X5ZEX{QXW0>sYpq@rjuO&K^m~L* zo_qwm06QuX@z3kaP9|z4Nue&qE+F@lv)9E}!0rR{VXv66^g6hjD|y}wIuU&aY4lnX zaW)Z+PLoD&@`cwG#6mV_a@cLe@*ET756&cn!=@O_o2SdG$!J|bb{my?@B8pGW05v+ zayl&9^nK|z%}LgtJ~}5|gQEr`(RI+o(Z_p=x-VdvK0G}v;t~m)Nix0P?4+{u-Zc)o zNP{7tQ`(Kzc3NUiW^Sk&3r~b+H^u!D*s*c=vDpRsBocA{JAZ%u0ht<{^X}_dT)~8b z*nbC8iCAgQeDbTXID<8PKIt*QTHU0k49DhaD-L!9j9dyE8U#6F)$boT>}d2rY)=TK zw-xR~fLRT`J7%AR=Nha&VTI+=ciOIx=W_~RdR6FoZc)9CA?J|6 z(35IA^Dl~jVRVzJJjQy-;h9tQc0_YQzrT6?*fM9%MZdSx#qM&=u>s>Rt8F!t#5GBrRiWY`+D&7&d)tyGo5rpb9K^^RtQ0C4 zyYS@y>s7SHk*bq&kaD2NsHDI2?|VvYf#cIph_H!3YO6=B!u(17-Me>bt=9S(U2(Da z+kvn`oyM1bK5%I+t{AVKtk^#Q6aPc|S{#XBHuv=q7ZMt0KLZix=;w!LE=5jbWzy;p z#+T$n2Fhzf%#KO2P!h=2HMF#nT?A~yvGpqcDoA3wEl}Zg<$mt3JOzoM(Q+3;!gjon zpuO`(0*}GfY`SNyk^Gb?dzY}Q+}50~RUdwvz2G6;nE_9h{geLk|MhJVqgxxoPrQu! z)h-it4sAdA_pXG?MJ_t{R?hl&TPHg>Z3YFXLQ>>%%wn<%uV(B_3Ap9l7Wdt5Sk6$7 zH|BRiD8|#&Cgol>g)!su!OssDLVCf($})mQn^+!bJ84j3c*R1IiOr{vbnS}n59pFr+PsB<)XT$ zxjX}HdlvDt)^;kgepac!9+Bq%=Mk~6?~>ZYRC%XuJ$8a@ng1|oA&Oo%{qj+1`L%1i zjV0u|QG&=a#8gMSY0GZ+g>Su-R9Rj6=kh+hgw;2lTC1Ds{IWpv(rji45Yb8~`LGSQ z-hlz>fIJBvIv9~HjETclop}+^ta$`SuM~WguR4e%0aC*7U^*+nla&JVs>1KN^xeU* zSwhDKx2d-<0v=n3B0S{}$OyPsrp`xX5f8$nUHm3 zq-j4D<=Uz7kk#t}f=cCW0-Z=jJ>v*aj7~-07%TP@GMg%U_{D8-$NOdT+bXgrUr}Q? zo*0^sBiZd=P>CC2fJQ$&k$?|!?sJuCd)ny7j~o45&x=qo+EAIFf~npJm@h@ot?t^9 za^29Y)Em5?*r)0TmyOb~$|L-c|9Sji6axR)fu~$A$RdLwq;o{@pGql8ENb(#IT8Od z*`?k@8^*+J?%8mT0>I-jy_w5{k-3rqMgn0)Ftcj%V6%uOWy!9S)v6x8iXn*(%f6dI zef9|OFmc4fS0||vZ+!sQi)Ay?b|nVf=m@=TrlZMkXFf1J_Bn)yv5&0 zuL?US_EwdVDVtwllfd)TpkNxfk6!BdSr%bF!92D>F!7br*4Hc$7|MtX_`~uEv?^Kb-|jIeRmjs(|vAh5wcbH)d4DqEo7W~ zcy@}NitqQu5_W4=9kl|%;!f+P#u5w|Mh0{$e{THVxZ}lomI$CXRr(3UeYW!d>hM!7 zxo+ED5cz5ks4O60c-CIi1&nKF&Q#(9*yKth+frD4l{E5H@gru*ET&Pg z9rs4U`zGQr*nfu7_$fsdMx0cK>mxnf9u#Yhd~Wj+@C)P_E?*gnZWX_T%*bU5K6HUaT*J>GgDEcP z>wLvU$!k((=;rwO<%_WqD7$javmUV}@Ett{7XhBy-&u-u560M^i2J&B2Nj~3_Lx3} zdp~XMPI%_`b1m_1z+ee;t|+$v^PiQ3DZ*)*t7jv7kPqcZ$6w&p>kSLzmz6bBu(2;- z&w5fa6Z>UjO@zI5!@5Iq(9;94$mk*QF|g%KW2g0SGE;xEK!##0jlZ5C@HcF&exO*(?d?IXF0s{;=f*%6yHU z(JoaRNo0^+hxfAa*{6KzbC$6ek@O1@!Tx#W-k8d^q}p7C`$Oe7F<+UGSVBxPGqFaL2J|sSe-9cMG&am1vJ58z?XC>cI zVq{yTwN~o;aRqAk&k#G`2M0UhP%J;fL)w4H7AxXX)~hnm{A$0@i8g8j^dJZmm_h$@ z7p6Q$;h9QTYR_jt%i?hFWX%P0_9jI5}u8C>@G-y+9V;Q>GeaolcQH>7mY5R zN6{0*mw@g=(YoTCM3VpB{3r`nD{&uL+qfy`YgsK9In4rrFV;SZjd&sa>^mC`H} z!h&P)cXuTh+xc$XN77|~PJYd|VoMo?i*rWG4&q^mWgOqnc7vUw-m7^XVqGzpoHK(! z#)c~*%!1jH_lMk4!VKE`KpQk_r@}Il(>*^FwL@CE9qaR^^8(7IX8f2zhJlvZe7C6!bD zgsO9><*-vZQZa#cJlXY@`gh^oTtCF_d!D^&hcUZeTeA@8=O-x6_Cm057U5dqr6C`Q zAj1wS<98TkeyRj;*h(uynvQ-V?C*wMOo9IKM4}^h>VLf~26QHL-utBbS$E_rmcKFr z2UPwv4`5#^^Y!_IKtHmOGd*FvroK+StGTdQtU924KCo?+=2(aI)%Q+s2qoo#C7e>P zq;Ksnsk$HT(BAawYZ`!6;2HAusN1$)LyEs++nW& zH0;Ys$Nn6uOJ;Ej>Ssl^Y4vWK!3!h3h~H44Hvs7{>1ix@4_=J{%r_c1d(fpIw7bgG z{**(jC@PaeFh|>y=;jUbU2^yocdOl+bgcBO`+&wU@ECb^Q@32}wa4?lL$IV0fPOID zfj*Z72mL4ih6q^1iu7lRJU0vzMOtTIZMSkim3@t12^LDqzfAcN2kpn`{((PK!m?=~ z_`lwmSQ6()KVcK~rID*C%>nz3CCt*ncdKX?){YE_oWBM0@YHzgmNL{&uW3G27=i6~ z^E=Vcm;-ma@Dz;VYB=0{E{>FIvkAQy9%K``3_+Z&4e2*5aEePyKc#t+QnG8H#LZu? zRvoOoU(+GKPrslPymMDKqX5Wy24fj_#}i03N~!}dc4>cDBw^0TG=F~)?ATI!U3uho zGtHk|mmtzpxPj**1~%po?XTC*(!2tmGKzvR_rVXtNlL2kJ|2$S{%3x8*ii3XUJ!NH zzC6@3wD{T^P3Er^@Cr7f%3#wJ!&)Ncu=)ux2bUjvqMwj*>MkV?GAtH8BKIUY@n{(= zCq~U`=2dK*dw1wn4Qw^?F81)t(%jV`y!IE$q36Y?x$At*=caT}d}Xqf?bVGm~Xnfusz-DBq_7gb|{=gycK56$ZQVjf!q zWp?uGLBuA+&|S+~xOhd11C!J;u5%+t8n@2ZXDHO$yF!&jK7e1pxcv^csk}1cY;wNUx zRoukvyByzmP221#DZo!q0kHcx`31lsm&=ruJcPZ6^sb54T>l1~!!UG%4nSx2mi(D! z7JB7pmHM`iwVDJ)pN+sJyn@=Akt2f5zky`q^Dkch$xTj{ad2>e7vi~bb9G9SO3hWI zXPIfcaBbyU6|SCwr3yPFFc!mMb=ZuZq)K=OF=bxxIyzoAi(P~P7MJG`r@)vB4Kx~y zcR#PT%j(+N!*_2`HIx1uB~gnH)2-U)ZoPJIZ@H%NdjDyM6oWVJd?b5Zi#eCvhwiT# zS(Ox!bY>#s$dsWF$fW^M+`l_ia#;A}mzFwP_$XHj6*KccoMJFa4oS0SCqQ`}_v!M zfr_{xNzEo^JJ8{I)tIj~+r;X&?Uw&EWS43?I#%S4ktQ`tU&gK35jc) zzh?v2S&`dSnoa%YL9BOM+%5d!W`1_w%7k>t(=L_*pj?dwpMv6`o(1gtNq&WBTHfyV zNc`RdN!sf7O?E@HgEO(MFVc97Garl5zV-jGr$X$7)f$q9!X|d7#{@UsT+V&(I8kLd z%+HR6Q$Px;2TqQZ0|M+S{O2YP17wJ$;NYP*j}AvZe=%RU!uU^#X2wE)5$-3HYwAX% z^OXE`P2$U(4sao3_jx{p{&N0DT<_CE{EA;>; z%X0Bh?d`?f_k7>iMHQgO=v@!*szG(eBKZ3%THU`BS?tYN?$5+}-LU(oDUafY%U>?- zx)C0bfM_I1nMOg#7{A+hv2cLU>B7!a=H&0PM6OF73_T2BLp1(k;q-q8M3NVdxc2>2LRC&T$LRNe4ZLc6B;fH8c zjcNlh&e|FLHrbgdT4W1jaKHx&Uh*GU@QD(cSl=N*EQO<7jd)GG^AaPng3{NQ9sivO z002Uz;}=EmeEMiSrXa9Ub?l1iCNT}A15PTZZt|NanVFo$gV7&L)jz^DI>?AAy=9iK zW0JYeNcV(~fP5u2E--@;%6UGn)Urk3_?enAlQJ^OZ5h6i;OzM%C@$5NrBPX~>Slosy+McW_9qM{=E%w{&VzTddhPU8>0VR$jP zk7o6~4A1>Pued%U@EOP4tw@8FmpOv&K0=zu9{>ATSD-XFx(ukF9lPc~o*-2En-H>$ zMit0}m+6#irDSJUUEG8;p5Bs;@xgoZ#1D@bBct=qZxq0Sj35}LN>cgm>lWEO*8W z%a*mew?vI(30MG5l&ihUR#@WS zXttLY4KOC_4OmcI+yb8gK~PCyY*Qbhe}!+p+@5Ry2u1iJRo3%7K?l;6cVwT9CH8I{ z0dpMl=OFKFI77gW@DI%fth-~(k#(A!4OOLFW(57uPLDQV+lT33u?N;&BjRZ#mBDKG z+TLoDem6{hieAKK3OPYy@QV)q2UfXz?g>{zFM3*!6w0UM=JGaXe(WbS7#dxP%0kTE z=^ZSNd4Z0oi^hFVSnE5tbYHGzZRzJ%i>)O3U?CFx3VSjHiryH(AI2X_tyRU?`1kz# zuMH{a)uq?dWkg8&*Lg!{vkt?I_Z?{u3_;W&r|lfbY@>l$DmCmCqO<2RZ4-p_ z+YH{e^}=jg|EAvwhpphy8-khL+)gS0rger@k}GFDhVUlRhpWq#9o&+tV;hd(W6=84 z|Ba;&|@kbK;a78orP&|XX*gCY+&`2&QIia%X z8`I53zN>MCoc~;Q!6+0?hJ&2qS&gmTutX*CKhJ8wtWm5?LCtTCTQFd?cM+4j>WxAD=urtN3;@x8y{oK3g+;*!HnZoZq1psmL z)mPqJz?r9CXixfZyf!Su_la`!(a&D{uFU^sGj9-!zKmi&D4I>oOWwF?#Bw3&?}sTFSA@h8+5#VipCgVX^Dsx4+I}_ zn?|(U0r0?k-S`7hCIyuao#xjvc)Lqt5-$m0>)9$PD-(ED)=txGw@!8N)=E`?NhR)S zRb3XwE{TQj;XHJUDloTTH>_i(;5N81w#4wHClg6UK>%O6T-Lt_Ic1afTT=e5f2syt z8yxlDRz=?`cI%PK#FGIK(R)L$wVEAEdFwj=J~1}tQkKq#n=;cW8gt`7Bh4-(lsd&7 z0RP;@%8StioD-|_U{&Q0`e&i=2&liK0M73KBJzxTW%k0q0Ya?5p_U*?Lk(P7i$PS; z?utjTl!vOC`!Ygb!D%c1RJ~oQCih!e?-Rm)!aeU7mbMKr2-6>~6@(kWo{Ag`t$`0n zGgaqDKcs0YIp)hnlcoT7a+~rx{`{7m%h31l-<^pvc3#0y+on7RfI?!MFVPt;?Sxz! zFCHfbWQC$4G;B4_`jtr1sEGfg=)2WB%`)IrGo0i+eNBvPAXUy?yW?1G|Df7$oVk#5 zO^#tqkp!Nfh;(7?9*-A22>I!-mI;cY0V1(& z{N?YGzQ?;3zhG@dP{7T&$$3O=;~-%Qk{X@qx*rH%N9{_`Mf95tw^4D(p&Pnv#c(j4c1HyxMaBptp$-eImTwxPZ4EK(}G3^M!KKOS~+(#z*qtu=Cp3(MO z_1}S$A8*wWbNq9pu+Z+L3+Hq|i8E{lxi4%|H5A+?_g7LeVwJBjMy@IB1Be#Ih+VbD z-u=N+U&FSKQS`e0r@vqchuVZ#7{~*}mKiS~W=ZzpHDy2#P{6b{*g z)j_5Q+PM^@Q#ojAgR3GeEUzbSqJVj_Qcgm@t#0EnA6$fUTJf=8EA#Z^nxy|&(>BW! z6HBhh8#mC|Mr3G3@-wl$CbKVjqPB^@9LP&2O@D8`jBr3Uy7~!^rJRIQud8Ofc4jCn8^=?41dT z4%dn1+s}G_9me!`aJUo6sW9Mv zN%bcLzKj&ze*B0Nklw_s>Xnd(OkjAPkHLdL)IYV=nE)0BdP8rV5p)Q1p^P2 z$BSEUXZ--p2-%S8vETcRS)>?LbzohYZ7O*33U1<}rtYM^Xd#mLdzCJ|qigmZ&-7DS zM(K$?5&Yr*Z`tQXPH_kvJ2b^oYLNfc_k&!CUJmD^;}~uLHXo6#qCSl}>)UT-oiStM z)02ZCh95DOa3lax9Ma%dBsr0#Q8B(1wlI!=!#acEqJO51GEj2q-AQ;e5WxK?X;427 zK!IAeXwp*?jEd4yz%Yqmq-AZa7FOIoeuqt~t4oTs_rs2hBcXLS^knGDs)crU>~-1^ z(FK9oEqGOx`<)wlIO;RjxTNF&=3OGX7ztyfp!X2>juIA<1 zyT_4rzowH!+{_%fryW!r+3!SW8<1I0w~gK54vf#lBUj%XmvDu%$BJJxN~e>496Mz9 z=(_i{k?C`CKl`sR%KdkD)+y0{As^wZtVHb%{x+sGmJ@=mgPj@p)~@n#(dK^lrhH?|b24YL}gXaNQlJbwmB=p$ePunr`CO78Vdil0<_U>lt~_7 z-Qv(6kmCaYAn{3W3*nbdNBNg$z-@q;N<-~uG*-fxM3n~>sky4;wA|_KzJi$;!)bOG z+uyy6SstnE^wicIQLL$OOLX?COl!}z4QErNP-Mv6dXT1O9`U1p~SxZ40H8=gx0Mh}e+8 zhbhCTVovks`WdyoL7s(9v|0p)jKW}$nZjYJhGAtW?{2btOyE>Yz$roiaopgDpxH}O10zC*U9H@jw}+I zu34rwyIev-x&dp09zOU0uvfUfyEmNx#q@H+;g$h$L}Al}TCT)xr-`bh8R!whn2O|z zCJ~=Kx%8`bfd#1T& zVvKRR?qnVG&TCaqg>QM(aW-R_8;*L&#)URLX% z@hQd#onq<=djVR=!5C;<*o^0V82zYK!=?Z^l^e~U&u6E1+2xsxBIfSMamp>_YQOEz z%H669r7(SYvabT;`E(s?U_Vn*QkdVh`FyLG(@wsDvV>`jmf8)Qi0OJer$1!+RR#>> z0Wi*RfSxD$jw(on9{oDnaf`y#JGjsVMpuz%!+&6-V33i?Rm6_zBWw4jGZ=e(2cNeQ zh-=|tf!S)&17T}hTk8ArG2{g>!J1WKBBe$vJ;VPKxHVpR`Se%i$%!Y7k-x*Jn>y4= z+vNGHKQ}R1?Mq)C-%0>XV&{~@MxFQ6`I`lpi3&dW<)y5mBJjpJz9u#u@GA@jHrVQ? zqS{TLE68JR!YYK;#$`^Wjw=V$8Fs!T*DkLL#`dO-^H>E&{o5x?5Q}o;x@7Bh+&<ovT&)&}7bis42F6h;R z1-oJ-ocs}(G15p*B(3B36DH+ncDg87lEu9mqkHgOQ_<09o7?~&{ftvI93&MVuldeO zS2zTCUYidC2P!v*`HuVFK|q%{$8n+*KO#Zdd{0W~blNVj^%4K?LfVOzQAv$=qKW99-UlnL1r6GE6 zNiLt;^4XF`5132{YCCq!&rG-871`=KrW_q=qnpslY~5=t2@ILSwvG==?jak6&y z6MjW!uoE#X&Fx7ynx{Qrl0-IrxU%kcrgRkdgDet@ZdDAl0G;@IFJVn?##98XEHwI-4*DlLR4*3X~oAIul_06wAnRBelPeTIXSyV=O}Adr`^XzG>)6&9>?PHvh`EsnUWu@+B6KB z{~^Wf8wtwe^13f{Vt<9SJlnKv^+#E(`UV!{eSV>_T0emP-EHlHo-$miSg11cuj=o652SnJ(c2x!t|+H}y-L-_%PDs+Ki7V(4&)E4sCuw*aJ z!S4QW)qPNCGX1eucB=3EgCkSN>AK?OKj_A2*~XfEAT%5aWLIx#88_HVO9uR4)|)ZE zxB~Vm!Djv&Eah%$AVQsG3Vz31RV>e#M#PJIGlKMwN6Q2+3_V!17T=rg@ z6RN*l@t5bB#||rwLVe-JBSu<}lJ^$_NWEYa;W)7qPHZHL>CXQCVeO1TJMPBrm>`Jz z7}2QGnEELb89f0yn%jN&5e`~g)Q^e!((}l|${q9vYgk61bhKCWgx6A6XQrN0mbfR$z1ksYJ54v1+|cx+ZG*vQTk1W!xBl^G zPU+96j_tIQMWt7sm>J_m|HkRsMD&(BzG`WuxeJoz50tEn^$L!!V1y7n2uSTZjy5G!$vhukef9ZKLaDEi>xXHA-K5 zGD9|jbl$x1_*9`)g}1f$x^PsZhs<7A;=!qyc_-GVU)-~GHnBQs@6X^)$9}&_we+@8 zC*d~{%blE?Z#9bK~7Ie_3uI!KeqM?Yn7DP*QRQ@VA zaR2rzjp~s%C1-ZIY)`)U`!I|hI_h8MCn8(;{LVAw^M|FQhL%ZU1omFDy!-{a)zr4} z9+8VB|Ee9Q>D_Jfj=wF||M_zq?zr?;af_tp!G|^;j;kUl>x%s$;f`!7xro4}#{Qug z%SAKNt9gbYS1(w-y^=j?>2uuD$}#c$?vCH}3o6)upckZ8&6RpEbA8t2_s?m!8onS9 z7%RLmNa+2YDSU9z$25_ywed23)2h=|?)XERM;cozu>lGMH3RgEsBU1jKX#gkjD4I% z|60^uTXhG9ylo=NCiX&L|5PA#?rCRzhF3ei@Jr^$>RV0dLDYGI#Vc0%O$<-)<-X)T z4q(>$ZLN=-#{YUs+V{Hm;hzJ#U!d39fTg4FP(etuJ}h;nF$_pQ$+Gb`xh8Hb*osjF zZw2qj;LG%15LAFv=-M~b+YSy}1g9pjIfn-(NOJ1RJAqne&g@tDo2C-yhAuqiBa$C; zau)FxF@ZfY)xaH0=YbJ2kj8KuJ$C;2hUWVwCpoPWdx6qXs)prL;Pq9ububNNwQF?c zCL>qg;TCK7vmF|7p8@;4j~GTOX@Uv(xsOjXi}O&dSmXD?>nI;ZkO28p!?NI{+TJUB zQrsQc?3@;*o}#y9V~i*B-?_uXd8aMUy7O3Rk?yYe4EycGe-9*aA3t`J!Y27t#WqB{ zM9poxerk&AMe{E`z%T9eR!E>#C6d?;*5+G=?lLR`RaLzPZmLeM%OcKGs1u1C(|VmU z4fNC!-z$tbjT2NG=?=Yio*K4MLz`(#PcI#@X3jB>#W4#%+!2m?+}LY7a_A1JV3>8Y z??o%7pF2JZ9G?plNzv6O^A8ezQ`Q{rYlj1^)d151UG`Ox{-K9;PX`i{lB7X`&F@R% zelBlEM9*^JrfoClmj7#-PGY7K6T7ru3H6?s3^%OhXtrEf5^#_f=CZiJ<2lZci&vuN+CJ+uYG>&&mG-! zpPjh$s&4dloqP`e;tAU7`I!WX7auV~O;3ly(!rhX{)EM^VRH;C@1G21KYKk_rLSUx zVPCTGj|J-|>-d3d3g!Zp|dw3iE2C?2$0%J%OYy7aLt%17T6)XM?A6W0K&RnJW@%LCL-3Ev?A5i zXC1k*!{+xiI016xww&u0fH**VjMwwTlRXHkQj%nthp0pS(7TO}sWt*!931cN$xe2l zOMTyDYIJwJ$`zdN;Zu0ZCl^8*^`66#w*D!~9-+xULhD7a@#kXsF`WRssa5Px#l ze}V3G)wI*44hI@`=;<+r8~KG1*^mKrVU#8$XUp!cLpM!kYm9ADS5+8;O-xu@D=b{= zSIQ?|LI?XNROr_2iw*o)8XM|*b@s>DB{QsC!Wb&4L5JVH@{=Sc#i9l=lK3Ibmw4gt z={)OaBw>GC`gRH16lotYREI_1+CLpSenN%;%=&>C4d<7o^}pSJCqprXRf=mrEwNYW zA25f76eaExmG@en?NKe)!9ArfPQtwGXd1@hmk$DzZ{L1bFhArQ&~^TDwmLm74ln-h zfmTyP;#_CsC>S}Qy5so3SyZBZa7F2KtIDAXwTE%d_+pC-sy%^GS4M-6uJ#30a1x%` z=~bC;c1!SjF04&D&t)F5PBZdFVn{wN8Hl?HcLm>1&wE3!3kDWiRdO5gOG`>f-Dt(m z&tviElqG7DyzRhD?tTUmq^mw@#}CwcqCWYxK&x2U3&P{qZ-4R?e)1f5aYvcLVZ;5= zNRtJU2T=$pFsI_`>ys4=JzkCXcq*0OQn~cwK_)<2m6`N>VvTHNyi#9ee~_dpFy9qb zzwyW10&>WcRnkCPN@{HAPS8=#)4@(9?>JUQ^8I}ZMce>(KcOvk!Rx1utj7hRjwE5K zMi6hov49$pjr#GYc^gAApS9O#Lu{*2zpR1o+ znLX-Ng7d`UukdwGmG!v>7(RRW$)))=FLV7&yio#?VQ)Q)TGjc(7=M95Jl?@-?@&$i z_bpqTEg)fwxbG)5{AMWai=?%kYUR*){YO1fQOgIBp_V6&Br2O3rXOoQ{T>*&gTD*` znB}c%yv9x3dqHv2MxvjaF?va0vPiik{De>$U6uOtD;+<3ee*8{y(q^-K_%F3m;3YZegY~`VPsoD=n3z`>nq`Q6#r5cAw!-*0dd%RA4Syh8 z>@lb<5+BPB|H4ScmY*9=ik;IJ_mhqgiS-R-G!bQpq9?!c$X^$Wf1`CCCTTWW%=ZduyO+RJ$ef0<; z8h;@_{kWb=eTmou+glr5I9W9PY8oOfvu)|=;=a|^spz+(fJpOYja~+}7T?K_YuSn) z86`nrW6HFE?}w{JRfo2h7WHi1J@_oP^%E*VX(lD2llWO z@{739_fNMs0VV*dLCg$xCoP>WA-9ooPO4=`h=-K*wyoM4rl~c$*(iqgyr-C58o>7D zKevL;+^s73WosJgFJi$*hhBi$VkN#1&Cd+>gGZe5)QEo?{LXlkEw$#pj*}Xz$sL1& z-uSQwCj^^R&l>elr{zvKC-{AEqmtxug8lvIzn()y!s{=MkN9+A-Q3$T4Ux`RL zSR64gibz)@3!UYLZlf<7*Cl`HKl@(o=yuyd>KAQ4 z%wTrbwiLq!sxA%nk+(gpeC^yf+P{}U!*ugmSt(o;V?(_@-)6)<9Kyh#vY8w$^{Qu2 ztS#u&-$5eK1%w+M?`B!y-JJUoxlg}v8>iJFH~olYkJDn{mh3T}1owc=lOIi=Z-wG9 z{&Pnx$Sv%u%Wc$YJ!cP!N#4*eb?ke8A4jIZ;!;BQj@xTDOkWK@d}_gYWMVLx;1soC zB7#>r0@>T`|7YjnnF?kFMFS(UjGhas?D{Zs^5fg5am!p~*^DmjC*n?zL5hh7+W5X& z4+t;5b`WaA3ye5_($N3M)pvknz5nm$vG>Y~tc(=0GP6k;QARQ%**h!QkG(S@L{^Ao zlRe6w*?Shs$ck+L*HfMIJ^$Zzxm@SUIj6_x^M1d^{kre_<(2Jg6P0JP9hfAtd3(W- z?e)Guy_;es%)gdQ!(D}2%>@F3FQKaZm2(=-sz)# zxxawL`f{J*DwB?t?=)^MrJ_!Slf-pnGhXJX1r><7!|p}!jK}Mh$Mp-x>Cat&;T8!A z@z&r;sl|nZx9e-9^uM3|>PDf+g$$J6NiI6E#(TtH?CNas}4BvKLw zk<-d3o!}UG(WIWh%dfI8Y!SLNi|CJRiF%9Q&3Yc~-35kBF{TFluRQCmfYc_3i>KjF z9GKQjr!M0oLZldTM)`FEsU^K?Fnk_mEU1lscD+d^~@R~duAC#J_cd*1xYP+e&TK2sbgG{ zW4!Pyd7{BG-N%#Mg#9g^l(pBIFFshC-`;h)p{s-QGjkW1Bc)}up?rR#W;5qaS&4Fr ztVX#xUg_Ty4j{dBQ~Ow|1vH?8$qvld;E~1=0&Zxd9DTMAU}So9MVJTl7BLAYSlhpR z^>NykFJE;@kWH)aHt}sKt$FBI{@erZy*KXucl4YWbjia62`s*-@e@pMB4SuNH5Z51w)}QH3Fqng zZwdyD(qqQPPwOTGa7l4|pekm-h4mfY(fa|2;ilUzSxo4EfdZ1xxOB%i_b&?0Yg!Y zO5n;y%N^ZNl^+cdvfmskjlQaZJ={Ov?)8Sva44rr{AN|;;@H(<9hMNcQD^l=6_Erl ziJSdMF1HxI7R;uSln_=KS&9!LnYc@P8?$!370tsv!_a#S-7|$TLh}U=OQ5MDS&T~M zj{UR=EgABz`r5PuzWHe3R z>Xhg4Qr6%c9T};+xVZu+xT9lT7J?2SQEq>;yc{aOOg74n+&clt3F+3PwVnL znF#^XQ}xkTiAt6D@b$;h5VP-ZUjS5kY?L!>0J&qvwek%ubzZqYqe=KdiK@@Ef!G6t z$*)G>1L-$LYB$Zn_V64Ow0JKO^ajx@5^?xN$-f@WxPBq*D%EgAH}KfPWMyY3*ZrzV z{2{R`z`?LbkNyeCEn`Ig!#>k^kope2i0#-m*e})`iiV;n^j4K{H}{H~z~?;M=^S#-#B4Tme`KZ#i9EZt^8j ztN`3~ht5#dSeZyu0V}=N{-e@h6bsi977elRvw!5h>F6FDyNAw|^FxH+yXHu5dQ9jg zF>9|xPj{1tH%tfb?ilRB;QCb<^^?<*1%~V;`=4)lu3F;3C?Xvy)_zdRD-SsK!z_n} zPs`PiiYouTl?y{7rUoWasmy1XA{IV)s-~~~mXZP38K&cmOH~H2GN#!oO664|dg{+n zblqoj45JWqS$E6o9=2L-h#ZJ?E8^z4RJxsXk3OENd?Em9S6K~co8UBkqaQ^I#WE4t0CRec1HKcsvq_1O|l%_9p5oE{q6k#|+}3y@)%NZQMIgFLMz?yiiG zT*<`w+p4-oHGWT#Q&`qvC!!+PVvG6*GpEKWTI>fh*<>;6jFO=e9#m8VL>-$u92JH= zv^lkHjkGaKjRkrK50dn2;Fpu^N#K>e+Uw2-{vAv;W`lq`4({;)V2~cSXjZi(``FD} z&Ghr_xr-RV8FjCFx`A(p?KluTi<*?`MSNS;jeB`TQ<37C*m|nU?NgD3w5)@2=cutS zQn#EC3oN1xg@oT(>+?^HKJi)K@)t}WvI9xq@xq(@k$RpF8x3G$7jX==!+y*6hf1?I zUNg&wQZKr4%gFXuB=d~2#6h>80Io8f^VFGD;>{u3B6%F8gYZ(PXy;G%T4-U{NUx$P-MP9C7E#I_IfOB55EO3 zGs(V(d3fFpCLh)DeU7Bu*vB&S(pNvNDw;DYEz(}Md=t00%O#it-SP(RG%T=CI^Xa)a5#*511gW3vcaw|&z;?Qoq6m+*^tw65<=qRv+w@6 zJYamJX{IiS$MG+|WY1jW`F7ea!WpgJlqvEN;?7H&XMd`;VtFY##LfhE}%(}q-Ype)x#>~ur7!-%T|9WKb4uC6M@$rNlFEfwI$u>X?t0hxU`yk|7pn}I^GeovZH`lBZ} z+KJht9pEh4n@P3I#p^JG#ri)p+@q_5`9z{r9|}W^-YOf zrKmcowZC!(5;~ow7xoBj6RgU_;D82GBSQkC%uqn8R= zS;~Q$o!qvz6|ES6wF5mEqOWxD&R1UCVzn?X#XSYBt}_42zU*H$Pd9A$v&ddm3>6wk zk|jN75%h`>MKZ(7Oxvqt>-+A0XSt_6j8$IcVr$(buRmh z-aIAv+}6J81yfX({DfP({_;3(V1>p?!|oqLdQrkdJ2q@^v$=jA_(nMqUA>$aezjP= zmb~wu5s)Q*4G-QhP78(Z8pv%#aC=}Dgg48-#wJg^=HkCP?fNC6x_wJrFEw-Fx!}T4 zHyxA;svz2aCrDp9NhUbkHmm>fN^n)>`TL{3le<^02ZFi~G#nD-0eo4$xwHYe>yoUb zWPffcbu+e>_f2`deP5z;;xTb-U{>yKQ~zV+wU+QGIhDn~68aV%3p-;5@I0gF3+zb!|Y%kDC@z4Vg>Jze}`Y{@fZ zpY;9iN37`cY%;N1-cW4TWpk3;vS0=ipfPOnt6blZqeq~?O5>o$xk^&4?dA>2yE5mW zVE1}UQ4JjIM$ZQV9>d6Szv!$I#*xqcj|ehT-WAp-!M3G4$C37tQg;eN4t4_i+|&(O z&p(>DQ$ws`#MSBD3`Ks#zQkgXN_wEZXJm;3W?><)XkzSBWjy?s}g!|V(r!>EVlf5*TAfo<(Xc`RVy@zQojfJ#~7 z5l+!|tv&6ka4NKMVoR3*Ljp43Z6D%BoTpVRcjT>(h-iiIqB_-NPRwQmyzumpVH=v4HS%)6e5M&M3~KgS+jt`Nvu1&vs00KD}(P zcf!QT{KObt=l;v`7muzuNV~Cc?_WzQ$E>mZ7#BLiaAsc*G|uwf$M<_VA1`8Sna^Ek zSKd+w--lb3VLVj0)T$~f!-tbqj_jc6lV9bJLSXQNPibyzIrQvFB^r%^(WK4@;7a#% zUP}-&4a2u4HG5UG-Si9lYVFppiIQOcH#WVTgs_^>pdH5-amFwdD|~?ch?Qwx@mA^O z=1y5#8R3z4uTIqjkvP^$%o7sYucK12!c?%LktD%BXF}Df4*l%+yy&-12Cc8%E%1n; z{s-~fWI=WdWHekM&Fa9G@VCiS@U;2nsmd3!bGwvdPB>L&EJ)I7gmXbw^W(3YJ!_eD zpHzGY2?VOnYsN$Ky4l(%^MEugb#%M*VMT&&x(wz5ASgsd0er+-KT)&ptjim*MMmv>zRl4b8miD zkGbV{$H~KVr=G1zZJc0v=a3{-?{OeNm}zf8v-KQzCLG@k=NI)S5y7&>*NtFemZfNI z8_adl|56cx@KwcAK0Ahd2Cag*`k>SSDBn zk*A4OUy;w0S98JW!=-tG`C~eh#fw@W=kL3~bG33gRM02cq(fs+p>(9b`fyo=A`Abi ze8*rgckQsFFwdjEB~1~D@Xz;}(a3L{DH(-aCQRp$!mOz6V+&Mvu7Hioln8dH`(AGo zt}CJz#9`q=R%W&T@If5&;7oXF|S_v(Fb>^z0~Nv*J07)fR^ z#_$-ALM2@{(*&Z`h*WP;{Qh^57>p#h5|MEY3mhnwILe)6lNqgIDg#SG`93(EY365* zPxIHveZW^UCAq1$-Uym!$Q2J6KST9lk%S^MK}&O+k435xA7c$ZKaX8Ec*!va;TnI= zi6^By2>3RH6-ND!_s`~eB>B~l$PT==iDOTJR&_cu!;-b$QOTg=gSEMExyh*4(RHS%yYW1VQGP?7k&7z=Vwr+z zskpV)S*)r=fA@+z9v0gwq`>D4AG9y$LVvtmbq=L{I*+5L&=4wlTQKysn%O>B2ZY-cg+->_ORr;ZbqZb;-7k!KYNl&_=sWXwv!OuKdkI zm|yTNEZu^j*3;1(No6PBe2vsYGd5;lpZIUXB*i$7OF$UgMSzz(Y%ew5`^*BM zK6kN^zlhX>S!MJm@cy*fN!nX5rhYUZ#(mnCl45IYkOUJCE?lJ-mzlD5o8gjwhlDT^ z+bX8}7^IJOwAC`MgoqrXvL*>${PdZ~@4k&9h`cb| z_-*3%Flk9ejU)_FVi2Gy-pbkfXgzg1H3R= z2joy*KP-4?S|#^owe*xdK$UCB?@uU&Plg)g8YM_EngqhxYTbOxD*$ghPO98;Sc8;; zOJ|7raTDZc%jg}nDJI!wX$FjlbDaFYeO5LJkf7frwQA4=`vNhW-lZzw)!l-AZ-)R~ z`8<`u**h(4mI5**B)$(q;V&Jn{dOX1ECVR-K&0 z+o}7uYA??6vl!8km8?aEO^B|s=_^dh;_mSpCMeqvb@*$zY{0n`j#lw4pXqf@y~@un z=3vj&f?&dF=0t)>MW+>Le!{hx{$MH-l|nWMj4oAF#d2?==JsWN*IyHlml)?n5V7p4 zdsE50kHE2MPV(%4cnklxZHz-yqj~>_{BHXdhKx(4!HQ))O^?#8{@#ROKIEka(ygay z7d-5F)0w43pWvVz8?_4bI8=gFFI8)rq9o>k>?o~Q+Q{aG|9&ZQiRWG=F89e@f5G`6 zWf;ZrK2L5fh)pTGZsk4h z{aeER890($!tk&$kDv}6tGoO0$KB9#xBq^;TlVl6$5b|j46Jm_UZT<9upp42Ul?qr;7WfB8rG(*1L{nP|JAB2NhG>p*(^R66m_G(`rS{iqpSX))|CYhN zMTEeWHH0ztPw=4{g#82GV1nZW``v$Y5p5>gy&V$SJ;2S@2Hgfa$ubz}tpDUG9EV=< zGt9tM>>UlWDH2^7O|Xid6KuFzQLKuBaH}6B&^SIaTk1}1of~J3U!e@+cUKI4OJOu^>iZQ{HFM*Mdj zRhpa+elqKt7U2Wayoj8(B!X5*Cb20EOeaZ%!5UC}tC_sniiKbOSlk5f>}dAv8~k=z zP&y~Qk*hTYV+$1&?2^6DXaWn3pg4R%v)1VUpu#q#jw#zPHRKUJR#F%*LVJ|gMOu#_ z2SD|oaE%6mJ!1}g8z=1O8)mPSH=3a2KFd<1biF)h-xck=2lA zdB=A9Kh$B=>u=n_s9+_vVc>kcIcWHZNXMA5l8Nz@=sWu<5zUq14)xHtFPyuf3Cb^Cie<-BHV{wX;u$oM-d)CU8Io3iKC#=?WrJ z2~bjc8~j>n-3&QgyzI-5bjMk~3dY37KB}H>#^~Q?Yowh=dEl68GJRGaBkKU-J%JE6 zjMv`4xEwX8MQHYaT_ak`dspDV>8A|wc9XoZnn-dm**co*=GOIT@Z$T;NYH$DGfOdK z`5_J^;$wN_n}(>adx(E(fq$7t!4gQ~4%M72+rik7y@>yr&PjbF9XlyVVV1p;l5xg5 zrN^7@&)PUbJw{xX*WO5*_Se050Q3nK;9NKa(>6NOLB%6E6b^N$ zY&4UG_GPLa3?OsgK^gC5GNY^djuO9IY_v6mdrhVh&E$Q0tO1L0-xHm}>>-#QEr0b} zaio^&T6GS1XE=dT(I6~j#+ApSnu9?Xf->|kqX$m#?=QJPpCZ#Ix5|#QvUV?i?D03? zt+C<2K>PdCW)395BYFcz-t_QU)h`}G1Jd%LEI7?9Xo9KTUaxzY!0v0P!P{G>$Pi5& za+^~N#eKE$5|dmAO-UQyfHVIpjP}Wlo+i;3nrCzz08+YSMC&VkyH=wfk?OgZ7+v_1 zzu{~X{YK?2!{}gLg!T1whzUtqq^~gwgP;)~0BM>{9Ld-PbX_BGKZwgVY<5OX^`ZC4 zh7~YAvG-7h7ux58-`qrzWb9ddoY#Y~6X{u^+WZ__U;LlYWfU!iRd@T@H%TM&<(`g| z@tbAd%y-9ai-nECaZ#ZSzDg7fH-h9WLVJBv2U;5D$E~Q#r5Qyr*pw5bRBvqgXYsp; zJOR@T-LIVOpmIlbun#@9G0$VL>~ z6U+tsms(CoP!bYA^@j#v~ zi?JG`s35b8wys42P1`j1#LA!a|9go(>^S=cnQ=xL>RhqYypV?5%YvEy9#do`TU6rB zGAsY-^AbDys$y$=jOa*hW_yqU!rP$&hdc{yz+hda~5m_|DpuRCMP(1REKaOCb7x0y8BzdrplJ~$(tA1D~Vxu5)?b< z@?7|vn*TiMO#!q=DUXWxGZ(4Q(o-;ujz}`_19FXKpAnzGC-G0nBT>Z|Tlh?*n-Z`Q zpc&}BU~z@|oe(Ylbx=1&Nc7P$Q$Do_dfMd@&YLZ z7atB_R=Ar72FfDyMq6cQ&-1UFd92^W5gmqKn~u+QTP ze{g@<;Zr%smLs~##Qm38Gk>1`zu%4g&*_>$5iu;+p{~I(@V6`>NyQ-YRCrN6Bf95A zRv!l$eGeD4I0YqqCoBUA;a>y(Vi;#D#p{OQSx-NLV-yd26b+>jNNikG5p{ad_E0D? zE)wkf*QKu{@ZmHIL73XF_oDV3K^d8>uBrV7X8!#wQj7=;f9Jqw8RN5DDs~@Mgta@R z{uD^(5L;4)K@o62y-e`sw|G3l`zJB9L*_sXTqcEbZ{x|JNTPTTH}k=Ykkv>xvm;p% zhlgf*#;s^(r-Q|pVw$q_IIh0u3ZI9E6Ljgp3{@>dy1G4%6N+t4=4$LjDBM_M)*E~izTPc=1rRdJ=d=`Xz0jg zoU!-Bft!FQ8%4POux<1w4>&jCoS}ES`x6)PhV9j~8-HW;zaLO8$wxR*bWk#gGQ{hG zlIJ?f2#un1)W2q^Qd1 z8BUo5l(Gxb7vN;EV8O|1u8d5OAAzEc$!bas|8Hpj=kNN6(TqEs`;l`8Pf2JJOGl36 z@AH%DCg04#e@Ad_jL@;TkWi(#%ZC8ND54QmKO7<=<9zvRSNJQPd5dkQMMXXKiv);V zu7Pxv{5Am7uF;KvfOCLAmTwmbcumZHs9#C~Vn|z?^T*4VWWn4hdOlR^|Ky{OvW=~x zemse8n>Cwdky`(JKpl)duUk%TvE?5KR?H}55QyU2hCmRjKu68ttrzpay)m)Oarus> zvMm??SXBawBTT_@uX#WagK_RU!euoRSeOO+Rjv`VOZbRl<#Vf|@r6XKlont$7cB6o z?9bripXI$8JdDNAR8_%{s!)^wXXffP1LFXhvn-vXAaNeRp>@4Fi+Obxa zdu3=SK->p}#ZZ5F6I1G!h$5qEc~^h%_u_u2bS+?Mi}U2E>6oKd*)# zh>!vcn*1V)JPxPC^>rg`1F#r)a~mrNI-1`#`R@-X7vv*cd>HE;>0((PG4usCEQbBK z)TysccIh9}NnUxzb)Npn(f6|3gDhJYKXfOd;o>xNBZ0!#H3xHDCj%x(qkk}|5T)c&uNy>u@NYnvSfCILS+$Ls_P-efA9A%vRRp9DaA~l5(k&c4{@M zn7oJ5Y5oor?W71Z7t(cllhNIv`jN5@Zgf5pFv$RgND09;aq$%G2aIae*DcShB=9h) z34&q0qcE6FLzfgeFfh<vrez7m;ewzY#CRph;y zwuX{Eoq5KN@@<+p0v1oi4RNF872lmPf)5wQLO-G0zt4_$h%ZWdC5t}&kxwZNb!2EG zDo5-f!*Ed;4`Tr;z6>vZG`u7Q@rBeUlu8V5|-?!^1c&r&8 z3Dog>u=Iv26deGFEC?(X@x)_E9|5_ufa}y+7D#ka5jaHTm@Uz!Z&?n`Qth~&}Tod^8`+qfj%Y7&!yGpO}=E%zqJ zsGavpo@}AkkWAXvykDz7`r)t;!rtkFhJInfr*}B;+-%FlKApMzI=+3u(5aamKDgUS zB8Pal>xHDbU%t?znOLYhlAmEHWFpFx9qtBNCIW|1jRLK!z36Mme!BR9^a{^|^9E9c zh}}z@5q2K3C>4R^7n!1Az-`F!rEn;_+6^Dj*6)=U9t?|Z)Rrp49j*+0GiY%Zms@7z z1ujip_Ztp}l{LS4!h+VHesuKDzDBFnVG12D2{qsjkB6@pzJIU64b1N`zAGf=EJ!H) zZj7Y7ub5DTH+kV*4gw8hMb>AhT0(+d=fOiXUz*OB%O`V^W-bLE~mKoTa$#e`9!f?RrPL(tu5?kYu)SLo? zuY?^ciphR(O<9Bu5u4t*IN2|{o{>;8M+L!IxDlm|iAFJ}H*WovP)q z^4KJLeBPh1*!eB^fntzIrA;CDe}3P%zVk(4>+7+`yZpXq3Ixz)lU1T}g@hcEw@iF&M&@iZy*&;aS? zJL!f)F?HSLt2Q+({C*s24tO^XzQ#HL!M6)c!(>;XLACXAtAOp%>Zt8j@e@7wDlpqB zN`E@!Bs+wTNgV$CPCoHEaRCDymM^3FcuxK5vdW7u%&&IUKWgyND%sS}e+@LL2^7Y-U{#MdMHD>6WH2xPfQ%Hn{O#4~f8X^?v%q zV&sr{BK7ZUa@8Iz=R)f^&F5i@LOhH+4nOrx%o2bY{nZ%n(GluTxnZr%3)sLiri!%l zHkfi-eNX=3370S|5MBh+GV-INOmXF1;Cple9%=%f)6v9Un|>mba}x~CI72m!AP-D+ z+d!-x+VwzxG+Rx${BW+gjXrkvV}{-`F97_(*Z# zjx|V=-FDyTxYyIR*`s60W+R>U0)Yw3}=4fyF*IoW|?NE0wf}Nyy zdD!(NQ(>B<=v-jz9nsqvNCJcXk5tEV7+tL-T!N-E@>DJSrxV&w&trsA%++Y;(@kD~ z1(&+zQCo38^y^L*^}J{RDm z|32=2z9IEsa{hI|V%tP8ys_)rYUL-S!)Iyij%(frT~aC@+d%4#)T}=}Tm|poH`?M9 zIA1R{KILU#9g&vucMPzDfv7;P)GJp$v5{Y+rwR<=?|&5O6${=)qGIajN`w_vnDUuR zt@rC%X7$D8EzF@m(e`)9MSXK(Et0rsb>FI_HWvehLREndU)J+Z*1hdjgEJk$F+)E! zPp8(_52en~#gXfvoV3RQU6*z4KF?hQCbgYo#gFl4i<)&cR!YCtXfF^)PG8T52lL6R z%!vREqnhMadFMU5WDm`tS2#IF+V8gr`FG8-Q@kcx=(WR9pV3DO7h`QU`TEX+*!I`s zAu85m?3jI>|NAk8F>p-n)6`fUKsWiFvH4-}fwTXHz7CP-3Ff`WUkhh2stoRJ7pq+}Gq_Z;XeNKSL7HcZg$h;YkTH0i zUW#AeTj|#vt6P?ZJ94EF{r>Qv`3mlh# z{N8FDoY|HYFI-AaZ4%2ywpZ57pkXH%9_|V!bSynPJ?esnux9!zYF1VOlAeq`M_}-T zT3vpf;X*kU?8xfV!~(CswDDl#p#N~ChF-exsL<=SabKIfKo>RU-_MC(X4Uv}h~N3c z>c^Try6NBb(a0*%qMOsiN3Li4vaGwQx31)ei*;x`5GdjEWq6o^y=joKC}#`?U$cweC5m#lf*~B$@ua)N%MRnBeUYSZrO__h8fCwXXD}Zuz`1U>5&OS5fWt^a z!U*g7nv8=QLApVeDZKbX-N^Yi1(3pj+9}_BIr*9VJO&==+wzSJOkO>38+8C{f_CtB zB&%`wmLAmVO9L1$Bk(x$yluhOoVn~hru0}n7h{H2C(PJstU7~9FBv(vXwbn108p?! z-IOb$&BO;ioEGu*68>54dh@L32W8KJ(YNW+#oVp0~eiW2M#~cNngi5%(!(`*~7ZuCnM@1K1yjL^8k1orr6f54Ej>e}|5LzCi?h&bBTzeTT$# z4vF2oQ~P-UoJES$UL_Q;SK9>XWAba~J*(lh{&aV273hjY<(ZVGE9nvCa~h$78ap-A zHfJy*SbUoqiHFHI1q3c5p~bccT6Q3w-*<_H-GK+6RE)2RviCx*SDep^N`;wWUo#m_ z%<~H-DEbYt@V{QxEcPL(L6d}MHpZ>=EP1Z?`Hk&ck_pWSMLf?G*M*QK{Sd+$^=}e< ztEo>~Y0^F8jI!f<=zD}Sz2y*DyhwYL3+2zit0Rz$SI8-3fzUB`LX=nn-Un(DtJ zxsTLcL@D?1JZpBdQ29LT&9KMP9-5XH45WgZ6x-Y^J}ImHH>otTl6Xno9%bbB3(E~z z<=k~IqjP9y>d#BwIfIJuahLn<3ai*!l-Tf5&s(${=?6wL7=2XRpX|a2?;`*yHEBK~Y7t1<>+z zYP8J&>5F5@^Z~W6v}`$bx8cbch!9FWh2g4~3}z*aIdpNg6mU@z#BjtPr0ad^AG`i` z%&HTM);h`}<7_6yGr&b7BT+~6zT!~Un62gf1g*7(cuH533x9-^I&OX}{+l4WJkJ@4 zp$(eZb!yv;*`j|-=<_T^#xd=*k}ftG_{~*~<$TWn=OSD&L~HQVNj{IkFZ>|s!vICF zJqSOjFcv0)rl~LT7)te?{B97qu8opUr2Cy0R>!wF9&I>frP6o0i{!d?vLd!#rvAW> zMsGG#(UZzg$8o7U8d@MOT@xhE?0&0I$&TP7A;Gi#Hp#%17*kloNds47*OE73KJ;qA ztreM$l^$ko&M#XTsNghnv&kS$QD++S*v1t=(5KzL6Qn5jyc+K#WlCrq^r~&P?X?cGCJS$Pywcvtwk3rlw z+WpB5qj^N#|j zvFoOH+apFmDxv!taIP_lHH)8miw$Qdh2ac^x!z(*03n*d%&nt)q}ly_8SlQFw@TxS zbkgBt?wd7^AT0?q`P6>Ciw#v7n402yr3!S=4?Qs8ByR$X_L2Jn{X;0F|B4M47Eizh zZ9o2Ur-Bz@Ia&pE{ijd37332X()KU}T7C&%v|XSoj^{cklu$YPq@m?Vq$$fdY`9T?sEpzzi7FhoqsVg~Z&xIIc%DYTz3f5DD#v=u~r?9M;;!f2*EkgwpHVc){*kH<<2m zvp*!i_VX**$%8jpA(-DpQ6c;9Tj)UWPsY6b9&^00v@}JCPb=LRm*)Y+1UdfA_MEa0 zACv?rDs9YizpO8*6^%s zNj*DWhUoF39b{nVb%znGks?dfRbcdVOq*X!QI;`f<8+ZfRkQ}zb;2TNvDEUxXz?Sq zPR%O^`r&&`wzd;$^pxiuLCV$%(O&i)SdnT?Oiai)r=I@IIoZ&8%lC*a;CN;X%}V-u{Xvgir>fU>teU>ll7i%6w8w_~Y?qc@aSJ6Mh&k3g>=*D2_I;;- zjNX*DR`#xMJRlNR?96NTok7QQ@8ieC)?EE1P*!&-NNkf9Tiwha;}Kvu{2atrTD#1l z7U9@ZkbAZxd3FiZDfVyUU}H&$2M{v)M!57U)lN6gPE`x)_H-UpJ6TlBu&rI#g9b}W z5VDk7UC)c739CeU?uKKBXOQ=d!Bw;Pi*5knMaQv6K^>B(TTm;#e})gKwwrg-G2@iI=_~*nLu2%cO~Q``Mwz|2NfSG)}NGU-P_2)fV_tJ2TOz zcOYD<7&GL^NLn4&+-<7tC6UuJH6_Eq+yCukg7J(46N=mbJQ7v8aR#r7eZ;+HKEdu8 zgxHfDz0B3K&7iFya<)6kmko{=s50LLF6n>b&?fOU(QMZQM2!{p@07@?k70i(e)S$U z?0&cbD{LHKlXLK@#gRh=?^wJ{&N}Wl6yo%`XT|bF(TggR2j#%%O)jCsko??Bs0Oc~ z!`ezUQNfSQ0h9lC4~ow2D|EttgxY7hIHaz+o9fUvBtP53JErB%ERic`Cx|^d3GJz0 zsYnhg;RCt7V(eLh2B(r=oo@>wXAO%liwCKgdKGG%G-obttwkUza;3AGE*3i-Lo#=m#|o7aRfABx{M&ib;ft6K0` zD$xFCl_R!>V7ZinOW<9;H#Qjr!#OPfX=ia585iiixGuh7N?>~1-#i#^JNa5GA|BPx z_g`7A&7%3h?wq-(o_hF#&grvszon%5CmOTl`B`&MsOyZ7*p%UOZbqZ`rgg7uzA+NY zT??5-Ej+FXomLYA%;or1eLUI%7~}4s(^v-p{H$HyI@XRpr!8V~K z%C`DgiW#4>^Mx#th#Z)7-Y8BZ?5Z#t*X6GJInceZR%2qHex6P3yW^sn$*Mu0 zpaYQOy|7}H1E%K~K-FyxY$`s}Zt4d`hRB9?Uj%1t7tGN)N0yWi;KD7L^^QYFz^{4g zE>chz-@gpomG**S+v%(U|9fHQeKdLcYtom$7eJ@a4f0ii;vK*Adg;QR0$_uNv~b8J z64>Ik>3Env(tVoPp4}EoaDn3AkQ&ukJNuo1B$=2(ImW`_nYq3xi9JrfcS*j=8KvNe z*QWWC{F@a2iLE`XMLZWb3falO#33(uTIQ;#r#?#`4KH@Hix_A}0OKq?$G~q)DfM;F zyU9Zq7i--*y0qs;-TdUDOFxBxNmY)JQt8patD+f-`+D*=U>n6R?1(kD7};9D_JJlb8~L$;o!E} zD6ND?WsVT?WLIF5N!M<65-eTYtLlEk7U|A5C!j#$d~(w53sV4|vXqOGsSX=;zuYc3 zkxpm@Rc`)Va<+C&w-Ewn&{h$_sV z8yYrMqlO&+xhE=&Jf&PSw`pH<-J>>TMhxTwqVlUDMgMHK(0hG$u2^8JKkZzN$WI6- zcYoCG^nR{L)Yvvlan^b(VZr96-aarzrSp^HLGk(JL_XNldeHye>qTNjotZ6>8D%X{ zFH4~83L&~4{orA!py^(VyDievi^`9y^jzY71Hv=?Jh3JMm~FE#Co+f}!%au03V)l4mc=T~MMyepo7PhR^SGq$zTG9SDe zB84zdVs_PslF6HZS7*NSU#f9mKN(|jWelLOgVIE7s;Bw zlIdOS7OaJRjfA+$FWwoypu5Cz;$6f^$8OXhwNA}|$5DA<@OPo7!p3t#%iJ|`cAT_c z$KEjb2;H`;mmeuLkJwX~C=Re|rTkw#h$Gfl~=*2%V637*!hD(^k)qCZT{|>!(KqrgGi=qJ2|Tc| z?JSwt)>w5J+mFY$%UaY8{m2KMD7fIG2R)*H!`wa=t_W8em)p7rAY#5jyE%4Migkq49ugxs6KA zPwEsApCVm~rfYAabgEm-_MeMD$ZSYVAs=RH@;Pm53;_k2nu@TUP`IAzC>F;W?k>nK zIJGHWRS0~&ulGd$rI4f_?afrq3U;?li*~=&n(XxsUH5x}>Ht8xi_H6z^Og2KRSIL^ z0;n!h>tH9#JkzmFqZK0$iflnPx6HA?n>biha-$oT64edRG3f*n&ztB;5% z-vGnFw3}Xo<@NR)zUB6sUOAos7MuCu^Cg+oK?Xg9)=k?xn)lu0j&4Ly> z+~4tj0daMfB+>;T(;{H`_tg;@MSoIWPO?x)})z0K&80& z>s8l&m$A=e@sX0+P&_CXzxYh8~*mQSI=_Eb0; z(QNfIcmF-p)GV08aA)>}%71)!J-L(JknvFegT%5Z1J~w!LCud&`Zl&+sH~DFwHfufR}mv>{{Iuz|zvA46*WR)vJ?Q5qSuZ8E2xMH^=T zqmfSR!mN#)mYbtY<~9G_BbmO%8^oV1p@;k%w6N%KzU4h$+NWbWZ-2WwcVt>H@cBnQ?%zRry@WPBK8sOX z@nUzBRdC4M6aK+|!$7^+9dpJ&@o3uzs?}GXi|=Wh#}!QML#5XV2KKU(uv?N%fFY#{ zpnF-N#t&mg_{G*lFFs;e&g+GE8LS=F0p_u^cFT#Hyz5P@Mz!PmMOZG10{YYSVe3|O z`hk&I#-4r(dkcNgI$U%F!lFv=W(P7RX0wtk?2L==i%fKimk#nKtlmZCx+L<1QT#iy#$JIu(&imX8;zz+=ma3&TT(a6m%o+y|-6$s2wVZb=2kr>C2NU z^L^5dXC-5144l)rreHmY98=?Jis}0$P+Fh0^qhz``f9>F5&l z5AzrX+IZ&gNwd>d8$-G3vNpwi)-XQ69j4$PoOY}NxR4Nm^Fozzm~DLq4f}qdp>lSfji=0Z`s_6wN2Vc2@!Tq9qDW9-qsU`-$F-MZ45y(BZyZQT?MGXJ~8*6>0bpsX~q-*R8 z8q41olF6*8c=fo+NEP$f>gY98yv+HLsd_wNf1%)Bs9^P;lD^*WGTACaC{$?NIM#L= z#}h~qGC?RhDGEPLuh!$=^DH;?Qy%x&=`jKRcVBnl3>PEQg?&R*AOz=jU+vFw+20s6 zstE|(fO60O?r6j1bg^@Y&#afO*!l{*h)t@d-FLhvxvHLuYnoTHD0?ZaWCh$iK%n73 zmNoc&Z2L`G@{fn^nX^5BTDL^%E6YFaTQrL75QN+M5G~^H;ao(l>Mi2Lw`hd(vk}>^ zSqHq|+?pkR@qu*h;_u)mh(RgI!MCooRpDBqg+6;n>Efl{C+rKfkYFAmQqTzZu&0Vo zO?T}XCRC~WlRwz> z^8RL!B%OX%+5xJTDm$fjU;7PR!C)#x0u2~goMVleUdLY;+VXaAO@sHwZBKPb6bkqN zhyH9t%g1~lWn7z8S0`SkD2~DJe?bqOA2Nw{k@V&)?C5f9hfVHr>dOEJ;&DlWDyHSM z?nxXx#96>|KHJ7dlIw6hPyF$^>lvM|Cfr**L@w{PUES^7l(FWR90k^ z9obu1A>*Wu5FwO(WbZ9|q>y>+y(*iijO_3G=)QA~`+1)0cU`|f?)y5OPM^>F{o4E0 z$Ep_Rdk`|oNpuiLd3%C3DmXlo3D z3d!ox*eU(d8NwV?G284OCgKZeKR`n+n-7RE$&L?$MIIjw4I(Y!QBOvG1IPK9>u84#mD|txSC0Cbt<4)$1D_n}+sQ6dsm9`rUb`d&^o{3_1F;_tm z!LUb_D~RyOw0GX*s`OkTF{dl^nzPnhAF2RWr@-8ZcpHFSCB);kVsm6@U6XF%GH%!# z)!taWX`|{P56z>Bm1k_rH#GeI?4chA4qZu4@G;Urv>`Wi;m9 zl)l6Kprw5`a*qr}v;U;gC27tIiRF?}nQ4fm=*pdUuM)r!cDCz7+Q=m~m37`-OdA42 z->;W}q8As-33I;E15o6^+5MwSm=~;XzAC&>cQbK^l)o<;bc8^URBP$DQE6-z26r3T zpf>9UNcY!(&?1Dl!*?48r1eG|UQc)`H@!6^)Gu&df za1xG3VA|W2u2=18<5F*gcc%e+TWcBx$EEUnN)1$QeW#2*==e9VPQrF^ML&#VItlxV z=VnAssG!&_!G#;SL@B0EivyxESub$pJDuVIb=LoaJ3|4cB*LJRofyynp|9S9(B*LD z8Nr=%f`T}kvN9}*u?JNOpCq&h$gcu5l#ERj^spwS6P0a!UEa$k{RG-p?pP#?zazzPaAy?YRFWZLTAh zR+s;G4XXRi3=B$*1Xk5xl8cZq4L+*O-Q0YCOFl?s#RZ`pEhMk2t7{6%b73PPpMfQF zNzVeCOzh()0-nJ_Z;;LnjF0U$ zHeI$lP%h*nb6o*_DKDA(LNn*21g3UE?k-l^|;kjT> zM`B57so9(X<#^2|UY%aqA=tLT<(+R^{w_I0QW5x?x3`;R4p>noLD>r425y}*ALEMv zaIJ5!xV+^HcfG9O6Rk^c)2AoX+mbQ)Z!t-R6Kr_!MYUIPHl>qUY1~^$T@kxexG+L| zoCLKXCOdNmY?4b5h2Dk%X$LoOCm^5Kq z(13Ro9awA?6$S}FWZsH-FTm3LpcJao3An3m_YX(HRtB3)+#*r2HPp;h15(4O{RpXxb%Un79-f;{Ghpvm;+=3WMr3g3RDY*N*nBTM+^99G)r7pbSprc^e3S! zrmeTHtVBewPx8-ynUh& z88Dtv2TZn8HkatAGM+ol%)GfDf;%*1VL2IQS!fh*)I`o;_@VansBN)|b3gJ4@Hx7D zn)xKYfcU|$uIP3cXt5IjlbDLN;pDqp&4DD54;rNNdd8iGOH~)?sHL=D89 z%L6{sw;O$ZRoX0hxcEkXVzu1xW7MCj9W@__t4iVl$-Sz3OLe$hsibcV=oEWyFeG>Mp5eWoKc~B%K$F^ygW84B zw$0?RK!ON#`2}`ZKA9itE;F6oNPCn1HJ7VQ$v*&p#R3&g%d0!^P<7aG2Z4mO^-LoN zFyD|YH(B-4ZX1Q$NCiUEZh29nqqcfv@)M%YOI&DD%5#tbq-*3=97_A|CV5%=K24`x zwT;JPeGE3~Jo!>9T6NuZTzPceJl|q+qrIARdrE4fPa^@3)O1Ox%^BYp`NMlXnfX)5 z28`?pOG!Lw4aG=PNV+W1v$9?+d{w`I>!ZV$P-*zCk=K{K!FI745GO$9&nTv}?Oi<*LJQ-_Cw*&!Zz-ofI~cza;9_`@3%1HJ4d1F?-49qvexvqWLZs z$~YN+9G}^^;ilm~{auWiNBv&Lc>6job<5dpxpY^3+wu>E3Yf@ z$Q}3MmiOE%u`5TE4Uk=HiX_|nb~#-(qOZGynqTA)ScnAX-o!=m4s?Yur`y&^&|BGb zrRs-RSzBpL>Rj4yh&6de95hS5nXFTE`I+b^AyMT4Int=ih9AEyXLQ@bjsw9?t|g8%Vxdf0) z?(5WexR?+4yq_kk)XOXXLU?(~_tf>O!qsnJn<`6;3adpyh}}AfagLKAyNP@(nZ-V~ zf|=QTsKkJ!7>-YLt_k*)?xYK-;@!e&y-vxc9Hg|6u$`S9=z9o{-k1bAHQA$}Fqn=^ zeT7#46Fi{+>zSpIdZM(~UdaJSc*HG(+4aLJbs#Sq$WKI591Uh*Pj(qe^*)5@O58gK z>Wb7F`JfyNl6DO^lRE?NKX&GVHe7gFo2W*hO+M~jlxDsyy=_|P!}addQ%}AZ#$asY zBaoP#6^&B{4vQtjTL&y`Z)tm9vP(B<(>_;toM&ayGx?Mal9KwT-)U3o$^F4xFi}Z< z9~AyoE~;+XdLBpN^PmCB@#}%mYYz{YLH+jmVhF6?X7~s?9$u>GQJH)EZ@>Z6eDWB{ z_~=6IX3b=R;D^%I*u7jZbh4bv6n)~D9(R30aIvO7AtnfRSl z?8)*_VPSfuUt8NHJMmX`#G4?1*30A>I6*Yv4XuK2vz9$Br3To25R* zL-y@7dWz#s4@SE)M*Cacet1sqCNM9DKP+1fAq{5nnC{S|-me8MVTBC4z#^hm(OR7x z>Q7=I9GL`>hyB$}U_H*A{!Yv+J=Rq=H4;Et_=UL(a@uv&PQ36}R(8+CgaB$-MVGuv zg&+v6`_DU`F|eBzybm@``&0vM)LU0**W71(F>W*_6W8eN5@@g~f}QP*JW|p6Km&#( zy0Z>(gVckLv$-`Mc~!{L+W7+u8RvPm!;$R9ES!6SmX(@=6JY(zXW*!RDuS;s4XQc= zRr>Ngji#pMg-;dw^Otn$;JC}DH-9q+_a5{*i;mgGMAi9jsnYRzeN(8ZD0Q;U&aGt; zS{^L4{$r#7f6<0E`dx$N>H+@*RMosya#-9YL!w9YDAcA%IKOt!Zl}0qNZ{4w5*5>1 zgV~=pDG%Mn$`lQ$Pu-;6djHj2rgj`lTjY}9mpESPrAh8)@wf-IhpCG)&4QhiV~Q>$ z+_S$NQoho9l5j_Y@4ipgs~h+QZa74Ibs`b?P9L}Q>%CbtMOTHQrMs_Zn-TL4@kJlwQ+ zUZJg>$BkEYQX)?;fO!c`wCaA)6?}<9Lk3&J!Up1~{v@i@q0`g!GrvFWjJ^&w7-w&p z`(?EWZnXNr{#&PQ+f9ir$qw%)y>v4(VvqPsS8L->rE+(8@0r)(-K#q6@FvM~J&fI_ z*IzkU64hs3=IJJj`0hg<4v)Uod{^)jnO0%EIx6Mg2lqAmg^wqsoh^Ov(97B%MCNp` z+dh!klr#5cu%dU5;-=D*tsPL^&@m z6{SGoS|dG__@)_!T|u)FxJ9z6mTNgS&-;*KyXT9>sY_;}@9Czm$Y1-{PzBK7gU~yV zsz))oeea`dN60XyyHwPCc#agd!Z`HQ&UqpG@FSb9RgE?=9aFM zJCh$ltw~(`>fYs%ikGcB)JR|W!;zg|@lubqvL`EgD_$rbsc3rgw{FGKE}n(itSdWv zNV>hzm3K*bcX9a;4L-xY4m5q_1hxe~0yg?-9_zH;_vDBQ-`J=a82IS-O&fW!IkDSS zA;g|jpeKiTA^fb!?vMUUXD&BG?z|8GEx!AM`qF8qFKpdQHx%|R*Spd}QBY)5g(dH*p~Gre53r6|MZI0(Uppqf@gHD4&;vtpJRiuK)!vv z4wa>Pb51JmhvD$XTPPwL2US!5k2GgP0OTkzW7d}UjM`;6L zK3>p_^BU!&dcK2Y9hF6(d^i%+fGpzj{pS)Jl*A(P94Q`TGe>$DbH#Chw6|L{vNN-XTYsX83? zqVe*caijTGdY5NVH(@F9XBcHmXDCsy83?1%LeAfK849}8o{dz3VwBII$&ULpqa!pU z0Tkqt(}5}D`W_3Zw*$d&p_WE6nPvSO91xNVMzYEhPwCX7zQ5^NgG?CoXo$U$SxWc1 zo>LqZZ>+kn=X@AeDE}u4qN!%IEu!*1r@_o-wwY}4b8Zh$Vz*C;lG84aH;(nO*>Utq zkv%jnI_WExm6y%J+vPnOS(RrJZ?9p&cBqD21`Uqdd_bE@r_Z~wrwr1Yg|8Gq0pC)w zYzaL~hzR6jm_vVcxslO2%#q>3(kL8pgyfoe?Rj2tF4-<>>dZ;8Bu03@t7tQG&e&a| z3ApO>ti|wE!20;*k{3tC$1p;IoVNl(I)Yt!P`ogo&m!B5FkSBp7kB+})Z5$rg4UcK*z-W&t*=Co4Tmxk82CHpFDhs%T%74N|da zL(rTdvOHqBe{2AxGK7*zMw+q(sXXbccE85oOk;uLmGz8$60TUU!&V3@m)H>B-`C>j z#HGqAX|wp%Hn$P$@G`3~@i;Wo-tqf;$)n0QwQPt~rS02;hhk-&A4*=} z&r~c0JMkeo;!*pYx>t1&TDkNsYY&{y{KB{|z!*p9J#)D2oA3nb*fE}>i}41*1Sneg zC+4?w4z@}Rx7QxOG) zdmNhb4jl9){j4^(wCv|Q=g>@A>2bSR<5OGOzvvBRL3Ke`kbykd<07^E3jZ@Lp++~4 zV6paY_VxY9qO(LO5^NH~`x!EP@MG$&qLGihV zPrd!hf|xFsCbihVUofKL`t%P;BOt$4Y23G+&p2qsHNJK)x*f7sJpDhsdk%t2CLFeq zPbwZZ5Rv&=HLask&eUAb++_kiq-%h^TOn#?NnqoWm*{Ep{$5GW4V-ni^g+{7#&?s+ zI)$9>yz+*U^51gr{{wSi1CKFm&~FC-gKvSp#gPR@ta8YY{V%~DkuJS9VpR3bg2oL~ zOUo%ie1pHVM{ijF^z2xpqO-HXDY`xA1l%9tH$$^_z;UGo?`Od|3);EDvcs~L5E%*u zm)${;x$l`M6J=3xilpXT=Wl!U*NKBzh3D5N=iC)|)zE6ckMMj5;n?MkezWNxXQ{w$ zJQN^DPjf6=G_TbcUD6gS{1!ft#39lHdkS5^5jp!rczo1L^4rIQ&KmoNF)anL@*eB1 z0%>5@8oA|T>@G_8eLc2fpT+N4T(1_MnuvxPCwbypqk#{_r}XlOd1KCZSO?4=QVIt) z1^Y>l9|R_Gy>o0o79N^+!M&H~dh}t$J5KyC;>dmNm>EltUv@CQ$nKFz&v-BIIoB}S z5DJ|tFq8vjx+0QrM_lIP--u@LUJjl1ju!L>DhoLYk@Wv7kWiFpio2d)p31+Iawk^W zOr)%>COk7qi4BYc`A2?`b0_(da2R6~SHL3*8g*D?vk#P&t|49?86#z(_ZL_@AD%N$ zmXecRCkvAvm7vuK08bT(tIw)n)3{Bp^E0B#CSfd#sI$i>d9d;xg67}n{&@qU8%Ob6 zyAjm=w8&5^O`nmOW$aLYOr;a8ztWb(!$lJgsMjNOa9SzaLmPOx zGaAI```CmWHJhtcG|=BF$MV{RHIbOOVV~k7d7n|`2gr_RZ?s3tq@|i{{!o(VIxU@d zP|GY=(;uD^c~-wyC2zR>^49JnH@B-5+QpajXAN)hqsc>*aym~GA^Mi})ICQvS#w!W zGCWaj1cu;}r)lV^c!s}GnXYi<^N*x2;vzou)qV3 z9KCBKr2-1O$5&od@MVE}?3Bc$v^>710fPw+!lK-TZi7BdZDz?=~5)>4&(SG z`Bc#!Wi3|-9ug+B_d?;6BB9ftD;Fq9PP~e{0xIuEm$Tsw^;1bP&8fL_{5JB8P~a}y zZVshj>ZO*v=zopnKK8zsRmMS{b6s9tl~ZhOqx_9Mm6XYg|KK);R$EH-+b2r7DE5yR z9{-HXiTbJzI`c#)Y3VVI8&oqw;AvqT&t#tro9=CzeUtcLX5(ITX{~s{i0&-sB5AMW zgllEQ74!Qj5bstIhX&8k>`N)EZeCVY?PxC;wPTAt_f`8As)kmr5u(HypxmuAn)OlQ zd0i{dW!Y17QT>jBmh>l3{2V8XzMXu}p&NI#OvY&fN>l3ELO>v9sS4a$T*^XZrH>^XS?2d@n5^Pvi!wN82^mK_)bdzra4ar7f{LiKCXKr}o2m zjBCa1s5Wd5JI}@%2)a@}(i<;0c~g%S$kC_1U|(@V>Abqn`?v4!#|M9}$!)QhT;D^` z%zX*JE1>MQEbuE~*nM^=dP`XfAW@AD)_ytKy^B%Q>9+ZV@J6b~eX&#%?SL6Uu9b%Y zaFh6CItyo8Ms&OD=u#E9_~tDujp}GN3J`m6wX22ohptDbMW-Bg(Qki^jo#IP;s*8=k zBy&Dq0RNSwP-mb75E^eoS-7DWW!3zwFup|G#;HD?^2(N5ChXkL$d8;3PY~u--bYStd=3T*vS!BO4hKV^aM9KB(bdmFHDd9x)?qT`|4~|4FjGK6;da`XT-f*A z7f-}C8=K_)qDo`bM?42$5OfZ{u$>BVe_>}KuEh1eDN9Kgcm^*>WUm2Vdk@!odQ87c z&F^e2&(GT-XK_{2em5#Iath8(ULH7(to#umMm1QVE|~xAu`7f6e(`R-O|K*T&QGZo zXe^%Sy0fdPAHSUIw%MQ#y?L1nG4J%40z!!LMW}lhUr}uCW@~zADl2LR074c))7i0g z?nsg^z7Ln7R`nKCnyKz9EV}B>5p{xUSwi(dpZAzFA10eFDgw=P+FHADWn`X-f}u8= z^s@0ak{Z!+R^7@<9cvg7jjs_0{YT^-hJ(W#B{|!~3PNm>3TJIN)E`>f)#0KfpXQ@s z`kmx+;tMV!=jn>fyW37b75PqB`(OqPfc8(P2mF}yWRIfeTMVAb=9a%7KS=gmaHJGl z{um{>Czs~zQtWIwx&wu`i*47sQTIWivbEtkgD)Zd(&DFsyKRXr0&|{7=}f+^Ym?l5 zu&CyK<$t9%0yM!)=^dETY-I&=R>!mOXXYK;k*cA%?~8KDPDJ8L6eD; zm4ElDq;{#i|5Yc_H0=bO*U_4{x){I%q!UOxe`hbv7G`B_PEzr+BpHs{T&HD!#qS!BuKm@JP4 zQzvA9w)N&G`7)S+jV>V&G&mwkjpidEHliASxMJS~W-4{)uV$rozNgyh{@_RnhRMWg z)kj7q#muF7ktST_TzO<}{ZiRs;MH^QW25-!+?qE4X|_r@p=)>`>&$UPGNy6t+Fj`T z3bY<1^N5c5Et#+U6U_m_J}R;SZW<6}oe=vDo7ietv_EZ{81oSTMy-7H7b(r~%(u`% zm#HhKd_r0w0keSv2waiUNMjUJTdg_)b$@{JShFgWMY@tc| zQ_Eo12m1Y4c!03deCCevC4vA)y}4yKM!*|hoGsfVncb#t)m)@Z+@pgFj zyB$efY|L=#r*VmsZUi}ZE+E*{4bu}55ig?h#HPF_iTbLk_iCvb&%j2`WJEd%b5Or@ z&0_M_#gt(zYVLs9cpN&<92+j+;II*L$MFb8iL9S?PEvNa&Ec&+q>P2GIQk7&OCQ-2 zM$=i(1zlG{zabW^i3ZGhDU`YIf$4Ba;8JLOs)9dp{GRkI(p7ozA%P_~t58)I11x2= zW{L5FTRKH&$H%3c6{C#@7-s;(ToeJS({{=i_%LWTR2D%muH+=*FArk=7wRiM-^0d) zHi7`CL(30j@dB8%!-J&In1Wlqd`5`FBQVw4USFZ};}z`Gk%xZF6PowITNi!sN=|6G zdY9q>7Y*7AARdw9yENvEE2~we#Yl4U)qtNZpOQrjoOtT-^U*%3se+j25Naa)4LrlDzqwpRr2R3`44o%{*pTTBON zgV>YdIcZYNCxAPR|1z<`6hhx|t4kgi-UcD-aS?utLMZ%+#@tpP6c1Dm)x(2D>OtV6 zhLEq9m3?-6?1%2I;dD*Lmx^fXZY0-#tWatSqqhNs*POtV!@ph-N;KoHTP^lYo%OdNrlu7xqtF^EasO|h7)7Bk_%;bB7=|4-`z zs`pQaZ7~nX>e^m6oMDn8^G>!g%$JpH&kJ)B&efsQ{beo1%Zhdkl!jGf1C*$U^VuF9 z5?C+~nl(;K7TH!Z%NK5qJtKNhzZ$pnA3|1j;>gA1!>afK%;CqJMVqOnt#rKHG}YugUjZs`IUNq1!*x`!b6p zi;Z*RV(GQRI!gjTUnv?Q0NwGh0zJeLaTyOopsIa^Ty}VCJ~9!{NAh|2tmJ0ih2;%% zE4^8@%{)8$q-!K|i{I#W=#5<5BPT{J%iyXmIAZIQ!ucxN1TcF7zG(KJolHBG2BsTv zdu}tI6+1pS`qz4g(VwrRbX)kj$2V7DrSd2)@q3gyT50`}83GUnte^i0VGdDcR#Q;# zuL?uK8oUN1GTmC7@Q@m3_hik{f&2IRBZz)-*nMqjH+QeuV#=dq2%#Nbmk0}%x!we? zMrvm&!CY11j3!szRu>-UdK9;*u<>3^#fJEVYCZB?yyY%f)P6a3@UrfrlkgBlN8EK+ zI1B!tUbSOpoIy#ou@B447;KyI5LRs;&bh%Vy#B0av}0VE;I_r0{wnSS%f`7VsT6-a zv6OEbJ0qQ?Z67O1#AxZByu=f?G-~AZ4_F`LeyBymLQbjutG31S5IOa{ZfRFWetZ-}0%Vy9%Q7uMdC2{M9f*^_Q+PAOtLl+AW`zd05zVJ&ZYVXJL zmzEU4#f4?m2hzn4(~9^SPbXJ8+c^j4&5A;Zg`ib-lRkN{N=HyrvB4uRF%v{$yq z;|*}$_lsGZKDCs4npil^?X}9JKe17AV13o-K8dZwUDHrJ3F6-cw_cG{PP`PQS2Y?J z3X^|jGxfBIu-dJzrHD3g;j1+X-s5^QBkc=Y=^+CR?5%38UiO@L$`QZ!_)n2kop7xz z$pLH9fAdRWIL0>k1Wkejh|4mG^HL7-e?x@q(-gT&FL@kBH1yoO-JRa5I=DgBckk7c zGNRmGS^vbd8L^~2r;#h7M$BQF`AD92kr(xqGi*VJ_wKh5yvqt@cThDe4m zEQKL5=7#hSH~GV+%#Ggor_wy3o4k@w&q}5x+eAE^@YF7gNQfz9CkZGFhPDLFgS~^!2UU&;2sxw^;!g;V2(iEodmuG=fZSO!6>I!a_@Vf#o|qp{tnhC!sV=S&O1VreT z=T3_~rXLFpKSLVyAxmuRjvu{9K?9k60hWJ_yax%7EpyJe6ilo5$|+ir$Hoinpc~7y zV|08p9WFiZQz^S$mGjyYHRx}A+*ry!@hno!XiYPlFlEJetImpPDpt6*!bh_C_8Sp zydKbH&}|yNjSA?aOWH-0>0h~1Zi#97Bc)&|>Dd~tvEdYyYa`AV15cTyFpnW+9cDcq zy-*~1nLD&XII~*p#Zqo;5kGlD__FQjc5UgcSJLCsiYhBc@p$Yx3F-M2OrRI&7As|r z5jL^Sq~JX#{@&|1XK}p-W7qN{m!QFAlOK(^oos)c@}c`9#@V7b_G@RnRz|n!^RP*!eoZJ8@^1EY3+`c^CV4cUe2uhb zWgK#Dy|iJLtwv-?whMcwXUlLWVNq*2EO4|_kE_T&x;Dz1<%1_Z0&$JdMnt<|LiidH zO?hav;rsIPuT({mT-ocz?uLG9oD1fvZIbSC!h*`YrnET>G`Ffl$dTmmKXpCFP(_Ra zXTupxn?+|*kWXes=+HS+J--FAz7X}_Ia>%@c7~P&kKU`4V+HxQSF=19@BE%nESD!^ zDq~|9L(>>mgly z!aMPvK{00e8PA--*EAamQ&)usskn$xnGN(n$a5kvy11N95A$0Wj?nX=u=EVx-wM{c zFC(sUk19COrTH_ZiGTICLip9JW+}NMD;LclWgz4Y{fm@|*n0cTW9WyGXPOfec>zqAfNUq?C6 z^5(h&u|vjXT-Pwe;VJmaM(^+qhc6Dj+qJ2+TRCB~QCW00B|Z)|b+e14bBpW5GL{Wb zX-7mIr|-B6_orz@#8BTNj-UX=5#}kY}2PgqPb#wJYl4}9?`hIX6izg>jN;(IJr z?W@!uO#XNe+gx_OY2avR8k{$JCc~(KaLIY3cgxjhBvF-#w=KwyceFnr8Zynvr~!)g#>S0m>R^2Ozy zFHO^uHRh9*EqCtw!J?x59Mgxyf?dV+_*QW)`-IET7Y3zmR7zRO;1!_;Ek647U9||h z`}!TrjPMVoFC;OE1{+LPCTB`~?f)EukfC0L68>+Hc`3S*{!|%V_TAPXSYZt7tbZP* zO-VPxdE@#~J@j8#`$7jHrsnfUqfCzl1qeS<++s9joBq5|xyTqS4|MkUC^x}wX^-9E+Z`I(*g-V#$f#IR7 zZJ3OHB?fVJaD_%hUTZFKu#qP509_H0G)Nm2U5cW5XHwZa3!D$Ho_LBRHF(Nx8HpHB ze~5MxIov%c{8XZXXcQw*=xsKz)RBNGt7TPaS?5`paYW?$|*9+s>2xFY} zOpC65I;Q4?$$kBcePHE-4sLj!f=8# zVKw*6c@=>|zZoqOfjf6eUFcfA&i@2f{zEG1SkZL9c}_@3zEG+bgh|hR_2%(+3m_Z^1ts-njVdW720bi%@Yolb zC)pcZ$sDhi5CiB?CZc)Du7vDuQ!~d*H4r}Rr&#l&QTK+!A5&hn+VD?#A(-@Mn(kjn zA_4$-ksq0*4bZ_0+o4%27AmiV-o<-E=mKCPk1eCkx-jzXEN6P4$_x_YtymCIHmEtA zew)Seggcz!E6jV!1WDW4vICxR4ZFr(1=(2ok@RyZ?@10qE(4`1vMcU;fG#AU286$L z=onRng{$QZyyHaZLD3`&A}LeaUJ}yQdx8Y1s?= zqeCAcCnEX`TbWSs6D`zS*e^D3tACdxM5@~k9sX^gc50#;|vlJYYnu@a* z`y|gVEV3c>IuppqCqXUhuS*WB1HClzVB=4r?=j1XRAZ^GiR|4}K8=13L;~*KrLP)7 ztOKV>OsA}7-V#y+vTcS03|?7!&6Rr)O64*p6N8yrX8~$9nvU=nMrS^dT0Ic{JWzWx zrJe)|I~R58$?o22l}M(eU#xH_IqLjB7G?-X;71dRn-F2%lMPbQ6qa#4KGAFWwi5^MeC`38nUDV_!vc5x!Th_`M(% zPa15Ff-myMpU6Bq5|X!1nI@r%xzL`H0L`1`*rQtRx1qm@>h$1YLkvo`seJ9co{UXG zZH06C#8Wb|fB~a5GdG2SXnwgs&w(AT22FU6dNM>ppcdr!M!J)uK2ASoN=QD>X#cLiLP0&$&&ljRuAKsK?Lg&`C0P7) zd{U7dU*?w3@WMjU#Wjj}s{xasBOJ%bBZLvt6!&~YlY_6=_}Q49jE60v(t(xR8I~9l zO{4^>c%x$H=D*Os0qa~c9_ikS`q(ZO5}AofY~&eafRY!#I=zcTI}i)99uhA;t^2^B zFNe5L@<>c@(%dS&)#FHMvY|WD^HIbZlOrr8P=TfI=Tl|TAEV|5Ar`#j#!N?Y0}IAO z_&5NYgjoE}r&rHq(P4-MNky4Jeeaer<9gg+@sjnrGj^3RYLE-U3Z*iG|(^nfH` z?-R~+yc#BJ#YZZ@)y zUyKnTK$bwfS!5sO6UVFQJ>)@IZX(+Cpwy&5Yu;Hi`8>z1_Cg6AbWi{;?%+Y@>hRvE zIi^MM3Fb;$!2d#OP?UVC%p{Svh9w13M_Z#yI#SlmNBd68fUfoowr-fp=oH98%46Mfzl_?J*6eDZcAg@1q| z`tzZq1ngvjm)@;xMlzvD#hQI;t2UGhXLctc_l^|zN&UM6OaRTv2nuwWj0Mw#eKe#S z`&ySC^+_3px@@8$U-+2;MDH{*?5Ed~b{roXL!ypqjtcojhv2!O%M(}`?HY4Vg)Y%( z)TfuL6Px~tOmbK-dIIOAHUHT8m(&h*4qy&?+I36|21x-K*L@eIi7Ck}DY5f=ALkhq zl6@(c{s+pYRKU`zjPCCIBqbc`dnuhb8Xca#&|X5Mux#%+`XluI<<=F>(4lenp2w>z zc=ZRUZ$-glsKaYj4Wz#G6$nsa+TSc{6ieT~A->BaCFAjsIT zA@nixMrza;??Wg9Rl`jg(vL}dIjACi&de+8Ik?4)D2e2&$Fi{7j~6+-`i*o;@f$i} zph)~RAUu)YZiE0jppa3J&S{S2gdSd5y>s@iK%BY82rjQFaXbV5ml%aZvQK&Fm(&@O6kE^p|dd8T}02WM_03ayW z^a5tVK>?)Gkj|FH?=X~v74-^O4AIc)bhqnyrb3b*28`!fPob*=a!XEZ4|B`3@;LM+3PIGpD_^NRS?TxyM`8O~i8}lNnupvmmuVLU}&xX2L$Yu7Sl6 z{)Q26UemEXuRzxSfGGS(pQ({xGe8HTFOEICl()KIZ5ttk{$7-hr2(15K!OVxFe86K z1iagVAeW_rSDm+x!0$qPi%2NBIj)kKR1C8&G=d6Y0S}ks!?b9>QQo3AWGggtto@>M z6u(0d!zNGBNSb6Hh>d-Gp)E~^j#ZS^L5+}CliePcK)-8Atgp<0hq$n0*?rf?1iA-J znk^{oF7kx_X3EulY>tp&mp8Ea*mUAj;V7Idz)#qPS7D@s);L1tnk}fJ_eI{R)4W4d zLUVWR$VfV$2w}(+VCb*`7LXkp#!z-8$xDmCVcnD4=(sZTloh3hED`mqFe0Wxo{M~HFP{-( z(&`z=dOH%|r3}MFu;W58;|Q0!&ut~%I zI(knM2q!4j`^?YK+r$slR50qC1xLw9_hqubzP~{1Y~Q;P|6M;6B~#I;1MmTe;PL9T zV)#IL?8xa69UpzyLE@iBKaaUSwdD&B3u$%UlO}#{LUU3*ZIBy{4R~-Dli*YqKvQfw zLj;r?^8V4&P){P~xAx}HZUK+kYgfi~BAVqmAhoH7Q-$H`+JOCH6cJW=bXCzS{O#3e z;t0y-;BDvK(e#!`Iy61&lcVbuVZ&-;Z*pOM^StW~M$rJb@0lq%M);-$qm{ipH zvyN>?WIxVT;hyi0xN;pUM_?8{u?D_2P3AJ!`~F%R^Ar$oBfoJCaD$K2=3JvFBh6JZ zW|;`$LWMsbx|s3Re3%N2U{Y~yjZR64MVLPzz_GqiFKWW?a)IHmP(iR^=h5+gF)_ng z1R)GrsnTH@h#gBkNhSu}A+>PIuk~w%i0lr_LG*>ix*HHY$NXmnA(jnGYrG0XDL4z~ zWpd3^Km>8`!uyn&Uc1^gVZWaEh1Pury9&84^Lw=CzRRpm+uKQa;>Gp8&{x5!XIgl{ z=?pJL9lNR-Sx~dVm&7u6`u(mM|C5K&QKx{|@ftZO245U+4~uqHYasfyS8FK|%G%q? z_w3lHhTeb?ocGpSTL~M!i+BUKz0LO=N(T-f1#3b(dnq+iG#SEJg!j>@Lwoe=wFUeL z6da-*bKuZ>XVxS;EUWUFGItqpE|OP=wthS&W}wmn{`ygh`g0KBq1XqqucxjEmPHyg)XWX&2XngO$E(w*1fgqWe&iH3of6*&RnsKTFt zYkDSPEdvkr&y$gh5FLB9?&p@-%w~&|P*`}wD!|{`e8F)+AS@hGY0k%lBh!P|Fp7FB z@)pWG&g?2XV|g70Y7BodTD4c7k13edJesHZhnZn7pqUJsI3jZOZ{-jrfWT9OwU7D_ zkNz4Vk5g`L8+J-ww5kX=Sq9tl#jC(jRCiG9F5-OmiT&T>q3k-#p@6y5ocAJqkKjRm z=2ne)Uy)DAdFsWHFm!xo<44q*o4);U_i|KUF+Zv(s;#m0Hf!kp`iY3ei{meAeto-D zNtYU{&WclRet^&*$ zNyv^XWT{O6q2uW{I}R|yZV=6j=X#G;s`oe;4;Fo61*$Y$noD2UwO{z@u{Ir1V%d+m zH2t8c27ibFC*}@d3iGQxeFQK`qQ~(BQbO}-ebwg|^%=y_Tx@7C&iberD}wch_`~3= z>xK9w;l47}&$2)03s?S8IDhbOsRjFH(?kmBY6GFZdY2Yxh`TaR`cKJ)C(0i1(n0`6? z4)#k7-Tw2ibN96PB;MT60jUAoG*OtRAH-8y;F&HX~Dg?y8!XOXIJS(HIJ9XQye27*-_do2o z5RHMoK>6Z-2nmr19%j4PC_4k<;(*-0N)*4rry|&!(6YRF zCyjHAzxP2<*s+lJg^uEv`<&{%0;abbxu!6}`V8q>FNC_F-&hMJJ2qPGtcsxm3@W7T zRWPUk_B|!xeSc-5SpAoL$HRw^c#y?t*aY3k6tsv)iyDZbtT?J;(;N8#T)u3rA$f5H z75u-#D8GpfV}!gsj6D@3e{6Aplk?kEHa#>KJpK+G2=)kYGT+T$7xIiiRJg=Ag&ap$ z5pn*mo`>0su+Za6A)f;ybi|nz;z|vJPHh;{%S| z2DDptC1qL72IU{pHwWmlsj0w1w2ey&0?<$A>q|Xv^j56XPkCNI22Vxc3HaWbpPTQ$TfNv==&rRsKx6JPCx<0OlKC4%Nmg)5Q4#% zIQ8NkXWp^dp{j^u&uP07!P+6tv@y|vxM*kA_qCTD-NP#OG$U~A&_>DbzqYUQ8EiP$ zGTZ5xaoY$0?ci{WDtwXa^&5#G@Suq zlV?HppiGYS>Exfs|5}U6@lfo0#uplHLf8PMqmO{DgA9Q1@ed*i%Y7K!G9?uv!*7E=T)TasLZOl(~ z35W^@@=SKe%To_@(5*ET#GDmiAwz~v9cNGaS5SmEfLC8v4F54sj$kcP<-0p=ceD6j z?9Pe5>AMCAKve~^9=W#%c+RR8U_ zJ?5zYeu8m=t#?TTq!6uRJ@KbC>KrrH5$-}>c8g^fdPV0aU{>c@JddQ=J6ku zL9oO)9C!70OuOW#+`Wlm87^wkvEznO8Wy}K|LhRZFM5ebL+c#;Dxk{K46no zJJ{M|cK{~h%n7~~{oer!gfS<@+jjwr*R}rq$WjpoRv6Y-gE5!SD?2VW63^xocM1u2 zh-k3p8#>rm|G4oBY?w;3w@+}NRfz9W#?e)P)&!$w=egwn{_qa5 zQNAeE8^7?08N}MF_kOm?Q|nr1rnZA2ZTb<-9~~fI`&t#9km;@Q%eN&L*nfcqbfw=% z&HVls1Y$wzfFP-l{zInFr}aaHCcz$Nj;U5@(M^#CzpBBugds-PA!Bo}o^9*!9fQ78 zhiC_1aXUqad8Go5^zW%tPGIj(%zdBz(u^CZMW`|655nug^cZogcOk3vXL2-~$4xZ; zeLGZ%aGri8`IR(J_V@8h7vV*-e6a28VDhDKrzL0Nlb=Pct;IcHUYSMHD`p3<4O8v+ zE?6$o0mi>BY~iHsw2O_}nm6FmuKQD^q}TLRg4u;RVaAyvU`>52SnvG4??Ln|7%sPq z+PS7=aJNxL1}?wxFPY+Rvv@kx4{ z9AmI%T(^JHXVT}e2DQ4o1I+F-N&qYB^f$DO8WnkSO~Lj9?c$*aK{LAVdHDd3$?P^S zXXn@QR8-w$eE1BtXyx;aB0pm3W$A!!TQzFHHH|2kS^rj;96>*q2Vnps$yT-~EC8P~80Ij}bHy-$G^E!5b6H zjgtN7oK09`oK0jK)k0(LLB&OqGg0h0?lKr`E^LQO)iOoMt*%Og8VLy|#;Stx=~qoq z18#kW$6ZG|7Rtb)*xg@R2~l2v`h#^3oSmH|-f)$F{Cxed!HiVHmijnwzcd`a>bbL! z-tqrvd+&Iv|Nniw6iP|MD6%TaR#sL>l)blOX79}rl2DluvbW4*9S6q>A$uKrC-Ye0 zko|o;z2@iheSbc`??1nPx}8wYc|M<1GW@n2|1EXPHvjhi8%ZnjbOe;XPn=wuYM90W7 zOvpDA-D{*62HxSFAo5kKM=>ZG{Ki7gL|lliK&7?`)0sIEpWotxMre#9Vwz-vt{SBC zbRhfXsY>nH3jI;wmJgSJki?oPW!qGL@(-PNKhsP8Lj&~KpT)O>%zNj-GqHP)K$k4e zIF$czvT9pLsXcE1{l-}2a6>$ueoEK3g!y@m+YvB(<*qy241z4iU=lGMRX-zSpbKe? zsnX=mi75iHPrc^DqQaWqAid;}{BX88B#3=k0Si|l$&+$8i~nJ9bQub}I{6_Z+m<8A z=@)QObwgBJS+NZ6t2m*{)LxlqD2c4aQ+jJJL6sR&Vf=l#B6D8zjRtQBglKH<4t}Ho(7^|MYnGec zuGpoK!Z#QZL3yiRA6ycYscmAii}zs;O=K?l5<|nEh>hOyk5Q?atX8iY@v296l?tv^ zq~~vrdNBx|e{1F^^r3@X|CPf!Fg3Pc+Xr5Vvm;bK67B77u_nv*j7myG-2gAt}^6Ywo{3(`wVl$D1B(IxT$N)S?(D~$O z*VB)H_(MS+&QC-zSi&;ue*g+)>cXlT8*>9os+1}q>U)Qz! z9!}rovK;FH40qumaJYJ!`%R_|Gri!!|AR_epOK;WqFY6MP7c1^ag6N?4vMOwx^fNdrEoBn&Mtb zD*sYQ;kifo zou}kxv@y56um()?fC$ zT=>`O5OpGYTR76d)3GJE25e)u15+*((pD#P8b00N#i^0>yHID*vo^HuY?}!P3d%+A zFV1YM>)4R#O)Sp1qz8WP@pB@EU6t3`VXS^tFf|2ESq45ODb7YQLh6^z!M7sOAtEPx z-S@j|e5Jc!uuBZoe(QB7leF-=WhJhreLFm|0eJS7A4m6`lrM3#{W){tTLDg4hZ~Nk z#TGi{wJj#QOg!9KManqS8zftZsn5883hqVqFE!0{Xmrv!_J#Sb(V!NS~nW#;3}MSY|JHDlU> zFV=k-jYx0?9`mtWU60d$<~9ZpvsB8u`HcQ2XrNBMU-u|4;BzQI|z>dXnSR({}nLTE>6r<3-TWhxP1%M zAvJrrA^o8F;fEZZAP!j(5D_MLP*8(0+eg(qVt1n|O<|sGOkcJ|B8ik0l?yK~_ugU3 z@zfg%X=81%8iKj;EIWuZt0MMDQ|JENI7zujJJJ8PYxpNkcaaup|gA4E9274XJX>Eb?f z+KyK=z(IB_50Rn=$nhicfUC%NUuG6>R3MNVqMxX83&Te|Ffj4OC%-_>vKQyP=IZSH z@`;;StO@yot7@I{0<3R`?6ICDY(Y}yrK^9p)I~hgzqptv2#L6Uuj%w)Kq6kF!(M@k zib$}iUtTaagm?6NvMa75{0mj<(^Muq%$py@Ei_>9Yz2FOr|u};(lQo7h^F*&0#UqN zRt06zQ)ouGic}MGxFJY{#g%V>gEba7h;y1)HI+g|2G>>x7lqj<6Ye;LE=<{?`Ieci zjZ4PYO8`$PL|b`*8*~&e`bgI_)}-UB@3|1$U^$)v)lA+fi(0h!z{b_l&|_ymB{n%{ zV~0U4Gn_Dmj`hRzU(PT#H}AbU3a9V%fuB|6`592cQR&ms_F-px{gDczp|-D=2;$KP zhu}Co)TkNKw-=Wa7CFy%K7O^rqHfu1r6Jmq*loll#=lf~s_F@afS!23Whs~fmrMGi zSEYSrPkGp!Rn|dv+ODTp?`}}n#hPV>g@9TT)nxFsIz_U?7|WU6HkQKQK81D1=(>E{ zS1ea&A%FT9Ie~TOEBB)cy!sc8AYXKm^yQqVQx|2}%0Bff3eLbunMvVW<4)as3+N}a zmL8ewY&iSW8BJ&7=)d7tRWN8xW%~}_E$^`U?3sNzT54O_xy2bIRzPg)({{YHxC{um zIxY|HsRO%osFtBW*;D?J$z`!aVE&2a$e)6Op>0uQR>)5Q`0h|%v~)LC2kA(BKdtVB zhxPg63i{!f_N!#FG4KKY)PO6=KF1!0y&w&2#KR_H=-$#e8xN~q>Ybes1fi}}-Gs|p zMrOw9!bBL-=KL1AVd37e1~|n3ftgaI@g~i=8nDRj7Y#mk)E3FH;a3+&@j!U!=1NQc zh8gX+2dLp$Q-;3=G*lb#ADlsR)Uc$B zPrrsr*&^$09LOwS&Znv_7CVzvb1fq0wr4tV!(R#3-`scdsHI-3OJ`u0$1%-Hc*kpF5kA+S=%{VTo(%>;-*ST-U@qeJ=C}*rd5wsQ4pSs zF|%{ej5gNUb*Fu`A!bMsKUb0S6er}0|C(s>6)2RGsP`G>YnF5K)D@TQ@&+5yFqMW? zu{h@^%p;Tcg~nupzpQ6Y-wvu}uUaXoQyV#Jc1FIu zgz1hAx#SR|*eh)(@uAyXc(ENojqv~v9UdH#Skc{r#1dQ7H!M$|Q$@zJWuqkJ-_>_N z#(++(_ht^l@ILcutXRQDnBdxDW7=uKs%C}JBamIo$ZHjF%aqe+zdxC1Z#jAj9L1Kc ziP0EKapTflH6?DZ%~nQl5&}!Hq-ac2CVG*C(FNzz@v!B0Bi^T6@uDQ{kUYNgHbVn< zpB5!x)e3gOo%BghV%@xt8&BEKo%^U=GTfe9EkVjA0@5#U;ha-{3aHQc4{$x>MEY)0 zd5xcmu?oY^+`yPa95pyZ0s zhG-pQ0Ma;Ro!P4vBfzjlNO_|BI>$KZ zSR5sURW0(pEWL-*jtWXV`=p4H2^b0(-@y}^IzKNig)fg&7G7e1k``x}Z zT;i~XipBIKfq@(M{vjNJRfM@oziDssihL8^GCmk}9}cscS)cYJ35~L%WhI{{Y(M@L z<`c{o!!rv4=63gSSPB-Enj4VE`a1Pjy>}kq3PqdQWx4Yz(xa>uNoqh;5~^4(uZ1v* zLXZ8>8~lxBnT$^Tb!X9p)qT{a`Uvm4uXa2%dP7$tXGVe=hh0~j(x&hZd!N(dbhgKV zb9K^YgG--h8By_sX+V(dD?2+q57&fI&09N?Z{09T&pmR3v`Jcf2&}(F_wQitz!vH| z`KDG9XbbdnmdshUDfD0UVnLKx=?XZC`A~Kt0=Bs?gu`IgEi4_H#vwdL%~43s+(3!Z zN<%w@Cn|ns_z6Rz&pzv0744B7(|kTCaL9yk3|X&{L@d7v(%)xB4DUG>*xBU=;GAdg(il>8{G*ter=ifhME(~=)Cl{k6~@`_^7 zdMurovg{qE!x)T;M*2=W!6Es8UAztIR~{62PBCXiI|$$=*#V?4E{S;ffwq2KyUnzx zlHRlb!{d{l(=)Soe?s@5oEhI(=2&X8SlhA(UL znLo zARM~bMU~`e0wj@k;dJ>Nk=5wvj=1fw!o%_#mUl;M7?a%ZsK2Fl8jJ_9Ur@A$Tj6pV zX3kP4z+FX-N3C~rG=TbmTiumgoWu&`kloVJ;45wre|sM51%MzuB$cv_3`wxxHb@8c z+%FY&a3^|qVTaEh1XG_Zcauar)hsugb629z4shb3wdSR9Pi%9=g~Jl9l37|yYw~`J zan?H!=@DMJt*Ks&L%tB;Q`dE9YS`nO4d&X1&5o)+?Mmg2s14FJX}|xeSfU~g`sy*g5H2X z*G1px8`I67j($g=(?BoDwrevm$Gk#$U4DoZMP_^d(wOvA|2pNtUg!~1kq{fx=drRR zn^kPixT_Ru7m;>^S#JyzIj*p9Rj6MCJ(`&kwVxpctj^b_`y2xn@hCmy!W%;>%rrfG zy!~PVVtaf*epv5JqKgNp`J~sj;4DNsVhy) zmpU$^IXr(pxg`q6s;xbt^6q8|*_nXZaSb>5#JXx4RjnetC3wm>u za+V=v9VB(3Aa4?Fo@FrxGs&+;4vOJ~Y)5_JG*z+&{1gIU0T|Z`*g_FVPSL79mxd~@ zB{-qQa+97o-(Y`%pm|36X-?Y0gk|xC@mU{G8yBmshF_QOPSJx8$W?9-qf)Nos;}#@ zyBEfH-aD@?|C&l==dSD@i+{XaEhcy4_pjN+Ivz}hxqVxnr7ImXQDJdK@zjU(zlm-k~ zyTA)f+6rRl(2_AN;5#sG1(W$2TE(TxJnD++H_P?K(_ft=W5!5m=3`XpdOYOej98T}%ry%3D60l~;ao!kQ@x0N2uQC4jwh<@sj}8Kh09w98{x zo4FzXn0`mF5kwBT)aWx$cAM%C-3Qh2YIenIyj4(T<7tFb!D_vqZo9xRYcBg*5llq;PI7F`sd1BBNV=J4_bU-Jo9?=~tvvBAoctQ>`q; z$AcuPPe!+d-A^P$AT|fceE3^kRGtZ|XX|{+Pz{^SvKiVMW>~e#+=GV~67KgpJ)JO} zmkKGePox?pA$HgV%n75q{#FuvndwA9xT8W@d4#B=k2nRHVc?(rC5tTB1we>d1Ludiq zhK^p53`ks7hwlGTCAmdla4hPkU8UzN7R5t6gN|L1^qTRcsHno^|K%P zZEqrhDs}Z9vDkH(hyXRUo0; zhA)y7A#!(abnATo`G4V8-xLy6M_gk|F4H4K%5*op^ng(hS@ z@t+`;A#DyUsD*|!O49}x_BrANc>y;bXPul3Nn(V6d0nW$a<+2b!B!-y4n$s>Xp3N< zh8!nkQxH$kUPz{eC!5^7vN{X$^+z&i8E};<^~+46!#biFye?d(@CQm<1uBO zLjC4N1qNw)SKb=!HZKv6xH)FY5(je}ML9kMXQ#e(C5TLY6iv~UjdSQnEqkmq9H_7E zB4$nO3~qTEb8cl(8h&)PT|BS=8+xa%+t3>+lQDJhq%1lXD1S0*p<|C5e7ow^v{(bTf-bmc?}Ns^&VKXgz2!$p!t1K*l0Nl7E0Q{(p)$Lz zq{fneq0A_Y+O@M9a1Rn{94@QMNPB7sI2qD1qvirx#T2L&baV;v8KSL5fVtkf|4MHu zPGOhu%}Y-)?tL3-IS;vpu?sPC%q;|pH>qA*JIg17yt{coxLqjxQ8$F^f~dFcgC{a= zrAMrD3~c@1yIUbb6r!}#T1KB#@Jiu6tOa^;(sXWqu_^q}bx5bOD{tL>rT*p20737~ zL~(R-#3$W_nCOAsel-o4iE}q26IAau?@(TgxADC59{6Et7fhn)#jW|A71-Kq7$+hH zf4sYUh_kU*%)kAI>_=OiZjb4O6((8x5Asb>`K#o_K1Dh2esxhk`(^Nk4jPcy5cVrB z;`DJP1YEa4wzlNh6-Ioz;lb{ie9SH=;CWnwJBA-2;%R66LFvvdqMf`UfcA9(cpPbo zv!Q#x9N(?6OV8*z+?~J^*wPl~(0nU|`c31jy}6irqxXd<8I1*@u{h>vpJ`vhX4|VB zptO^$9gs$!b!a5!`}Tol?zYa%?&V4eyR5~mZ-C{GKz>Np$IfPjV77)c*R1imR_!l# z#e1SMKP~m9KcA1FGMo|Mb5>Dy$>FYS8}u-%8fVi&;?-IQRlJF$}!e{y42UFd*N z`@J(yWO&#Cx-@Kr(s45mbPbyOVzTI#zUs?i^RZ-pX>|OGr#jMh`zHF0YTD3C=4o)M zF%uDdb{(}iI&-|?Qj4uvAWj%WMRdW&@?t&aW6e51{ccXWSZL)9o2{S;&b!SZcrU=` z$>^kncl-B^LaPHUgbWucOf#c5`NmjerU$m5oCw!THQ}Zt`t>qbhvKJX?itH(X(q5; zzXlVijhm4=sd;h^KycC^QsRy4h5BK1+vDTkvnuCqAv*whlMo%P( z$$DP|wa);*pv-nTwPZtuHb2R&N8g6m1QX}bk6?VXE5jjK>FJGS?(`f>GseoCjX=g? z2@&(86NO0>x@=8k+XhvQ9M&xE>4RroFZUZf+Ra<828bJZ0ps(D#rOB|(;lRxYD?9a zf2WIUBQ~|C<}16c{PnqZ?wmuvE7oJBZ>O}+W5C=H^8nQ6hJgW`fXdRyWbKO26j(2L zrM_Z+wRjaP`K7j0nxX7Rf9J3{^X}FSw!pu%){AI7qP4-s1#{)_4JpP%)$fDi2KZ`N z1(InHze;n$0hikaJoNsMaGZ$!^w)-c1RsE(>XcrKax=NRf;i3~Q@&T)V%Z4tjyd&? zD^njm2U_pPcYqlC=9f0~NiOGlu^@|-@uoo7IyT>K!PhZhp;;o1I}fab&WhK zfhyBAzpo^nZ2~Ebul2IBH^){weTt1}zs*TVdX;wTPvkkpim1YSSITI($PiMx$UXz7 z0t~oev3*hRo5@f=CK|>x79|OL2AGl6)PV~Vl;w93KS(lg=F&XRvRYk}Zb1PSMhjyB zg@5^Sd`J=@Wg9EygQQHe004%zEYkagqbA{2rGOB@k37!v9t-fJeiN&OSa@TCqbzW8 z*BkZ3-A|d}BwH+`VIXskW{?lbI=ei_g%<5B|jA7`Y@mNVpz?%>Ft4U`|771niaX zJ-`lc9RGd)Ru3v&s^zX#IRe;){5{ErcXkN(+S_=sTrwwvhLoE$T6>}nw3re>;1OQ% zKC?-#W3BXwHt7ZYMZC$_jN|ezy_(QD>g#FbxKaB?;*bEFY*R;AVWsyMG$OO1QFf8? z2x0|P7FSX8`>06U~3QkVO)80=~2ZT=F) zXHfhnxv-S*`9)KMZLu2lLE_l6=LksKa+;}JR+k)Av7aonfM%zgA7won(urKH3LS%E z3(T@Wk%1gI^e7XsWv)(YLHrC=FM6FY{m0+#b62k`+PnRRvwzMEjM54??%xz+vB3#Z$Uy*;)P85-uN9tq z65LDmap)yG)OItK9k)c3E!4(AU@^V0j4f)WwEEU2 z-EJ?8RCCMLYChMosd(r|@?^skaRH6(7g)B`D|i|afo%@8J%d}H60wLX*;)s2&kN{f zWn^3{?y5W3et0gi7KyHPLK4Jzz&eH&#Ne{@!0KUE?KY-3mu(b)bzkC&ovZg$43X4S zh1;HYajQ|+MzuTp5%Yd`PFL~0TJQEkk*}<|fB)}knUw2b?S!|r?~e%#yMi|DO}RuN z+&=LWW_c_MgZDT`XlN7HHU>WbA^pWoTZOuyKyb1m<7kMY1$8yKAE#tbKVnR)QzMhT zPHps5q6(3T;qt?yM2124CzXIP(K$k?{lHw>@^koZf{cJ$RSM#=M9UoOLP8d8ph7v<&kXU-qD-l%_1|FQb*x zCaj>YN<{N5U5m`9H?6~Y zw&!y$)%{7lw9=!k2Q2{AOmibm%9niL_9Z2XD)je|`}iTBBDv*lw(Ak_^R4>^<>W1L z2TI+>8lgIaF5D|0n!g2S<>7}zW3-hHN+)hGdSmX0lqRl$@VK-_j^Wk&+G&4uj z!haDySIO;%^07R)TylA-Iu)2O*CKj1T>Q#IsC!L)2UL*XRzT{b|I!B45dRm^2_fHp z=5Dpnt_kGe@eEvbf3nOAM+!m^b7TLwHxJi^Y4Iz9DS4PB`^xR~OWT<9zHRzljWPleNg|)14vPW*OnHY^jAIA>UrHy$3C+QzrMsx z+}8O+&7Ok7gNNsT^@o&~Mf4$pP?buaRa$$oeEdRTBzZ?l(0z@J-HlZ)XEFbQ5`&^;8=2dY!%!-Lss{*uV0sU>c%qwXpgL_|-(x zGYJ`U%lvZfxnyq2((o5mK(5DF(C4^E^iAW^yuGqwF3lvLOrLoB22NW$%vx%p$>K|* z`fwunaAUN0io>tL(>*UBUhh|nbE$5P6RUA-jb*HAU`Oxjew6f*kDxJib9R24r?FWB zr+45n8UatnbNyxopQDFR)Hr&2^Qw6job}540q^Cb#^dC}Hga zPZpwiA|y+QWYo%GTs0Zs-iyZ?PNTTW@B7oj+0XB?<~rD($A#5G+mz}=dX`r=zKbH= zv#WGuKXWMJSiz39S^f9)7W>&DJ5sioh4QlKA-|148u$x4V+)`W37W&dg%Bm`#10ke z7rkh&n*@OMJfyN64Xdt|U>P#F1E+aVBwBX@YaQSccjxC}uK=ZtTo{dT!-)SbWv+5f zg@lS;q=4SMb{FvRcq%pg13(smEqv4I1U)`g>#k0_%4ikjExQ>mP#v3EB`e`js zW9yfN?giR*L%ug3+ovemWl`SeS@r%D5XW$;96|T_PtWb@ zUK4tqV1pVty!2I>&s&uOU#p@Ii0W6{5Dz#kbjZjkwX_J(z7%q3PIg8M-M+6pnsq?o zJFnogufnOd2|eLda+~>0#VnPPChQxaNTb5Nx-O?hTTO`N0qD_!q~gh<6@v*>$wE-V zJM!2)`g*7U)X`{3kLt=BzVi3f7coVTo#j#U`z$Tf*rxo-`MBN{%o0#$4!;6A%3>2B zjNw#Nt;DM52rM%&*oP{ znF~b&gofzR6w{W^iW{#8-t-~|$EZ;2DAu1Pg$~P~g&npeAsHsZ$qQU*8|t~iQL`O! z3kay3iZeXnUH*TZ=jN8sAjkfO;;9#)Z$2JWa-N)FACGuS;Xgxjp(L7&br+N(=k0>@ z`e_aW%W3s;bjodKq!v6UE)3h0Pb82G4_KOmBiaSA&;~6OhjXBNm+>JhlkJsj;Gh58 zsP>GNgXx|4+3OyCppP&`ib%2jR8vmSL~de%Tqwri}>XBB3x2a7womgxeLO4M)-$GN>U@* zLu9WbO<484yay=n$8B!Y-GIgX)ryLW;_Zk00w{Y6=k_#ncaIm*RkyNg^?E{TXT|(O zd&y{Q`Nn4E_`mu6;o6)no?w^p`|eZ9^Vni!Xs$A+nTi-ao#*mr(our$mcwTBhud zf`I9%ns$GvPIlDKjaX!H*@Ui8s;q0mksn)gbZ<-BTmA{qpfiTOft6$d3aMTQi0q1+ zy7?CpFrF{|py&i$UtfQO5pCf)ub5=`{Q8IcrxpV#Qs>9ru<;p=s(;v+_$uR}XCQe> zY*$xTnglMKi!H6z+_2>v02k+ObEpJm!9H_T{)`Ci3-$oeT?l({x>SMx_G>*p446Ua zT3l&xo?LqpY0G>b-xlhOfLQU$eXbhjje4~;a01SggaDXc|8cug`$c^AY#H?samsrd zTJ_c7zaO}znqq)AFy@jjH72r@5qYRb+aN{{1}roct$VQmWYXZKmR*$8Kn8|@5a>?U z+TTA+0Sn~r)s8==5shpd*d8W z#ImAY5cIk$_opzQu75?Fx~N)o5*el`Cr@X9b&6*{pXnV_oPdBpTrQ73(p&Z~k>u8; z2d{^5qUw#aOXZS-&Y;4L1=T1&>Nl9Ov0w*Ub`0T|wp85#td;Ecj5hoCe$KbxP1-2g zX?oCx6Pl=P@lZbC?3d-5t98vW^-43bxaq5uE=gt*vwXILEIiNZ5iaG+79(3Fd_?87 zSM=0?Sf$hg;!Vz0LY0_@hkpMlJv+z7-oF|0JK-Ka+e-_B)Wp6Ge@e%DahQ!X4F4e1 z6e-uZaJ(2o?ziv5a#G(qceBMo`gC*he;M+3vQCHa-mRARJi2}CiG4A~8TnQeO9TpZ zzRBr&KjDLspres^)uj2DNb7ZcJDJ@1wb1uCZ1B^)Fy>ARn}w<8%)dOOObiVwHal;l*S9J_O0)~M3;vW^{) zc_ghq9e_Y+p)sz2y1JR8o&z8!=sH*sInlCINg}3sF&<4lH8=*2AfYjy3^UJyjd@Tt z4ws-VX|DsUW7m4x&q-(JtcP`C3gXcyC?JH#g5H(`@|T>=4Uh*EQD*1#Kg<)F3~wNy=Y3@Y1!H)(k$z?(~S0WFk(mg>~>9$z&&(mnmRx{a<_Eb(8ew8=C^4qs-Hk z0*J*8vMY+7x=8hGaC-RcOQr$iq))4Coamwp!ou1xTm$`$zroPa`x&4bt~@II6FbLG`~oCcz; zNBwv$x1D4S6Th|+ESYJ?_+NdX^V&-Cz~Sy6nrlUqv$nEPNBwWf;uS?nez?>xq948# z5%3TC47D%qPU(wa7;eK_#;hA9&GhLSJBNFr=tZj5_3Bt7oo!E)XB7tL=h#q)X3ARS z0LO5b3)xS=B9XYG%EadH8tJ5dYb%+Eg}c5B>@ayofn;+R0#2<%!rh)!ChH@EQoTlr z{V1K`lJax|LP5Rk(s4Q!lGZH@to}aAMT7ovgXFO%B&&JWXl!xk-dndLpQ(5a34s#3*abIXTy=NHS)6IfefXEO@K`RJ+f@Z? zE_n8B!M6rQnm_VZ@!Nq4O(XyESgGa@0bE?_RKsZ4hCa6ekSsXZEP(SqSok+YKf;2@ z=SC(Qg~rZ_&Dj0km`QrmB*T!%Wf(MFLmg3o$@Sl}SXhJF08NCtT_37&YN1RTY$uE8 zXS+OaW=X5Nz^RdQ_9E&zf!W;)6Lm^OZaEnNRQ0Gg8WU8|yq!$JvVIUD_N_kIWo70O zLtf!k9e3fHDFZ)Z>4N2MyG%INTgF|)#SWo*_sqmrzl*uM@DXjw0kDj)WQH_i!~fQZ zS?+K7H(*)A2aTq@ojRQt#DH`UgCe4<5>9QkfY6IJQ{86j6JfC4ZSWurNx%+We*KBxBAfc%%BHuGwyoh*G zAha^=ee$;584%0jurp>$9#4hcIsi2v+g3giL@vxZR9k^sZ^Mq2+eJ^Xv8v?FW)Wks z!(^^m9ehD-C{5F;uj5x@!>z2!N{V3a^SPc$uLFckx}iZP8p~Un%$)a%FnQV=Z~Ifk z{2r~xXkYnw)!YKQ?}cZTH6YiS%(}|av`xfd%TDRy?wu6%UAQLo_A6e8AI}u(FDVG) zKDhxl)gA0F-bQwp!7&kH78w2ABqOkq=-c2;r7-toOk>iU_wfwGRJ{4w-nn^rzE&UE!Ra`R!ixu^S$DVYM}i@|IRTAF6bTOX z$kh|D(Y9(zA2&g$0g-5L<*c;j<#d`men8kaI`bq^Y`XW+pSX0MLl`B;Jui_Q3L3%N zvMUw2zph;b259!3=5^Bl7V+?$Tx=v*zLyfOu5}K7rRDp2+8O{ZShNIPGz#ao4s+(* zm6bJCA=P`SkW2g*56h)%YZhV?ccMp+bzE--B?!@k6-!zVmY z)p9pX=sL?LhtFP+Q=<6LN|5B2c32cfdZ|2i$DI`B;W?QqzXLv6!wP(L4>E*og5HYC02xG^_a`Zs z60jABKAyK6X~UL>BNCl$3`N}3PE-n2EI-2Jm3wjm_+I431g!SCywiTdQ9kb26`0xU zQo5t8A@5b1pZCcLKHEdDn6m5L^N}+4!lu z6Ys9w71Btl)?UNXd*rB}oa#IVb*4fPvpGW4;<)jg>GVtN6<-B^Py5wCVlPy6?HK0F z>$jpX;3tBlRe&bg7S-$IiopPzdB$s7*;w28?`Z-Z#*7QpN}rm}0}Eq~uU8}ttb-hk zbtI_9u78DQ`bv6;c zoIAb_YF6f+8C<;Cqr>zw9bmjeS7~G?wR{9D!H`w_v@}O#39mx7nU781co}5mG`D6&+j6=tCYrtHhz~rONhs}ul*0Yz4J;Rtf0J9r*v9NI&XcxvR6MkBXALIUZ7rLJf@MC{ms!jh~OkE!M0=Ik%b_7DWy(3 z14w1277K4F3x~$)-S&>=tC0-#{cyVDjQ=uII%(K4Pl4(F`LvciX9B=7ecjO4uMm`}s6LWg&ge`@NOY?!!80 zqWh$Fz%35|#1GXtqwzvtT;P5N_PKzf(q06MX!m8oc`0R6c@ z{NJyT2#754T*>DDAFj@SdFwV1aI8b44mqcu&WiqTe}G33aQ`l0)I5K$$p7WXQ~vRE z`!B!ppZoFeS3PK;6+x~Qr_cZUtN+tKL}r2yY2%?e!u0?BuFsy=hLYP(SJ3~?nYe8^FVM@9+dul(CA@_)RR|NlPRS>_YDD*wZ~Hplk`iq!UCDn5kb0MnO*zRT1C z7)Ze7FEk0Q!cOL|ZknYux0X*Yf6p3p#SX5Sp8Urv2BI>7K*|yUoFdY+Pwgjw-j==1 z7@BD}mpohB822k%k@0k-9zh2ZA7C>NO2bYs3h}kyQzyq}l3_G9G+x`i0oKVOz?sVP z;-cUjWR?SAymGE9OJ9KNj-?`VHXC5f7vR z{VM?Ef*f;9!4y3w(A2oRFZ&EJG_u!+0YcRY^a93UuE`8Y4cVbV*LYw6GAjG9iS*bC zU@0Bn-rry>Q1o{L{Cv&`=$kqqwjSW`-cDd^qni7(Kl)pt_}2yUmAuH=@htu42s^X^ zi0WL-F53nunzj7nne$sfk3_zzgN)h&gMn7HU}PfAME3?fzkkY$gHZqgl&h42JFa0Q-Vo8Z3whBN)2_8v3yhjhpXvk#sAHeh115 z$}VZI6Q7e^pLsB|#RtxZ>OH8ur~}1!e=xP+p!51<&BhSKHZ~BkdTPj^Pvjo>q#dB5up_-0^@(&G7z}aU>ttu~sXN-) z65-I+O5OviB}{^IpVT~XPvD}mK`LpPl>FOu@ZW1k@jcjYq85w)xTG-@J?C1WYM8`G z@vG?LC9wi+0TR$og%`{X==jkr#z_BV0h7}t-8IzM`#OTt?@4t@^);mFAgH0w>5c>5jfagA2+E)U6p4*-)#il@-9M3(w z>jn+ULS{F{ds4TEerN*KOQh)#tB9R2jI8&e`bHaQN;?3!cQl(y7mkSjY2PbBvA2K$ zjKIwJ5m4Fo#p+AVk6IPl(14-W^w$Y=Ojn;9Yhnebzui#S_6)c%|LZRshJRl_Xh+0o zL#B~zvU{iBuwmKw5yY`i(y$8#Ms8er`59=ZIss)mzpzwaOQTM>1lG`Q)q9exe!YQq zQ2@niFo%l?Gd*}`2)LLKn@0uy?(T+Bsfb^I_tV}7#0Qr!X(CW1cU%!%3BUP$+9Rp* zcprNdxC^iF1Xe;niC!vFy9lk-9@w7n6m%g@rw2!%>L+wJRB+9Fli*RdERMzjDyP9r zDnIICrCC|BbxEu2^XGwFba&M4qqsIPTsHm83jkbv2#G+nfL8J-iAxVa5lFRB$0WSR zK+-Pg1Q4VH0z_iI1gGB{L~{t0nLPg-X&vLo4Og63RubhO_Syga+JJQ`3`&CRFT7)? z^@I7Pv$*s2V7VRmUgoIYcxsruy>SE#l6H0a)P^E^nwMDX5{=6>o?g!ZHg76iyzXSrLqdx)M&v`&&;ymYHP<1_B2%(vL9vSK% zu=^Nv%IT#<{X}$1d3#_SsYgVTaz1^MNk+KrB(DG!H#PPq)c=jCZgQCYM6%}^{|Q$X zy&&31nlje}kOAF1+tM_G#Qte|(xfj^^^_W<#O23pQ9u~*oYo8J8nQbaGH;CsH8u)NpgjRo%K1^`+{#-01+ zfH{YnB#@Y8Oo>0G-`eE^T}`A&^*cPtG%0VXF^#ZG0s*;I0hiuq0hQ74Z7`+QyMk8u zGN7`>{YnhN;Bm?v)I=9C9!6bee=-{mmldh5WOiMeW!JJShm`S)5Uogqg7lHQ#Oz#n0;9t&8 z?^7?pR%gHwz(A@?lt9fEjOg#4;~~A5Za}Jm28>>C0O^wD60lw15jp;CY+YWOR9H;r zWV+}?e#8pgB4XzCPxdvsA~gS7-q_#e+5;$7baPzu5>y(b1YKMdE1+o$@Vkm>d;`DN zqliaINw0tgB)kl}-oMRVh;f`L^jAtUprQzz#~j5HI0H4YYaWsC3iqEMpBv>_xQAwB z%m_o8($nLctptn<#Y4;UXA%A~J>5Q?CQ1JgkP@VA>fR7o)0RKWHp2}XTZog>tei1< z1K`6tz|lcsRMa}9(VglZu$x~?Vy&Kn_P?%rQz5zq|zq+{9DQN-%4FYE>N^n=IVKXMQQwOIQyb+pVud0XEy8J?4dP@HR4Z1k!y$ z!M_0udpea=?_ymU-GLxMFf`oCt$Li z%rcL#71}eQ+8jY7yr5%rgoQl^bsuo;b$uUo3DYfrX3Z*Z{P)1~sS1nS;HSs)?Wkbou{^u>C zrwo4+_kN@8f;EPZRbN6I4mbr_*M@Kqu>BIbcOu-WCKfBY2EooA|1hoC;;i zo2BKtDL1+SDjmdQg`yF^P6DzY0r9-D?DJf_K za+r@MBXC#GMducsl;%eLaOpp0$}ciJMSD5utO!o0R$63H~| zm!+J#=s!pAXq0N^e3GNMEa}k{_1Lw{tI+MdILS!YN2{}#X#?d&oP*f^xn|{`LttnG zA)v@ppJ5+)%aHO?P2W{#J~_r!b8G5q({A+o`gvrENVWKKK_~$5XS^3^{Ug&Ez$;L1 zX$xs;BU4Y9IA-#B|E@0q6-C?0zZ5Jd=hc6>f=*o=(QQQg|A{vl+zU>>GK=S%dw zFCU&)a!o3<(r^PNd6YT%mwya>l30#$xmBeEU7y1LqA{FL0bx)Yev)NB%5f16t?xk{ zbxm(_bJc=!nBR$O4_pQ_NNKt0dX``3M4si|g^r#{*P-{b3+XoBKEM7nYIlbMBLSX3 zPoBc0Ml6c!krPP&MZEe%)Wyfo$U&3C3!IYLKfMGL;Q$6m(Q}U7?s8k_19g8P91*m6 zB{9_z0ckL@YWH)vOZNgJWNhye-lu$B*~|;Tfg>)9CRYr=^-WY$S+wm)nX;P1V&QZO zA_FiTUwyg;mK1FfJ0#_~*%#=9KWG7jj`4>%B8w+v{=Gs?jjtU=UG$$r(dBgZv*zJN z`Z@TU-;A?tA{b-$4B!9tvp}pL8H4WRas|Er_Vf@>(0l<;lKY_VP-|2`wcm2hNFRun z=PBhLEyKZ=?2u^G_S-G_>hntd4Pb^G)4xiA{lxm8e=!2v((J`tx_$WznEfBad_XKQ zmLr&LjmKUFep7rAAp)A@!u$-r+UF(Bnwfers zuV70Q@X_DuN{x8&Xp2u*ftuW~c{i*wU2$oM2gb^OPU%I2BndGAy~EC)1318lK*by& zKAwePRUWHgH`9p8Bss=1YUah6VIlRP$#)9ipSRVc$=bBIW|*%a~Dc`jf*Or(a%5)Nvr9WImyta@|MHWLRrXUbhJ(2eD>dk#E2^??as#<$U`}^_G#| z{31xljk`t&zEyiE9wN$m+@h6z8gX__2UWY$nf9wb*N<9yzy8qIH3WttGgpOp-5O++{oI#G-!t82!D{{(GO8{HC!E;6W)0W-J(6)=&(_x(mvmY4WZpdMkEi zMdjDIcFOBUY0W5}SS5R^p3yVS@G41{4eJk~J{gVeGmU zY%*hi$uQ@^Fa%j&?=@e{o#Sj zI#`%>*!S|S?8$;p8!q_XPR#3)3d^j^ttiCqSEzaT!-Pb&lAl+=Xb|GRR7L0Q0T_Hs z86yh@xN5CpDh7qbwKH>9!AJ4qB9oG`c2eT(IE-7E*BEAkyROp^jr_?;&)!pe1dB(l z{MelC_Wk7JKjBVOQt)>5_>~GpVbcv`if7YWc_kK0L!#w_0Wa1%jynMB5o4s+ET_)o zV<>Qu5%{bZVi%jq@*?D4JF4Kbl_fb8_b1fCcK&(7p6cE@!ZR{a0* zg&pNqu!|U#Z#uz58jnXBuYiZ19(*tLGg8w#EK{S%TFDdK6&oC?1=k2R`|whKLScI6 zG6gf#GC|l8bk|U!flsFriZ*`CE`yALPVM6%2^kZ+2zWDUvBf==V<%R2CuhO0!dsx~ z#%_Q;%};P?ypx9?VP~(yE$d~X2_oc@A)7a*mHqK-q28Yd*sgSv=g;JR^XiOaigu*= zC>MDHr}6Oe>qjY;W3{gL^4@ndD4N@Be0n*STXpfNr@dl5Ex+Prl8MtDn!9&7)U~Ew z1eF%?Hb9SFd3~<;aqqRok59ekFRe^GZCH0kF)M2&@n$n<;QUZ*SC2_JbH2QH`e>MH zu^dhp}p9!103Rv{+Gy7KeAZ zuP7bun=$%NmB+a1Yr!(^=PYPnJRcbicx=_mNeT@5}<+CSaMw8xGq z$38s#iXtw{B1Cf!s>TDPc5_`)_9kP!FMXRS}!>JEnZuGn)RFEVFk}f5)fz4`A|%puTZi*Y+R{8Nc60A~=Z6l* zFthp#S@NQeuTFG7kb71n>N*2_=Hf{C)x5(R#2jQfqyCk`DRvqEb3?)6B`C4F>hnDj z+&_SLq-`jCKdIni-yCb7`s7Zu-I+S^Sd{gNhxB&!#Kj*kD{B<^i0J)>Oi-xVopdr$ zRI$VoD`($Bck@Lf-m6$))ZB{OrRe_1QXp&o$0h+TPjt@+qDD%&%K^rkk_@eswtK zy>qF8-LEob2$@fOUmkZP6A7s*q7$;Rn4wfA=7GZh;w+R_`RrEZ=lBU?7l(__!E)-j zD&b!^3~h(vlawR$D;Fk=&wIkuI;p&y=k_8W!rGk*o~ZbT*K0P%-Qnw9VQtJpJBoDRem-$#p$y5yr;}- zlkT7KIyRiL0G38wgL&t4{8#bxDs5F0-^a8#e;8jvSJ&=VJH6=vJ|U06f%RJ}9GdEP z`MZ!=d1;4DCRaE~3h`g4;Xi?r{n)V0xz<2@a!l-CnWap-dwao!!fMO2ud_g45x8=s zMS`4BR7HfLvVl03!qifoWD2j9;dJw^xXW7s1Chr|RBA+OcTQcl9*nLgsf*`2E*HgU1+?0H>RCJ<0i)d#A6u4H6awr>3^GsaQ(*tZGkREINq4ZTGk^UUBCej?-p z_zFG+YdreMhUg$c$Wq~Y8$m@dByJdXgx93vp(aagixhs0rdm#~pU?wFXx2QgUo`$T zP~&2}@c#L;48vMPi}`9)@uhc0g;8(8IEYfxVrAmzXQMBcy3|ZI-E1-`JwaJ3cOpkI zl&H$%qZ$_{ev77*zmp#*3}%6Oj#mmNGxfRf`emlfh$!WZ!eFkZUHZ|(d242Rcc{C5 z8Z3o*xr|(>vvYWvVHADjRMGvW^O@3`iQGD9B0j6xu^N{Ump8mt?*vRL6s+%C)4{4{ zJ6Ic9`vf)X5{iv6W^{-o_UVr|MJ7;jWG=JUZ1^J)3c5>zE?y6_KK51Ke6n7gJ8!e0p z(;}wYxX>^@l73+*{~#io=|s2aJ_2*-JyACqjO)WVqo&C*C3%Zq$BnPG2FY{RB(jq{ z#}lHkZ1ARL$T}hGH&AsZOeRg%zl!ETOI7d%Cr9YX`nWvfvw3Y^8HumJ%bljwI@;_&3xf;>rl&e@{{E~0V1>x+f(Hby88?qAdx;1Jneq?E|sNZLC=&o&hbfHdq z?dO|dxo*Gqp*>%gci=JNT!xJFGD==avolYj*h(^CdwTgbyFu-U2iv!3JhxT>?oO^^ZxhY5YnQz?32C zX)%Gt^Ncdu`-w#X)vdZ zBNn>o%bZPz+{AE|Ac8V2t(~(D^XmQ?yE8v;k@}a^uCoi1n7p|Dw2*hAr}MOtFn*z5HONw}DO9h~B2suj z_`S=R?{v)!ba#$^vj=&`(Il*9yavk?~2v#Jp;RVtxhFkjF^8E2N;plvV_d_P97irZcBjykD@%@~tzJle3B?01 z{Gc~J1RZ={_BqZij>`$;a1$?ig(7u;WA-Ceo=4Se5r6yjvt$j?ugHTUtOIC@$Fzig-` zLE6XJY3jvu4E;BG?$fu6o0qvSmdc7*JP$B>0KJ<`5gbs>Y`$#5CETji(QNT+*2(qB zXc@g-Xh^4JJOkxso^G}hcaT1Pd(6>P@N@60VyvI6ODr|7GjlL1F?p@eBA7S)v-R93 zOsQG;QoLf#J(^t=1|kL)#`cvPqA5=ny#Sr&xzDt8JW6r$lZIyVSy}(ZQNYu-oHpNT zt%p~^-z8a>zN0ZW_7IVjPKLOMB%|&&pr4QdNj1c}E|xF<;F$oe=-B>i2~=>eHr6AOlE&J7YAS2W;kok}UePBv0$)eAZ1X(n$MpDKP{jvXVk-Ifs(P zaQ!)?QXEixNXg8cIKV!*-z3&GwR(#~g&crQC5l3|INr-Doh?WZu~Rc|MM2^&_yJwa z;^8?k=cB1_tfSHXX4^8t>%R82_K`(t=!-Ior~0r7NuFWp-(j_EZ69f8h<6)OWD~b= z0p6~;a?D2bOWUh7d>evt0%X1pg&wB%{i|7T`c~v zDIdoFJP<4Ib;bcFkFSw=#M9~KhQ zytKFbi-{7v$)%c!iC(E;XUY=QrFUKg+eqw(NF^QEak{kjfyn7a;Pg0Jzw`U5l=PoN zU%?$5L);t<-Ch)(`>HDwEnvbLooa!meZ0qON1&-RWYFG0Rg~XT0)F zO3{>;T8=#6o2F>tWs>m~FOtoB{T})m-u5$2c!{Mm32z=>FqgE<)1p5omxNJ65AZye z3Hn$Q@=+Oe<|gkQ`pxO_9CgaMHz4yXQ0C5FbYx`;iKGpq5+Q5k@}p)Dy?^$ml`$Q& z36-%l19l`7SR5mdd2#{{nM%=Fk(8Pb3*YV!yj9n};)fO1dN>kWDL;r&Gaw0?8BPp0 zx;J$90D<6?ruE(Tk-J5FD+MICUt>)=$1n?QO&hFEqlY( zZO;v&!BoOSM?TBmFYuYEX^9yPm+sCw+myOkYj3Sx>A&HiI7-KV=EhEa@>sbHR6R6x zW93_4=pWIOd$ zCQo3O>%UvSa_^(Wun<_ce=A!J);*GeG`780XVec;>@E>QS2EK79@?kql8GmFp5lhI;~hqC zz#r$DxIo25U-0lp6VsI;A?;{0|Dz<{EpL0m1sZ9kn8p{$v*ZFXU@&~c;4N^Dm(MK3 zgpl0DcM!&VeGHp+%tIG(p!1yVqBe-)$+|w6`-bn!sNLN4hbSQw^iyh^%+f{eIWIhr z-K!pc1r)4P-eKN0G>utqiih|$T$t=>XEAsdF$7OlhId^YXZGiCnf26UF~57ETAh@hl$L!6~n|m*M~nE zn?BsUW^G9Dkvs=(2h+yg{&zomqE~~ig11WuXSZ?0YRb*$!`F>_)ylrh!kAL|I1 zRcoAoAiH1;Sh>6Dt39#DX=>YMUM6Yp_#>yl&b*;|(m(QXr~=w9f;!UI=-K$JDECBZ zEZY6m@zZW!ey)NrEDjg1#19rX1(NBxLCQMVf;L>aiVCi&@eB&{&PL0Lvu`Yg6E$Mauy4;`guvWiO+tlelf>DS766z_vz=x z{B1tVj0cR~QfWwVOn^hK7-A>!PjJi@ArM0Sjg8j!eY#+37GIGj>6&qw=cAZ?&#beF z^&Au43>g5cPoq4st$YrGN@gaE8d4HOBfDzNoTs>5o|83lSnrrA-XcEzm8QdJkzpj6 z-?;ka=t_nBYL#okT+ zfo51iFRBtQ7ly2XI;7Kiuqg=qnp*Lay)}Lf3@e3c z5ITpI0l+h%@@=1g*e?ZO1{)6_eET5*2XYaek7^p>xn$hODz&Wm-J`Mq#+3F{XGyqV z+C%XgNv1bIt<($Ej?P>x8UIx@FKvM_(a=f#ToOs?=IEU>UY|aDW1_?yUy;b-!&NH= zshMobM2ZZtC(d|&{x*hAd-*`lyjY){7jjI_8&fW8aa1V29WN@(H~4W#2;m_!qc;o4 zRYR6$I=JMf%8Skqd}G$KDeThX_>79A8n(iTiA%!jXv(edr}5FuB(com;BD(Rd_`CK z@eQEch<${WxwI?HyMg7f7q!BwCqY{gnkSV`Z@7z{qYta`U_^tqy)%t(!;}kB&em`xwmt?6fbpzl0U%+Ta0eZ+wqc`YVX(_(!nGNV7yQ~~vX zG9#wBSc*Jw)~ibjyS7jFO=-Gx}>nF^#; zLPuPVI|onv266!mrpxZpX#M(eHzym7f%1!I;nul_WkRYkN9eVwpy6xz_%wG#$b09( zEx0xijC9ZS2ICWG^S4sd5LmGaJ6#di#`2JSE7VRG?NBz&x49LZywjU1bb9XE!k}u! z`W~1{49wYtu-I@-Q{#7#V?H><3&XW`2y>=wWP=w%>hC&AQA5=cv^aFKSw&{F#pto# z^xQ{4IO0X2=f^i`DV&v^(Ls2T(Zb{vG!{Zod+W8Z;Lf!|C_(COvg1kp02-16O!a}_ zu_s2XDyuelgL!_jV`QAHq-c>(>BBygM17Ha9G@Fnw%5>$f5cQje4l zlFv`>S=-NoxKSXcnO3tRL|CX}NeW_ONZ^29KXD-WTaJIC&4b`f{3o@letj)JvJ}un z32syKjn~ylXVV1Ep(?fTnpnUGLx_}=^rUp#`{#j&9uVK8gLPYQsI7qw@fpBk{h#i{$aBPUCe*I1-+6&mc*mwAkMeg3F zB7taE=4z?!XPae9!v5qh19b_JFSG{Em{=3>6vm)^<1j-ZKYPq>;f5h;8Ty`OU>)u{p&x z0EM6;R7Ts_8`_aFEbTy1lD}Rrm~zUT8=BIO5#<|7P}sH^E|4(8Qj;}udoPax!|~nY zyiJLCz_-h$4c>((S;)d1vDb(uB852z$jE)&v!p=_^*-?D5ojR!nC$gnTH*d zaL6_4e+j}Kn&6q-f*F?pfMDT#al7hIe)>aaziCmfA@o! zKwj5I{a-*MgTCBHB{QGE!M49WNP>esQ1S-(+9BOt)@a%rV5{@k)kymY`y~`Pb47;y z$q-+-R`cr^8dG%o>U#@tusOakRG8@EPlrdqU?=W2VS+HsMPASiN{13fv)pK?%+eg3 z=0UF1T1*`-q7qgs_rgo|2F5&y54L-$2DHn-9ra@2e z0#$?3aa*DYv3mL2{Q@`Jf3Hk$mzY#Ovb|*Ev&`6AyntMN@ogvMG+eylr+BL6)ZY}N zKt_W@pCsqYL_c^6Ev_ZQHc?8O;fc0$qmp%!Y0x&IYC`zfDX;#!fB0C10tA>Fa=trA zQwoOEIKJP&RB!!?O@8BEx#)wkDkZsqZ-o)hmW@Rx@VQlb)+|4U0`#nD@vzMDQGSee z`sj5Cd6(8_yQ6>I@Z+hVExFQp=oQj6i+UXsPl4?LO|Cs`$tvezzPZrUA}JO()K@|% zD-B{9tX1wBV&}s*mqz%Dw`Z)Tx)oFd1>6PAYfG%Z2wwp&WOB9U&AGYo%1SF%CqGK) z-Knw~Y1@Y`IP#Xc8mM0{TXseR~3L@UV$(Fnx~ z_dEBaKSTSB<8-3yknDp7_vZ<7}nyT&Np*h+T1sgpCTg<0yF& zC;XhdkjQyETSeAL-zG*FoTWaZ%p|jor%Ye6+F65*@(eS?5zkvrvgd}UzOeqV3zH7# zBk$a5)L5Dad|MDE&>U0??8DrB>rESxtQm?GyW-&xlAHKnX&4T1&#!q-ZDFaS`SP@q z*R!sDrh1FdtmwWx6S@(xamVZS(QPU2?L1dy*WUVepM)gGHe|GTE6mJ2sFh0jIM4A$ zD&~A3X|v@|jQ<<*6M|Eta6qvW4mTXNBP#xCrpOeVtijqrnsm}Y4IF1PKIDX@)@abk zx3Vu>sSx@b9Ws&VkU4+*`2pFX!H!?=+Syis5q7%mPMGwY8pkPz37AQQF6PHasxf?7 zv(ov)9;+XYut?BhjTjdb?%n??IWj1E?2Kow zHze8O5C6VyG8%;ve<32`W(F}%PEOvL(us&jQZtBk!7}(0{BWkdgc#Jh;8^7(zRXTI zAQ==GNOKR{jbHiOs$);Vvz<6T*9qo7MQO-q$ELBn|6I4{5+HABxJ;NWZRIyZSQ@?0 zr0$o8C$;e+$VbX7y}l5}-l>^+a^jGSsLkl|Tgh0>+iVr$I_{~+20MXNIJYAIR)Y#C zW~9R5;5~A%(6>&Z5K5Bb@=QxOHxoUwh*bROGOOo&pp~j)$npVWmPn-Fs}$9a3;nZW zZ6olMEIIhCTbdjBN32Ik?o0bsHbE-5pP6!}NDd{K}d6M((wZuH;qRz_U%KsqOXSKMjv^EYke zx%R(Qh_E^8cN%;-8SEk;F!chBW;vcO63Wkh1b&J;9Pl)P6Au3rm42XkAnGqLiQiv8 z*0&@ua$~tZ-(P}AeisZqG^GZhN39_aGzC@Wl9hPHC|@vDt9 zBOZ}|Lk2W1!d#3tpcefsJxar+b<60o{7AIG{=WFlg>z&OBK-U8SY0X-*#axxxSl^6 z0u5Fa$!t&ZxqO{_+=w7Sd%JKQ1c2Rn=TdW_wOkav&*|UdLm?V>+07w?D&T}Hn~IIg z1);^2Kt|8^UrWJ^g$UW;Tcm(RmDyJl($LT(eMOmN4$HR~=)ucqq^C(wC6mqt_#T>bd{>NThIZlShG@9EAsh)*9&gHetwWIa z#vY4!D+;4d==qW3mFbOU6Uts+A$`bfQ-T zemk5y8mL?OPp15{aPI67gm|R;{`nh7!1>QFEFF7r1!bYZc+-n};!X*i+}vZ|SD+7_ z#HJiE`iX-BwoiyJC{4ba#O-GjDqe`PD6Wq6e_pp5>U&c}ELZwJZHE8)`TM_1LlQ-@ z>kbxA8HNS8D-i^N+FNFLPWM`x?!36SO=$T3k_H(dykf895fGUU~`3| zIgTHIepYNfcf4eodVljDl=?R`{}ZF*70|Q{^4IHkfUigNAH7yLAdRg^=}zZ0bEkf9w1*1O%6zKTW*P8~*di|NenW z2d$MVc*zPDY||Ri z9Zk=V)W?I5FR+Z*L2Y`D4^#5lERGO)Qq~D5Jr13=sXaKB3aCm|;J7M%aI)YY>pUYb_x7ct?t=kr@d8;7Wjc($)Up1zyND&7vtrc*Ij zCL4@?Qur*7N%UUlL-r%oZ^vFW{sHsppx=sKT+ zvT1kj#qYhqr$Dz=L_(=#Ay`%VNufNuW&GOp+{iw_S zW;u=m*b^`|oSq#n@8QCu)ca61kfKiHOssHe=OJJ@P#lAWT_n-#anoRl2PXOo_q{#Cd@m z{c`H(|15(Wl5=lUZ5#wH{a8HAk>LV8Wm9aa#HGul71_WTsvc&N&ea|&Gpo^3yM2tS zqx&7eEMES4k)Mow@SIm?;r}VHYDO@Oey%@)Tx###6UXG_IeLCf61?h&qH`YO=e4X@ z+=|~T`RF?j7ty;Lz_4K*OoADcc0Q806nr!kKKaEpRum=i=?QmboNFT*jw;-E2Y(#OZF;jLyGl;WN_W|19J9MFh#l?Ji2>9+MXYX%P(C+5q7hAiQDtUjgN29SLG z?l}cP{%N^tA1gfJ!JP|2-R&rcib<3*5&j6SyF6_<0xyW8-i_Lm{iKzm{-S(WGJh&L zv#iOq2O-gUGvsRaq+cgN@>l%npSlq~9}^bHVd>9b1I~EP#zR+D!ISsJdLlNcNOzQ6kpc$8-07wbK+>N{7&fF!+(&M{4)?H!K3gg_Q@pM>4 z72pf$?MNT{g1j{^tX1J`dAms=*^?oEEtRWoMDryANs5;h^?AqY~wdjjA6_6^Ackxh8s;whGRUM?3x^fQeXQ`Vz&ri=a`@MX0h1H6YDq;@qNu=@H1^TBPpwML_#bx!u zpm;$`>2?y52Mh=TW9bJ$0G~YoR$pekQlU6c6) zgv!8foJVQf!Q*iw(1ujt-d~vtFSqq`I^k&Iwz7k?-F6rYomPEP>NO}8$uKQ}cY(Rk z%9QreRz2-Dae46RNkANV3$BF1bh*Wj{s5U#z*H9>a zNo-{~aw9Onqn3-g4}9{-`ZryD2*)Frr7ojXdjrr}nBw@%FOa#NgneX8u2_1bGmTI6 zA!@3cCWV+C;awng@(JN*E<&As zVXbG{a9@SdlLERd6EG7H>J;rGXTs`QusC0(8f(_9a_kp`9CeNFQCPZyA4yI!ynJe@c%6oHCTyASS=VE zO)^1Kvi3f_!h$3+@@qMqixj}x0qfU~7V=uVKRA8WYLZxO74|W96k+zR+*gm=ulHaZ z{~z@}do-SvGvgc-rE1Bp+aO#;jYI&jsyY=K3vHENNGMFbtbkn-LNr}vG_|#ppwl&K zZ@^-U(tIvIv|{npQqQhVE91-NMBNa zE?T8ENcWKnsR6@nNc>#|dN~QA-f#cxFl1+@u?2rkZjYu<1d2@)2J_Hr1QZpto#k~AIy<6j&+FYJ!8P9bP zdVJ|-umK{fhfHR<`MBg2L?JX(6f$-v<(Ks|4yzBSYm#AR`4SWk(oD4f1{MH{$~XpF ziYa27YQ?AvDB-p}FaRUW{yrTZri8kC9#l*kXO+9h&jMgbJ$*+Fo-7Dq34`Wi?&E~briMroAhQ~B!W?i&-> zDy!R!h*Ux`_TnaiGO5ry%y?!*16CUb6)&~U4}9_%DoQi{)eI5OS&jc>@8HvmsNkfi zP2d&~hGR^TG$dHiplL$Qh*$xpfKtHFI!T8OtHA?%DmN>8c8LWNU1x?m2u?xH>y%Ts z^ySO}_0ULD@4b)!7|jwoo_uL(Y67aI=XPPdBYgn3*?1IjUAY8u**H}%(7}R$*GIsv zPo>0+C)*L~LF`=k#`eJXwykx=dyzq5sjJR?)+I|^6n12GEP%E+Aj+)kPgZJIPZEyhL*&+w z`3`d)y)b9XkFZN+lb}ZbJg6P7z=^;ZntrYI?@|2~vvV(VfKX654LpX*kDqp5_d%~4 z2?B)u@v9tO9z&)~aD?|T)A>#Ch7i-Y=AtV(enHtY?fk7ULC@{>;7DxeS1+4TRRS&+okj7kg`rN92KX0g&U1xB`87U>+BX#8cZ491w|l zV0Et#qegS~?Vw(>7rlg2un`ajAX%uF6Y;@nP-+Vh9HUYGw~41-_P~vDKgVFB92vyW z?9eq%h8DOK=IQmHRd(`XrKFtmo7v0FpbQ{9tB~3Dil1N+QCkn@>l9x%t8-f(inV7Z zXM+T0(-RsQlB*^Do>-VjBq9Tkm|snEu66rAPZ<2U`vTtkBgBw109GWPG{`qOCi>8L zD^nFocUTP$WS<^j6V?op)p1Ml%v0Z!k>~mli-SU*kY5V8L2ZuCq8}jb(0U;2k0v^N z=!gIn1BG?M4^UL(uVb=#V#C3R!Y8LE$!)B<5W35Qx!3fqjiMXKrphcpf3aLuP{$TO zF%nsOykqt7n5<8e+hoIHncFXbS7b^y1tD8ySTflphH|g4w2F)Y%If9I&nZik4iq<; z()LSi9J-PMf6Ghc`Qkw$#Iy}`U(Y;H0zF5peop6o1-Ylz)3~)R8y?UwiZ6`UmAlR!oS}$!CNO(Y$=ltES9P@Rg7y%sxI?_t+{?{v8yldg2e zR|1KlFkc*r#2PfVU5x%!?m)qNO9{0amkRQb?o63Fx|8}U))Q7W&PIr`6h^pt^rYB* z&rqROyB-I~ggHh)0ZXPtVH;ARpQqK3Kdo~YVyYSvQ)M{1$65Z3sewpL4V+KnKs52# z0-oTgMd+;Z^Sf4uHs^kYo^mb(AG4aOJF7HTpKqpUmh+Jv5hz$ObF=AkM~YinqjdS8 z0H^WV(u{A7yI?$M+1s$$UxSt6i>Dchq%OZFq6`grZJ}Xgq-2#DkCMG;TGbpN&QR|B zGK{`x+aDLdYz-qRw?6&CUHzp>juLq_;srJ_6;=n^7&RI_u4JYBZd zqh0yn7@Hgu#v2U#`T3yON8->sbOq`r5N4j>mBdH1;(XQS#FaT}otN{Hk|a2j9jMb| zZ#TQsd1%6GtCwBgQ`}(uq~6SgU~iIW;5tHDI`wY7<6ruZKl%LS%IvGI@`~J;B zDKr_9D#RTMte!U*c{IK1<#YqSeP@o~-I{G5m`c9Cl|KZrhKZb%s(3`xdUX{nBBbf{ z)9T|^xhrdC!+7{ESDBMB5${9@m(=8;eoX!z-(kF-^5Hc-a4&P^SrN?MK3^VgU#IVW@ zR@cs!AZz=9N*G^2g6IU?SBIUp{#V)y`p!LZ5;d&T4S0u#g-^3@gwp~TYlE4sRWX`}rW z%tlvRF%GnANH~~)nByBqR|!rN(+2a?71_X)4CgU_DL1J(6_n0?H_w~ACa zzGT2gY0kBmUn_yk>k{Z6z@qMIPm$@9e9gy3C*-f2+^e1ty7;0oSLgUv(@S_EX0?xv zK<%jk(XYhFV@iRQtDI8Vxb3Bw@?CaS@3O0uC!|;b)h5I33JV!ZPpvVRGS7Iajl^YW zEM5wNC-EhJ^+4{N9nq%J%zuo|K1#xCJbq=oR%hgwx!Y6K)VL%x`3!8MlSnrW%F&rry2TunsaWcV;Gv|1HRooqNiRic^qhKjWza z&t49yjJ)?cKLJfP(BGAG9ji}?`hhy>A9VR0A{v2`JrCw>E2YV^bqouk1_!yF{-^#d zA&3Y@!!)tRM1~!0je6GPiH4zRCiRqGVi~`XE5x4oFOvP11q!Cg?*UUZIn@lnPqU)w z&CcyGCt|i~Y2Por&^;Yu6jLeUiBb>-HvQSKy+3HnDvbIrUq|2)nfHgBVbw(EU&wMRx&!_%_ar42mgzu$t!Ep5 zY-2K(v4os8))l>xa0Q4w+qsi1qms87V^#VVk{SUo|AcD+YczX(osr-1*yQWVIT zm*SpeH4Fds0S4Web`!28p_M>z)l9tRKT)BPltp*CFj3tLlFr^u@ z&|J%{Gcc*acE5z=KW?P=ZN}{G9^h|j=A-El@Zv#8)sPbN1BIEFd|F0%DzJ`Qfacb` zqX~@Oe3^V^>>pwI;<_o>QNfZ%O1+~T)@x+3xvO!UPOc*sl-4@^UPlE{b7Y zGteJ9wcj7Z;*jQ!u$90;V5A~K`T1YsazDOkwA%U^a;G@`$O{D#`#oc9D#k5=-&nAY zy;X#4mXC}8E%csgqDA-Nj6#d0wa*ri zAUeAud_?37tRVasElQW90@6x(`4!+tH1p~EoTEpJ&buDlh!rBb%R*)sfrLb<)%LN51wJ0M)*Ok^h?IilbVBn!iF=(ET2yo zZNJ!2PtfYa@Jh&{UN<#-z!Szyv}S>oUR+0!ieQaeDdoP!Yu}U986g@w>%X2mC{NB? ztwtx|JT#D8qZQ`7g8``-suE!u0v=pggFPbvo>HkRriVCq87gOfl71Crxw%tWCMVx| zAdH7oJV2#U`fgtEPoT>s!C#KybR}*qp;Py7bl?8==0c58@-CD&nq$=~zJXTU9v~>KS6pj1 zBOe@ik-yT`!6@$U;1&`?uw9%LZj;iMogpIZ9b_}GyxTk@6)1+4j{f^+a_Y~jUnVhv zKriSX>Bc4O4xgjuRs0O7nGMHO>!Y8`mt&FfEu&56K0M_J&;yt|cHv#67G78TwaSg1 zYEbjAqbXWDey3L)5kX=70wS7ov=LUI-z@BXx?Er_?lO7PyHwDbs(mE^7K%2>3xr(v zKMUVMB8}}m0C~2*c09fJAqqVjbQ*T=F33EuTm^kV(oq>Rxmbo(rS2HN^tSeM96~T= zq=&RtTwa3-s3xcjK0rrNToGH(HdiZUVG3!syqvVr&~>4uc#3QS)q-Z7sdze zv-BT-|9+Kh?MH0LQN7cX-idmy>2J$$l>*ZSEgH=+fMnH8-+eSZQmKQm;E&n+-W;4g zG7O~@sZs=DmQbjnvA)_uU&eTQ9by5a5q=)*uL|4Xqi?KatI_C~@PAu4a0-(VLc^!C z^_r|v#fPI)truZ%$|!FfYI{vYk~?D4zS?z{gA3yG4JO~}h{{!(%l2Pl(pH?WUmIEg zb9`C2NN99e6F6$`1(*o;+b&&&{(Tq7Xm)*OW^5qr?o~I03LhW)T2Jyc`E~&f`>~Z<u_fQnn!UbzF z)c|Quc!6zAihmK!$F|;+COHO;)Ckl5ye&p&xmTz7fpIbBr52MXlPpD&(8c#dE1>Ms zZcw1BbZ0RUsfUgmK{KxE531Zx1JoKghVx&56G=!*7B2QVDgpj6qY7{I0Wr%1A%-mmWgUF`{wVxJy94~N(OK1;jFhS4A1lZH+%5DhB#le zDY3S%zw?oG2;EV#-#*wi+E9|XQS)45%gwwrpdC4-jOl1@aiZ2%$y=Wje-t&U zxYhYtRsE&dv1?L{Vyle#`Ui*PLn@rn50Fb{_s^&ZdK#MzI4FM)nP}zC#9-+eFxi|D ziZj-m{(`q%h6WC^eu_3|){;qp28jMKbg zi-qXIv0~C~*YR4er!!D7*1AnP+ge-DJN}0i!v4ml6%LpT1ovrr0x-9}X^}nz2j!XV z>;p*BlO|NpnBLPFBBl_tuQJVb_-4r9II1ev*TbX)e}Lba&m-koO|cD6+-H>DJ@tB= zL06r8%3SiM*|R4AJ6!b}8N_f8ZK3?#3jh|hYV(uzl_@uN;Uq9@76|vX##I6l*{nnm z_8oV|dEg=TtoN<4Q#;(apVy%Xp9SSdGUN|3S*!cmsG`2L>B#W@w$o88OqIMF%;60FIN1#( z(9V~&8?Agv++-ko?SvVS-8+OokTPKZ%Qi{j0}W%!)|0C?PrpBLEVvp6IOVS5G|-b# z@?Xi_gQkCP{eN_Q2RN4P8@EI#GMdONWQ0=LtL&LwWQP#4H=z_lHksLbMCcI@**jbI zNO-Jd@9(_z_Fv!k9Y@D|Q13JDbzSHATj$$TWUwt6W0E?(`Mb(akLfDp`Roc!pHguy z@P{q$+U`iu`ysXA#{OA)j;`4$_EiXomstde(A>FqHE&TG%w05<4n8w%_^CfOyw>t6 zPc3djU_pb}xU5>h*lV3v=6!?2iL=3Vz(g|c=ykc!fMdF@uC7d(opj*rvF{q2{>4hu z&`R6jmV4Bupd<4&k{+Dx*qcdAxM!HSP9z3wiX~2>miKhv;UWVyNxT!N&xXey9$2oA zHlZBc!k+qp3sGkMRj!nP_0&rvt*URtaCpRrF5AmXT!m$9XVN&>;?TJqga*kRi(VFw z?xcXC@Q`orYE{&*U#Chenr#rypK=qH`X!nyiXOx&?Eke61JtrAPe$;3gcJg6?GTGYmmFYL|%$EDGbK zWFFJt+fH#kR2m6&;l+auoOJw*)Qk%iR0>bpqPt{2CACawif0lm*< z&LEd$TPOOx@9SBDe5!vLR*>b!?%Xf(_%l;QhIcsb34#?@aYyJGA?pV!1`n>~vC82B z7U+G*BC+%tdldn!PJBWzNZ&vW;hPBBeO(m%cZFs5QVTit$~t zJ;?_1Rbv$zasCtQ*e5{*SC)mA)DZn%f!9e}Fi!Q%jG zK07HSiH{jVDh8bpwa=r^#Rm1Kk+1+*+8Z|>K2LmvDDrLCNCdoqZ<}Di5iUJ>@btbg zj;}>$^EB9v5vqCGK!_Sl~WvXEg0Vt z0MJPf#Xtl!-R8yN?l056wZM;`q>X4K5OLexdD2@-5zq*CbZ=iqTbCI038@P?F7(Uq zxF_Gx%21%P=tk4DtnN|HfOHJ@4*VIxD+HaKFQ;FMMyf9+=w3h9vT7dOd|W!ydi{@t z=zPIwjcn$-4{u7 z6%+u+)GY%kr{|pbbqK2<_QLap&6?oejcjbIa}EvbaP?gT5UEvYu*mh5$>+kg2w-xO z%XRaaC&i>kg3}a%*K7|MdUrPlMc=6N6U=Nc1&RJslGu?*QBsXQ;rOS_r=1~~NIS1% zlLzX(QZA?Y63A7X5f3XO8JPe?d+T=_SLd9LDtYTod9d0~)XzW^aGtHv9)JL6)CInP zE>I_yAoFqRC>|oKgQ{$y;e{)kR_U|t_|o!MF&OQbbW7D zvB1|r8$>5jNYm2Ljh0~F_k|UP;A@5`h_UHmiPqQ+9!~@xl^S8B>*ius3GPWkDQNU? z4U0mdQOkBmye3~|X~VakmnZ5&AJz-@ALF;xZ%!D#C{uIFuSxcSDc-U$Mp4J}I|w=V zwgQnSugK!x**5S7^zZ(bBq3f%Nz&=c2Q>gK33fr%u37TRMaO+`Bl6z_;n+ZKXZ%wG zu+&DlfV5qHy>hO5hD#K?hU{161Nn9Nk%fHe@iXmWiK}vj@~HlR@V)9SG7ZoN#a2bt zrtfvyi~O;NXku>%H#X2~9*C3>~ z0(A>^Y3rBdEvx+0Z(^>2#Qe~BeRSwNnUoNyqeI+sov;1bzb@1M0?E#M^{&gWako;T z+zAB`>-6J8Lxv6{-3Uqr6@L&#se#1lNp8b!0vz!qr`4(75dS`p+10UWfn=(b^3Q!n zFRzmlY}EkK&H#0DcOZUFH==oLj^iy>=oE7zh=U{yzZ(aLr`B(JBuj;Mi`cYtv)I8o*Q$+=dh*UXAm?F5B&;&_&dr^Hw8Qd%cpdU3HT9tS00l`ny3= z$t{@v!9cv+Utu>t#o=A}{LL9*zyWpu^pQ_EKS75*c@^E<^4R zn1DXK#nquxy_xT;zJa+LKCLnQ0}u<^9sHNSsn0fYEdq-D8$gNcB5tQ1DMJz&RUE9# zvcO7&a|+>;^FxNDrw0N^B$bY8@_?!+xBxXe7ONPk>;?kRdH)Z>xQwtcWe!%D)nQi<;Zd^lBh64aR0hM(R zUs>1=B-mQu#1Y@)ye!_f4gSL&qU#SXHe4ccYy6#Ob}zKbo)N2NF=>blw+mEZWAt1$KLK@Ctrh znNiGaSebdqb{{(*%J{Aeq-WlVTt^%z;kUnEBoN~~Q5Ee!5muWoPK97Y zg~nj1k0E%*3B~uxW3bdbZF<)>%c^r0FPH|F-8=H#LsWLq@=8Piws z#4+CYg>#_kR>Xa>is}C8Ye5t~bSED(Q4*2E1bc;g_ur7;hcjC@NBRS9QJL*9Ktt+D zZ=5|Aqx|{kTmo8w_bd=r5`_psHEx-~~%qGcz)MGDWvA%Fr@4aQgubF<* zgvD|8mspAnaK=xYlDXi+$A-V0dnhPM7{JGp34h9xN%`n zJV+NA)b|>+T-xeP{@0!6%FnL&|dWG zQinBh+GPT5Iz@%VXC8h@U3|GF!~@0z#AOdsC5zLEU8&ONzL%JWt-}N~=24STEc^)y zZ_#EDr`Ub^HB#sjqXVH~u(tW0mLYWn=FBmI6uppD;<(^@8IScP+yf%2 zYydrK1@bn{yldZ+MmXtKYao+h!)vUP_n-X8=vH{rXG5!;R`I+dzr6X zzOnjku(-%K5I)7HrYh8Y{U1%_suV4#&|4ho+!Py4pu+S1^&C-`KCgB<5$4HBeM2of zd>!lqX8pP3!I?<=3knV_V*g)}Z*Ya2eCbLs09WkO5rFKL9-Jo{3HsaG<6uq_&6N7v z`s2ueV8e^d(D}IZfbJQ9x?9+sW^-W1`^^hwv+4#lhr}dWO&{kBL_3Ey*> zB1U#yvm}%7GN;H+k`KNKfzK%?j&{h%6x)o6{}Vj=MhJV|o$B90*5l?riQq|Vr2H?x zi#2CRC~l?hfB*P92`(2dME$(U==cOuG}UFWfPQ@T{Z?4Q@nVj=435eK8~;Y^}E<-8cE0^M=fJ35(8nu=q7Gz2*vAuh&)9REHa$css^K@oIoBNbtli2o{_ZW~YB`JcbklbwK|_mA#-4k{iXr;_927l%dlMlpn97-VJq zCl~#%$NTROPur@^|Ni{ zKJ?u?lE)|Mvim9=txUm1@S58mD~;UlY?rLNEHC_Ur%p zkbe&6|N8PG?{JV%I#|;lz58@u15&Uo&uAO^Mj`=8-}XJhn3Nmo!fyn3-i>1AA(Fa} zDv0*y|Mk|$U#9Y$@QsmLRjEGvU;l^553)k8yZF6a_kjwzf#(dWFImBee_iJP`-fF| zLtv5N`2YIye~-3K5YUji0|pT+h|iI++>@fs+KL{! zaG48(cdvh3nt?efdlMqVwhgLi-a>k8P>;me8alWc?b zAq_Sd=wSP<5yL~)r<^Jkk~D-^&Qk#e7VR@aycJD!h%=?N12GGL$+6#cLS9R~`Hk+P{!P%%GDO@YWHuM##%NGbF z$NtdtZ^lXi=iZEa>-^Mfxv-H-)upk1pvmy&MHW@dJ>m zcwK@>YUHl)Cr#Qz{YpY3xpq+MiryImAzREH^GLKM2(XX``p}M+U)nvZb>S+>5V(RT zuOcq;)`lW)CK6tU#po=agyTu4(u#i*#;@UdA=5OFu?e8vKfV1ty4MY?v=I^C!9oxQ z@->{1Q>@1yE=%o~bu%vLd6CeF zCg==uv#*o^^N!T0em#5sh_Q(9E8`6LwPWPOaC_!JTwgJkZA*WaJ{y7s-VsZF0mL^M+TI} z@Y!dl!qb-}|MA$qSI6cBn~4;NOpt;#{ibnqs5s0mz5#QtqM13ePCHqd_UFdz!!sKx z55q_`Y@q3S^fRl4#$#I)Nj#8*9HCqXEpW7ahFUKT!X1BjnI$id&2R6)XbR;bjEYh!Qci08eh?uWqO2P=;FLHY}f$tTikjM1CNV z@)smWtp8Rr*O&m$g2BtJb5@3!2_I~GH#F}dBP zwf`WZN2J?;0^cAae?SjF6s$dI0**PVS-;UTbb|{+pK|i(PQFLteRFQsLSc9EE@1cg z)0_|kH{)}34)m5}mwB8*5Q(emA^>6kCKE|oah|eMx0*y5{%0xyKI;jltz><{=eO=! znzR*y0ll-52ShhBsicole=}fcKjR}5qh{$d(;Z>yrIP}uYRuudD1ZDGT zjzrDg5_f1dXJCC`6#SEOBaOo;8Z@f~0e}~-vN=T|3u$Ot0C>)x3!J^Dh$yXsjIi5Z zp4bHp;Yv=;yRY&S5gb_`Aq1)ZVQrqUwBXr=e=utiHl!tDi;sM}kq1KeW8ZSvQE&*D~Q1L9vN?$O}2LQ*eCySVMJ6D)tVosIYc2-*iTmTODZh55ie zLHPO5yOKtUzaQ3*xJ_~s^@ENzHh=&284f(0*PcIm-|fwLZZZ^l8rB@;B7Z4XPGcU| z&0MG?myQN2&N(f#I9pFB-g;$u`;8QH+U+pLZd!4AvSiUYiL}S}8ir#hC7J{rLM;97 zWj+pFh*0(o?apoPrDa)zk>0aqgNWt|QITFixrB+M+vwjbkjW7kaDXU|!KvhAHm(y$DhhQ(nvRNnbk%^I>0l z=Az&UgXPl8Dwix&5+yf%o4(n?kdQPEkJW&A$ogS7KIy-b))Zt76D{TB7l_~u;afbo zNYeJeK}q%n)}P-+^pdJ!5L2YkMwTMc6|4MU62*`GKIHU*76R=Xxh8_<6PS4G2J%2_ zgM?Q0if1W0xW})%oFZ!KC_EIcapox)TF?*2tPc(;^EyJm(cRYjD82g#*1-_J$8yn$PPzOpn<)p{C7 z+BQ2zW9nx&FBaZM?TxOCeJ`2Xo2wA~*{4(BI=(m77g+B^m0q{vMYBJFHxDyTGCi2;?7A)t z2(CDH>N6_BoUG2AB|C_uO~=Rao2TBc#!H8&W^Qq;A4wA-A1;PU?v8C$Y1qfw*Aj83 zjvsE8G3OfaPL|YP)FR}MO_&~q2exmsd>OUS4g0)+{kc0sG~9}SZ`8JIj2|wIyL6ac4Q~pjQzM2ME2Wh@RBV4aPy<#iY;&F?Zdmm5W5is8Kl# z>T3t^K;Tu6>)2WXgU~6riIX&Q?{x{q)O}0uPuG&?zA;?w*RqQ50O3u3THW0sLzBuJ z|NIcfC$vFbUF~}Nm16ItN_Y=Tr&nhD6x|H`*(6i0Tkr1=4OB&X{0>V@{k79q;Gq_8 zp8v6hh51nYm+V9K=HF91n*qA}^an69ydX$F*zL#F*N6H3_5p^vIBRfKJQb|l0#>r{ z1m{ErHC)Mp#=FX`sTT}9gI4@=@=@8lm&SEh-ANzi-TnR~tON={^~Ugor$WdB1h?VE ze31o8Qzdijs{C&|E4x?M4)Brt^@z-=zc`_?1&z2;l`(!MjtF$tx$AgdHJ1tGu{-Bj+u5pyixhQ^IGoCJ<{ml z*|$6CR@1K6tvd5jSA%#AhF*VgJ6UE{T%7@8o%W;8@|A7$UOBbuTiUyS)D)z6f~>@>%84No5sCk{BS{>!$pY|FYI>u z(t4ZdyF6{j(NFfUpGO(1^v#I)HweIWsmxjh{o1;E)%X6(8MeHM=Kz(u zaPds@=&MgK7ra`1**k6J)=(>E<9(t`qM7Y{yWHYH+Dys9MF@>=;TGG=SJ^-KGD&3h zbS5^k@5#-_x@foJPlBx>l^u0L5kA@!76k>e=*dC~HRs))#IdKc%f25jEX1GGrAVBy zyd3$oi`OaZW`Wz)FGC41k-L#HQE;bOk-9uid02NNTZ1SrU+~`kQdAtW43tTleqmp_ zMG4XlLOx%Qg9Tly8LZC+!@~>b#$8uOOHvu|>OW2KknU+7L=Wgl%r3w{n{k)JwZ!oj zFW&nczjKQ_%yRE{2=T11-!wQoHuqlgta8@oXt$j4-Zqs5#cUBfUbh;WSH`n3ewe*V zF=0KG>C==;O&og%`I@oX@lss<2KVRmt1R9FAIa%PMiM45UJS|}0))zJa5e@b`86(G zi`I82i#xD)IJn_nD6*K6cE)VSMAp9c2{2lgm5W9k=JoO+-Qt;09((?}_f}M|6!&p7 z#cIr(@9h`7b{smQ6FrttlO*G68=kA$5sB(?{YtY1o(JZu{Q0(Htht3>OMdb$HX-GL zdAxnUX|Ck#N7h)skcaoN-kA#;Qkn4I$TOMPMtfWj|3F3e%`D$IFUw z-Vg5{H}5d}C>9X37PPx*hF8$8&RHudyxS!an;ghXDcxz$=1gS!Y?-(AD_+{P9L~#w z)nLzO$TH4a`q^@rI>@$Wn?F{eS@BodlxwzN@cfsCz%~5ql<97p#fcVdh8gh=6vo|I z*6HVZRc)vSR$WILQE^2n5nScB-S#Y6;~u44I6-4u%45F(R`mntcp*gWbi9(|g%hcag+?EBf>!9N4Ypa^rB+^Sr0XC5Dn&R+99-aXq= zMXEmEe7*@0v4)y3G8?vS-g~KEmiL1e_Hz#oPY^qa>@79vtOc%=K$)&F$|>iV-i0e_ zw?{I+F3TY7o}jL2n${q>8t?JlDC!Po_(GRpy{yjf(9MZnbGz?_z;?(JIacvP-e zZ>gB+7;KSScTokHILFyNDOu{_Ehq(-bC>soP|8pi31*M;xl6Xu{ERJ7yYyJrMS)yf z=?7&Qb0qI*XfcDTwxQUTZf)dzu2$QcZQ|Zzz8~-IXRmM7vZstYRUR4}+o0J$F^xZ! z-;qA@@~n;TX&A3Iq+W~8-V5s3RYE5=yPHXOuR7-I?nlx{`sFiXX5f)koK0}GtVFFm zx5NDfr7X%kjwiVJkpv3UAV{K387@;|(cJ=~w}gaWFPT!WxZTZ5uiujBh<;!{&Dr)6 zGJo}577b})r(DgXW$d;{297Y=;cU%c8g91p1dSI_PYd*mbCa#QUmVqz&-gL%?Ywr> zb*MisOC;@`{=uno`StNtkjoTV*NRe!NZT(_ZZU0BSy!WEx*N|R91<=B8&ie`dbh8S zKxKU+gCv^h#NNWQOs${SagwdRaaYdEGiSKwTS}}i2y^c$dCbe|zPE9Rn`#eIDrpor zUPDB*C(2+;dBBa1{L!FZAT+~2{I22==E`Ht79>xb3_;@)z^yXPI31~F`DoNBsM;If zsYx;DZX`K(`tPt7W^57+!!B*`T84`sB@rH#-GfROGbbPlnKQN%D1x|G`RRn+NAO+V zcG5>r*%!EKf4LDEeOH($Zg;1CC0Ji(gtLV?moh~Hy^&TOxcSDq=OEC^ZMr&kl=p_4 zYiRyJGzjqu5^Y*`OQ2>`9Zu1B-YU;M8gAq8I)DB(InVE=t12F$6*n=N{WpdpDzHr3 zgjs&~^z>Pw@Lx2Of8NzBH6h{dE`wSW?zjcHl>wDAyR)(TKc2gEx{j$7z&Awu;9Ihc zZU@t*$FRdPAP6-lTmC@d!uetMD@L!W9He;7r45y=Q+%2pT`fDqihDwoeaK1 z%9F^2yYarguD5f2m0#Tsp80L|Hx`dLVxN7KzMx7f`j+%@5p*i5Xi~n$fN}EE|CF$L zGI~{Mgzc9upT}0g?izoZdZz6XIz6JB?Z`{j^xN(Rn(xy)+~4U5*>)96rCPM+LREbw z=%CJ_(1xqO_QsE@F1UAlOrFtwI+H=86~f1->?(=s6?L}@&Uh7hPwzdpvadq?eW&6= zoS!|s(^1fFc+KK=b=uqA=Nu(FC~I^ULwk0*FWYAD4_8e~%B*((c7V}-{jg+qMc*C2 zY_^12$gejY+wCvA+hCX_FDy_FlN=x2 zur}oJXkBOJT@xFy1D&%otMW`#siYkZiF>o82JwcGxc~Z~PG!rpT&;sR&Y>$7!Z3}G zvi)u8%BKOHzzu(onOgep-m>sBRP{T3;KcQKgXSBEG^NphhVsbA$U zo4j5T6Fu+FG9H(Y>hsID?|^op0VUqDwVk`p(V;J?CNs)dQW5jU=*v#Km1W5dR2q5d z@V7aCMidNvQdkx0P#ZG3i;8Iyb;tvYSY=oKYk?nhOU4&;Wp!4|xLR(tSu@Jj=B=&l z438f=u;R7c?q~k8;(4@J_PbWxJ$A=m<4ZD+#j}+-zYL6bQgN&Q+%iMQ9oq>dYt$TB zdhZ3L3evD|?PBieD;tg~xeGkO_DJP!tyMQ}H(P3M<#Xk7KOZJDB^|ZII~!*^MAKfW zG5glpQT<;vL(i+M&sz%;0Vc|{3aF`<3`fm&2g5KZU5j_rdEpXzTbxw^27652uyU4V zW!LO)blP!c5E~& z)^Rrk&j=i@23yTAt@BGRxzKQT*j#*}&OeuQp6%wG#N%b-!U!&9yY@0c5^f$_|1F8( zgzY+}Qc3123G{5&Vcdn3h{RLuK`I6NiAOt$4K@bU<8T{eQWS_f8&DCJ=M$cPB~Y=* zMmH<=m(tUwRgJGNG$M$bMQ0gn)FvwERx| zc3O4Eo@$h=)N^=PS3Ig82puT*A)i{I#qO42@P2|3HL=(#XB^0>8&{aGxZzDpFO{tO z8Izr(Ns800qcCg!Y*@#r+nd~Ys=^djmQd4Wo8$q;=rC~vL$)FIHPh^ z=-kbU&-9#$9pq($1_AlDFTW5`)UH062o0AMm?8~y?wdX8aICVGHr9?~bX5EHa10Ek zL)`B^2&UkRG-)f`P7h|%agB(>uDI!utGn!=XBLX@lwV*=c(ihKxPmMY^xGfSQ|$!O z1m>DYH{EQ|7`3}q3;7lK+IKdFR(y}>USGNOjm1%I@L{48i{}Hs+><@bt98$rNzl4U z)#U?Av%*KtwFcjMI`p46kel32cOB|U=npd)*|Q(Hox4&DCF?zV6|~9KHM}CA{>P6h zZ(6wOq!1dvad>N(=szQ8!+F2N3cv>Q=*A+)Xyq1}unK{>_jV&Sgw}!C=aWbW(&<~} z(YJ9pSOGmbaslL8y&nDcnK)Ux_G%0NSZwucu|Jg7K zt8st#^fHfcQ)j;8!X~_uA#`s!l75Ei-a=pCB;W~UN@dqBAfQT%V(DGRcJ9;49gN>H z3pDCh+K(}&`26xIe+li1sXGPE@+~n7Lp~}6T_y*PRcrC#DqOeFc7mDw@YNvTo|QHjZTxU!G)e16(slq+$k?fgL|@khSJ!di-+ zS&pyY34s~# z8A`g@P2cbO4;4wAMW)`Iy|azYbAahYQU7{@CwI`QsJ{{4 z@cN9F-rcQ%S2wJd{(`_E~U{gX2fN>DO99xB;1G7zS1Us zNq!zRlh*fhbPRZR{I*;+ntV01EIx%-*k5)!$9}OKYJP6Z{@J&r%qFFmAGr>3wPaD) z={Nn&0_YmH%FvPJ-b)u^`x4V^&x)&Fx3iq|5f*iBiO)T2_jNey8_qIob}b*fk1=A_ZtzJcW5n>qc$t>voRV%>mP{kl-e)T(*YDi zjN$&S?7j(b&YhXIJ1R#B2a6^dae_?s@?*d?7NQi?G)1n&+LrOA1v4hI?+dTRbf(5B z8Dwt$ASA0_wTZeFxQqY$`hon)GF&7AAFm37+(Rnq{2T}f>JvFds^$UfGh^pVdE zlREiC*f}VU!SKS{*xuxk5}hRoR>--4ZGH0w{fezjPYHMkiTe@|T*YbiR>l^k+3u#) z`2J7s9HwF$8n3Jaj*P1oo{JG8_mS%-VW@h%8p9*|^FuYKVsE2L>T1cdo$T^^%ZX#K zGduR-Ny#e)FGmvX4091!RB51qK0zlh3gkhhI(>JpaBodZPq{QR7w5!kZR{Eb@iHO^ z*SnkOSC(JoZ+QjW1bRJ#XtJ6c);?J}zTcxTJ%kh;2Q}OhND0!Pxu9_q zfG(yK*|Z;=a)si|_{kc;8ys?m?b$1w7=PG@k2MG=JHzok=_Kw8KJGu zikd)yst7*NAJp?5RF_pw*7A2LBMZCI#yM&q*KvHa=j48r;oF|@>A@4;u4%{?f6)mx zy2RwYcTIJw7~`y)A*X?>SSKNA`*u>|4szbwth}~sv^gph6taRZ)p|k!qbwLrm#}&D&joBtXgOrOg|T&0>6O$+g10$C>JGzyxw0A4w*Vi_s$1^A>T~D#l7fU`t`JuBq$hf6sf%O42o=-Ao zjK*og>twKEpr3IjeHG={M!sX34!NxH(;~xoizmZbcwFJs6xqS&0@>`NZplrjETp-_ z{IF!iSWw*-h@akiMzt`tO*8sj+j%A;u+PstBjnY!!+NWNIT-|&V>e!XDwpdGCyNRj z6*E`;D-t@b}y{V{=8sOqN4( zZTXOS-Y0bzsN98~f6noOca$MAp?6mCdDj7%_#eS||ppGNU_ee`$WWAZ_FN z=nJ!8;V7MHf(0iEkh-0f8Q*T;PN!<(G+m(he(5alk9QKqI$NlLw5TsX80NXrJMFzD zA*0iPn;W?Frcg&<9W-mv7RGDO2CtlPa+xw+8Y(~_1?9%=VECBd?4Nkw+r zw)zC#ZALWs$$CD|CJd_`jjnib_Z26I6{Tz>e1AW(AlI z8a?+V`Cw+5$`d9Y@Z8SY)86Ni95}zB+=8Xr5?7dd!gAG7mxHvhjHDILvH@k?_>+Cc zISuxT#g8oTqRGpQJ!)v@owrzCub61LYQGNu>^~;cK8gsKs&b^gS>5Hix&K;tpRRJ| zhLx$?wdfC`$p&KBvc+hJyL8FWyCG10Y6J0w_OYGR0S^1nvE2=?UNhDK}KEd(*|En}r0{7_7^xPF$|&{hl z*gf_)9D5BFPyQpZs6@b)U6rqN8fbX{bv?f;a7qXES8j}ITdoMR)YHH_{ZH9$V=&sH z$hf$`Ro|>k6S+Xs+7mT|J}=!AkbTrvf;Qn|@LPfq15Smv7{y!l#CzZp{_KBvL)>f&!*!Qk zN5`g_fmyi}q1pu#h5UMB4`c-T*rPure*3!nEPZnTWsth(3dGw;1X-z&0wEfR(R_t# zvIq2%GT(P`Q#S>sz8*l4J3zfIa}03flSHd|A%}f3N*5I~WS-1p1q&F;_PtZGi*(Nu z$Bv9O_pEqQ5y3V-hYl0zG0CHTh+)~RmY#mh>><2~QKmH}Sqfa#i#KA&E&A3dV|Mnzsr1y(r{q*0Ha3@?%V)D-1LoQ1i zx>fp&adhXND%xE?x-&V)Z};<4oWaJMm^2P|)7`;Fp-7D4kyT;MDN`WusEjuo$Ulf= zE%YmM-hblgXX(t>3N-8pFVFhk+}9MHDgAV1W-4=8V)O9y_%rrlHo0eJnFQyh`_O|- zh*m(|izuy))H;tDl>rij3qP@o?vl}R24FkJ)EjBtJ}y%(`-!9qA)8&|-Jz!`ENS0^ zfq9jnz0Y=iw=*!qH(9Y0$Q*RhVmY%G2Pp}uyKjTaymeRB`gDbmyeUt@DE|yG&D#50*RievX6h?3Q`EKEB6E@$dCu_R_ap-*S;D z>`gx=3A{u1BJczzPRy$nxTv9(BIk;`RU*?Uxn*y!37r#nd=g^)L1#(;lkaFDbjkjk zDBgu)$4?m?7GWNv8YLOAy#vaO{&O}<;7gPPT0x^z#8+B7vBSX~;YOa&@&1f9>;}+H z<#9n159u&Bwq`z<6zzZ&RnvZ8*HL`v0UiC5rpzd$evt1z_kGPdF?(3iaXTc~?a9!y z;&zHKagB4R8t$VB>QE+W5^|p)~X~!miJMJ;MIbRk2(D`VnB!b^8E zzS+Q9RG`_HOI5#K1wMN9-CTEX==^svK3M@UU{&oos5_%^u)Bx=fc3=`zh$?>5KGkY zr0MxKwLRdl7KmAvUus%@aQg0IU%S}I!-6Ot4P&MkL4JEnL!=vSN~v+e6^01BpDLVY8gPQ$dC}fA^Gf92stDOluJBD}9+K zPXj3V4qBkQjn?_)eQBN2$WNH{pu>OMwySepE@EzWH#JPbtWC(9ZyB)P4DfbcsOWi< zu_ep>#84v5ws1~M@S{#!y|PRi^aB9gln*d{ z#MK`bH#XOmN=IwmfN6zFfmz}2e8A8?SibuTzQM*X~W;?aiXJIvbKNni+5Fn9GseaIOP9$p$qq2AQ}-kenA0wsfLSwi2P|2F}FX=j9% zOs6&=bG(4PSKNQ3Nv@mSnvN@>Jb;OqUv}Q$$?WXd-%_lBnS@(00l>K}?I-(pDOH1n zcIVDBZ|y~t(_Lon;J4b|r1V53il=`@Hp#$dpIT&>jg!z%CY*;rZ$+$~v+C-IEhAY0q1A@+DszxWhwJy&>=l0( zcf!V6nY3l2acJzrB@(j(k{Zenoj{YM&?i3?`}uh;s}W2|KVt0Ch=r9<+iO`jvv~NT zEYq>pqVUxN&LMsh?k96`bu@kKnR2X^U39^iFYPKr&;b9GXPNCmpc z3s*r)$huV0qo6%qFJh;$ISv;paQ9fyW!1EcXPx2%SwjleVQ3A%XRZiE=+c*?qb5ks zAD@51rQOzaLH59-QK+wAWn%@P4pZP2Gd~{oMU7kO<_7ck)aq+<97`kv={kz1o5$tv ze!pf>+nsdgel`=1(QYh^i^z^BKC>RrzJil?<$6~O_jW~ke3qA+ z&zDcTn4C+);Z_$v*)({ocgPL2F#n>If3p_5UuVi{5)%7fD4jXNt+q76ic>!D(gOLB z%_WMoE-gef$7+&j{x-3gsl@X0Ok~4t-V_~HVWP3i!jYssbuK>+OSForc$1PNtei@I zx3@~f9;3=X<9)0t*V<2f30&P(dxn}%4|HDVLkk>kPg71excrFS%Y$Un59T*{^->~y z%r&j6hVJH}e@;y2oATi`R9f<$ygy#rKdeL1K`yJgCSlU~#b0Vz4#@091Yc0?s%*T|X$07gz*>nn? z#LJV@G)GQ`LcE1W6D4V-vm!ZOJvZ9%rym;6^YqM9tk`m@EaX|ao7=kuS&;5Yb?gqd zXye^xGu-zf7IK;}vx1XWZ#sRMET6Y2$$qV!TVrhp#CKyY+6J2kL7+<=7knO*5oaS~ z9mCrMNT&uGWe?MX)Wn%M0k|`_Rpp^}q z{zoRC?hfktiza^tUxn#oe^J_eVbZ_Mu*tcJ%u&VVm17*RZWWAkm ztcK9qKG!#^God%;*1-8W%xmZ|gpN=NyW}}u;<+UN&9P#@eF zHH6b68AH?v2)XNGm+z=>ay;Q#maZ)K>}vw!H@LIQo-U=lj&$Ezv}XwWglY&!!=EU3YwRT#qjz zRT~M`G++NrzI;HC^`b67TAGLyiPhv9hWs6QKBx*<@hpq|sA_l($6lWHS|yk>HgN0z zBa^yH@(`N;B6QXMkGgNR!6lI!p+4t#(~|>NZC~2%P7-@QOuXIx#O!;+SJgIVukRze zd!jQ;w7c#C8;??ngAV;J{4MGppL=K$Z=Vfmvj^4%WLZaR?M2~Ko8Qn|mE)Z7PWNdcym&Lrk~P{oK%tGzur>D9_< zs+H>rvtks#)YKoP@g!!x1-Rg$VPL~-NTh)9tM8e?Rg1FBg_Kt}vzo}ccis3Bf|FxE z1-rul^jF84dfr;G2pfLkzJ%ZHefg`&vA!;suBWZ_9N}?sacYhVNP}5wi(T48^ir7?*dd+ZwvqEOo`iFtJ6kBO?MtTbUb8 zHT?GjxqGf6indscP*GJbWZQAi#Exy4)yiJ~h0x{9COt6m$cp{aLdvZmtFCyZvzO<1 z7504Ql=gt8IQu}m+n}%=?qm-PJ7G^RQNT}mo$#(q#c5OEXc{SxtXFb?dTwWqMlp&{^ z#)VrxdsCVWKi$c8S-0;!SD13T52?KXkt*~9iOxXvsb4Xv_xv~W#9sUExlZCnZ(J*2 z!XY7{ILWhrG3mXDE1+(Eg$JOou}-P>y#fvNbFs*e<_TO+yG4#fpPNCZhNJMPe){V?ykn(cbCzuQZO(>*9b)#fdI%hjb+zK! zPya~R6kZL-?6UO)=8=uZk)K}LoKhXR=#gNoqTox z;jy2=Zdbolwyl28BX84Z?`Qy@(iGjr9t<(Jr=3~ZNeV*i}NS5kM@~XOP z4S14gu^AN>J_XJ62spBks{TgKClku-p1GoJh}wbn4C`TW60berXn57CIcIzv9g^_Z zDJNJq{FV2N~|p;KpCQRAOW32ZJ=||R)potG@q^hx%)85#N<}@KYa&t3Mgk=;3Tcu2+aG?d;?))$ zH3gRRWb;955gN;6$YXMUa=cgaJ)zyMNRc0RjJ+*W;;G>6uQoT$)tusle+)a`r0nOh zJklGAuqnOT2W(<~7~Cqe-u+OZMpMdVTlq7j~D`@jv@7uCn z5wG6Gd7E$Q^O+5sj(~(hzwv?XexiWnaJPq=Sn}UKq{&!a$$5g$!>mD|YC?R* z^#cVzjW(A7NSEY8O8DTFjb3%vzuw&x2!!?n=A%IvuT$SUSq~J%N+zn{(?c)y;%<|y zgQ{?xZyz12o~d$+yuQ_hQ11r2Y74_*9l@^t9-)Z(+?GD72+c(|X4A+Hcj;7H{^lZK z_Zo%g4~<11R#TPt1o{f7vT7wEbT_Mfs)rg7&rArr6fpdazUnnEWeY#wiMq~8@)3^~ zbQ~9*J^I44Orbee`$Dhfn}Q9Tv*!pO6iv_WoP2H)->q^H;GU<@-dAxR^1k$m+CHAM zP&MP5_NpA|LIg7tsq(kU@B5>u=oMph}j;uEJ>_^a{p;6Wk$i}3D~CwB#5r168*2RAf!Oc2OQGjSDJ zDy$7pGs8V@&nx5^+PMY|=T5lRS1oy22g)yY#hR&YAbca^h5BN0 zpSs&V2pdk$t5*s47*W5!Ml-eh$n5ZrCe`%Z^xP(^M)Hhrqf1AEw+B^4TsdCn9hP{u z$AzAoz<^+*N$jR{yu!COuAsX?LL+UBeMkHR#8qEM3psZbIPi-*;k$tUa{GtMm0<38 z>JP|nWwZ~tt|b;Bo1gK#sDP`?6FF~kUGv3CxI{8LnNG%9xy84wT{w*Ug|r{ia+}!u;vZ9GzdA zH1|Uk)$W8AjDV^ilMe5nd;Ksk*cMmdL`|*Q<~96};~*33&as+>)v#3Ir#!D&SN{z! zk;7WokQ6>_`nibwhuMKd84g1;Ex(F+y<^I)!)G3GF6{t&aws4zd9b&gKrb8MFm$qU zd&OLVzu4MO+BkH%+Gd;lR;I50&tA8rndy>|{Gy7-8r~+7b7!Ukei2@f2gzY(Qp!Qz z8&U)aBD^dzb)Ga$@X&qXlzbh&;5)R70p+fOOB3i zKLM>Vqfl4X-b_O(*rM9ER7TLmePkO1!swdjex$5$|c+8;aYtHVkY&5$`X&yv#>1EwiW^090c(yole^i zkKPlmITmLjRZs59{Ia-NYUo>wiyhnmbtU&uw6(vFje7SEgsk;tia1&Q{o3G}e+ zL_Hn-fNoKRZ6#~97EcA*?lOeyRz6Q44WDmV-kc+pmKPLkpm6==U&+y4W?z8yd3d2p z*+I6n%?M3-ssUP#hEzHT5a$TZ#T$OA&4DVdIrb9l*IeO3(1N6tJkh$g-lH58JlsUMf%YE77rFLG#lX6J4qaw{JTM4YHUl5xB%|4w|a7{7ehSI67zEWxJtRB^r%R@n&3q-m(OD^%{0)28 z0eYM}mhM@anq9r5xI}8{bS^&c$#YiON{Wl=Uo1ZnqBhrm7Rfg$*EG>%t#Vfzn;QXmPkSd9AWG>%R`D5_2^KR2#=P_GHiUX@ zN1zNjdM6_wK4L+6TPB;IKVnk2kEv`!L?Q_jQtgvvlLodkv=|z|g!_gZ4K#5yrU_^S zO@Mw$$#7bAc70CYOMKBmw#kAw>NG@TnS9qED+{-IaY&H2V7Jj!=4SWhWk4jFh1)Wz zY6F`Dy<&>jO`1)>VB{;zBc!{rh`{|x#?!D9tW9&ms7!5Pl7vD(kxYUNk~~ENJPHj127)ut#o!xPLpJE zNQWMZKe^)EXEvT7+yeR>Uxh{~&{55M1*|V*RWM~_B`^Csx`vtf=^I1s%{}!xh*6** zv(fkab<_RtjpaBUiL`_Ay;6+dp3>>Ag2_EY}K}EH9^#R_dpdGxQW+A z7Emji#Mhoe%7vkfk@5zFh4U4qo{ZiG6U>aM7>3)?i+41v)HNvJ?D!_ zJTg{MNGOn+AB0)koBnRyVg`!1BT17<^^NPZ;`Skz!)XWuIRPGRYf=~>Y|lo*v~cNY zX_#L9f*n&j_wb*Q8mz#?C*s6R8X(h+g@+Ojzflc3v;>qv65z;8P|C`=mW-Is#l8*MJ%#WxEU>+rNlkJdVk5}rJ1whH;K7p#@2#)co+2lQB_qq|z!P+O z3s{^E3660iU?<h{_g6yT=X46X_`r~jYp#ywr4~+>w`5hNPPgX^zH_wD{;>)S~5EqeE>p}Elwt2z$Z12!5d?57|bK2rL}kIy%j#DUSW$iBV%I^r%m=UVkc0cxPZZPvhU-4zIv zAGf@C88r5l>&wDCc_9V?)l=a58TT#hZE1l*7poffm%u)B(VJ^niG5Oo*^lNg1b1lS zxs_3eMfbG<^&_q;0as-B{2eRzKxQWMx^IkTP%l9L~Yzy2$ z4EKBCt^n5z+3l^PL*8L2FM-2VpxSn#>#s>fJF6{41zL}~wkOZX4ZY%$kuX;b!T@W% zZo?Z(xiFX!d2n0N{XVlrzzL~DOUmlyy`AoTv?$k}=rqcJzV?1I<>DF&4Fx*eVPQ?+ zE;t3$^(~F}@1Aq6*`V;rLiQAaIsgG!8;23S&&dV`c@7X%+ixje71 z&H%ts(t)70z3LRmictIlitMSU+hzwqQ%qBat~lLQb~OGf4r@E40uM)yogMd7s^JM9 z%Np@~wlc;3`GFX>iK9mTZc&c`GN_`wEg-$6*=vl(5&HfJOa=jK?tQcw}%;jfA@Q2cVxQi0bA+@;1fXNsYN#| zAu7?KAHhR}uRwYmXuQV>m=XriBp6==MRqz+lbii8*lSbP8Z(t`fCXJAZ&fO8bB(rx&G80i)I2 zaxSjLrei<4V}ZMb0D2rr$oZ9TH$DX znAGnaNF(=TCct5d|1t$IcQh3oQaIX_jygB-{QlV)UWqv! z*`Lu{uD+IzfFeFjkOTRtx!ddXaro}*fAYGX5&D~rc&_<(RI;DiWv%=B3cudsH!-0m zmJ+60+w!9JSMLtwsVNCu63j|Dced8JJBd0Cnyi&}!Pw3eJA@`J=Z)|k6oXZN|H$$UiK?O^b)M7)5vPFrtUjwi4+in_*c z)gRB@elAQ@{te8GnvwO-rkRpOjRx~2%e?h%DfGB3{O^5LjVW=jD9Ee z&^e?2Ln+!0pRb5!7G&*Imu=fyX?k$^YOse`8|BL#^DV!oc|6|+y3W9Ck$z<@O}xbw zB=?ENJEFD6+qoqz zrI3@)1TGh0q$3JR{Li?(1pFBvJ9~$SXDfbZDS~?1@#j;)BK8}vowyv(Xn)tg|F0$r z)yZ2|nMb<2QzZ5jEiY(-gD+6C|IZg8rrisAKvD0fhZ38+0y;rl5PE9nM@$oGwm zjNd1iP}#`uoBCv8DT8bUDtoR@L}QjsAb$#_F00pn`t+Z9cY8uD5-k`?IvRrHT@a{$^TxBHFIc$JlAa+s0!+2PZ)^VerPZ>_a;| zdoOV7w{a6V`$j%k>?EsZGw&g6L41={uwIJ8@-)$gH^sN$upW@WV;gU*+?-pMgY2PC zTvzyBR?lE6k6x0+Hn!e>7;o%(AuL;D*@TQJC7C*L2H#{}XV9z7>BG2@rAtJiC_)M` zEky}Skek+PD@HDXVH98-pgxNd1a=k>+OGrc&7-;Tcz!Wci3Ca5<>bhZ66|x6YpBlgTZt z^&yrqey8&G{$$BxS$UzeL>qXaNt0@KBYl5L5K}aNy?0Fj15s7B&MKiS(&E%?rBj=- zg)`Sy$(7$Q^%Q8HjF38DPg=5gXq+pQg4vVqqB^uw{xs1W?=;!nA;cAbXn;Jo1FLl#G~(KX^whND+o@}ii) zx#bTZmez?5sK>>#ue@jv`S&J8mY*iI-${zyiSTc;4Cqr7t4? z{ds(BuTQ?Aj=G}z69WVE2op3iE?>g|?#Y8td*!l;@ zO_0&!85MdPXXMz2(V<4TzR?O5!_ZDo#_ysH%Da}oP9&n}s(yrzDiL%vQ>6^dz_zYv zUEButbv}`#obVxb^T=0-CT=M*$cIuMbj&qdzGjh`J0Co;um9i&E*rL*2ujYo&CHZ{ zb;b*LHa(Lysjz&Td1foOO!f~kx?70NabjdwF+52T>>w>~6L(rurn~K2x2B2K!SDYFc@4(V-&Kcy zsV<_g5@6uKPF@zC^<;li*QQ^J=A9;05wEZb|XhMQ`^`J6;ywac~&nJh}GGMV(Cx%m4 z7{9x_jyftV*%K#=mVW4z#4shf))if22KI9Ic z3rbZOzdN(Rm6!jx`r%&` as a placeholder. Also see external fields for how to specify these for fluids as the function names differ. From e137df4d5f39cec78cb78077ca74c717d2e730c2 Mon Sep 17 00:00:00 2001 From: Marco Garten Date: Mon, 2 Oct 2023 15:31:32 +0200 Subject: [PATCH 038/110] Bump up HDF5 version on PM to 1.12.2.7 (#4333) The previous version cray-hdf5-parallel/1.12.2.1 was not recognized anymore. At this moment, 1.12.2.3 and 1.12.2.7 are installed as modules on Perlmutter. --- .../perlmutter-nersc/perlmutter_cpu_warpx.profile.example | 2 +- .../perlmutter-nersc/perlmutter_gpu_warpx.profile.example | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example b/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example index a722c9a9104..d25b4825dde 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example +++ b/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example @@ -14,7 +14,7 @@ module load cray-fftw/3.3.10.3 export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3 # optional: for openPMD and PSATD+RZ support -module load cray-hdf5-parallel/1.12.2.1 +module load cray-hdf5-parallel/1.12.2.7 export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master:$CMAKE_PREFIX_PATH diff --git a/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example b/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example index 629a2073a9e..af3e69c1190 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example +++ b/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example @@ -12,7 +12,7 @@ module load cmake/3.22.0 export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3 # optional: for openPMD and PSATD+RZ support -module load cray-hdf5-parallel/1.12.2.1 +module load cray-hdf5-parallel/1.12.2.7 export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master:$CMAKE_PREFIX_PATH From c894651465d192b6594ab35ea5334406405a55fe Mon Sep 17 00:00:00 2001 From: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Date: Wed, 4 Oct 2023 10:30:42 -0700 Subject: [PATCH 039/110] Merge nodal synchronization with `FillBoundary` calls in PML (#3534) * Add nodal synchronization option to PML functions * Add missing #include directive * Remove calls to `NodalSyncPML` * Same changes for PML classes in RZ geometry * Remove unused functions * Update checksums of `divb_cleaning_3d` --- .../benchmarks_json/divb_cleaning_3d.json | 18 ++--- Source/BoundaryConditions/PML.H | 14 ++-- Source/BoundaryConditions/PML.cpp | 61 +++------------ Source/BoundaryConditions/PML_RZ.H | 5 +- Source/BoundaryConditions/PML_RZ.cpp | 8 +- Source/Evolve/WarpXEvolve.cpp | 21 +----- Source/Parallelization/WarpXComm.cpp | 75 ++----------------- Source/WarpX.H | 15 ---- Source/ablastr/utils/Communication.H | 5 +- Source/ablastr/utils/Communication.cpp | 8 +- 10 files changed, 52 insertions(+), 178 deletions(-) diff --git a/Regression/Checksum/benchmarks_json/divb_cleaning_3d.json b/Regression/Checksum/benchmarks_json/divb_cleaning_3d.json index cbb43a81004..17d56dcc906 100644 --- a/Regression/Checksum/benchmarks_json/divb_cleaning_3d.json +++ b/Regression/Checksum/benchmarks_json/divb_cleaning_3d.json @@ -1,13 +1,13 @@ { "lev=0": { - "Bx": 30128649.582633037, - "By": 30128649.58263304, - "Bz": 30128649.582633033, - "Ex": 137776481658696.53, - "Ey": 137776481658695.72, - "Ez": 137776481658736.47, - "G": 7382627392167406.0, - "divB": 6914882882637.787, - "divE": 60880563.38401385 + "Bx": 30110529.73244452, + "By": 30110529.73244452, + "Bz": 30110529.732444517, + "Ex": 137103615680453.66, + "Ey": 137103615680454.5, + "Ez": 137103615680494.48, + "G": 7413121805692223.0, + "divB": 6944252335584.075, + "divE": 60882624.59445304 } } \ No newline at end of file diff --git a/Source/BoundaryConditions/PML.H b/Source/BoundaryConditions/PML.H index a925cd6c9ef..4be1be146cc 100644 --- a/Source/BoundaryConditions/PML.H +++ b/Source/BoundaryConditions/PML.H @@ -29,6 +29,7 @@ #include #include +#include #include #include @@ -187,15 +188,10 @@ public: void ExchangeG (amrex::MultiFab* G_fp, amrex::MultiFab* G_cp, int do_pml_in_domain); void ExchangeG (PatchType patch_type, amrex::MultiFab* Gp, int do_pml_in_domain); - void FillBoundary (); - void FillBoundaryE (); - void FillBoundaryB (); - void FillBoundaryF (); - void FillBoundaryG (); - void FillBoundaryE (PatchType patch_type); - void FillBoundaryB (PatchType patch_type); - void FillBoundaryF (PatchType patch_type); - void FillBoundaryG (PatchType patch_type); + void FillBoundaryE (PatchType patch_type, std::optional nodal_sync=std::nullopt); + void FillBoundaryB (PatchType patch_type, std::optional nodal_sync=std::nullopt); + void FillBoundaryF (PatchType patch_type, std::optional nodal_sync=std::nullopt); + void FillBoundaryG (PatchType patch_type, std::optional nodal_sync=std::nullopt); bool ok () const { return m_ok; } diff --git a/Source/BoundaryConditions/PML.cpp b/Source/BoundaryConditions/PML.cpp index 5fc3e2dd3dc..811d1aa469d 100644 --- a/Source/BoundaryConditions/PML.cpp +++ b/Source/BoundaryConditions/PML.cpp @@ -1229,103 +1229,66 @@ PML::CopyToPML (MultiFab& pml, MultiFab& reg, const Geometry& geom) } void -PML::FillBoundary () -{ - FillBoundaryE(); - FillBoundaryB(); - FillBoundaryF(); - FillBoundaryG(); -} - -void -PML::FillBoundaryE () -{ - FillBoundaryE(PatchType::fine); - FillBoundaryE(PatchType::coarse); -} - -void -PML::FillBoundaryE (PatchType patch_type) +PML::FillBoundaryE (PatchType patch_type, std::optional nodal_sync) { if (patch_type == PatchType::fine && pml_E_fp[0] && pml_E_fp[0]->nGrowVect().max() > 0) { const auto& period = m_geom->periodicity(); const Vector mf{pml_E_fp[0].get(),pml_E_fp[1].get(),pml_E_fp[2].get()}; - ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period, nodal_sync); } else if (patch_type == PatchType::coarse && pml_E_cp[0] && pml_E_cp[0]->nGrowVect().max() > 0) { const auto& period = m_cgeom->periodicity(); const Vector mf{pml_E_cp[0].get(),pml_E_cp[1].get(),pml_E_cp[2].get()}; - ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period, nodal_sync); } } void -PML::FillBoundaryB () -{ - FillBoundaryB(PatchType::fine); - FillBoundaryB(PatchType::coarse); -} - -void -PML::FillBoundaryB (PatchType patch_type) +PML::FillBoundaryB (PatchType patch_type, std::optional nodal_sync) { if (patch_type == PatchType::fine && pml_B_fp[0]) { const auto& period = m_geom->periodicity(); const Vector mf{pml_B_fp[0].get(),pml_B_fp[1].get(),pml_B_fp[2].get()}; - ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period, nodal_sync); } else if (patch_type == PatchType::coarse && pml_B_cp[0]) { const auto& period = m_cgeom->periodicity(); const Vector mf{pml_B_cp[0].get(),pml_B_cp[1].get(),pml_B_cp[2].get()}; - ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period, nodal_sync); } } void -PML::FillBoundaryF () -{ - FillBoundaryF(PatchType::fine); - FillBoundaryF(PatchType::coarse); -} - -void -PML::FillBoundaryF (PatchType patch_type) +PML::FillBoundaryF (PatchType patch_type, std::optional nodal_sync) { if (patch_type == PatchType::fine && pml_F_fp && pml_F_fp->nGrowVect().max() > 0) { const auto& period = m_geom->periodicity(); - ablastr::utils::communication::FillBoundary(*pml_F_fp, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(*pml_F_fp, WarpX::do_single_precision_comms, period, nodal_sync); } else if (patch_type == PatchType::coarse && pml_F_cp && pml_F_cp->nGrowVect().max() > 0) { const auto& period = m_cgeom->periodicity(); - ablastr::utils::communication::FillBoundary(*pml_F_cp, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(*pml_F_cp, WarpX::do_single_precision_comms, period, nodal_sync); } } void -PML::FillBoundaryG () -{ - FillBoundaryG(PatchType::fine); - FillBoundaryG(PatchType::coarse); -} - -void -PML::FillBoundaryG (PatchType patch_type) +PML::FillBoundaryG (PatchType patch_type, std::optional nodal_sync) { if (patch_type == PatchType::fine && pml_G_fp && pml_G_fp->nGrowVect().max() > 0) { const auto& period = m_geom->periodicity(); - ablastr::utils::communication::FillBoundary(*pml_G_fp, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(*pml_G_fp, WarpX::do_single_precision_comms, period, nodal_sync); } else if (patch_type == PatchType::coarse && pml_G_cp && pml_G_cp->nGrowVect().max() > 0) { const auto& period = m_cgeom->periodicity(); - ablastr::utils::communication::FillBoundary(*pml_G_cp, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(*pml_G_cp, WarpX::do_single_precision_comms, period, nodal_sync); } } diff --git a/Source/BoundaryConditions/PML_RZ.H b/Source/BoundaryConditions/PML_RZ.H index 72c901bf480..ac4b6ff2c4d 100644 --- a/Source/BoundaryConditions/PML_RZ.H +++ b/Source/BoundaryConditions/PML_RZ.H @@ -22,6 +22,7 @@ #include #include +#include #include enum struct PatchType : int; @@ -45,8 +46,8 @@ public: void FillBoundaryE (); void FillBoundaryB (); - void FillBoundaryE (PatchType patch_type); - void FillBoundaryB (PatchType patch_type); + void FillBoundaryE (PatchType patch_type, std::optional nodal_sync=std::nullopt); + void FillBoundaryB (PatchType patch_type, std::optional nodal_sync=std::nullopt); void CheckPoint (const std::string& dir) const; void Restart (const std::string& dir); diff --git a/Source/BoundaryConditions/PML_RZ.cpp b/Source/BoundaryConditions/PML_RZ.cpp index 98ed12e7d67..d04c6e53bf8 100644 --- a/Source/BoundaryConditions/PML_RZ.cpp +++ b/Source/BoundaryConditions/PML_RZ.cpp @@ -134,13 +134,13 @@ PML_RZ::FillBoundaryE () } void -PML_RZ::FillBoundaryE (PatchType patch_type) +PML_RZ::FillBoundaryE (PatchType patch_type, std::optional nodal_sync) { if (patch_type == PatchType::fine && pml_E_fp[0] && pml_E_fp[0]->nGrowVect().max() > 0) { const amrex::Periodicity& period = m_geom->periodicity(); const Vector mf{pml_E_fp[0].get(),pml_E_fp[1].get()}; - ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period, nodal_sync); } } @@ -151,13 +151,13 @@ PML_RZ::FillBoundaryB () } void -PML_RZ::FillBoundaryB (PatchType patch_type) +PML_RZ::FillBoundaryB (PatchType patch_type, std::optional nodal_sync) { if (patch_type == PatchType::fine && pml_B_fp[0]) { const amrex::Periodicity& period = m_geom->periodicity(); const Vector mf{pml_B_fp[0].get(),pml_B_fp[1].get()}; - ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(mf, WarpX::do_single_precision_comms, period, nodal_sync); } } diff --git a/Source/Evolve/WarpXEvolve.cpp b/Source/Evolve/WarpXEvolve.cpp index 467a4268d97..a1b536f6c71 100644 --- a/Source/Evolve/WarpXEvolve.cpp +++ b/Source/Evolve/WarpXEvolve.cpp @@ -472,10 +472,6 @@ WarpX::OneStep_nosub (Real cur_time) if (WarpX::do_divb_cleaning || WarpX::do_pml_divb_cleaning) FillBoundaryG(guard_cells.ng_alloc_G, WarpX::sync_nodal_points); } - - if (do_pml) { - NodalSyncPML(); - } } else { EvolveF(0.5_rt * dt[0], DtType::FirstHalf); EvolveG(0.5_rt * dt[0], DtType::FirstHalf); @@ -502,11 +498,10 @@ WarpX::OneStep_nosub (Real cur_time) if (do_pml) { DampPML(); - NodalSyncPML(); - FillBoundaryE(guard_cells.ng_MovingWindow); - FillBoundaryB(guard_cells.ng_MovingWindow); - FillBoundaryF(guard_cells.ng_MovingWindow); - FillBoundaryG(guard_cells.ng_MovingWindow); + FillBoundaryE(guard_cells.ng_MovingWindow, WarpX::sync_nodal_points); + FillBoundaryB(guard_cells.ng_MovingWindow, WarpX::sync_nodal_points); + FillBoundaryF(guard_cells.ng_MovingWindow, WarpX::sync_nodal_points); + FillBoundaryG(guard_cells.ng_MovingWindow, WarpX::sync_nodal_points); } // E and B are up-to-date in the domain, but all guard cells are @@ -746,11 +741,6 @@ WarpX::OneStep_multiJ (const amrex::Real cur_time) if (WarpX::do_divb_cleaning || WarpX::do_pml_divb_cleaning) FillBoundaryG(guard_cells.ng_alloc_G, WarpX::sync_nodal_points); - // Synchronize fields on nodal points in PML - if (do_pml) - { - NodalSyncPML(); - } #else amrex::ignore_unused(cur_time); WARPX_ABORT_WITH_MESSAGE( @@ -927,9 +917,6 @@ WarpX::OneStep_sub1 (Real curtime) if ( safe_guard_cells ) FillBoundaryB(coarse_lev, PatchType::fine, guard_cells.ng_FieldSolver, WarpX::sync_nodal_points); - - // Synchronize nodal points at the end of the time step - if (do_pml) NodalSyncPML(); } void diff --git a/Source/Parallelization/WarpXComm.cpp b/Source/Parallelization/WarpXComm.cpp index 9b7f16478fc..8c87cc9c772 100644 --- a/Source/Parallelization/WarpXComm.cpp +++ b/Source/Parallelization/WarpXComm.cpp @@ -561,13 +561,13 @@ WarpX::FillBoundaryE (const int lev, const PatchType patch_type, const amrex::In (patch_type == PatchType::fine) ? pml[lev]->GetE_fp() : pml[lev]->GetE_cp(); pml[lev]->Exchange(mf_pml, mf, patch_type, do_pml_in_domain); - pml[lev]->FillBoundaryE(patch_type); + pml[lev]->FillBoundaryE(patch_type, nodal_sync); } #if (defined WARPX_DIM_RZ) && (defined WARPX_USE_PSATD) if (pml_rz[lev]) { - pml_rz[lev]->FillBoundaryE(patch_type); + pml_rz[lev]->FillBoundaryE(patch_type, nodal_sync); } #endif } @@ -618,13 +618,13 @@ WarpX::FillBoundaryB (const int lev, const PatchType patch_type, const amrex::In (patch_type == PatchType::fine) ? pml[lev]->GetB_fp() : pml[lev]->GetB_cp(); pml[lev]->Exchange(mf_pml, mf, patch_type, do_pml_in_domain); - pml[lev]->FillBoundaryB(patch_type); + pml[lev]->FillBoundaryB(patch_type, nodal_sync); } #if (defined WARPX_DIM_RZ) && (defined WARPX_USE_PSATD) if (pml_rz[lev]) { - pml_rz[lev]->FillBoundaryB(patch_type); + pml_rz[lev]->FillBoundaryB(patch_type, nodal_sync); } #endif } @@ -761,7 +761,7 @@ WarpX::FillBoundaryF (int lev, PatchType patch_type, IntVect ng, std::optionalok()) { if (F_fp[lev]) pml[lev]->ExchangeF(patch_type, F_fp[lev].get(), do_pml_in_domain); - pml[lev]->FillBoundaryF(patch_type); + pml[lev]->FillBoundaryF(patch_type, nodal_sync); } if (F_fp[lev]) @@ -776,7 +776,7 @@ WarpX::FillBoundaryF (int lev, PatchType patch_type, IntVect ng, std::optionalok()) { if (F_cp[lev]) pml[lev]->ExchangeF(patch_type, F_cp[lev].get(), do_pml_in_domain); - pml[lev]->FillBoundaryF(patch_type); + pml[lev]->FillBoundaryF(patch_type, nodal_sync); } if (F_cp[lev]) @@ -805,7 +805,7 @@ void WarpX::FillBoundaryG (int lev, PatchType patch_type, IntVect ng, std::optio if (do_pml && pml[lev] && pml[lev]->ok()) { if (G_fp[lev]) pml[lev]->ExchangeG(patch_type, G_fp[lev].get(), do_pml_in_domain); - pml[lev]->FillBoundaryG(patch_type); + pml[lev]->FillBoundaryG(patch_type, nodal_sync); } if (G_fp[lev]) @@ -820,7 +820,7 @@ void WarpX::FillBoundaryG (int lev, PatchType patch_type, IntVect ng, std::optio if (do_pml && pml[lev] && pml[lev]->ok()) { if (G_cp[lev]) pml[lev]->ExchangeG(patch_type, G_cp[lev].get(), do_pml_in_domain); - pml[lev]->FillBoundaryG(patch_type); + pml[lev]->FillBoundaryG(patch_type, nodal_sync); } if (G_cp[lev]) @@ -1473,62 +1473,3 @@ void WarpX::NodalSyncRho ( ablastr::utils::communication::OverrideSync(rhoc, WarpX::do_single_precision_comms, cperiod); } } - -void WarpX::NodalSyncPML () -{ - for (int lev = 0; lev <= finest_level; lev++) { - NodalSyncPML(lev); - } -} - -void WarpX::NodalSyncPML (int lev) -{ - NodalSyncPML(lev, PatchType::fine); - if (lev > 0) NodalSyncPML(lev, PatchType::coarse); -} - -void WarpX::NodalSyncPML (int lev, PatchType patch_type) -{ - if (pml[lev] && pml[lev]->ok()) - { - const std::array& pml_E = (patch_type == PatchType::fine) ? - pml[lev]->GetE_fp() : pml[lev]->GetE_cp(); - const std::array& pml_B = (patch_type == PatchType::fine) ? - pml[lev]->GetB_fp() : pml[lev]->GetB_cp(); - amrex::MultiFab* const pml_F = (patch_type == PatchType::fine) ? pml[lev]->GetF_fp() : pml[lev]->GetF_cp(); - amrex::MultiFab* const pml_G = (patch_type == PatchType::fine) ? pml[lev]->GetG_fp() : pml[lev]->GetG_cp(); - - // Always synchronize nodal points - const amrex::Periodicity& period = Geom(lev).periodicity(); - ablastr::utils::communication::OverrideSync(*pml_E[0], WarpX::do_single_precision_comms, period); - ablastr::utils::communication::OverrideSync(*pml_E[1], WarpX::do_single_precision_comms, period); - ablastr::utils::communication::OverrideSync(*pml_E[2], WarpX::do_single_precision_comms, period); - ablastr::utils::communication::OverrideSync(*pml_B[0], WarpX::do_single_precision_comms, period); - ablastr::utils::communication::OverrideSync(*pml_B[1], WarpX::do_single_precision_comms, period); - ablastr::utils::communication::OverrideSync(*pml_B[2], WarpX::do_single_precision_comms, period); - if (pml_F) { - ablastr::utils::communication::OverrideSync(*pml_F, WarpX::do_single_precision_comms, period); - } - if (pml_G) { - ablastr::utils::communication::OverrideSync(*pml_G, WarpX::do_single_precision_comms, period); - } - } - -#if (defined WARPX_DIM_RZ) && (defined WARPX_USE_PSATD) - if (pml_rz[lev]) - { - // This is not actually needed with RZ PSATD since the - // arrays are always cell centered. Keep for now since - // it may be useful if the PML is used with FDTD - const std::array pml_rz_E = pml_rz[lev]->GetE_fp(); - const std::array pml_rz_B = pml_rz[lev]->GetB_fp(); - - // Always synchronize nodal points - const amrex::Periodicity& period = Geom(lev).periodicity(); - pml_rz_E[0]->OverrideSync(period); - pml_rz_E[1]->OverrideSync(period); - pml_rz_B[0]->OverrideSync(period); - pml_rz_B[1]->OverrideSync(period); - } -#endif -} diff --git a/Source/WarpX.H b/Source/WarpX.H index 034c321ba14..1bd3f5bf46c 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -726,21 +726,6 @@ public: void CopyJPML (); bool isAnyBoundaryPML(); - /** - * \brief Synchronize the nodal points of the PML MultiFabs - */ - void NodalSyncPML (); - - /** - * \brief Synchronize the nodal points of the PML MultiFabs for given MR level - */ - void NodalSyncPML (int lev); - - /** - * \brief Synchronize the nodal points of the PML MultiFabs for given MR level and patch - */ - void NodalSyncPML (int lev, PatchType patch_type); - PML* GetPML (int lev); #if (defined WARPX_DIM_RZ) && (defined WARPX_USE_PSATD) PML_RZ* GetPML_RZ (int lev); diff --git a/Source/ablastr/utils/Communication.H b/Source/ablastr/utils/Communication.H index 9be9fec1e58..85f58ff9f65 100644 --- a/Source/ablastr/utils/Communication.H +++ b/Source/ablastr/utils/Communication.H @@ -60,7 +60,8 @@ void ParallelAdd (amrex::MultiFab &dst, void FillBoundary (amrex::MultiFab &mf, bool do_single_precision_comms, - const amrex::Periodicity &period = amrex::Periodicity::NonPeriodic()); + const amrex::Periodicity &period = amrex::Periodicity::NonPeriodic(), + std::optional nodal_sync=std::nullopt); void FillBoundary (amrex::MultiFab &mf, amrex::IntVect ng, @@ -77,7 +78,7 @@ void FillBoundary (amrex::iMultiFab& mf, void FillBoundary(amrex::Vector const &mf, bool do_single_precision_comms, - const amrex::Periodicity &period); + const amrex::Periodicity &period, std::optional nodal_sync=std::nullopt); void SumBoundary (amrex::MultiFab &mf, diff --git a/Source/ablastr/utils/Communication.cpp b/Source/ablastr/utils/Communication.cpp index 3ae18ad8d1c..c33e6a1d154 100644 --- a/Source/ablastr/utils/Communication.cpp +++ b/Source/ablastr/utils/Communication.cpp @@ -114,18 +114,18 @@ void FillBoundary (amrex::MultiFab &mf, } } -void FillBoundary (amrex::MultiFab &mf, bool do_single_precision_comms, const amrex::Periodicity &period) +void FillBoundary (amrex::MultiFab &mf, bool do_single_precision_comms, const amrex::Periodicity &period, std::optional nodal_sync) { amrex::IntVect const ng = mf.n_grow; - FillBoundary(mf, ng, do_single_precision_comms, period); + FillBoundary(mf, ng, do_single_precision_comms, period, nodal_sync); } void FillBoundary (amrex::Vector const &mf, bool do_single_precision_comms, - const amrex::Periodicity &period) + const amrex::Periodicity &period, std::optional nodal_sync) { for (auto x : mf) { - ablastr::utils::communication::FillBoundary(*x, do_single_precision_comms, period); + ablastr::utils::communication::FillBoundary(*x, do_single_precision_comms, period, nodal_sync); } } From 7d4a5076ed56ab6ac3d490dd1f223f04062a5fff Mon Sep 17 00:00:00 2001 From: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Date: Wed, 4 Oct 2023 10:31:20 -0700 Subject: [PATCH 040/110] Add SymPy notebook to derive PSATD equations in PML (#4316) --- Tools/Algorithms/psatd_pml.ipynb | 607 +++++++++++++++++++++++++++++++ 1 file changed, 607 insertions(+) create mode 100644 Tools/Algorithms/psatd_pml.ipynb diff --git a/Tools/Algorithms/psatd_pml.ipynb b/Tools/Algorithms/psatd_pml.ipynb new file mode 100644 index 00000000000..216be77fd87 --- /dev/null +++ b/Tools/Algorithms/psatd_pml.ipynb @@ -0,0 +1,607 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "import inspect\n", + "import sympy as sp\n", + "from sympy import *\n", + "from sympy.solvers.solveset import linsolve\n", + "\n", + "sp.init_session()\n", + "sp.init_printing()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Global variables:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Dimensional parameters\n", + "dd = 9\n", + "DD = 12\n", + "\n", + "# Speed of light, time, time step\n", + "c = sp.symbols(r'c', real = True, positive = True)\n", + "t = sp.symbols(r't', real = True)\n", + "tn = sp.symbols(r't_n', real = True)\n", + "dt = sp.symbols(r'\\Delta{t}', real = True, positive = True)\n", + "\n", + "# Components of k vector, omega, norm of k vector\n", + "kx = sp.symbols(r'k_x', real = True)\n", + "ky = sp.symbols(r'k_y', real = True)\n", + "kz = sp.symbols(r'k_z', real = True)\n", + "om = sp.symbols(r'omega', real = True, positive = True)\n", + "knorm = sp.sqrt(kx**2 + ky**2 + kz**2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Method:\n", + "The goal is to solve a system of second-order ordinary differential equations in time of the form $\\boldsymbol{\\ddot{X}} = M \\boldsymbol{X}$, where the $d \\times d$ matrix $M$ has eigenpairs $(\\lambda_i, \\{\\boldsymbol{v}_{i,1}, \\dots, \\boldsymbol{v}_{i,\\mu_i}\\})$, where $\\mu_i$ denotes the algebraic multiplicity of $\\lambda_i$ and $\\sum_i \\mu_i = d$.\n", + "Assuming that all eigenvalues are either zero or negative, we can write $\\lambda_i = - \\omega_i^2$, with $\\omega_i = \\sqrt{- \\lambda_i} \\geq 0$.\n", + "Then the solution of $\\boldsymbol{\\ddot{X}} = M \\boldsymbol{X}$ reads\n", + "$$\n", + "\\boldsymbol{X} = \\sum_i \\sum_j^{\\mu_i} \\left(a_{i,j} C(\\omega_i, t) + b_{i,j} S(\\omega_i, t) \\right) \\boldsymbol{v}_{i,j} \\,,\n", + "$$\n", + "where $a_{i,j}$ and $b_{i,j}$ are integration constants to be determined by the initial conditions $\\boldsymbol{X}(t_n)$ and $\\boldsymbol{\\dot{X}}(t_n)$, $C(\\omega, t) = \\cos(\\omega \\, (t - t_n))$, and\n", + "$$\n", + "S(\\omega, t) =\n", + "\\begin{cases}\n", + "(t - t_n) & \\omega = 0 \\,, \\\\\n", + "\\dfrac{\\sin(\\omega \\, (t - t_n))}{\\omega} & \\omega \\neq 0 \\,.\n", + "\\end{cases}\n", + "$$\n", + "We remark that\n", + "$$\n", + "\\begin{aligned}\n", + "& \\boldsymbol{X}(t_n) = \\sum_i \\sum_j^{\\mu_i} \\left(a_{i,j} C(\\omega_i, t_n) + b_{i,j} S(\\omega_i, t_n) \\right) \\boldsymbol{v}_{i,j} = \\sum_i \\sum_j^{\\mu_i} a_{i,j} \\boldsymbol{v}_{i,j} \\,, \\\\\n", + "& \\boldsymbol{\\dot{X}}(t_n) = \\sum_i \\sum_j^{\\mu_i} \\left(a_{i,j} \\dot{C}(\\omega_i, t_n) + b_{i,j} \\dot{S}(\\omega_i, t_n) \\right) \\boldsymbol{v}_{i,j} = \\sum_i \\sum_j^{\\mu_i} b_{i,j} \\boldsymbol{v}_{i,j} \\,.\n", + "\\end{aligned}\n", + "$$\n", + "In fact, the second time derivative of $\\boldsymbol{X}$ yields\n", + "$$\n", + "\\begin{split}\n", + "\\boldsymbol{\\ddot{X}} & = \\sum_i \\sum_j^{\\mu_i} \\left(a_{i,j} \\ddot{C}(\\omega_i, t) + b_{i,j} \\ddot{S}(\\omega_i, t) \\right) \\boldsymbol{v}_{i,j} \\\\\n", + "& = \\sum_i (-\\omega_i^2) \\sum_j^{\\mu_i} \\left(a_{i,j} C(\\omega_i, t) + b_{i,j} S(\\omega_i, t) \\right) \\boldsymbol{v}_{i,j} = M \\boldsymbol{X} \\,,\n", + "\\end{split}\n", + "$$\n", + "where we used the fact that, by definition, $M \\boldsymbol{v}_{i,j} = \\lambda_i \\boldsymbol{v}_{i,j} = - \\omega_i^2 \\boldsymbol{v}_{i,j}$ for all $j = 1, \\dots, \\mu_i$." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Auxiliary functions:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def C(omega, t):\n", + " return sp.cos(omega * (t-tn))\n", + "\n", + "def S(omega, t):\n", + " return (t-tn) if omega == 0. else sp.sin(omega * (t-tn)) / omega\n", + "\n", + "def Xt(eigenpairs, a, b, t):\n", + " \"\"\"\n", + " Compute X(t) according to the formulas above\n", + " for a given set of eigenpairs and coefficients.\n", + " \"\"\"\n", + " XX = zeros(DD, 1)\n", + " # Index used for the integration constants a_n and b_n\n", + " i = 0\n", + " # Loop over matrix eigenpairs\n", + " for ep in eigenpairs:\n", + " # ep[0] is an eigenvalue and om = sp.sqrt(-ep[0])\n", + " omega = 0. if ep[0] == 0. else om\n", + " # am is the algebraic multiplicity of the eigenvalue\n", + " am = ep[1]\n", + " # vF is the list of all eigenvectors corresponding to the eigenvalue\n", + " vX = ep[2]\n", + " # Loop over algebraic multiplicity of the eigenvalue\n", + " for j in range(am):\n", + " XX += (a[i] * C(omega, t) + b[i] * S(omega, t)) * vX[j]\n", + " i += 1\n", + " return XX\n", + "\n", + "def evolve(X, dX_dt, d2X_dt2):\n", + " \"\"\"\n", + " Solve ordinary differential equation X'' = M*X.\n", + " \"\"\"\n", + " # Set matrix for given ODE\n", + " MX = zeros(DD)\n", + " for i in range(DD):\n", + " for j in range(DD):\n", + " MX[i,j] = d2X_dt2[i].coeff(X[j], 1)\n", + " #MX /= c**2\n", + "\n", + " print()\n", + " print(r'Matrix:')\n", + " display(MX)\n", + "\n", + " # Characteristic matrix\n", + " lamda = sp.symbols(r'lamda')\n", + " Id = eye(DD)\n", + " MX_charmat = MX - lamda * Id\n", + "\n", + " # Characteristic polynomial\n", + " MX_charpoly = MX_charmat.det()\n", + " MX_charpoly = factor(MX_charpoly.as_expr())\n", + "\n", + " print(r'Characteristic polynomial:')\n", + " display(MX_charpoly)\n", + " \n", + " MX_eigenvals = sp.solve(MX_charpoly, lamda)\n", + "\n", + " # List of eigenvectors\n", + " MX_eigenvects = []\n", + "\n", + " # List of eigenpairs\n", + " MX_eigenpairs = []\n", + " \n", + " # Compute eigenvectors as null spaces\n", + " for l in MX_eigenvals:\n", + "\n", + " # M - lamda * Id\n", + " A = MX_charmat.subs(lamda, l)\n", + " A.simplify()\n", + "\n", + " print(r'Eigenvalue:')\n", + " display(l)\n", + "\n", + " print(r'Characteristic matrix:')\n", + " display(A)\n", + "\n", + " # Perform Gaussian elimination (necessary for lamda != 0)\n", + " if (l != 0.):\n", + " print(r'Gaussian elimination:')\n", + " print(r'A[0,:] += A[1,:]')\n", + " A[0,:] += A[1,:]\n", + " print(r'A[0,:] += A[2,:]')\n", + " A[0,:] += A[2,:]\n", + " print(r'Swap A[0,:] and A[11,:]')\n", + " row = A[11,:]\n", + " A[11,:] = A[0,:]\n", + " A[0,:] = row\n", + " print(r'A[3,:] += A[4,:]')\n", + " A[3,:] += A[4,:]\n", + " print(r'A[3,:] += A[5,:]')\n", + " A[3,:] += A[5,:]\n", + " print(r'Swap A[3,:] and A[10,:]')\n", + " row = A[10,:]\n", + " A[10,:] = A[3,:]\n", + " A[3,:] = row\n", + " print(r'A[0,:] += A[3,:]')\n", + " A[0,:] += A[3,:]\n", + " print(r'A[0,:] += A[9,:]')\n", + " A[0,:] += A[9,:]\n", + " print(r'Swap A[0,:] and A[9,:]')\n", + " row = A[9,:]\n", + " A[9,:] = A[0,:]\n", + " A[0,:] = row\n", + " print(r'A[6,:] += A[8,:]')\n", + " A[6,:] += A[8,:]\n", + " print(r'A[6,:] += A[7,:]')\n", + " A[6,:] += A[7,:]\n", + " print(r'Swap A[6,:] and A[8,:]')\n", + " row = A[8,:]\n", + " A[8,:] = A[6,:]\n", + " A[6,:] = row\n", + " print(r'A[6,:] += A[7,:]')\n", + " A[6,:] += A[7,:]\n", + " print(r'A[4,:] += A[5,:]')\n", + " A[4,:] += A[5,:]\n", + " A.simplify()\n", + " display(A)\n", + "\n", + " # Compute null space and store eigenvectors\n", + " v = A.nullspace()\n", + " MX_eigenvects.append(v)\n", + "\n", + " print(r'Eigenvectors:')\n", + " display(v)\n", + " \n", + " # Store eigenpairs (eigenvalue, algebraic multiplicity, eigenvectors)\n", + " MX_eigenpairs.append((l, len(v), v))\n", + " \n", + " #print(r'Eigenpairs:')\n", + " #display(MX_eigenpairs)\n", + "\n", + " # Verify that the eigenpairs satisfy the charcteristic equations\n", + " for ep in MX_eigenpairs:\n", + " for j in range(ep[1]):\n", + " diff = MX * ep[2][j] - ep[0] * ep[2][j]\n", + " diff.simplify()\n", + " if diff != zeros(DD,1):\n", + " print('The charcteristic equation is not verified for some eigenpairs')\n", + " display(diff)\n", + "\n", + " # Define integration constants\n", + " a = []\n", + " b = []\n", + " for i in range(DD):\n", + " an = r'a_{:d}'.format(i+1)\n", + " bn = r'b_{:d}'.format(i+1) \n", + " a.append(sp.symbols(an))\n", + " b.append(sp.symbols(bn))\n", + "\n", + " # Set equations corresponding to initial conditions\n", + " lhs_a = Xt(MX_eigenpairs, a, b, tn) - X\n", + " lhs_b = Xt(MX_eigenpairs, a, b, t ).diff(t).subs(t, tn) - dX_dt\n", + "\n", + " # Compute integration constants from initial conditions\n", + " # (convert list of tuples to list using list comprehension)\n", + " a = list(linsolve(list(lhs_a), a))\n", + " a = [item for el in a for item in el]\n", + " b = list(linsolve(list(lhs_b), b))\n", + " b = [item for el in b for item in el]\n", + "\n", + " # Evaluate solution at t = tn + dt\n", + " X_new = Xt(MX_eigenpairs, a, b, tn+dt).expand()\n", + " for d in range(DD):\n", + " for Eij in E:\n", + " X_new[d] = X_new[d].collect(Eij)\n", + " for Bij in B:\n", + " X_new[d] = X_new[d].collect(Bij)\n", + " for Fi in F:\n", + " X_new[d] = X_new[d].collect(Fi)\n", + " for Gi in G:\n", + " X_new[d] = X_new[d].collect(Gi)\n", + "\n", + " # Check correctness by taking *second* derivative\n", + " # and comparing with initial right-hand side at time tn\n", + " X_t = Xt(MX_eigenpairs, a, b, t)\n", + " diff = X_t.diff(t).diff(t).subs(t, tn).subs(om, c*knorm).expand() - d2X_dt2\n", + " diff.simplify()\n", + " if diff != zeros(DD,1):\n", + " print('Integration in time failed')\n", + " display(diff)\n", + " \n", + " return X_t, X_new" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### First-order and second-order ODEs for $\\boldsymbol{E}$, $\\boldsymbol{B}$, $F$ and $G$:\n", + "Equations for the $\\boldsymbol{E}$ field:\n", + "$$\n", + "\\begin{alignat*}{3}\n", + "& \\frac{\\partial E_{xx}}{\\partial t} = c^2 i k_x (F_x + F_y + F_z) \\quad\n", + "&& \\frac{\\partial E_{xy}}{\\partial t} = c^2 i k_y (B_{zx} + B_{zy} + B_{zz}) \\quad\n", + "&& \\frac{\\partial E_{xz}}{\\partial t} = -c^2 i k_z (B_{yx} + B_{yy} + B_{yz}) \\\\[5pt]\n", + "& \\frac{\\partial E_{yx}}{\\partial t} = -c^2 i k_x (B_{zx} + B_{zy} + B_{zz}) \\quad\n", + "&& \\frac{\\partial E_{yy}}{\\partial t} = c^2 i k_y (F_x + F_y + F_z) \\quad\n", + "&& \\frac{\\partial E_{yz}}{\\partial t} = c^2 i k_z (B_{xx} + B_{xy} + B_{xz}) \\\\[5pt]\n", + "& \\frac{\\partial E_{zx}}{\\partial t} = c^2 i k_x (B_{yx} + B_{yy} + B_{yz}) \\quad\n", + "&& \\frac{\\partial E_{zy}}{\\partial t} = -c^2 i k_y (B_{xx} + B_{xy} + B_{xz})\\quad\n", + "&& \\frac{\\partial E_{zz}}{\\partial t} = c^2 i k_z (F_x + F_y + F_z)\n", + "\\end{alignat*}\n", + "$$\n", + "\n", + "Equations for the $\\boldsymbol{B}$ field:\n", + "$$\n", + "\\begin{alignat*}{3}\n", + "& \\frac{\\partial B_{xx}}{\\partial t} = i k_x (G_x + G_y + G_z) \\quad\n", + "&& \\frac{\\partial B_{xy}}{\\partial t} = -i k_y (E_{zx} + E_{zy} + E_{zz}) \\quad\n", + "&& \\frac{\\partial B_{xz}}{\\partial t} = i k_z (E_{yx} + E_{yy} + E_{yz}) \\\\[5pt]\n", + "& \\frac{\\partial B_{yx}}{\\partial t} = i k_x (E_{zx} + E_{zy} + E_{zz}) \\quad\n", + "&& \\frac{\\partial B_{yy}}{\\partial t} = i k_y (G_x + G_y + G_z) \\quad\n", + "&& \\frac{\\partial B_{yz}}{\\partial t} = -i k_z (E_{xx} + E_{xy} + E_{xz}) \\\\[5pt]\n", + "& \\frac{\\partial B_{zx}}{\\partial t} = -i k_x (E_{yx} + E_{yy} + E_{yz}) \\quad\n", + "&& \\frac{\\partial B_{zy}}{\\partial t} = i k_y (E_{xx} + E_{xy} + E_{xz}) \\quad\n", + "&& \\frac{\\partial B_{zz}}{\\partial t} = i k_z (G_x + G_y + G_z)\n", + "\\end{alignat*}\n", + "$$\n", + "\n", + "Equations for the $F$ field:\n", + "$$\n", + "\\frac{\\partial F_x}{\\partial t} = i k_x (E_{xx} + E_{xy} + E_{xz}) \\quad\n", + "\\frac{\\partial F_y}{\\partial t} = i k_y (E_{yx} + E_{yy} + E_{yz}) \\quad\n", + "\\frac{\\partial F_z}{\\partial t} = i k_z (E_{zx} + E_{zy} + E_{zz})\n", + "$$\n", + "\n", + "Equations for the $G$ field:\n", + "$$\n", + "\\frac{\\partial G_x}{\\partial t} = c^2 i k_x (B_{xx} + B_{xy} + B_{xz}) \\quad\n", + "\\frac{\\partial G_y}{\\partial t} = c^2 i k_y (B_{yx} + B_{yy} + B_{yz}) \\quad\n", + "\\frac{\\partial G_z}{\\partial t} = c^2 i k_z (B_{zx} + B_{zy} + B_{zz})\n", + "$$" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false + }, + "outputs": [], + "source": [ + "# indices 0 1 2 3 4 5 6 7 8\n", + "labels = ['xx', 'xy', 'xz', 'yx', 'yy', 'yz', 'zx', 'zy', 'zz']\n", + "\n", + "# E fields\n", + "Exx = sp.symbols(r'E_{xx}')\n", + "Exy = sp.symbols(r'E_{xy}')\n", + "Exz = sp.symbols(r'E_{xz}')\n", + "Eyx = sp.symbols(r'E_{yx}')\n", + "Eyy = sp.symbols(r'E_{yy}')\n", + "Eyz = sp.symbols(r'E_{yz}')\n", + "Ezx = sp.symbols(r'E_{zx}')\n", + "Ezy = sp.symbols(r'E_{zy}')\n", + "Ezz = sp.symbols(r'E_{zz}')\n", + "E = Matrix([[Exx],[Exy],[Exz],[Eyx],[Eyy],[Eyz],[Ezx],[Ezy],[Ezz]])\n", + "\n", + "# B fields\n", + "Bxx = sp.symbols(r'B_{xx}')\n", + "Bxy = sp.symbols(r'B_{xy}')\n", + "Bxz = sp.symbols(r'B_{xz}')\n", + "Byx = sp.symbols(r'B_{yx}')\n", + "Byy = sp.symbols(r'B_{yy}')\n", + "Byz = sp.symbols(r'B_{yz}')\n", + "Bzx = sp.symbols(r'B_{zx}')\n", + "Bzy = sp.symbols(r'B_{zy}')\n", + "Bzz = sp.symbols(r'B_{zz}')\n", + "B = Matrix([[Bxx],[Bxy],[Bxz],[Byx],[Byy],[Byz],[Bzx],[Bzy],[Bzz]])\n", + "\n", + "# F fields\n", + "Fx = sp.symbols(r'F_{x}')\n", + "Fy = sp.symbols(r'F_{y}')\n", + "Fz = sp.symbols(r'F_{z}')\n", + "F = Matrix([[Fx],[Fy],[Fz]])\n", + "\n", + "# G fields\n", + "Gx = sp.symbols(r'G_{x}')\n", + "Gy = sp.symbols(r'G_{y}')\n", + "Gz = sp.symbols(r'G_{z}')\n", + "G = Matrix([[Gx],[Gy],[Gz]])\n", + "\n", + "# dE/dt\n", + "dExx_dt = c**2 * I * kx * (Fx + Fy + Fz)\n", + "dExy_dt = c**2 * I * ky * (Bzx + Bzy + Bzz)\n", + "dExz_dt = - c**2 * I * kz * (Byx + Byy + Byz)\n", + "dEyx_dt = - c**2 * I * kx * (Bzx + Bzy + Bzz)\n", + "dEyy_dt = c**2 * I * ky * (Fx + Fy + Fz)\n", + "dEyz_dt = c**2 * I * kz * (Bxx + Bxy + Bxz)\n", + "dEzx_dt = c**2 * I * kx * (Byx + Byy + Byz)\n", + "dEzy_dt = - c**2 * I * ky * (Bxx + Bxy + Bxz)\n", + "dEzz_dt = c**2 * I * kz * (Fx + Fy + Fz)\n", + "dE_dt = Matrix([[dExx_dt],[dExy_dt],[dExz_dt],[dEyx_dt],[dEyy_dt],[dEyz_dt],[dEzx_dt],[dEzy_dt],[dEzz_dt]])\n", + "\n", + "# dB/dt\n", + "dBxx_dt = I * kx * (Gx + Gy + Gz)\n", + "dBxy_dt = - I * ky * (Ezx + Ezy + Ezz)\n", + "dBxz_dt = I * kz * (Eyx + Eyy + Eyz)\n", + "dByx_dt = I * kx * (Ezx + Ezy + Ezz)\n", + "dByy_dt = I * ky * (Gx + Gy + Gz)\n", + "dByz_dt = - I * kz * (Exx + Exy + Exz)\n", + "dBzx_dt = - I * kx * (Eyx + Eyy + Eyz)\n", + "dBzy_dt = I * ky * (Exx + Exy + Exz)\n", + "dBzz_dt = I * kz * (Gx + Gy + Gz)\n", + "dB_dt = Matrix([[dBxx_dt],[dBxy_dt],[dBxz_dt],[dByx_dt],[dByy_dt],[dByz_dt],[dBzx_dt],[dBzy_dt],[dBzz_dt]])\n", + "\n", + "# dF/dt\n", + "dFx_dt = I * kx * (Exx + Exy + Exz)\n", + "dFy_dt = I * ky * (Eyx + Eyy + Eyz)\n", + "dFz_dt = I * kz * (Ezx + Ezy + Ezz)\n", + "dF_dt = Matrix([[dFx_dt],[dFy_dt],[dFz_dt]])\n", + "\n", + "# dG/dt\n", + "dGx_dt = c**2 * I * kx * (Bxx + Bxy + Bxz)\n", + "dGy_dt = c**2 * I * ky * (Byx + Byy + Byz)\n", + "dGz_dt = c**2 * I * kz * (Bzx + Bzy + Bzz)\n", + "dG_dt = Matrix([[dGx_dt],[dGy_dt],[dGz_dt]])\n", + "\n", + "# d2E/dt2\n", + "d2Exx_dt2 = c**2 * I * kx * (dFx_dt + dFy_dt + dFz_dt)\n", + "d2Exy_dt2 = c**2 * I * ky * (dBzx_dt + dBzy_dt + dBzz_dt)\n", + "d2Exz_dt2 = - c**2 * I * kz * (dByx_dt + dByy_dt + dByz_dt)\n", + "d2Eyx_dt2 = - c**2 * I * kx * (dBzx_dt + dBzy_dt + dBzz_dt)\n", + "d2Eyy_dt2 = c**2 * I * ky * (dFx_dt + dFy_dt + dFz_dt)\n", + "d2Eyz_dt2 = c**2 * I * kz * (dBxx_dt + dBxy_dt + dBxz_dt)\n", + "d2Ezx_dt2 = c**2 * I * kx * (dByx_dt + dByy_dt + dByz_dt)\n", + "d2Ezy_dt2 = - c**2 * I * ky * (dBxx_dt + dBxy_dt + dBxz_dt)\n", + "d2Ezz_dt2 = c**2 * I * kz * (dFx_dt + dFy_dt + dFz_dt)\n", + "d2E_dt2 = Matrix([[d2Exx_dt2],[d2Exy_dt2],[d2Exz_dt2],[d2Eyx_dt2],[d2Eyy_dt2],[d2Eyz_dt2],[d2Ezx_dt2],[d2Ezy_dt2],[d2Ezz_dt2]])\n", + "\n", + "# d2B/dt2\n", + "d2Bxx_dt2 = I * kx * (dGx_dt + dGy_dt + dGz_dt)\n", + "d2Bxy_dt2 = - I * ky * (dEzx_dt + dEzy_dt + dEzz_dt)\n", + "d2Bxz_dt2 = I * kz * (dEyx_dt + dEyy_dt + dEyz_dt)\n", + "d2Byx_dt2 = I * kx * (dEzx_dt + dEzy_dt + dEzz_dt)\n", + "d2Byy_dt2 = I * ky * (dGx_dt + dGy_dt + dGz_dt)\n", + "d2Byz_dt2 = - I * kz * (dExx_dt + dExy_dt + dExz_dt)\n", + "d2Bzx_dt2 = - I * kx * (dEyx_dt + dEyy_dt + dEyz_dt)\n", + "d2Bzy_dt2 = I * ky * (dExx_dt + dExy_dt + dExz_dt)\n", + "d2Bzz_dt2 = I * kz * (dGx_dt + dGy_dt + dGz_dt)\n", + "d2B_dt2 = Matrix([[d2Bxx_dt2],[d2Bxy_dt2],[d2Bxz_dt2],[d2Byx_dt2],[d2Byy_dt2],[d2Byz_dt2],[d2Bzx_dt2],[d2Bzy_dt2],[d2Bzz_dt2]])\n", + "\n", + "# d2F/dt2\n", + "d2Fx_dt2 = I * kx * (dExx_dt + dExy_dt + dExz_dt)\n", + "d2Fy_dt2 = I * ky * (dEyx_dt + dEyy_dt + dEyz_dt)\n", + "d2Fz_dt2 = I * kz * (dEzx_dt + dEzy_dt + dEzz_dt)\n", + "d2F_dt2 = Matrix([[d2Fx_dt2],[d2Fy_dt2],[d2Fz_dt2]])\n", + "\n", + "# d2G/dt2\n", + "d2Gx_dt2 = c**2 * I * kx * (dBxx_dt + dBxy_dt + dBxz_dt)\n", + "d2Gy_dt2 = c**2 * I * ky * (dByx_dt + dByy_dt + dByz_dt)\n", + "d2Gz_dt2 = c**2 * I * kz * (dBzx_dt + dBzy_dt + dBzz_dt)\n", + "d2G_dt2 = Matrix([[d2Gx_dt2],[d2Gy_dt2],[d2Gz_dt2]])\n", + "\n", + "for i in range(dd):\n", + " d2E_dt2[i] = sp.expand(d2E_dt2[i])\n", + "\n", + "for i in range(dd):\n", + " d2B_dt2[i] = sp.expand(d2B_dt2[i])\n", + "\n", + "for i in range(3):\n", + " d2F_dt2[i] = sp.expand(d2F_dt2[i])\n", + " \n", + "for i in range(3):\n", + " d2G_dt2[i] = sp.expand(d2G_dt2[i])\n", + " \n", + "# Extended array for E and G\n", + "EG = zeros(DD,1)\n", + "for i in range(dd):\n", + " EG[i] = E[i]\n", + "for i in range(dd,DD):\n", + " EG[i] = G[i-dd]\n", + "\n", + "# dEG/dt\n", + "dEG_dt = zeros(DD,1)\n", + "for i in range(dd):\n", + " dEG_dt[i] = dE_dt[i]\n", + "for i in range(dd,DD):\n", + " dEG_dt[i] = dG_dt[i-dd]\n", + " \n", + "# d2EG/dt2\n", + "d2EG_dt2 = zeros(DD,1)\n", + "for i in range(dd):\n", + " d2EG_dt2[i] = d2E_dt2[i]\n", + "for i in range(dd,DD):\n", + " d2EG_dt2[i] = d2G_dt2[i-dd]\n", + " \n", + "# Extended array for B and F\n", + "BF = zeros(DD,1)\n", + "for i in range(dd):\n", + " BF[i] = B[i]\n", + "for i in range(dd,DD):\n", + " BF[i] = F[i-dd]\n", + "\n", + "# dBF/dt\n", + "dBF_dt = zeros(DD,1)\n", + "for i in range(dd):\n", + " dBF_dt[i] = dB_dt[i]\n", + "for i in range(dd,DD):\n", + " dBF_dt[i] = dF_dt[i-dd]\n", + "\n", + "# d2BF/dt2\n", + "d2BF_dt2 = zeros(DD,1)\n", + "for i in range(dd):\n", + " d2BF_dt2[i] = d2B_dt2[i]\n", + "for i in range(dd,DD):\n", + " d2BF_dt2[i] = d2F_dt2[i-dd]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solve second-order ODEs for $\\boldsymbol{E}$, $\\boldsymbol{B}$, $F$ and $G$:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false + }, + "outputs": [], + "source": [ + "print(r'Solve equations for E and G:')\n", + "EG_t, EG_new = evolve(EG, dEG_dt, d2EG_dt2)\n", + "\n", + "print(r'Solve equations for B and F:')\n", + "BF_t, BF_new = evolve(BF, dBF_dt, d2BF_dt2)\n", + "\n", + "# Check correctness by taking *first* derivative\n", + "# and comparing with initial right-hand side at time tn\n", + "# E,G\n", + "diff = EG_t.diff(t).subs(t, tn).subs(om, c*knorm).expand() - dEG_dt\n", + "diff.simplify()\n", + "if diff != zeros(DD,1):\n", + " print('Integration in time failed')\n", + " display(diff)\n", + "# B,F\n", + "diff = BF_t.diff(t).subs(t, tn).subs(om, c*knorm).expand() - dBF_dt\n", + "diff.simplify()\n", + "if diff != zeros(DD,1):\n", + " print('Integration in time failed')\n", + " display(diff)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Coefficients of PSATD equations in PML:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false + }, + "outputs": [], + "source": [ + "# Code generation\n", + "from sympy.codegen.ast import Assignment\n", + "\n", + "# 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11\n", + "# EG: Exx, Exy, Exz, Eyx, Eyy, Eyz, Ezx, Ezy, Ezz, Gx, Gy, Gz\n", + "# BF: Bxx, Bxy, Bxz, Byx, Byy, Byz, Bzx, Bzy, Bzz, Fx, Fy, Fz\n", + "\n", + "# Select update equation (left hand side)\n", + "X_new = BF_new[0]\n", + "\n", + "# Extract individual terms (right hand side)\n", + "for i in range(DD):\n", + " X = EG[i]\n", + " C = X_new.coeff(X, 1).simplify()\n", + " print(r'Coefficient multiplying ' + str(X))\n", + " display(C)\n", + " #print(ccode(Assignment(sp.symbols(r'LHS'), C)))\n", + "for i in range(DD):\n", + " X = BF[i]\n", + " C = X_new.coeff(X, 1).simplify()\n", + " print(r'Coefficient multiplying ' + str(X))\n", + " display(C)\n", + " #print(ccode(Assignment(sp.symbols(r'LHS'), C)))" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.10" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} From 9dc9d2a8a62e5139485be4d174c34beeeb2eb4d9 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Wed, 4 Oct 2023 14:31:15 -0700 Subject: [PATCH 041/110] CI: Unbreak macOS (libomp) (#4341) Something changed in the base image again. --- .github/workflows/macos.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml index 2fd979feff9..c81ee598114 100644 --- a/.github/workflows/macos.yml +++ b/.github/workflows/macos.yml @@ -45,7 +45,7 @@ jobs: brew install ccache brew install fftw brew install libomp - brew link --force libomp + brew link --overwrite --force libomp brew install ninja brew install open-mpi brew install pkg-config From 01f90d89973f4c8f777333b50a716d6e5c746628 Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Wed, 4 Oct 2023 14:31:37 -0700 Subject: [PATCH 042/110] Close #4331 segfault when activating nuclearfusion collisions (#4338) * Raise proper error message when specifying product species * Fix case where 0 particles should be created * Clang-Tidy: Disable `modernize-return-braced-init-list` Buggy transform for non-init-list constructors. --------- Co-authored-by: Axel Huebl --- .clang-tidy | 2 +- Source/Particles/Collision/BinaryCollision/BinaryCollision.H | 4 ++++ .../Collision/BinaryCollision/ParticleCreationFunc.H | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index 42f4fc36065..f40dd987add 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -33,10 +33,10 @@ Checks: '-*, modernize-replace-auto-ptr, modernize-replace-disallow-copy-and-assign-macro, modernize-replace-random-shuffle, - modernize-return-braced-init-list, modernize-shrink-to-fit, modernize-unary-static-assert, modernize-use-nullptr, + -modernize-return-braced-init-list, mpi-*, performance-faster-string-find, performance-for-range-copy, diff --git a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H index 5473f0093f0..9cf2f91b5aa 100644 --- a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H +++ b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H @@ -97,6 +97,10 @@ public: const amrex::ParmParse pp_collision_name(collision_name); pp_collision_name.queryarr("product_species", m_product_species); m_have_product_species = !m_product_species.empty(); + if ((std::is_same::value) & (m_have_product_species)) { + WARPX_ABORT_WITH_MESSAGE( "Binary collision " + collision_name + + " does not produce species. Thus, `product_species` should not be specified in the input script." ); + } m_copy_transform_functor = CopyTransformFunctorType(collision_name, mypc); } diff --git a/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H b/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H index 2b43f76e89f..7fdf04de6ca 100644 --- a/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H +++ b/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H @@ -115,7 +115,7 @@ public: { using namespace amrex::literals; - if (n_total_pairs == 0) return {m_num_product_species, 0}; + if (n_total_pairs == 0) return amrex::Vector(m_num_product_species, 0); // Compute offset array and allocate memory for the produced species amrex::Gpu::DeviceVector offsets(n_total_pairs); From 1ad0dbc9166dd204a2a40c932fa8e366bd0eab67 Mon Sep 17 00:00:00 2001 From: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Date: Wed, 4 Oct 2023 18:39:51 -0700 Subject: [PATCH 043/110] Momentum-conserving gather for MR ratio higher than 2 (#1650) * Momentum-Conserving Interpolation for Arbitrary Mesh Refinement * Implement 1D Case, Cleaning * Add Zero Padding * Add Langmuir Test (FDTD) * Fix CI test * Abort when using PSATD + MR + momentum-conserving * Apply suggestions from code review * Improve docstring --- ...gmuir_multi_2d_MR_momentum_conserving.json | 40 + Regression/WarpX-tests.ini | 19 + Source/Parallelization/WarpXComm.cpp | 32 +- Source/Parallelization/WarpXComm_K.H | 681 +++++------------- Source/WarpX.cpp | 12 + 5 files changed, 273 insertions(+), 511 deletions(-) create mode 100644 Regression/Checksum/benchmarks_json/Langmuir_multi_2d_MR_momentum_conserving.json diff --git a/Regression/Checksum/benchmarks_json/Langmuir_multi_2d_MR_momentum_conserving.json b/Regression/Checksum/benchmarks_json/Langmuir_multi_2d_MR_momentum_conserving.json new file mode 100644 index 00000000000..72c11245e82 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/Langmuir_multi_2d_MR_momentum_conserving.json @@ -0,0 +1,40 @@ +{ + "electrons": { + "particle_momentum_x": 4.2339125386796835e-20, + "particle_momentum_y": 0.0, + "particle_momentum_z": 4.2339186493637336e-20, + "particle_position_x": 0.6553603030161138, + "particle_position_y": 0.6553603026454923, + "particle_weight": 3200000000000000.5 + }, + "lev=0": { + "Bx": 0.0, + "By": 44.73189518299763, + "Bz": 0.0, + "Ex": 7583674122515.894, + "Ey": 0.0, + "Ez": 7583672042436.007, + "jx": 7283798122345162.0, + "jy": 0.0, + "jz": 7283804678733656.0 + }, + "lev=1": { + "Bx": 0.0, + "By": 467.8271651312273, + "Bz": 0.0, + "Ex": 7609732451490.526, + "Ey": 0.0, + "Ez": 7604891944279.171, + "jx": 7103859890231833.0, + "jy": 0.0, + "jz": 7099873215674540.0 + }, + "positrons": { + "particle_momentum_x": 4.233907500710132e-20, + "particle_momentum_y": 0.0, + "particle_momentum_z": 4.2339135644675305e-20, + "particle_position_x": 0.6553596934061604, + "particle_position_y": 0.6553596937869579, + "particle_weight": 3200000000000000.5 + } +} \ No newline at end of file diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index e4c502d858c..9905f77f34a 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -1225,6 +1225,25 @@ particleTypes = electrons positrons analysisRoutine = Examples/Tests/langmuir/analysis_2d.py analysisOutputImage = Langmuir_multi_2d_MR.png +[Langmuir_multi_2d_MR_momentum_conserving] +buildDir = . +inputFile = Examples/Tests/langmuir/inputs_2d +runtime_params = algo.maxwell_solver=ckc warpx.use_filter=1 amr.max_level=1 amr.ref_ratio=4 warpx.fine_tag_lo=-10.e-6 -10.e-6 warpx.fine_tag_hi=10.e-6 10.e-6 algo.field_gathering=momentum-conserving diag1.electrons.variables=w ux uy uz diag1.positrons.variables=w ux uy uz +dim = 2 +addToCompileString = +cmakeSetupOpts = -DWarpX_DIMS=2 +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 1 +particleTypes = electrons positrons +analysisRoutine = Examples/Tests/langmuir/analysis_2d.py +analysisOutputImage = Langmuir_multi_2d_MR_momentum_conserving.png + [Langmuir_multi_2d_MR_psatd] buildDir = . inputFile = Examples/Tests/langmuir/inputs_2d diff --git a/Source/Parallelization/WarpXComm.cpp b/Source/Parallelization/WarpXComm.cpp index 8c87cc9c772..e5ede95272c 100644 --- a/Source/Parallelization/WarpXComm.cpp +++ b/Source/Parallelization/WarpXComm.cpp @@ -181,6 +181,16 @@ WarpX::UpdateAuxilaryDataStagToNodal () ng_src, ng, WarpX::do_single_precision_comms, cperiod); } + const amrex::IntVect& refinement_ratio = refRatio(lev-1); + + const amrex::IntVect& Bx_fp_stag = Bfield_fp[lev][0]->ixType().toIntVect(); + const amrex::IntVect& By_fp_stag = Bfield_fp[lev][1]->ixType().toIntVect(); + const amrex::IntVect& Bz_fp_stag = Bfield_fp[lev][2]->ixType().toIntVect(); + + const amrex::IntVect& Bx_cp_stag = Bfield_cp[lev][0]->ixType().toIntVect(); + const amrex::IntVect& By_cp_stag = Bfield_cp[lev][1]->ixType().toIntVect(); + const amrex::IntVect& Bz_cp_stag = Bfield_cp[lev][2]->ixType().toIntVect(); + #ifdef AMREX_USE_OMP #pragma omp parallel if (Gpu::notInLaunchRegion()) #endif @@ -203,9 +213,9 @@ WarpX::UpdateAuxilaryDataStagToNodal () amrex::ParallelFor(bx, [=] AMREX_GPU_DEVICE (int j, int k, int l) noexcept { - warpx_interp_nd_bfield_x(j,k,l, bx_aux, bx_fp, bx_cp, bx_c); - warpx_interp_nd_bfield_y(j,k,l, by_aux, by_fp, by_cp, by_c); - warpx_interp_nd_bfield_z(j,k,l, bz_aux, bz_fp, bz_cp, bz_c); + warpx_interp(j, k, l, bx_aux, bx_fp, bx_cp, bx_c, Bx_fp_stag, Bx_cp_stag, refinement_ratio); + warpx_interp(j, k, l, by_aux, by_fp, by_cp, by_c, By_fp_stag, By_cp_stag, refinement_ratio); + warpx_interp(j, k, l, bz_aux, bz_fp, bz_cp, bz_c, Bz_fp_stag, Bz_cp_stag, refinement_ratio); }); } } @@ -239,6 +249,16 @@ WarpX::UpdateAuxilaryDataStagToNodal () ng_src, ng, WarpX::do_single_precision_comms, cperiod); } + const amrex::IntVect& refinement_ratio = refRatio(lev-1); + + const amrex::IntVect& Ex_fp_stag = Efield_fp[lev][0]->ixType().toIntVect(); + const amrex::IntVect& Ey_fp_stag = Efield_fp[lev][1]->ixType().toIntVect(); + const amrex::IntVect& Ez_fp_stag = Efield_fp[lev][2]->ixType().toIntVect(); + + const amrex::IntVect& Ex_cp_stag = Efield_cp[lev][0]->ixType().toIntVect(); + const amrex::IntVect& Ey_cp_stag = Efield_cp[lev][1]->ixType().toIntVect(); + const amrex::IntVect& Ez_cp_stag = Efield_cp[lev][2]->ixType().toIntVect(); + #ifdef AMREX_USE_OMP #pragma omp parallel if (Gpu::notInLaunchRegion()) #endif @@ -261,9 +281,9 @@ WarpX::UpdateAuxilaryDataStagToNodal () amrex::ParallelFor(bx, [=] AMREX_GPU_DEVICE (int j, int k, int l) noexcept { - warpx_interp_nd_efield_x(j,k,l, ex_aux, ex_fp, ex_cp, ex_c); - warpx_interp_nd_efield_y(j,k,l, ey_aux, ey_fp, ey_cp, ey_c); - warpx_interp_nd_efield_z(j,k,l, ez_aux, ez_fp, ez_cp, ez_c); + warpx_interp(j, k, l, ex_aux, ex_fp, ex_cp, ex_c, Ex_fp_stag, Ex_cp_stag, refinement_ratio); + warpx_interp(j, k, l, ey_aux, ey_fp, ey_cp, ey_c, Ey_fp_stag, Ey_cp_stag, refinement_ratio); + warpx_interp(j, k, l, ez_aux, ez_fp, ez_cp, ez_c, Ez_fp_stag, Ez_cp_stag, refinement_ratio); }); } } diff --git a/Source/Parallelization/WarpXComm_K.H b/Source/Parallelization/WarpXComm_K.H index b97773af251..4cec33bc84f 100644 --- a/Source/Parallelization/WarpXComm_K.H +++ b/Source/Parallelization/WarpXComm_K.H @@ -10,6 +10,20 @@ #include #include +/** + * \brief Interpolation function called within WarpX::UpdateAuxilaryDataSameType + * to interpolate data from the coarse and fine grids to the fine aux grid, + * assuming that all grids have the same staggering (either collocated or staggered). + * + * \param[in] j index along x of the output array + * \param[in] k index along y (in 3D) or z (in 2D) of the output array + * \param[in] l index along z (in 3D, l=0 in 2D) of the output array + * \param[in,out] arr_aux output array where interpolated values are stored + * \param[in] arr_fine input fine-patch array storing the values to interpolate + * \param[in] arr_coarse input coarse-patch array storing the values to interpolate + * \param[in] arr_stag IndexType of the arrays + * \param[in] rr mesh refinement ratios along each direction + */ AMREX_GPU_DEVICE AMREX_FORCE_INLINE void warpx_interp (int j, int k, int l, amrex::Array4 const& arr_aux, @@ -68,561 +82,218 @@ void warpx_interp (int j, int k, int l, arr_aux(j,k,l) = arr_fine(j,k,l) + res; } +/** + * \brief Interpolation function called within WarpX::UpdateAuxilaryDataStagToNodal + * to interpolate data from the coarse and fine grids to the fine aux grid, + * with momentum-conserving field gathering, hence between grids with different staggering, + * and assuming that the aux grid is collocated. + * + * \param[in] j index along x of the output array + * \param[in] k index along y (in 3D) or z (in 2D) of the output array + * \param[in] l index along z (in 3D, l=0 in 2D) of the output array + * \param[in,out] arr_aux output array where interpolated values are stored + * \param[in] arr_fine input fine-patch array storing the values to interpolate + * \param[in] arr_coarse input coarse-patch array storing the values to interpolate + * \param[in] arr_fine_stag IndexType of the fine-patch arrays + * \param[in] arr_coarse_stag IndexType of the coarse-patch arrays + * \param[in] rr mesh refinement ratios along each direction + */ AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void warpx_interp_nd_bfield_x (int j, int k, int l, - amrex::Array4 const& Bxa, - amrex::Array4 const& Bxf, - amrex::Array4 const& Bxc, - amrex::Array4 const& Bxg) +void warpx_interp (int j, int k, int l, + amrex::Array4 const& arr_aux, + amrex::Array4 const& arr_fine, + amrex::Array4 const& arr_coarse, + amrex::Array4 const& arr_tmp, + const amrex::IntVect& arr_fine_stag, + const amrex::IntVect& arr_coarse_stag, + const amrex::IntVect& rr) { using namespace amrex; - // Pad Bxf with zeros beyond ghost cells for out-of-bound accesses - const auto Bxf_zeropad = [Bxf] (const int jj, const int kk, const int ll) noexcept + // Pad input arrays with zeros beyond ghost cells + // for out-of-bound accesses due to large-stencil operations + const auto arr_fine_zeropad = [arr_fine] (const int jj, const int kk, const int ll) noexcept { - return Bxf.contains(jj,kk,ll) ? Bxf(jj,kk,ll) : 0.0_rt; + return arr_fine.contains(jj,kk,ll) ? arr_fine(jj,kk,ll) : 0.0_rt; + }; + const auto arr_coarse_zeropad = [arr_coarse] (const int jj, const int kk, const int ll) noexcept + { + return arr_coarse.contains(jj,kk,ll) ? arr_coarse(jj,kk,ll) : 0.0_rt; + }; + const auto arr_tmp_zeropad = [arr_tmp] (const int jj, const int kk, const int ll) noexcept + { + return arr_tmp.contains(jj,kk,ll) ? arr_tmp(jj,kk,ll) : 0.0_rt; }; - const int jg = amrex::coarsen(j,2); - const Real wx = (j == jg*2) ? 0.0_rt : 0.5_rt; - const Real owx = 1.0_rt-wx; - - const int kg = amrex::coarsen(k,2); - Real wy = 0.0_rt; - Real owy = 0.0_rt; - wy = (k == kg*2) ? 0.0_rt : 0.5_rt; - owy = 1.0_rt-wy; + // NOTE Indices (j,k,l) in the following refer to: + // - (z,-,-) in 1D + // - (x,z,-) in 2D + // - (r,z,-) in RZ + // - (x,y,z) in 3D + // Refinement ratio + const int rj = rr[0]; #if defined(WARPX_DIM_1D_Z) - - // interp from coarse nodal to fine nodal - const Real bg = owx * Bxg(jg ,0,0) - + wx * Bxg(jg+1,0,0); - - // interp from coarse staggered to fine nodal - const Real bc = owx * Bxc(jg ,0,0) - + wx * Bxc(jg+1,0,0); - - // interp from fine staggered to fine nodal - const Real bf = 0.5_rt*(Bxf_zeropad(j-1,0,0) + Bxf_zeropad(j,0,0)); - amrex::ignore_unused(owy); - + constexpr int rk = 1; + constexpr int rl = 1; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - - // interp from coarse nodal to fine nodal - const Real bg = owx * owy * Bxg(jg ,kg ,0) - + owx * wy * Bxg(jg ,kg+1,0) - + wx * owy * Bxg(jg+1,kg ,0) - + wx * wy * Bxg(jg+1,kg+1,0); - - // interp from coarse staggered to fine nodal - wy = 0.5_rt-wy; owy = 1.0_rt-wy; - const Real bc = owx * owy * Bxc(jg ,kg ,0) - + owx * wy * Bxc(jg ,kg-1,0) - + wx * owy * Bxc(jg+1,kg ,0) - + wx * wy * Bxc(jg+1,kg-1,0); - - // interp from fine staggered to fine nodal - const Real bf = 0.5_rt*(Bxf_zeropad(j,k-1,0) + Bxf_zeropad(j,k,0)); - + const int rk = rr[1]; + constexpr int rl = 1; #else - - const int lg = amrex::coarsen(l,2); - Real wz = (l == lg*2) ? 0.0_rt : 0.5_rt; - Real owz = 1.0_rt-wz; - - // interp from coarse nodal to fine nodal - const Real bg = owx * owy * owz * Bxg(jg ,kg ,lg ) - + wx * owy * owz * Bxg(jg+1,kg ,lg ) - + owx * wy * owz * Bxg(jg ,kg+1,lg ) - + wx * wy * owz * Bxg(jg+1,kg+1,lg ) - + owx * owy * wz * Bxg(jg ,kg ,lg+1) - + wx * owy * wz * Bxg(jg+1,kg ,lg+1) - + owx * wy * wz * Bxg(jg ,kg+1,lg+1) - + wx * wy * wz * Bxg(jg+1,kg+1,lg+1); - - // interp from coarse staggered to fine nodal - wy = 0.5_rt-wy; owy = 1.0_rt-wy; - wz = 0.5_rt-wz; owz = 1.0_rt-wz; - const Real bc = owx * owy * owz * Bxc(jg ,kg ,lg ) - + wx * owy * owz * Bxc(jg+1,kg ,lg ) - + owx * wy * owz * Bxc(jg ,kg-1,lg ) - + wx * wy * owz * Bxc(jg+1,kg-1,lg ) - + owx * owy * wz * Bxc(jg ,kg ,lg-1) - + wx * owy * wz * Bxc(jg+1,kg ,lg-1) - + owx * wy * wz * Bxc(jg ,kg-1,lg-1) - + wx * wy * wz * Bxc(jg+1,kg-1,lg-1); - - // interp from fine stagged to fine nodal - const Real bf = 0.25_rt*(Bxf_zeropad(j,k-1,l-1) + Bxf_zeropad(j,k,l-1) + Bxf_zeropad(j,k-1,l) + Bxf_zeropad(j,k,l)); + const int rk = rr[1]; + const int rl = rr[2]; #endif - Bxa(j,k,l) = bg + (bf-bc); -} - -AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void warpx_interp_nd_bfield_y (int j, int k, int l, - amrex::Array4 const& Bya, - amrex::Array4 const& Byf, - amrex::Array4 const& Byc, - amrex::Array4 const& Byg) -{ - using namespace amrex; - - // Pad Byf with zeros beyond ghost cells for out-of-bound accesses - const auto Byf_zeropad = [Byf] (const int jj, const int kk, const int ll) noexcept - { - return Byf.contains(jj,kk,ll) ? Byf(jj,kk,ll) : 0.0_rt; - }; - - const int jg = amrex::coarsen(j,2); - Real wx, owx; - wx = (j == jg*2) ? 0.0_rt : 0.5_rt; - owx = 1.0_rt-wx; - - const int kg = amrex::coarsen(k,2); - Real wy = 0.0_rt; - Real owy = 0.0_rt; - wy = (k == kg*2) ? 0.0_rt : 0.5_rt; - owy = 1.0_rt-wy; + // Staggering of fine array (0: cell-centered; 1: nodal) + const int sj_fp = arr_fine_stag[0]; +#if defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) + const int sk_fp = arr_fine_stag[1]; +#elif defined(WARPX_DIM_3D) + const int sk_fp = arr_fine_stag[1]; + const int sl_fp = arr_fine_stag[2]; +#endif + // Staggering of coarse array (0: cell-centered; 1: nodal) + const int sj_cp = arr_coarse_stag[0]; #if defined(WARPX_DIM_1D_Z) - - // interp from coarse nodal to fine nodal - const Real bg = owx * Byg(jg ,0,0) - + wx * Byg(jg+1,0,0); - - // interp from coarse staggered to fine nodal - const Real bc = owx * Byc(jg ,0,0) - + wx * Byc(jg+1,0,0); - - // interp from fine staggered to fine nodal - const Real bf = 0.5_rt*(Byf_zeropad(j-1,0,0) + Byf_zeropad(j,0,0)); - amrex::ignore_unused(owy); - + constexpr int sk_cp = 0; + constexpr int sl_cp = 0; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - - // interp from coarse nodal to fine nodal - const Real bg = owx * owy * Byg(jg ,kg ,0) - + owx * wy * Byg(jg ,kg+1,0) - + wx * owy * Byg(jg+1,kg ,0) - + wx * wy * Byg(jg+1,kg+1,0); - - // interp from coarse stagged (cell-centered for By) to fine nodal - wx = 0.5_rt-wx; owx = 1.0_rt-wx; - wy = 0.5_rt-wy; owy = 1.0_rt-wy; - const Real bc = owx * owy * Byc(jg ,kg ,0) - + owx * wy * Byc(jg ,kg-1,0) - + wx * owy * Byc(jg-1,kg ,0) - + wx * wy * Byc(jg-1,kg-1,0); - - // interp form fine stagger (cell-centered for By) to fine nodal - const Real bf = 0.25_rt*(Byf_zeropad(j,k,0) + Byf_zeropad(j-1,k,0) + Byf_zeropad(j,k-1,0) + Byf_zeropad(j-1,k-1,0)); - + const int sk_cp = arr_coarse_stag[1]; + constexpr int sl_cp = 0; #else - - const int lg = amrex::coarsen(l,2); - Real wz = (l == lg*2) ? 0.0_rt : 0.5_rt; - Real owz = 1.0_rt-wz; - - // interp from coarse nodal to fine nodal - const Real bg = owx * owy * owz * Byg(jg ,kg ,lg ) - + wx * owy * owz * Byg(jg+1,kg ,lg ) - + owx * wy * owz * Byg(jg ,kg+1,lg ) - + wx * wy * owz * Byg(jg+1,kg+1,lg ) - + owx * owy * wz * Byg(jg ,kg ,lg+1) - + wx * owy * wz * Byg(jg+1,kg ,lg+1) - + owx * wy * wz * Byg(jg ,kg+1,lg+1) - + wx * wy * wz * Byg(jg+1,kg+1,lg+1); - - // interp from coarse staggered to fine nodal - wx = 0.5_rt-wx; owx = 1.0_rt-wx; - wz = 0.5_rt-wz; owz = 1.0_rt-wz; - const Real bc = owx * owy * owz * Byc(jg ,kg ,lg ) - + wx * owy * owz * Byc(jg-1,kg ,lg ) - + owx * wy * owz * Byc(jg ,kg+1,lg ) - + wx * wy * owz * Byc(jg-1,kg+1,lg ) - + owx * owy * wz * Byc(jg ,kg ,lg-1) - + wx * owy * wz * Byc(jg-1,kg ,lg-1) - + owx * wy * wz * Byc(jg ,kg+1,lg-1) - + wx * wy * wz * Byc(jg-1,kg+1,lg-1); - - // interp from fine stagged to fine nodal - const Real bf = 0.25_rt*(Byf_zeropad(j-1,k,l-1) + Byf_zeropad(j,k,l-1) + Byf_zeropad(j-1,k,l) + Byf_zeropad(j,k,l)); - + const int sk_cp = arr_coarse_stag[1]; + const int sl_cp = arr_coarse_stag[2]; #endif - Bya(j,k,l) = bg + (bf-bc); -} + // Number of points used for interpolation from coarse grid to fine grid + int nj; + int nk; + int nl; -AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void warpx_interp_nd_bfield_z (int j, int k, int l, - amrex::Array4 const& Bza, - amrex::Array4 const& Bzf, - amrex::Array4 const& Bzc, - amrex::Array4 const& Bzg) -{ - using namespace amrex; + int jc = amrex::coarsen(j, rj); + int kc = amrex::coarsen(k, rk); + int lc = amrex::coarsen(l, rl); - // Pad Bzf with zeros beyond ghost cells for out-of-bound accesses - const auto Bzf_zeropad = [Bzf] (const int jj, const int kk, const int ll) noexcept - { - return Bzf.contains(jj,kk,ll) ? Bzf(jj,kk,ll) : 0.0_rt; - }; + amrex::Real tmp = 0.0_rt; + amrex::Real fine = 0.0_rt; + amrex::Real coarse = 0.0_rt; - const int jg = amrex::coarsen(j,2); - Real wx, owx; - wx = (j == jg*2) ? 0.0_rt : 0.5_rt; - owx = 1.0_rt-wx; + amrex::Real wj; + amrex::Real wk; + amrex::Real wl; - const int kg = amrex::coarsen(k,2); - Real wy = 0.0_rt; - Real owy = 0.0_rt; - wy = (k == kg*2) ? 0.0_rt : 0.5_rt; - owy = 1.0_rt-wy; + // 1) Interpolation from coarse nodal to fine nodal + nj = 2; #if defined(WARPX_DIM_1D_Z) - - // interp from coarse nodal to fine nodal - const Real bg = owx * Bzg(jg ,0,0) - + wx * Bzg(jg+1,0,0); - - // interp from coarse staggered to fine nodal - const Real bc = owx * Bzc(jg ,0,0) - + wx * Bzc(jg+1,0,0); - - // interp from fine staggered to fine nodal - const Real bf = Bzf_zeropad(j,0,0); - amrex::ignore_unused(owy); - + nk = 1; + nl = 1; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - - // interp from coarse nodal to fine nodal - const Real bg = owx * owy * Bzg(jg ,kg ,0) - + owx * wy * Bzg(jg ,kg+1,0) - + wx * owy * Bzg(jg+1,kg ,0) - + wx * wy * Bzg(jg+1,kg+1,0); - - // interp from coarse staggered to fine nodal - wx = 0.5_rt-wx; owx = 1.0_rt-wx; - const Real bc = owx * owy * Bzc(jg ,kg ,0) - + owx * wy * Bzc(jg ,kg+1,0) - + wx * owy * Bzc(jg-1,kg ,0) - + wx * wy * Bzc(jg-1,kg+1,0); - - // interp from fine staggered to fine nodal - const Real bf = 0.5_rt*(Bzf_zeropad(j-1,k,0) + Bzf_zeropad(j,k,0)); - + nk = 2; + nl = 1; #else - - const int lg = amrex::coarsen(l,2); - const Real wz = (l == lg*2) ? 0.0_rt : 0.5_rt; - const Real owz = 1.0_rt-wz; - - // interp from coarse nodal to fine nodal - const Real bg = owx * owy * owz * Bzg(jg ,kg ,lg ) - + wx * owy * owz * Bzg(jg+1,kg ,lg ) - + owx * wy * owz * Bzg(jg ,kg+1,lg ) - + wx * wy * owz * Bzg(jg+1,kg+1,lg ) - + owx * owy * wz * Bzg(jg ,kg ,lg+1) - + wx * owy * wz * Bzg(jg+1,kg ,lg+1) - + owx * wy * wz * Bzg(jg ,kg+1,lg+1) - + wx * wy * wz * Bzg(jg+1,kg+1,lg+1); - - // interp from coarse staggered to fine nodal - wx = 0.5_rt-wx; owx = 1.0_rt-wx; - wy = 0.5_rt-wy; owy = 1.0_rt-wy; - const Real bc = owx * owy * owz * Bzc(jg ,kg ,lg ) - + wx * owy * owz * Bzc(jg-1,kg ,lg ) - + owx * wy * owz * Bzc(jg ,kg-1,lg ) - + wx * wy * owz * Bzc(jg-1,kg-1,lg ) - + owx * owy * wz * Bzc(jg ,kg ,lg+1) - + wx * owy * wz * Bzc(jg-1,kg ,lg+1) - + owx * wy * wz * Bzc(jg ,kg-1,lg+1) - + wx * wy * wz * Bzc(jg-1,kg-1,lg+1); - - // interp from fine stagged to fine nodal - const Real bf = 0.25_rt*(Bzf_zeropad(j-1,k-1,l) + Bzf_zeropad(j,k-1,l) + Bzf_zeropad(j-1,k,l) + Bzf_zeropad(j,k,l)); - + nk = 2; + nl = 2; #endif - Bza(j,k,l) = bg + (bf-bc); -} - -AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void warpx_interp_nd_efield_x (int j, int k, int l, - amrex::Array4 const& Exa, - amrex::Array4 const& Exf, - amrex::Array4 const& Exc, - amrex::Array4 const& Exg) -{ - using namespace amrex; - - // Pad Exf with zeros beyond ghost cells for out-of-bound accesses - const auto Exf_zeropad = [Exf] (const int jj, const int kk, const int ll) noexcept - { - return Exf.contains(jj,kk,ll) ? Exf(jj,kk,ll) : 0.0_rt; - }; - - const int jg = amrex::coarsen(j,2); - Real wx, owx; - wx = (j == jg*2) ? 0.0_rt : 0.5_rt; - owx = 1.0_rt-wx; + wj = 1.0_rt; + wk = 1.0_rt; + wl = 1.0_rt; + for (int jj = 0; jj < nj; jj++) { + for (int kk = 0; kk < nk; kk++) { + for (int ll = 0; ll < nl; ll++) { + wj = (rj - amrex::Math::abs(j - (jc + jj) * rj)) / static_cast(rj); +#if (AMREX_SPACEDIM >= 2) + wk = (rk - amrex::Math::abs(k - (kc + kk) * rk)) / static_cast(rk); +#endif +#if (AMREX_SPACEDIM == 3) + wl = (rl - amrex::Math::abs(l - (lc + ll) * rl)) / static_cast(rl); +#endif + tmp += wj * wk * wl * arr_tmp_zeropad(jc+jj,kc+kk,lc+ll); + } + } + } - const int kg = amrex::coarsen(k,2); - Real wy = 0.0_rt; - Real owy =0.0_rt; - wy = (k == kg*2) ? 0.0_rt : 0.5_rt; - owy = 1.0_rt-wy; + // 2) Interpolation from coarse staggered to fine nodal + nj = 2; #if defined(WARPX_DIM_1D_Z) - - // interp from coarse nodal to fine nodal - const Real eg = owx * Exg(jg ,0,0) - + wx * Exg(jg+1,0,0); - - // interp from coarse staggered to fine nodal - const Real ec = owx * Exc(jg ,0,0) - + wx * Exc(jg+1,0,0); - - // interp from fine staggered to fine nodal - const Real ef = Exf_zeropad(j,0,0); - amrex::ignore_unused(owy); - + nk = 1; + nl = 1; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - - // interp from coarse nodal to fine nodal - const Real eg = owx * owy * Exg(jg ,kg ,0) - + owx * wy * Exg(jg ,kg+1,0) - + wx * owy * Exg(jg+1,kg ,0) - + wx * wy * Exg(jg+1,kg+1,0); - - // interp from coarse staggered to fine nodal - wx = 0.5_rt-wx; owx = 1.0_rt-wx; - const Real ec = owx * owy * Exc(jg ,kg ,0) - + owx * wy * Exc(jg ,kg+1,0) - + wx * owy * Exc(jg-1,kg ,0) - + wx * wy * Exc(jg-1,kg+1,0); - - // interp from fine staggered to fine nodal - const Real ef = 0.5_rt*(Exf_zeropad(j-1,k,0) + Exf_zeropad(j,k,0)); - + nk = 2; + nl = 1; #else - - const int lg = amrex::coarsen(l,2); - const Real wz = (l == lg*2) ? 0.0 : 0.5; - const Real owz = 1.0_rt-wz; - - // interp from coarse nodal to fine nodal - const Real eg = owx * owy * owz * Exg(jg ,kg ,lg ) - + wx * owy * owz * Exg(jg+1,kg ,lg ) - + owx * wy * owz * Exg(jg ,kg+1,lg ) - + wx * wy * owz * Exg(jg+1,kg+1,lg ) - + owx * owy * wz * Exg(jg ,kg ,lg+1) - + wx * owy * wz * Exg(jg+1,kg ,lg+1) - + owx * wy * wz * Exg(jg ,kg+1,lg+1) - + wx * wy * wz * Exg(jg+1,kg+1,lg+1); - - // interp from coarse staggered to fine nodal - wx = 0.5_rt-wx; owx = 1.0_rt-wx; - const Real ec = owx * owy * owz * Exc(jg ,kg ,lg ) - + wx * owy * owz * Exc(jg-1,kg ,lg ) - + owx * wy * owz * Exc(jg ,kg+1,lg ) - + wx * wy * owz * Exc(jg-1,kg+1,lg ) - + owx * owy * wz * Exc(jg ,kg ,lg+1) - + wx * owy * wz * Exc(jg-1,kg ,lg+1) - + owx * wy * wz * Exc(jg ,kg+1,lg+1) - + wx * wy * wz * Exc(jg-1,kg+1,lg+1); - - // interp from fine staggered to fine nodal - const Real ef = 0.5_rt*(Exf_zeropad(j-1,k,l) + Exf_zeropad(j,k,l)); - + nk = 2; + nl = 2; #endif - Exa(j,k,l) = eg + (ef-ec); -} - -AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void warpx_interp_nd_efield_y (int j, int k, int l, - amrex::Array4 const& Eya, - amrex::Array4 const& Eyf, - amrex::Array4 const& Eyc, - amrex::Array4 const& Eyg) -{ - using namespace amrex; + const int jn = (sj_cp == 1) ? j : j - rj / 2; + const int kn = (sk_cp == 1) ? k : k - rk / 2; + const int ln = (sl_cp == 1) ? l : l - rl / 2; - // Pad Eyf with zeros beyond ghost cells for out-of-bound accesses - const auto Eyf_zeropad = [Eyf] (const int jj, const int kk, const int ll) noexcept - { - return Eyf.contains(jj,kk,ll) ? Eyf(jj,kk,ll) : 0.0_rt; - }; + jc = amrex::coarsen(jn, rj); + kc = amrex::coarsen(kn, rk); + lc = amrex::coarsen(ln, rl); - const int jg = amrex::coarsen(j,2); - const Real wx = (j == jg*2) ? 0.0_rt : 0.5_rt; - const Real owx = 1.0_rt-wx; + wj = 1.0_rt; + wk = 1.0_rt; + wl = 1.0_rt; + for (int jj = 0; jj < nj; jj++) { + for (int kk = 0; kk < nk; kk++) { + for (int ll = 0; ll < nl; ll++) { + wj = (rj - amrex::Math::abs(jn - (jc + jj) * rj)) / static_cast(rj); +#if (AMREX_SPACEDIM >= 2) + wk = (rk - amrex::Math::abs(kn - (kc + kk) * rk)) / static_cast(rk); +#endif +#if (AMREX_SPACEDIM == 3) + wl = (rl - amrex::Math::abs(ln - (lc + ll) * rl)) / static_cast(rl); +#endif + coarse += wj * wk * wl * arr_coarse_zeropad(jc+jj,kc+kk,lc+ll); + } + } + } - const int kg = amrex::coarsen(k,2); - Real wy = 0.0_rt; - Real owy = 0.0_rt; - wy = (k == kg*2) ? 0.0_rt : 0.5_rt; - owy = 1.0_rt-wy; + // 3) Interpolation from fine staggered to fine nodal + nj = (sj_fp == 0) ? 2 : 1; #if defined(WARPX_DIM_1D_Z) - - // interp from coarse nodal to fine nodal - const Real eg = owx * Eyg(jg ,0,0) - + wx * Eyg(jg+1,0,0); - - // interp from coarse staggered to fine nodal - const Real ec = owx * Eyc(jg ,0,0) - + wx * Eyc(jg+1,0,0); - - // interp from fine staggered to fine nodal - const Real ef = Eyf_zeropad(j,0,0); - amrex::ignore_unused(owy); - + nk = 1; + nl = 1; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - - // interp from coarse nodal to fine nodal - const Real eg = owx * owy * Eyg(jg ,kg ,0) - + owx * wy * Eyg(jg ,kg+1,0) - + wx * owy * Eyg(jg+1,kg ,0) - + wx * wy * Eyg(jg+1,kg+1,0); - - // interp from coarse staggered to fine nodal (Eyc is actually nodal) - const Real ec = owx * owy * Eyc(jg ,kg ,0) - + owx * wy * Eyc(jg ,kg+1,0) - + wx * owy * Eyc(jg+1,kg ,0) - + wx * wy * Eyc(jg+1,kg+1,0); - - // interp from fine staggered to fine nodal - const Real ef = Eyf_zeropad(j,k,0); - + nk = (sk_fp == 0) ? 2 : 1; + nl = 1; #else - - const int lg = amrex::coarsen(l,2); - const Real wz = (l == lg*2) ? 0.0 : 0.5; - const Real owz = 1.0_rt-wz; - - // interp from coarse nodal to fine nodal - const Real eg = owx * owy * owz * Eyg(jg ,kg ,lg ) - + wx * owy * owz * Eyg(jg+1,kg ,lg ) - + owx * wy * owz * Eyg(jg ,kg+1,lg ) - + wx * wy * owz * Eyg(jg+1,kg+1,lg ) - + owx * owy * wz * Eyg(jg ,kg ,lg+1) - + wx * owy * wz * Eyg(jg+1,kg ,lg+1) - + owx * wy * wz * Eyg(jg ,kg+1,lg+1) - + wx * wy * wz * Eyg(jg+1,kg+1,lg+1); - - // interp from coarse staggered to fine nodal - wy = 0.5_rt-wy; owy = 1.0_rt-wy; - const Real ec = owx * owy * owz * Eyc(jg ,kg ,lg ) - + wx * owy * owz * Eyc(jg+1,kg ,lg ) - + owx * wy * owz * Eyc(jg ,kg-1,lg ) - + wx * wy * owz * Eyc(jg+1,kg-1,lg ) - + owx * owy * wz * Eyc(jg ,kg ,lg+1) - + wx * owy * wz * Eyc(jg+1,kg ,lg+1) - + owx * wy * wz * Eyc(jg ,kg-1,lg+1) - + wx * wy * wz * Eyc(jg+1,kg-1,lg+1); - - // interp from fine staggered to fine nodal - const Real ef = 0.5_rt*(Eyf_zeropad(j,k-1,l) + Eyf_zeropad(j,k,l)); - + nk = (sk_fp == 0) ? 2 : 1; + nl = (sl_fp == 0) ? 2 : 1; #endif - Eya(j,k,l) = eg + (ef-ec); -} - -AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void warpx_interp_nd_efield_z (int j, int k, int l, - amrex::Array4 const& Eza, - amrex::Array4 const& Ezf, - amrex::Array4 const& Ezc, - amrex::Array4 const& Ezg) -{ - using namespace amrex; - - // Pad Ezf with zeros beyond ghost cells for out-of-bound accesses - const auto Ezf_zeropad = [Ezf] (const int jj, const int kk, const int ll) noexcept - { - return Ezf.contains(jj,kk,ll) ? Ezf(jj,kk,ll) : 0.0_rt; - }; - - const int jg = amrex::coarsen(j,2); - const Real wx = (j == jg*2) ? 0.0_rt : 0.5_rt; - const Real owx = 1.0_rt-wx; - - const int kg = amrex::coarsen(k,2); - Real wy = 0.0_rt; - Real owy = 0.0_rt; - wy = (k == kg*2) ? 0.0_rt : 0.5_rt; - owy = 1.0_rt-wy; - + const int jm = (sj_fp == 0) ? j-1 : j; #if defined(WARPX_DIM_1D_Z) - - // interp from coarse nodal to fine nodal - const Real eg = owx * Ezg(jg ,0,0) - + wx * Ezg(jg+1,0,0); - - // interp from coarse staggered to fine nodal - const Real ec = owx * Ezc(jg ,0,0) - + wx * Ezc(jg+1,0,0); - - // interp from fine staggered to fine nodal - const Real ef = 0.5_rt*(Ezf_zeropad(j-1,0,0) + Ezf_zeropad(j,0,0)); - amrex::ignore_unused(owy); - + const int km = k; + const int lm = l; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - - // interp from coarse nodal to fine nodal - const Real eg = owx * owy * Ezg(jg ,kg ,0) - + owx * wy * Ezg(jg ,kg+1,0) - + wx * owy * Ezg(jg+1,kg ,0) - + wx * wy * Ezg(jg+1,kg+1,0); - - // interp from coarse stagged to fine nodal - wy = 0.5_rt-wy; owy = 1.0_rt-wy; - const Real ec = owx * owy * Ezc(jg ,kg ,0) - + owx * wy * Ezc(jg ,kg-1,0) - + wx * owy * Ezc(jg+1,kg ,0) - + wx * wy * Ezc(jg+1,kg-1,0); - - // interp from fine staggered to fine nodal - const Real ef = 0.5_rt*(Ezf_zeropad(j,k-1,0) + Ezf_zeropad(j,k,0)); - + const int km = (sk_fp == 0) ? k-1 : k; + const int lm = l; #else - - const int lg = amrex::coarsen(l,2); - Real wz = (l == lg*2) ? 0.0_rt : 0.5_rt; - Real owz = 1.0_rt-wz; - - // interp from coarse nodal to fine nodal - const Real eg = owx * owy * owz * Ezg(jg ,kg ,lg ) - + wx * owy * owz * Ezg(jg+1,kg ,lg ) - + owx * wy * owz * Ezg(jg ,kg+1,lg ) - + wx * wy * owz * Ezg(jg+1,kg+1,lg ) - + owx * owy * wz * Ezg(jg ,kg ,lg+1) - + wx * owy * wz * Ezg(jg+1,kg ,lg+1) - + owx * wy * wz * Ezg(jg ,kg+1,lg+1) - + wx * wy * wz * Ezg(jg+1,kg+1,lg+1); - - // interp from coarse staggered to fine nodal - wz = 0.5_rt-wz; owz = 1.0_rt-wz; - const Real ec = owx * owy * owz * Ezc(jg ,kg ,lg ) - + wx * owy * owz * Ezc(jg+1,kg ,lg ) - + owx * wy * owz * Ezc(jg ,kg+1,lg ) - + wx * wy * owz * Ezc(jg+1,kg+1,lg ) - + owx * owy * wz * Ezc(jg ,kg ,lg-1) - + wx * owy * wz * Ezc(jg+1,kg ,lg-1) - + owx * wy * wz * Ezc(jg ,kg+1,lg-1) - + wx * wy * wz * Ezc(jg+1,kg+1,lg-1); - - // interp from fine staggered to fine nodal - const Real ef = 0.5_rt*(Ezf_zeropad(j,k,l-1) + Ezf_zeropad(j,k,l)); - + const int km = (sk_fp == 0) ? k-1 : k; + const int lm = (sl_fp == 0) ? l-1 : l; #endif - Eza(j,k,l) = eg + (ef-ec); + for (int jj = 0; jj < nj; jj++) { + for (int kk = 0; kk < nk; kk++) { + for (int ll = 0; ll < nl; ll++) { + wj = 1.0_rt / static_cast(nj); + wk = 1.0_rt / static_cast(nk); + wl = 1.0_rt / static_cast(nl); + fine += wj * wk * wl * arr_fine_zeropad(jm+jj,km+kk,lm+ll); + } + } + } + + // Final result + arr_aux(j,k,l) = tmp + (fine - coarse); } /** diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 348b3125f57..6be8d517775 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -1198,6 +1198,18 @@ WarpX::ReadParameters () // Use same shape factors in all directions, for gathering if (field_gathering_algo == GatheringAlgo::MomentumConserving) galerkin_interpolation = false; + // With the PSATD solver, momentum-conserving field gathering + // combined with mesh refinement does not seem to work correctly + // TODO Needs debugging + if (electromagnetic_solver_id == ElectromagneticSolverAlgo::PSATD && + field_gathering_algo == GatheringAlgo::MomentumConserving && + maxLevel() > 0) + { + WARPX_ABORT_WITH_MESSAGE( + "With the PSATD solver, momentum-conserving field gathering" + " combined with mesh refinement is currently not implemented"); + } + em_solver_medium = GetAlgorithmInteger(pp_algo, "em_solver_medium"); if (em_solver_medium == MediumForEM::Macroscopic ) { macroscopic_solver_algo = GetAlgorithmInteger(pp_algo,"macroscopic_sigma_method"); From cb094d1915fe5d6aaff38d9d60bdb645032b897e Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Wed, 4 Oct 2023 19:49:31 -0700 Subject: [PATCH 044/110] Badge: Minor RTD Link Cleanup --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ef0551405a0..87cf2b012c0 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![Code Status development](https://dev.azure.com/ECP-WarpX/WarpX/_apis/build/status/ECP-WarpX.WarpX?branchName=development)](https://dev.azure.com/ECP-WarpX/WarpX/_build/latest?definitionId=1&branchName=development) [![Nightly Installation Tests](https://dev.azure.com/ECP-WarpX/WarpX/_apis/build/status/ECP-WarpX.Nightly?branchName=nightly&label=nightly%20packages)](https://dev.azure.com/ECP-WarpX/WarpX/_build?definitionId=2) -[![Documentation Status](https://readthedocs.org/projects/warpx/badge/?version=latest)](https://warpx.readthedocs.io/en/latest/?badge=latest) +[![Documentation Status](https://readthedocs.org/projects/warpx/badge/?version=latest)](https://warpx.readthedocs.io) [![Spack Version](https://img.shields.io/spack/v/warpx)](https://spack.readthedocs.io/en/latest/package_list.html#warpx) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/warpx)](https://anaconda.org/conda-forge/warpx) [![Discussions](https://img.shields.io/badge/chat-discussions-turquoise.svg)](https://github.com/ECP-WarpX/WarpX/discussions) From 7905cb99004685ff71e6fb012e09422b322f0a95 Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Thu, 5 Oct 2023 06:51:08 +0200 Subject: [PATCH 045/110] ensure that mass is specified (#4337) --- Source/Utils/SpeciesUtils.cpp | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/Source/Utils/SpeciesUtils.cpp b/Source/Utils/SpeciesUtils.cpp index 2ac5879f9d1..bdba97f10b2 100644 --- a/Source/Utils/SpeciesUtils.cpp +++ b/Source/Utils/SpeciesUtils.cpp @@ -42,14 +42,6 @@ namespace SpeciesUtils { const bool mass_is_specified = utils::parser::queryWithParser(pp_species_name, "mass", mass); - if ( charge_is_specified && species_is_specified ){ - ablastr::warn_manager::WMRecordWarning("Species", - "Both '" + species_name + ".charge' and " + - species_name + ".species_type' are specified.\n" + - species_name + ".charge' will take precedence.\n"); - - } - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( charge_is_specified || species_is_specified || @@ -58,6 +50,22 @@ namespace SpeciesUtils { species_name + "'." ); + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + mass_is_specified || + species_is_specified || + (injection_style == "external_file"), + "Need to specify at least one of species_type or mass for species '" + + species_name + "'." + ); + + if ( charge_is_specified && species_is_specified ){ + ablastr::warn_manager::WMRecordWarning("Species", + "Both '" + species_name + ".charge' and " + + species_name + ".species_type' are specified.\n" + + species_name + ".charge' will take precedence.\n"); + + } + if ( mass_is_specified && species_is_specified ){ ablastr::warn_manager::WMRecordWarning("Species", "Both '" + species_name + ".mass' and " + From 12414e432944fdd7024ce19455e0cec2060687f1 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 5 Oct 2023 07:41:23 -0700 Subject: [PATCH 046/110] Release 23.10 (#4344) * AMReX: 23.10 * pyAMReX: 23.10 * WarpX: 23.10 --- .github/workflows/cuda.yml | 2 +- CMakeLists.txt | 2 +- Docs/source/conf.py | 4 ++-- Python/setup.py | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 4 ++-- cmake/dependencies/pyAMReX.cmake | 4 ++-- run_test.sh | 2 +- setup.py | 2 +- 10 files changed, 13 insertions(+), 13 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 03a3044848f..1754549ab2d 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 2e99628138df3b5b0ecf50b0c1201d5547f821a0 && cd - + cd ../amrex && git checkout --detach 23.10 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/CMakeLists.txt b/CMakeLists.txt index f9b1a8bbc2b..5d00093e893 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ # Preamble #################################################################### # cmake_minimum_required(VERSION 3.20.0) -project(WarpX VERSION 23.09) +project(WarpX VERSION 23.10) include(${WarpX_SOURCE_DIR}/cmake/WarpXFunctions.cmake) diff --git a/Docs/source/conf.py b/Docs/source/conf.py index e2cdfe15cda..f9e06bcc8c3 100644 --- a/Docs/source/conf.py +++ b/Docs/source/conf.py @@ -79,9 +79,9 @@ # built documents. # # The short X.Y version. -version = u'23.09' +version = u'23.10' # The full version, including alpha/beta/rc tags. -release = u'23.09' +release = u'23.10' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/Python/setup.py b/Python/setup.py index 99f79cd1d18..568c3f83ddc 100644 --- a/Python/setup.py +++ b/Python/setup.py @@ -54,7 +54,7 @@ package_data = {} setup(name = 'pywarpx', - version = '23.09', + version = '23.10', packages = ['pywarpx'], package_dir = {'pywarpx': 'pywarpx'}, description = """Wrapper of WarpX""", diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index c6caa6b915c..1a1c1fc612c 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 2e99628138df3b5b0ecf50b0c1201d5547f821a0 +branch = 23.10 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 9905f77f34a..3eb8101addb 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 2e99628138df3b5b0ecf50b0c1201d5547f821a0 +branch = 23.10 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 3e0f044f020..a4d1f0fed5c 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -243,7 +243,7 @@ macro(find_amrex) endif() set(COMPONENT_PRECISION ${WarpX_PRECISION} P${WarpX_PARTICLE_PRECISION}) - find_package(AMReX 23.09 CONFIG REQUIRED COMPONENTS ${COMPONENT_ASCENT} ${COMPONENT_DIMS} ${COMPONENT_EB} PARTICLES ${COMPONENT_PIC} ${COMPONENT_PRECISION} ${COMPONENT_SENSEI} TINYP LSOLVERS) + find_package(AMReX 23.10 CONFIG REQUIRED COMPONENTS ${COMPONENT_ASCENT} ${COMPONENT_DIMS} ${COMPONENT_EB} PARTICLES ${COMPONENT_PIC} ${COMPONENT_PRECISION} ${COMPONENT_SENSEI} TINYP LSOLVERS) message(STATUS "AMReX: Found version '${AMReX_VERSION}'") endif() endmacro() @@ -257,7 +257,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "2e99628138df3b5b0ecf50b0c1201d5547f821a0" +set(WarpX_amrex_branch "23.10" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/cmake/dependencies/pyAMReX.cmake b/cmake/dependencies/pyAMReX.cmake index a36ef59d1f9..2ff354785da 100644 --- a/cmake/dependencies/pyAMReX.cmake +++ b/cmake/dependencies/pyAMReX.cmake @@ -64,7 +64,7 @@ function(find_pyamrex) endif() elseif(NOT WarpX_pyamrex_internal) # TODO: MPI control - find_package(pyAMReX 23.07 CONFIG REQUIRED) + find_package(pyAMReX 23.10 CONFIG REQUIRED) message(STATUS "pyAMReX: Found version '${pyamrex_VERSION}'") endif() endfunction() @@ -79,7 +79,7 @@ option(WarpX_pyamrex_internal "Download & build pyAMReX" ON) set(WarpX_pyamrex_repo "https://github.com/AMReX-Codes/pyamrex.git" CACHE STRING "Repository URI to pull and build pyamrex from if(WarpX_pyamrex_internal)") -set(WarpX_pyamrex_branch "development" +set(WarpX_pyamrex_branch "23.10" CACHE STRING "Repository branch for WarpX_pyamrex_repo if(WarpX_pyamrex_internal)") diff --git a/run_test.sh b/run_test.sh index 81237fb6971..1c948e48713 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 2e99628138df3b5b0ecf50b0c1201d5547f821a0 && cd - +cd amrex && git checkout --detach 23.10 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets diff --git a/setup.py b/setup.py index 91c8f1ac828..4636d6995f6 100644 --- a/setup.py +++ b/setup.py @@ -283,7 +283,7 @@ def build_extension(self, ext): setup( name='pywarpx', # note PEP-440 syntax: x.y.zaN but x.y.z.devN - version = '23.09', + version = '23.10', packages = ['pywarpx'], package_dir = {'pywarpx': 'Python/pywarpx'}, author='Jean-Luc Vay, David P. Grote, Maxence Thévenet, Rémi Lehe, Andrew Myers, Weiqun Zhang, Axel Huebl, et al.', From 3a06794120bd3c9af0f7d0a7c4c52154e1f10146 Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Mon, 9 Oct 2023 10:26:52 -0700 Subject: [PATCH 047/110] Add new PRL paper using WarpX (#4351) * Add new PRL paper using WarpX * Update PRAB citation --- Docs/source/highlights.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Docs/source/highlights.rst b/Docs/source/highlights.rst index dd9f6fd3e81..1e99629b9c2 100644 --- a/Docs/source/highlights.rst +++ b/Docs/source/highlights.rst @@ -14,6 +14,11 @@ Plasma-Based Acceleration Scientific works in laser-plasma and beam-plasma acceleration. +#. Peng, H. and Huang, T. W. and Jiang, K. and Li, R. and Wu, C. N. and Yu, M. Y. and Riconda, C. and Weber, S. and Zhou, C. T. and Ruan, S. C. + **Coherent Subcycle Optical Shock from a Superluminal Plasma Wake**. + Phys. Rev. Lett. **131**, 145003, 2023 + `DOI:10.1103/PhysRevLett.131.145003 `__ + #. Mewes SM, Boyle GJ, Ferran Pousa A, Shalloo RJ, Osterhoff J, Arran C, Corner L, Walczak R, Hooker SM, Thévenet M. **Demonstration of tunability of HOFI waveguides via start-to-end simulations**. Phys. Rev. Research **5**, 033112, 2023 @@ -27,8 +32,8 @@ Scientific works in laser-plasma and beam-plasma acceleration. #. Wang J, Zeng M, Li D, Wang X, Gao J. **High quality beam produced by tightly focused laser driven wakefield accelerators**. - arXiv pre-print, 2023. - `DOI:10.48550/arXiv.2304.10730 `__ + Phys. Rev. Accel. Beams, **26**, 091303, 2023. + `DOI:10.1103/PhysRevAccelBeams.26.091303 `__ #. Fedeli L, Huebl A, Boillod-Cerneux F, Clark T, Gott K, Hillairet C, Jaure S, Leblanc A, Lehe R, Myers A, Piechurski C, Sato M, Zaim N, Zhang W, Vay J-L, Vincenti H. **Pushing the Frontier in the Design of Laser-Based Electron Accelerators with Groundbreaking Mesh-Refined Particle-In-Cell Simulations on Exascale-Class Supercomputers**. From 57a7a01d894bdfb17e45f0e8aa1fcb70380db6a4 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 10 Oct 2023 02:31:15 +0200 Subject: [PATCH 048/110] [pre-commit.ci] pre-commit autoupdate (#4352) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.4.0...v4.5.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1c32c94c99c..bb8b6c8b33d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -18,7 +18,7 @@ exclude: '^share/openPMD/thirdParty' # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.4.0 + rev: v4.5.0 hooks: - id: trailing-whitespace args: [--markdown-linebreak-ext=md] From 45569111e4e8b8e8d4174864999d1b11992f764e Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 9 Oct 2023 17:48:28 -0700 Subject: [PATCH 049/110] Lassen (LLNL): TOSS3 Setup (#4346) * Lassen (LLNL): TOSS3 Setup Add a TOSS3 section again. * Fix SciPy and Finalize Python Env --- Docs/source/install/hpc/lassen.rst | 168 +++++++++++++----- .../install_v100_dependencies_toss3.sh | 135 ++++++++++++++ .../lassen_v100_warpx_toss3.profile.example | 54 ++++++ 3 files changed, 309 insertions(+), 48 deletions(-) create mode 100644 Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh create mode 100644 Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example diff --git a/Docs/source/install/hpc/lassen.rst b/Docs/source/install/hpc/lassen.rst index 5726254652a..ac15844a5ff 100644 --- a/Docs/source/install/hpc/lassen.rst +++ b/Docs/source/install/hpc/lassen.rst @@ -24,12 +24,27 @@ If you are new to this system, **please see the following resources**: Login ----- -.. note:: +.. tab-set:: - Lassen is currently transitioning to RHEL8. - During this transition, first SSH into lassen and then ``ssh eatoss4`` next to work with the updated RHEL8/TOSS4 nodes. + .. tab-item:: TOSS4 (RHEL8) - Approximately October 2023, the new software environment on these nodes will be the new default. + Lassen is currently transitioning to RHEL8. + During this transition, first SSH into lassen and then to the updated RHEL8/TOSS4 nodes. + + .. code-block:: bash + + ssh lassen.llnl.gov + ssh eatoss4 + + Approximately October/November 2023, the new software environment on these nodes will be the new default. + + .. tab-item:: TOSS3 (RHEL7) + + .. code-block:: bash + + ssh lassen.llnl.gov + + Approximately October/November 2023, this partition will become TOSS4 (RHEL8) as well. .. _building-lassen-preparation: @@ -43,78 +58,135 @@ Use the following commands to download the WarpX source code: git clone https://github.com/ECP-WarpX/WarpX.git /usr/workspace/${USER}/lassen/src/warpx -We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx.profile``. -Create it now: +.. tab-set:: -.. code-block:: bash + .. tab-item:: TOSS4 (RHEL8) - cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example $HOME/lassen_v100_warpx.profile + We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx.profile``. + Create it now: -.. dropdown:: Script Details - :color: light - :icon: info - :animate: fade-in-slide-down + .. code-block:: bash - .. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example - :language: bash + cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example $HOME/lassen_v100_warpx.profile -Edit the 2nd line of this script, which sets the ``export proj=""`` variable. -For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx.profile``. -Enter the edit mode by typing ``i`` and edit line 2 to read: + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down -.. code-block:: bash + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example + :language: bash - export proj="nsldt" + Edit the 2nd line of this script, which sets the ``export proj=""`` variable. + For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx.profile``. + Enter the edit mode by typing ``i`` and edit line 2 to read: -Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). + .. code-block:: bash -.. important:: + export proj="nsldt" - Now, and as the first step on future logins to lassen, activate these environment settings: + Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). - .. code-block:: bash + .. important:: + + Now, and as the first step on future logins to lassen, activate these environment settings: + + .. code-block:: bash + + source $HOME/lassen_v100_warpx.profile + + .. tab-item:: TOSS3 (RHEL7) + + We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx_toss3.profile``. + Create it now: + + .. code-block:: bash + + cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example $HOME/lassen_v100_warpx_toss3.profile + + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down - source $HOME/lassen_v100_warpx.profile + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example + :language: bash + + Edit the 2nd line of this script, which sets the ``export proj=""`` variable. + For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx_toss3.profile``. + Enter the edit mode by typing ``i`` and edit line 2 to read: + + .. code-block:: bash + + export proj="nsldt" + + Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). + + .. important:: + + Now, and as the first step on future logins to lassen, activate these environment settings: + + .. code-block:: bash + + source $HOME/lassen_v100_warpx_toss3.profile Finally, since lassen does not yet provide software modules for some of our dependencies, install them once: -.. code-block:: bash +.. tab-set:: - bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh - source /usr/workspace/${USER}/lassen/gpu/venvs/warpx-lassen/bin/activate + .. tab-item:: TOSS4 (RHEL8) -.. dropdown:: Script Details - :color: light - :icon: info - :animate: fade-in-slide-down + .. code-block:: bash - .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies.sh - :language: bash + bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh + source /usr/workspace/${USER}/lassen/gpu/venvs/warpx-lassen/bin/activate -.. dropdown:: AI/ML Dependencies (Optional) - :animate: fade-in-slide-down + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down - If you plan to run AI/ML workflows depending on pyTorch, run the next step as well. - This will take a while and should be skipped if not needed. + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies.sh + :language: bash - .. code-block:: bash + .. dropdown:: AI/ML Dependencies (Optional) + :animate: fade-in-slide-down - runNode bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_ml.sh + If you plan to run AI/ML workflows depending on pyTorch, run the next step as well. + This will take a while and should be skipped if not needed. - .. dropdown:: Script Details - :color: light - :icon: info - :animate: fade-in-slide-down + .. code-block:: bash - .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_ml.sh - :language: bash + runNode bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_ml.sh - For `optimas dependencies `__ (incl. scikit-learn), plan another hour of build time: + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down - .. code-block:: bash + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_ml.sh + :language: bash + + For `optimas dependencies `__ (incl. scikit-learn), plan another hour of build time: + + .. code-block:: bash + + python3 -m pip install -r /usr/workspace/${USER}/lassen/src/warpx/Tools/optimas/requirements.txt + + .. tab-item:: TOSS3 (RHEL7) + + .. code-block:: bash + + bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh + source /usr/workspace/${USER}/lassen/gpu/venvs/warpx-lassen-toss3/bin/activate - python3 -m pip install -r /usr/workspace/${USER}/lassen/src/warpx/Tools/optimas/requirements.txt + .. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + .. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh + :language: bash .. _building-lassen-compilation: diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh b/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh new file mode 100644 index 00000000000..f62c6018cd5 --- /dev/null +++ b/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh @@ -0,0 +1,135 @@ +#!/bin/bash +# +# Copyright 2023 The WarpX Community +# +# This file is part of WarpX. +# +# Author: Axel Huebl +# License: BSD-3-Clause-LBNL + +# Exit on first error encountered ############################################# +# +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was lassen_v100_warpx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your lassen_v100_warpx_toss3.profile file! Please edit its line 2 to continue!"; exit 1; fi + + +# Remove old dependencies ##################################################### +# +SRC_DIR="/usr/workspace/${USER}/lassen-toss3/src" +SW_DIR="/usr/workspace/${USER}/lassen-toss3/gpu" +rm -rf ${SW_DIR} +mkdir -p ${SW_DIR} +mkdir -p ${SRC_DIR} + +# remove common user mistakes in python, located in .local instead of a venv +python3 -m pip uninstall -qq -y pywarpx +python3 -m pip uninstall -qq -y warpx +python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true + + +# General extra dependencies ################################################## +# + +# tmpfs build directory: avoids issues often seen with $HOME and is faster +build_dir=$(mktemp -d) + +# c-blosc (I/O compression) +if [ -d ${SRC_DIR}/c-blosc ] +then + cd ${SRC_DIR}/c-blosc + git fetch --prune + git checkout v1.21.1 + cd - +else + git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git ${SRC_DIR}/c-blosc +fi +cmake -S ${SRC_DIR}/c-blosc -B ${build_dir}/c-blosc-lassen-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 +cmake --build ${build_dir}/c-blosc-lassen-build --target install --parallel 10 + +# HDF5 +if [ -d ${SRC_DIR}/hdf5 ] +then + cd ${SRC_DIR}/hdf5 + git fetch --prune + git checkout hdf5-1_14_1-2 + cd - +else + git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git ${SRC_DIR}/hdf5 +fi +cmake -S ${SRC_DIR}/hdf5 -B ${build_dir}/hdf5-lassen-build -DBUILD_TESTING=OFF -DHDF5_ENABLE_PARALLEL=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/hdf5-1.14.1.2 +cmake --build ${build_dir}/hdf5-lassen-build --target install --parallel 10 + +# ADIOS2 +if [ -d ${SRC_DIR}/adios2 ] +then + cd ${SRC_DIR}/adios2 + git fetch --prune + git checkout v2.8.3 + cd - +else + git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git ${SRC_DIR}/adios2 +fi +cmake -S ${SRC_DIR}/adios2 -B ${build_dir}/adios2-lassen-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake --build ${build_dir}/adios2-lassen-build --target install -j 10 + +# BLAS++ (for PSATD+RZ) +if [ -d ${SRC_DIR}/blaspp ] +then + cd ${SRC_DIR}/blaspp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/blaspp.git ${SRC_DIR}/blaspp +fi +cmake -S ${SRC_DIR}/blaspp -B ${build_dir}/blaspp-lassen-build -Duse_openmp=ON -Dgpu_backend=cuda -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake --build ${build_dir}/blaspp-lassen-build --target install --parallel 10 + +# LAPACK++ (for PSATD+RZ) +if [ -d ${SRC_DIR}/lapackpp ] +then + cd ${SRC_DIR}/lapackpp + git fetch --prune + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/lapackpp.git ${SRC_DIR}/lapackpp +fi +CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${SRC_DIR}/lapackpp -B ${build_dir}/lapackpp-lassen-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master -DLAPACK_LIBRARIES=/usr/lib64/liblapack.so +cmake --build ${build_dir}/lapackpp-lassen-build --target install --parallel 10 + + +# Python ###################################################################### +# +python3 -m pip install --upgrade --user virtualenv +rm -rf ${SW_DIR}/venvs/warpx-lassen-toss3 +python3 -m venv ${SW_DIR}/venvs/warpx-lassen-toss3 +source ${SW_DIR}/venvs/warpx-lassen-toss3/bin/activate +python3 -m pip install --upgrade pip +python3 -m pip cache purge +python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade cython +python3 -m pip install --upgrade numpy +python3 -m pip install --upgrade pandas +CMAKE_PREFIX_PATH=/usr/lib64:${CMAKE_PREFIX_PATH} python3 -m pip install --upgrade -Ccompile-args="-j10" -Csetup-args=-Dblas=BLAS -Csetup-args=-Dlapack=BLAS scipy +python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py +python3 -m pip install --upgrade openpmd-api +MPLLOCALFREETYPE=1 python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself +echo "matplotlib==3.2.2" > ${build_dir}/constraints.txt +python3 -m pip install --upgrade -c ${build_dir}/constraints.txt yt + +# install or update WarpX dependencies such as picmistandard +python3 -m pip install --upgrade -r ${SRC_DIR}/warpx/requirements.txt + +# for ML dependencies, see install_v100_ml.sh + + +# remove build temporary directory +rm -rf ${build_dir} diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example new file mode 100644 index 00000000000..979c27989fa --- /dev/null +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example @@ -0,0 +1,54 @@ +# please set your project account +#export proj="" # edit this and comment in + +# required dependencies +module load cmake/3.23.1 +module load gcc/11.2.1 +module load cuda/12.0.0 + +# optional: for QED lookup table generation support +module load boost/1.70.0 + +# optional: for openPMD support +SRC_DIR="/usr/workspace/${USER}/lassen-toss3/src" +SW_DIR="/usr/workspace/${USER}/lassen-toss3/gpu" +export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=${SW_DIR}/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/adios2-2.8.3/lib64:$LD_LIBRARY_PATH + +# optional: for PSATD in RZ geometry support +export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-master:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=${SW_DIR}/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-master/lib64:$LD_LIBRARY_PATH + +# optional: for Python bindings +module load python/3.8.2 + +if [ -d "${SW_DIR}/venvs/warpx-lassen-toss3" ] +then + source ${SW_DIR}/venvs/warpx-lassen-toss3/bin/activate +fi + +# optional: an alias to request an interactive node for two hours +alias getNode="bsub -G $proj -W 2:00 -nnodes 1 -Is /bin/bash" +# an alias to run a command on a batch node for up to 30min +# usage: runNode +alias runNode="bsub -q debug -P $proj -W 2:00 -nnodes 1 -I" + +# fix system defaults: do not escape $ with a \ on tab completion +shopt -s direxpand + +# optimize CUDA compilation for V100 +export AMREX_CUDA_ARCH=7.0 +export CUDAARCHS=70 + +# compiler environment hints +export CC=$(which gcc) +export CXX=$(which g++) +export FC=$(which gfortran) +export CUDACXX=$(which nvcc) +export CUDAHOSTCXX=${CXX} From b71b89c0fae093f9f552fc348b979d21e07b54c2 Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Tue, 10 Oct 2023 16:30:59 -0700 Subject: [PATCH 050/110] Fix issues with cupy-WarpX interoperability (#4348) * Fix issues with cupy-WarpX interoperability * Update cupy binding --- Python/pywarpx/fields.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/Python/pywarpx/fields.py b/Python/pywarpx/fields.py index fd647287a53..5eeeda50ea7 100644 --- a/Python/pywarpx/fields.py +++ b/Python/pywarpx/fields.py @@ -298,10 +298,14 @@ def _get_field(self, mfi): # self.mf.array(mfi) is in C ordering. # Note: transposing creates a view and not a copy. device_arr4 = self.mf.array(mfi) - if cp is not None: - device_arr = cp.array(device_arr4, copy=False).T + if libwarpx.libwarpx_so.Config.have_gpu: + if cp is not None: + device_arr = device_arr4.to_cupy(copy=False) + else: + # Relies on managed memory + device_arr = device_arr4.to_numpy(copy=False) else: - device_arr = np.array(device_arr4, copy=False).T + device_arr = device_arr4.to_numpy(copy=False) if not self.include_ghosts: nghosts = self._get_n_ghosts() device_arr = device_arr[tuple([slice(ng, -ng) for ng in nghosts[:self.dim]])] @@ -423,11 +427,10 @@ def __getitem__(self, index): if global_slices is not None: # Note that the array will always have 4 dimensions. device_arr = self._get_field(mfi) - if cp is not None: - # Copy the data from the device to the host - slice_arr = cp.asnumpy(device_arr[block_slices]) - else: - slice_arr = device_arr[block_slices] + slice_arr = device_arr[block_slices] + if (cp is not None) and (type(slice_arr) is cp.ndarray): + # Copy data from host to device using cupy syntax + slice_arr = slice_arr.get() datalist.append((global_slices, slice_arr)) # Gather the data from all processors @@ -528,9 +531,9 @@ def __setitem__(self, index, value): mf_arr = self._get_field(mfi) if isinstance(value, np.ndarray): slice_value = value3d[global_slices] - if cp is not None: + if libwarpx.libwarpx_so.Config.have_gpu: # Copy data from host to device - slice_value = cp.asarray(value3d[global_slices]) + slice_value = cp.asarray(slice_value) mf_arr[block_slices] = slice_value else: mf_arr[block_slices] = value From 419d47a7a369026acbc7d66f6078e6bbaf73495f Mon Sep 17 00:00:00 2001 From: Olga Shapoval <30510597+oshapoval@users.noreply.github.com> Date: Wed, 11 Oct 2023 14:58:51 -0700 Subject: [PATCH 051/110] Change default behavior of picmi.FieldDiagnostic and picmi.ParticleDiagnostic & fix CI tests (#4317) * Do not write particles with `FieldDiagnostic` Do not write fields with `ParticleDiagnostic Update implementation Revert "Update implementation" This reverts commit 77b473f4afbfe2414bbe5d1a8b4e65ec475a7bd7. Use syntax with argvattrs Correct typo Force writing of species Add function that can replace existing attributes Revert "Revert "Update implementation"" This reverts commit d946fe643b085cfec4ae90673f1105c8098ab748. Correct syntax typo [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Revert "Revert "Revert "Update implementation""" This reverts commit 69092be1e46da655075f5292931fa4ffe6d86d7c. Generalization for backtransformed diagnostics Update lab-frame diagnostics Update 1D CI case Fix additional merge errors Remove particle diagnostic Add particle diagnostic where needed Use latest PICMI version Handle the case when species is None Add missing `add_diagnostic` calls * Update 1D ion beam case Fixed CI: plasma_lens Fixed CI: LoadExternalField Fixed CI: Python_reduced_diags_loadbalancecosts_timers Clean up * Fixed CI: Python_ionization, Ohm Solver * Clean up: removed jx, jy, jz * More fixes in Ohm solver's CIs * Placed species in squire brackets * Wrtie FieldDiagnostics in CI: Python_prev_positions * Do not specify datalist in test --------- Co-authored-by: Remi Lehe --- Docs/requirements.txt | 2 +- .../capacitive_discharge/PICMI_inputs_1d.py | 7 ++ .../capacitive_discharge/PICMI_inputs_2d.py | 7 ++ .../laser_acceleration/PICMI_inputs_1d.py | 6 ++ .../laser_acceleration/PICMI_inputs_2d.py | 6 ++ .../laser_acceleration/PICMI_inputs_3d.py | 6 ++ .../LoadExternalField/PICMI_inputs_3d.py | 10 +++ Examples/Tests/collision/PICMI_inputs_2d.py | 7 ++ .../PICMI_inputs_2d.py | 7 ++ .../PICMI_inputs_3d.py | 7 ++ .../PICMI_inputs_EB_API.py | 7 ++ Examples/Tests/ionization/PICMI_inputs_2d.py | 13 +++- .../Tests/magnetostatic_eb/PICMI_inputs_3d.py | 7 ++ .../Tests/magnetostatic_eb/PICMI_inputs_rz.py | 7 ++ .../Tests/ohm_solver_EM_modes/PICMI_inputs.py | 14 +++- .../PICMI_inputs.py | 16 ++++- .../PICMI_inputs.py | 16 ++++- .../PICMI_inputs.py | 21 ++++-- .../PICMI_inputs_reflection.py | 10 ++- .../PICMI_inputs_scrape.py | 7 ++ .../particle_data_python/PICMI_inputs_2d.py | 7 ++ .../PICMI_inputs_prev_pos_2d.py | 16 +++-- Examples/Tests/plasma_lens/PICMI_inputs_3d.py | 9 ++- .../Tests/python_wrappers/PICMI_inputs_2d.py | 6 ++ .../PICMI_inputs_loadbalancecosts.py | 15 +++- .../Tests/restart/PICMI_inputs_id_cpu_read.py | 7 ++ .../PICMI_inputs_runtime_component_analyze.py | 7 ++ .../restart_eb/PICMI_inputs_restart_eb.py | 7 ++ Python/pywarpx/Diagnostics.py | 9 +++ Python/pywarpx/picmi.py | 72 +++++++++---------- Python/setup.py | 2 +- requirements.txt | 2 +- 32 files changed, 280 insertions(+), 62 deletions(-) diff --git a/Docs/requirements.txt b/Docs/requirements.txt index 1f21a414a19..86235035252 100644 --- a/Docs/requirements.txt +++ b/Docs/requirements.txt @@ -11,7 +11,7 @@ docutils>=0.17.1 # PICMI API docs # note: keep in sync with version in ../requirements.txt -picmistandard==0.26.0 +picmistandard==0.28.0 # for development against an unreleased PICMI version, use: # picmistandard @ git+https://github.com/picmi-standard/picmi.git#subdirectory=PICMI_Python diff --git a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py index 766e7693b6d..dc9428229a4 100644 --- a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py +++ b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py @@ -324,6 +324,12 @@ def setup_run(self): else: file_prefix = 'Python_background_mcc_1d_tridiag_plt' + particle_diag = picmi.ParticleDiagnostic( + name='diag1', + period=0, + write_dir='.', + warpx_file_prefix=file_prefix + ) field_diag = picmi.FieldDiagnostic( name='diag1', grid=self.grid, @@ -332,6 +338,7 @@ def setup_run(self): write_dir='.', warpx_file_prefix=file_prefix ) + self.sim.add_diagnostic(particle_diag) self.sim.add_diagnostic(field_diag) def _get_rho_ions(self): diff --git a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py index 2a365a3729c..fd66ba34a04 100755 --- a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py +++ b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py @@ -312,6 +312,12 @@ def solve(self): # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = diagnostic_intervals, + write_dir = '.', + warpx_file_prefix = 'Python_background_mcc_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -345,6 +351,7 @@ def solve(self): ) ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ########################## diff --git a/Examples/Physics_applications/laser_acceleration/PICMI_inputs_1d.py b/Examples/Physics_applications/laser_acceleration/PICMI_inputs_1d.py index 008311e6bbe..d8bdddfaca6 100755 --- a/Examples/Physics_applications/laser_acceleration/PICMI_inputs_1d.py +++ b/Examples/Physics_applications/laser_acceleration/PICMI_inputs_1d.py @@ -79,6 +79,11 @@ # Diagnostics diag_field_list = ['B', 'E', 'J', 'rho'] +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 100, + write_dir = '.', + warpx_file_prefix = 'Python_LaserAcceleration_1d_plt') field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -108,6 +113,7 @@ injection_method = laser_antenna) # Add diagnostics +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) # Write input file that can be used to run with the compiled version diff --git a/Examples/Physics_applications/laser_acceleration/PICMI_inputs_2d.py b/Examples/Physics_applications/laser_acceleration/PICMI_inputs_2d.py index fbc04a3e42c..b50e16bfc0a 100755 --- a/Examples/Physics_applications/laser_acceleration/PICMI_inputs_2d.py +++ b/Examples/Physics_applications/laser_acceleration/PICMI_inputs_2d.py @@ -112,6 +112,11 @@ # Diagnostics diag_field_list = ['B', 'E', 'J', 'rho'] +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 200, + write_dir = '.', + warpx_file_prefix = 'Python_LaserAccelerationMR_plt') field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -145,6 +150,7 @@ injection_method = laser_antenna) # Add diagnostics +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) # Write input file that can be used to run with the compiled version diff --git a/Examples/Physics_applications/laser_acceleration/PICMI_inputs_3d.py b/Examples/Physics_applications/laser_acceleration/PICMI_inputs_3d.py index b41d5c6df4b..ac3398e43fc 100755 --- a/Examples/Physics_applications/laser_acceleration/PICMI_inputs_3d.py +++ b/Examples/Physics_applications/laser_acceleration/PICMI_inputs_3d.py @@ -110,6 +110,11 @@ # Diagnostics diag_field_list = ['B', 'E', 'J', 'rho'] +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 100, + write_dir = '.', + warpx_file_prefix = 'Python_LaserAcceleration_plt') field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -144,6 +149,7 @@ injection_method = laser_antenna) # Add diagnostics +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) # Write input file that can be used to run with the compiled version diff --git a/Examples/Tests/LoadExternalField/PICMI_inputs_3d.py b/Examples/Tests/LoadExternalField/PICMI_inputs_3d.py index 857318baaa1..849d4d2be08 100644 --- a/Examples/Tests/LoadExternalField/PICMI_inputs_3d.py +++ b/Examples/Tests/LoadExternalField/PICMI_inputs_3d.py @@ -87,10 +87,19 @@ ######### DIAGNOSTICS ########### ################################# +particle_diag = picmi.ParticleDiagnostic( + name='diag1', + period=300, + species=[ions], + data_list = ['ux', 'uy', 'uz', 'x', 'y', 'z', 'weighting'], + write_dir='.', + warpx_file_prefix='Python_LoadExternalField3D_plt' + ) field_diag = picmi.FieldDiagnostic( name='diag1', grid=grid, period=300, + data_list = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], write_dir='.', warpx_file_prefix='Python_LoadExternalField3D_plt' ) @@ -121,6 +130,7 @@ ) sim.add_diagnostic(field_diag) +sim.add_diagnostic(particle_diag) ################################# ##### SIMULATION EXECUTION ###### diff --git a/Examples/Tests/collision/PICMI_inputs_2d.py b/Examples/Tests/collision/PICMI_inputs_2d.py index f1c41827093..99e217b0afc 100755 --- a/Examples/Tests/collision/PICMI_inputs_2d.py +++ b/Examples/Tests/collision/PICMI_inputs_2d.py @@ -106,6 +106,12 @@ ######### DIAGNOSTICS ########### ################################# +particle_diag = picmi.ParticleDiagnostic( + name='diag1', + period=10, + write_dir='.', + warpx_file_prefix='Python_collisionXZ_plt' +) field_diag = picmi.FieldDiagnostic( name='diag1', grid=grid, @@ -140,6 +146,7 @@ ) ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ################################# diff --git a/Examples/Tests/electrostatic_dirichlet_bc/PICMI_inputs_2d.py b/Examples/Tests/electrostatic_dirichlet_bc/PICMI_inputs_2d.py index 309251149ef..e4dd530c3bc 100755 --- a/Examples/Tests/electrostatic_dirichlet_bc/PICMI_inputs_2d.py +++ b/Examples/Tests/electrostatic_dirichlet_bc/PICMI_inputs_2d.py @@ -58,6 +58,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 4, + write_dir = '.', + warpx_file_prefix = 'Python_dirichletbc_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -79,6 +85,7 @@ verbose = 0 ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ########################## diff --git a/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py b/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py index 63b4ab8ea38..4da7feeff3a 100755 --- a/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py +++ b/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py @@ -70,6 +70,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 1, + write_dir = '.', + warpx_file_prefix = 'Python_ElectrostaticSphereEB_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -102,6 +108,7 @@ warpx_field_gathering_algo='momentum-conserving' ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) sim.add_diagnostic(reduced_diag) sim.add_diagnostic(reduced_diag_one_eighth) diff --git a/Examples/Tests/embedded_boundary_python_api/PICMI_inputs_EB_API.py b/Examples/Tests/embedded_boundary_python_api/PICMI_inputs_EB_API.py index c047106b4ac..faec3ed4668 100755 --- a/Examples/Tests/embedded_boundary_python_api/PICMI_inputs_EB_API.py +++ b/Examples/Tests/embedded_boundary_python_api/PICMI_inputs_EB_API.py @@ -58,6 +58,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 1, + write_dir = '.', + warpx_file_prefix = "embedded_boundary_python_API_plt" +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -78,6 +84,7 @@ verbose = 1 ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) sim.initialize_inputs() diff --git a/Examples/Tests/ionization/PICMI_inputs_2d.py b/Examples/Tests/ionization/PICMI_inputs_2d.py index 81b55d06600..86d24181a75 100644 --- a/Examples/Tests/ionization/PICMI_inputs_2d.py +++ b/Examples/Tests/ionization/PICMI_inputs_2d.py @@ -84,10 +84,18 @@ cfl = 0.999) # Diagnostics -diag = picmi.FieldDiagnostic( +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 10000, + species = [electrons, ions], + data_list = ['ux', 'uy', 'uz', 'x', 'y', 'weighting'], + write_dir = '.', + warpx_file_prefix = 'Python_ionization_plt') +field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, period = 10000, + data_list = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], write_dir = '.', warpx_file_prefix = 'Python_ionization_plt') @@ -115,7 +123,8 @@ injection_method = laser_antenna) # Add diagnostics -sim.add_diagnostic(diag) +sim.add_diagnostic(particle_diag) +sim.add_diagnostic(field_diag) # Write input file that can be used to run with the compiled version sim.write_input_file(file_name = 'inputs_2d_picmi') diff --git a/Examples/Tests/magnetostatic_eb/PICMI_inputs_3d.py b/Examples/Tests/magnetostatic_eb/PICMI_inputs_3d.py index 18c62ee523f..8f205724563 100755 --- a/Examples/Tests/magnetostatic_eb/PICMI_inputs_3d.py +++ b/Examples/Tests/magnetostatic_eb/PICMI_inputs_3d.py @@ -113,6 +113,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 1, + write_dir = '.', + warpx_file_prefix = 'Python_magnetostatic_eb_3d_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -140,6 +146,7 @@ sim.add_species(beam, layout=beam_layout, initialize_self_field=True) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ########################## diff --git a/Examples/Tests/magnetostatic_eb/PICMI_inputs_rz.py b/Examples/Tests/magnetostatic_eb/PICMI_inputs_rz.py index 693bc1b888d..1268f7a02b0 100755 --- a/Examples/Tests/magnetostatic_eb/PICMI_inputs_rz.py +++ b/Examples/Tests/magnetostatic_eb/PICMI_inputs_rz.py @@ -110,6 +110,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 1, + write_dir = '.', + warpx_file_prefix = 'Python_magnetostatic_eb_rz_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -137,6 +143,7 @@ sim.add_species(beam, layout=beam_layout, initialize_self_field=True) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ########################## diff --git a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py index 78d7c847699..b51d291e478 100644 --- a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py @@ -21,7 +21,10 @@ comm = mpi.COMM_WORLD -simulation = picmi.Simulation(verbose=0) +simulation = picmi.Simulation( + warpx_serialize_initial_conditions=True, + verbose=0 +) # make a shorthand for simulation.extension since we use it a lot sim_ext = simulation.extension @@ -256,6 +259,15 @@ def setup_run(self): self.output_file_name = 'perp_field_data.txt' if self.test: + particle_diag = picmi.ParticleDiagnostic( + name='field_diag', + period=self.total_steps, + write_dir='.', + warpx_file_prefix='Python_ohms_law_solver_EM_modes_1d_plt', + # warpx_format = 'openpmd', + # warpx_openpmd_backend = 'h5' + ) + simulation.add_diagnostic(particle_diag) field_diag = picmi.FieldDiagnostic( name='field_diag', grid=self.grid, diff --git a/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py b/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py index 8706ceb518c..ecad73d381c 100644 --- a/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py @@ -20,7 +20,9 @@ comm = mpi.COMM_WORLD -simulation = picmi.Simulation(verbose=0) +simulation = picmi.Simulation( + warpx_serialize_initial_conditions=True, + verbose=0) # make a shorthand for simulation.extension since we use it a lot sim_ext = simulation.extension @@ -216,11 +218,21 @@ def setup_run(self): callbacks.installafterstep(self.text_diag) if self.test: + particle_diag = picmi.ParticleDiagnostic( + name='diag1', + period=100, + write_dir='.', + species=[self.ions], + data_list = ['ux', 'uy', 'uz', 'x', 'y', 'weighting'], + warpx_file_prefix=f'Python_ohms_law_solver_landau_damping_{self.dim}d_plt', + ) + simulation.add_diagnostic(particle_diag) field_diag = picmi.FieldDiagnostic( - name='field_diag', + name='diag1', grid=self.grid, period=100, write_dir='.', + data_list = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], warpx_file_prefix=f'Python_ohms_law_solver_landau_damping_{self.dim}d_plt', ) simulation.add_diagnostic(field_diag) diff --git a/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py b/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py index 08f0bb22bb6..21f55fb1166 100644 --- a/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py @@ -21,7 +21,9 @@ comm = mpi.COMM_WORLD -simulation = picmi.Simulation(verbose=0) +simulation = picmi.Simulation( + warpx_serialize_initial_conditions=True, + verbose=0) # make a shorthand for simulation.extension since we use it a lot sim_ext = simulation.extension @@ -256,10 +258,20 @@ def setup_run(self): callbacks.installafterstep(self.text_diag) if self.test: + part_diag = picmi.ParticleDiagnostic( + name='diag1', + period=1250, + species=[self.ions, self.beam_ions], + data_list = ['ux', 'uy', 'uz', 'x', 'weighting'], + write_dir='.', + warpx_file_prefix='Python_ohms_law_solver_ion_beam_1d_plt', + ) + simulation.add_diagnostic(part_diag) field_diag = picmi.FieldDiagnostic( - name='field_diag', + name='diag1', grid=self.grid, period=1250, + data_list = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], write_dir='.', warpx_file_prefix='Python_ohms_law_solver_ion_beam_1d_plt', ) diff --git a/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py b/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py index 7b42e57d557..bdeef96e0a6 100644 --- a/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py @@ -22,7 +22,9 @@ comm = mpi.COMM_WORLD -simulation = picmi.Simulation(verbose=0) +simulation = picmi.Simulation( + warpx_serialize_initial_conditions=True, + verbose=0) # make a shorthand for simulation.extension since we use it a lot sim_ext = simulation.extension @@ -249,19 +251,30 @@ def setup_run(self): callbacks.installafterEsolve(self.check_fields) if self.test: + particle_diag = picmi.ParticleDiagnostic( + name='diag1', + period=self.total_steps, + write_dir='.', + species=[self.ions], + data_list=['ux', 'uy', 'uz', 'x', 'y', 'weighting'], + warpx_file_prefix='Python_ohms_law_solver_magnetic_reconnection_2d_plt', + # warpx_format='openpmd', + # warpx_openpmd_backend='h5', + ) + simulation.add_diagnostic(particle_diag) field_diag = picmi.FieldDiagnostic( - name='field_diag', + name='diag1', grid=self.grid, period=self.total_steps, - data_list=['B', 'E', 'j'], + data_list=['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez'], write_dir='.', warpx_file_prefix='Python_ohms_law_solver_magnetic_reconnection_2d_plt', # warpx_format='openpmd', # warpx_openpmd_backend='h5', - warpx_write_species=True ) simulation.add_diagnostic(field_diag) + # reduced diagnostics for reconnection rate calculation # create a 2 l_i box around the X-point on which to measure # magnetic flux changes diff --git a/Examples/Tests/particle_boundary_process/PICMI_inputs_reflection.py b/Examples/Tests/particle_boundary_process/PICMI_inputs_reflection.py index 18d91ad45fa..9ac40818e12 100755 --- a/Examples/Tests/particle_boundary_process/PICMI_inputs_reflection.py +++ b/Examples/Tests/particle_boundary_process/PICMI_inputs_reflection.py @@ -74,14 +74,19 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 10, + write_dir = '.', + warpx_file_prefix = 'Python_particle_reflection_plt' +) field_diag = picmi.FieldDiagnostic( grid=grid, name = 'diag1', data_list=['E'], period = 10, write_dir = '.', - warpx_file_prefix = 'Python_particle_reflection_plt', - warpx_write_species=False + warpx_file_prefix = 'Python_particle_reflection_plt' ) ########################## @@ -102,6 +107,7 @@ n_macroparticle_per_cell=[5, 2], grid=grid ) ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ########################## diff --git a/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py b/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py index 66a1c0c39e1..618b01b1c46 100755 --- a/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py +++ b/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py @@ -73,6 +73,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = diagnostic_intervals, + write_dir = '.', + warpx_file_prefix = 'Python_particle_scrape_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -102,6 +108,7 @@ ) ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) ########################## diff --git a/Examples/Tests/particle_data_python/PICMI_inputs_2d.py b/Examples/Tests/particle_data_python/PICMI_inputs_2d.py index a7387832a7e..877824715cb 100755 --- a/Examples/Tests/particle_data_python/PICMI_inputs_2d.py +++ b/Examples/Tests/particle_data_python/PICMI_inputs_2d.py @@ -70,6 +70,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 10, + write_dir = '.', + warpx_file_prefix = f"Python_particle_attr_access_{'unique_' if args.unique else ''}plt" +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -96,6 +102,7 @@ n_macroparticle_per_cell=[0, 0], grid=grid ) ) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) sim.initialize_inputs() diff --git a/Examples/Tests/particle_data_python/PICMI_inputs_prev_pos_2d.py b/Examples/Tests/particle_data_python/PICMI_inputs_prev_pos_2d.py index e9f53801752..1becd4464e7 100755 --- a/Examples/Tests/particle_data_python/PICMI_inputs_prev_pos_2d.py +++ b/Examples/Tests/particle_data_python/PICMI_inputs_prev_pos_2d.py @@ -71,15 +71,21 @@ # diagnostics ########################## -field_diag = picmi.ParticleDiagnostic( - species=electrons, +part_diag = picmi.ParticleDiagnostic( name = 'diag1', - data_list=['previous_positions'], period = 10, + species=[electrons], + write_dir = '.', + warpx_file_prefix = 'Python_prev_positions_plt' +) +field_diag = picmi.FieldDiagnostic( + name = 'diag1', + data_list=['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], + period = 10, + grid=grid, write_dir = '.', warpx_file_prefix = 'Python_prev_positions_plt' ) - ########################## # simulation setup ########################## @@ -97,8 +103,8 @@ n_macroparticle_per_cell=[1, 1], grid=grid ) ) +sim.add_diagnostic(part_diag) sim.add_diagnostic(field_diag) - ########################## # simulation run ########################## diff --git a/Examples/Tests/plasma_lens/PICMI_inputs_3d.py b/Examples/Tests/plasma_lens/PICMI_inputs_3d.py index ed64ae3e53f..6114fb05de2 100644 --- a/Examples/Tests/plasma_lens/PICMI_inputs_3d.py +++ b/Examples/Tests/plasma_lens/PICMI_inputs_3d.py @@ -61,10 +61,16 @@ part_diag1 = picmi.ParticleDiagnostic(name = 'diag1', period = max_steps, species = [electrons], - data_list = ['ux', 'uy', 'uz'], + data_list = ['ux', 'uy', 'uz', 'x', 'y', 'z'], write_dir = '.', warpx_file_prefix = 'Python_plasma_lens_plt') +field_diag1 = picmi.FieldDiagnostic(name = 'diag1', + grid = grid, + period = max_steps, + data_list = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], + write_dir = '.', + warpx_file_prefix = 'Python_plasma_lens_plt') # Set up simulation sim = picmi.Simulation(solver = solver, max_steps = max_steps, @@ -81,6 +87,7 @@ # Add diagnostics sim.add_diagnostic(part_diag1) +sim.add_diagnostic(field_diag1) # Write input file that can be used to run with the compiled version #sim.write_input_file(file_name = 'inputs_3d_picmi') diff --git a/Examples/Tests/python_wrappers/PICMI_inputs_2d.py b/Examples/Tests/python_wrappers/PICMI_inputs_2d.py index 2301a558b90..7104963c752 100755 --- a/Examples/Tests/python_wrappers/PICMI_inputs_2d.py +++ b/Examples/Tests/python_wrappers/PICMI_inputs_2d.py @@ -62,6 +62,11 @@ # Initialize diagnostics diag_field_list = ["E", "B"] +particle_diag = picmi.ParticleDiagnostic(name = 'diag1', + period = 10, + write_dir = '.', + warpx_file_prefix = 'Python_wrappers_plt', + data_list = diag_field_list) field_diag = picmi.FieldDiagnostic(name = 'diag1', grid = grid, period = 10, @@ -80,6 +85,7 @@ warpx_use_filter = 1) # Add diagnostics to simulation +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) # Write input file to run with compiled version diff --git a/Examples/Tests/reduced_diags/PICMI_inputs_loadbalancecosts.py b/Examples/Tests/reduced_diags/PICMI_inputs_loadbalancecosts.py index a8c74bf6227..0583a6fe1d0 100644 --- a/Examples/Tests/reduced_diags/PICMI_inputs_loadbalancecosts.py +++ b/Examples/Tests/reduced_diags/PICMI_inputs_loadbalancecosts.py @@ -78,9 +78,19 @@ )) # Diagnostic -diag = picmi.FieldDiagnostic( +particle_diag = picmi.ParticleDiagnostic( + name='diag1', + period=3, + species=[electrons], + data_list = ['ux', 'uy', 'uz', 'x', 'y', 'z', 'weighting'], + write_dir='.', + warpx_file_prefix='Python_reduced_diags_loadbalancecosts_timers_plt' +) +field_diag = picmi.FieldDiagnostic( + name='diag1', grid=grid, period=3, + data_list = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez', 'Jx', 'Jy', 'Jz'], write_dir='.', warpx_file_prefix='Python_reduced_diags_loadbalancecosts_timers_plt' ) @@ -104,7 +114,8 @@ sim.add_diagnostic(reduced_diag) # Add diagnostics -sim.add_diagnostic(diag) +sim.add_diagnostic(particle_diag) +sim.add_diagnostic(field_diag) # Advance simulation until last time step # sim.write_input_file("test_input") diff --git a/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py b/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py index 86699304091..8835e341fc7 100755 --- a/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py +++ b/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py @@ -62,6 +62,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 10, + write_dir = '.', + warpx_file_prefix = f'Python_restart_runtime_components_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -103,6 +109,7 @@ sim.amr_restart = restart_file_name sys.argv.remove(arg) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) sim.add_diagnostic(checkpoint) sim.initialize_inputs() diff --git a/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py b/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py index b7c97c6c1f9..deb3c6060f6 100755 --- a/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py +++ b/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py @@ -63,6 +63,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = 10, + write_dir = '.', + warpx_file_prefix = f'Python_restart_runtime_components_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -104,6 +110,7 @@ sim.amr_restart = restart_file_name sys.argv.remove(arg) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) sim.add_diagnostic(checkpoint) sim.initialize_inputs() diff --git a/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py b/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py index 240283e2a2e..7ef8e26d1d6 100755 --- a/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py +++ b/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py @@ -73,6 +73,12 @@ # diagnostics ########################## +particle_diag = picmi.ParticleDiagnostic( + name = 'diag1', + period = diagnostic_intervals, + write_dir = '.', + warpx_file_prefix = 'Python_restart_eb_plt' +) field_diag = picmi.FieldDiagnostic( name = 'diag1', grid = grid, @@ -116,6 +122,7 @@ sim.amr_restart = restart_file_name sys.argv.remove(arg) +sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) sim.add_diagnostic(checkpoint) sim.initialize_inputs() diff --git a/Python/pywarpx/Diagnostics.py b/Python/pywarpx/Diagnostics.py index 06562c28d77..19860e9b7ee 100644 --- a/Python/pywarpx/Diagnostics.py +++ b/Python/pywarpx/Diagnostics.py @@ -26,3 +26,12 @@ def add_new_attr_with_check(self, name, value): def __setattr__(self, name, value): self.add_new_attr_with_check(name, value) + + def set_or_replace_attr(self, name, value): + """ + Explicitly set or replace an existing attribute + (since __setattr__ cannot be used for replacing + as it would raise an Exception) + """ + assert not name.startswith('_') + self.argvattrs[name] = value diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index 9e88da7c66e..b17035d739d 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -2014,9 +2014,6 @@ class FieldDiagnostic(picmistandard.PICMI_FieldDiagnostic, WarpXDiagnosticBase): warpx_plot_raw_fields_guards: bool, optional Flag whether the raw fields should include the guard cells - warpx_write_species: bool, optional - Flag whether to output particle data with the diagnostic - warpx_format: {plotfile, checkpoint, openpmd, ascent, sensei}, optional Diagnostic file format @@ -2047,7 +2044,6 @@ def init(self, kw): self.plot_raw_fields_guards = kw.pop('warpx_plot_raw_fields_guards', None) self.plot_finepatch = kw.pop('warpx_plot_finepatch', None) self.plot_crsepatch = kw.pop('warpx_plot_crsepatch', None) - self.write_species = kw.pop('warpx_write_species', None) self.format = kw.pop('warpx_format', 'plotfile') self.openpmd_backend = kw.pop('warpx_openpmd_backend', None) self.file_prefix = kw.pop('warpx_file_prefix', None) @@ -2130,7 +2126,7 @@ def initialize_inputs(self): # --- is the same on all processors. fields_to_plot = list(fields_to_plot) fields_to_plot.sort() - self.diagnostic.fields_to_plot = fields_to_plot + self.diagnostic.set_or_replace_attr('fields_to_plot', fields_to_plot) particle_fields_to_plot_names = list() for pfd in self.particle_fields_to_plot: @@ -2157,8 +2153,8 @@ def initialize_inputs(self): self.diagnostic.plot_raw_fields_guards = self.plot_raw_fields_guards self.diagnostic.plot_finepatch = self.plot_finepatch self.diagnostic.plot_crsepatch = self.plot_crsepatch - self.diagnostic.write_species = self.write_species - + if 'write_species' not in self.diagnostic.argvattrs: + self.diagnostic.write_species = False self.set_write_dir() @@ -2263,7 +2259,9 @@ def initialize_inputs(self): self.diagnostic.openpmd_backend = self.openpmd_backend self.diagnostic.file_min_digits = self.file_min_digits self.diagnostic.intervals = self.period - + self.diagnostic.set_or_replace_attr('write_species', True) + if 'fields_to_plot' not in self.diagnostic.argvattrs: + self.diagnostic.fields_to_plot = 'none' self.set_write_dir() # --- Use a set to ensure that fields don't get repeated. @@ -2296,24 +2294,26 @@ def initialize_inputs(self): variables.sort() # species list - if np.iterable(self.species): - species_list = self.species + if self.species is None: + species_names = pywarpx.particles.species_names + elif np.iterable(self.species): + species_names = [specie.name for specie in self.species] else: - species_list = [self.species] + species_names = [species.name] if self.mangle_dict is None: # Only do this once so that the same variables are used in this distribution # is used multiple times self.mangle_dict = pywarpx.my_constants.add_keywords(self.user_defined_kw) - for specie in species_list: - diag = pywarpx.Bucket.Bucket(self.name + '.' + specie.name, + for name in species_names: + diag = pywarpx.Bucket.Bucket(self.name + '.' + name, variables = variables, random_fraction = self.random_fraction, uniform_stride = self.uniform_stride) expression = pywarpx.my_constants.mangle_expression(self.plot_filter_function, self.mangle_dict) diag.__setattr__('plot_filter_function(t,x,y,z,ux,uy,uz)', expression) - self.diagnostic._species_dict[specie.name] = diag + self.diagnostic._species_dict[name] = diag # ---------------------------- # Lab frame diagnostics @@ -2326,8 +2326,6 @@ class LabFrameFieldDiagnostic(picmistandard.PICMI_LabFrameFieldDiagnostic, See `Input Parameters `_ for more information. - This will by default write out both field and particle data. This can be changed by setting warpx_write_species. - Parameters ---------- warpx_format: string, optional @@ -2354,11 +2352,10 @@ class LabFrameFieldDiagnostic(picmistandard.PICMI_LabFrameFieldDiagnostic, warpx_upper_bound: vector of floats, optional Passed to .upper_bound - - warpx_write_species: bool, optional, default=True - Whether the species will also be written out. """ def init(self, kw): + """The user is using the new BTD""" + self.format = kw.pop('warpx_format', None) self.openpmd_backend = kw.pop('warpx_openpmd_backend', None) self.file_prefix = kw.pop('warpx_file_prefix', None) @@ -2367,7 +2364,6 @@ def init(self, kw): self.buffer_size = kw.pop('warpx_buffer_size', None) self.lower_bound = kw.pop('warpx_lower_bound', None) self.upper_bound = kw.pop('warpx_upper_bound', None) - self.write_species = kw.pop('warpx_write_species', None) def initialize_inputs(self): @@ -2380,7 +2376,7 @@ def initialize_inputs(self): self.diagnostic.diag_lo = self.lower_bound self.diagnostic.diag_hi = self.upper_bound - self.diagnostic.do_back_transformed_fields = 1 + self.diagnostic.do_back_transformed_fields = True self.diagnostic.dt_snapshots_lab = self.dt_snapshots self.diagnostic.buffer_size = self.buffer_size @@ -2390,8 +2386,6 @@ def initialize_inputs(self): else: self.diagnostic.num_snapshots_lab = self.num_snapshots - self.diagnostic.do_back_transformed_particles = self.write_species - # --- Use a set to ensure that fields don't get repeated. fields_to_plot = set() @@ -2428,8 +2422,10 @@ def initialize_inputs(self): # --- is the same on all processors. fields_to_plot = list(fields_to_plot) fields_to_plot.sort() - self.diagnostic.fields_to_plot = fields_to_plot + self.diagnostic.set_or_replace_attr('fields_to_plot', fields_to_plot) + if 'write_species' not in self.diagnostic.argvattrs: + self.diagnostic.write_species = False self.set_write_dir() @@ -2439,8 +2435,6 @@ class LabFrameParticleDiagnostic(picmistandard.PICMI_LabFrameParticleDiagnostic, See `Input Parameters `_ for more information. - This will by default write out both field and particle data. This can be changed by setting warpx_write_fields. - Parameters ---------- warpx_format: string, optional @@ -2461,9 +2455,6 @@ class LabFrameParticleDiagnostic(picmistandard.PICMI_LabFrameParticleDiagnostic, warpx_buffer_size: integer, optional Passed to .buffer_size - - warpx_write_fields: bool, optional, default=True - Whether the fields will also be written out. """ def init(self, kw): self.format = kw.pop('warpx_format', None) @@ -2472,7 +2463,6 @@ def init(self, kw): self.intervals = kw.pop('warpx_intervals', None) self.file_min_digits = kw.pop('warpx_file_min_digits', None) self.buffer_size = kw.pop('warpx_buffer_size', None) - self.write_fields = kw.pop('warpx_write_fields', None) def initialize_inputs(self): @@ -2483,7 +2473,7 @@ def initialize_inputs(self): self.diagnostic.openpmd_backend = self.openpmd_backend self.diagnostic.file_min_digits = self.file_min_digits - self.diagnostic.do_back_transformed_particles = 1 + self.diagnostic.do_back_transformed_particles = True self.diagnostic.dt_snapshots_lab = self.dt_snapshots self.diagnostic.buffer_size = self.buffer_size @@ -2493,7 +2483,11 @@ def initialize_inputs(self): else: self.diagnostic.num_snapshots_lab = self.num_snapshots - self.diagnostic.do_back_transformed_fields = self.write_fields + self.diagnostic.do_back_transformed_fields = False + + self.diagnostic.set_or_replace_attr('write_species', True) + if 'fields_to_plot' not in self.diagnostic.argvattrs: + self.diagnostic.fields_to_plot = 'none' self.set_write_dir() @@ -2527,15 +2521,17 @@ def initialize_inputs(self): variables.sort() # species list - if np.iterable(self.species): - species_list = self.species + if self.species is None: + species_names = pywarpx.particles.species_names + elif np.iterable(self.species): + species_names = [specie.name for specie in self.species] else: - species_list = [self.species] + species_names = [species.name] - for specie in species_list: - diag = pywarpx.Bucket.Bucket(self.name + '.' + specie.name, + for name in species_names: + diag = pywarpx.Bucket.Bucket(self.name + '.' + name, variables = variables) - self.diagnostic._species_dict[specie.name] = diag + self.diagnostic._species_dict[name] = diag class ReducedDiagnostic(picmistandard.base._ClassWithInit, WarpXDiagnosticBase): diff --git a/Python/setup.py b/Python/setup.py index 568c3f83ddc..e5165b17c41 100644 --- a/Python/setup.py +++ b/Python/setup.py @@ -59,7 +59,7 @@ package_dir = {'pywarpx': 'pywarpx'}, description = """Wrapper of WarpX""", package_data = package_data, - install_requires = ['numpy', 'picmistandard==0.26.0', 'periodictable'], + install_requires = ['numpy', 'picmistandard==0.28.0', 'periodictable'], python_requires = '>=3.8', zip_safe=False ) diff --git a/requirements.txt b/requirements.txt index 7d2aa5376ad..60f14e1282d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,7 +4,7 @@ periodictable~=1.5 # PICMI # note: don't forget to update the version in Docs/requirements.txt, too -picmistandard==0.26.0 +picmistandard==0.28.0 # for development against an unreleased PICMI version, use: #picmistandard @ git+https://github.com/picmi-standard/picmi.git#subdirectory=PICMI_Python From 807f0daf577a03d4b82cae3f5e110e28f49ddbe7 Mon Sep 17 00:00:00 2001 From: Grant Johnson <69021085+johnson452@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:20:26 -0700 Subject: [PATCH 052/110] Added extended writeup to the docs for PR3991, fluids (#4365) --- Docs/source/theory/cold_fluid_model.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Docs/source/theory/cold_fluid_model.rst b/Docs/source/theory/cold_fluid_model.rst index 0a98ede772f..971ebdba062 100644 --- a/Docs/source/theory/cold_fluid_model.rst +++ b/Docs/source/theory/cold_fluid_model.rst @@ -97,7 +97,8 @@ Step 5: **Current and Charge Deposition** The implemented MUSCL scheme has a simplifed slope averaging, see the extended writeup for details. - More details on the precise implementation will be made available online soon. + More details on the precise implementation are available here, `WarpX_Cold_Rel_Fluids.pdf`_. +.. _WarpX_Cold_Rel_Fluids.pdf: https://github.com/ECP-WarpX/WarpX/files/12886437/WarpX_Cold_Rel_Fluids.pdf .. warning:: If using the fluid model with the Kinetic-Fluid Hybrid model or the electrostatic solver, there is a known From d07e8bbd2fdba0b40c67d10768e5789a0d9c1f6d Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 13 Oct 2023 07:49:40 -0700 Subject: [PATCH 053/110] Electrostatic Diagnostics for `j`: Default to Direct Deposition (#4362) * Change Default: Direct Current Deposition for ES Used as a diagnostics for `j`. * Reset j in Benchmarks Esirkepov -> Direct Deposition --- Docs/source/usage/parameters.rst | 4 +++- .../Checksum/benchmarks_json/Python_prev_positions.json | 6 +++--- .../Checksum/benchmarks_json/hard_edged_quadrupoles.json | 6 +++--- .../benchmarks_json/hard_edged_quadrupoles_moving.json | 6 +++--- Source/Utils/WarpXAlgorithmSelection.cpp | 3 ++- Source/WarpX.cpp | 3 ++- 6 files changed, 16 insertions(+), 12 deletions(-) diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index 17504a8065f..06b3edd0f2d 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -1824,7 +1824,9 @@ Particle push, charge and current deposition, field gathering Available options are: ``direct``, ``esirkepov``, and ``vay``. The default choice is ``esirkepov`` for FDTD maxwell solvers but ``direct`` for standard or Galilean PSATD solver (i.e. with ``algo.maxwell_solver = psatd``) and - for the hybrid-PIC solver (i.e. with ``algo.maxwell_solver = hybrid``). + for the hybrid-PIC solver (i.e. with ``algo.maxwell_solver = hybrid``) and for + diagnostics output with the electrostatic solvers (i.e., with + ``warpx.do_electrostatic = ...``). Note that ``vay`` is only available for ``algo.maxwell_solver = psatd``. 1. ``direct`` diff --git a/Regression/Checksum/benchmarks_json/Python_prev_positions.json b/Regression/Checksum/benchmarks_json/Python_prev_positions.json index e6e39d2272b..55c886775d5 100644 --- a/Regression/Checksum/benchmarks_json/Python_prev_positions.json +++ b/Regression/Checksum/benchmarks_json/Python_prev_positions.json @@ -6,9 +6,9 @@ "Ex": 3710588.849989976, "Ey": 0.0, "Ez": 646727.8074440088, - "jx": 4745.078379617619, - "jy": 368.4331779923921, - "jz": 632.1508106460103 + "jx": 15259.034603501308, + "jy": 650.139263398662, + "jz": 943.0244062246846 }, "electrons": { "particle_momentum_x": 8.78764082600202e-23, diff --git a/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles.json b/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles.json index 80a2dc87964..f8186748f2c 100644 --- a/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles.json +++ b/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles.json @@ -6,9 +6,9 @@ "Ex": 9.882421146615367e-06, "Ey": 1.0440261046714249e-05, "Ez": 1.003739697324731e-05, - "jx": 2.9148662809570633e-10, - "jy": 8.46582291468749e-19, - "jz": 3.823492756863969e-08 + "jx": 2.914866280957325e-10, + "jy": 8.46605718473121e-19, + "jz": 3.82349275686397e-08 }, "electron": { "particle_momentum_x": 2.0819392991319055e-25, diff --git a/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles_moving.json b/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles_moving.json index 813eead7374..05b398e4292 100644 --- a/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles_moving.json +++ b/Regression/Checksum/benchmarks_json/hard_edged_quadrupoles_moving.json @@ -6,9 +6,9 @@ "Ex": 6.0282565190090465e-05, "Ey": 6.38479659567398e-05, "Ez": 7.880459213065183e-05, - "jx": 1.1659465170956616e-09, - "jy": 2.6115239381823616e-17, - "jz": 1.5293971084510288e-07 + "jx": 1.1659465170956563e-09, + "jy": 1.3057688751494639e-17, + "jz": 1.5293971084510282e-07 }, "electron": { "particle_momentum_x": 2.0819392998019267e-25, diff --git a/Source/Utils/WarpXAlgorithmSelection.cpp b/Source/Utils/WarpXAlgorithmSelection.cpp index d9cd5c138a8..ed8186a325c 100644 --- a/Source/Utils/WarpXAlgorithmSelection.cpp +++ b/Source/Utils/WarpXAlgorithmSelection.cpp @@ -158,7 +158,8 @@ GetAlgorithmInteger(const amrex::ParmParse& pp, const char* pp_search_key ){ } else if (0 == std::strcmp(pp_search_key, "current_deposition")) { algo_to_int = current_deposition_algo_to_int; if (WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::PSATD || - WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::HybridPIC) + WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::HybridPIC || + WarpX::electrostatic_solver_id != ElectrostaticSolverAlgo::None) algo_to_int["default"] = CurrentDepositionAlgo::Direct; } else if (0 == std::strcmp(pp_search_key, "charge_deposition")) { algo_to_int = charge_deposition_algo_to_int; diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 6be8d517775..2df53dc6d2c 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -1130,7 +1130,8 @@ WarpX::ReadParameters () } #endif - // note: current_deposition must be set after maxwell_solver is already determined, + // note: current_deposition must be set after maxwell_solver (electromagnetic_solver_id) or + // do_electrostatic (electrostatic_solver_id) are already determined, // because its default depends on the solver selection current_deposition_algo = GetAlgorithmInteger(pp_algo, "current_deposition"); charge_deposition_algo = GetAlgorithmInteger(pp_algo, "charge_deposition"); From a121b2aa9c51848a31c572e5a56c33882436f404 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Sat, 14 Oct 2023 00:15:44 -0700 Subject: [PATCH 054/110] Python: Clean Out Old Bindings (#4233) * libwarpx: clean out old bindings * Docs: Update Wrappers * Update Tests: `getistep`, `set_potential_on_eb`, `deposit_charge_density` * getistep: lev argument * Apply suggestions from code review Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> * Fix capacitive_discharge & ohm_solver_ion_beam_instability * Update `ParticleContainerWrapper` to `ParticleBoundaryBufferWrapper` in docs --------- Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> --- Docs/source/usage/python.rst | 80 +++--- .../capacitive_discharge/PICMI_inputs_1d.py | 12 +- .../capacitive_discharge/PICMI_inputs_2d.py | 2 +- .../PICMI_inputs_3d.py | 2 +- Examples/Tests/langmuir/PICMI_inputs_rz.py | 2 +- .../Tests/ohm_solver_EM_modes/PICMI_inputs.py | 8 +- .../PICMI_inputs.py | 16 +- .../PICMI_inputs.py | 25 +- .../PICMI_inputs.py | 11 +- .../PICMI_inputs_scrape.py | 4 +- .../particle_data_python/PICMI_inputs_2d.py | 4 +- .../pass_mpi_communicator/PICMI_inputs_2d.py | 2 +- .../Tests/restart/PICMI_inputs_id_cpu_read.py | 2 +- .../PICMI_inputs_runtime_component_analyze.py | 2 +- .../restart_eb/PICMI_inputs_restart_eb.py | 2 +- Python/pywarpx/WarpX.py | 2 +- Python/pywarpx/_libwarpx.py | 234 ------------------ Python/pywarpx/particle_containers.py | 61 +++++ Source/Python/CMakeLists.txt | 3 - Source/Python/Make.package | 1 - Source/Python/WarpXWrappers.H | 80 ------ Source/Python/WarpXWrappers.cpp | 172 ------------- 22 files changed, 159 insertions(+), 568 deletions(-) delete mode 100644 Source/Python/WarpXWrappers.H delete mode 100644 Source/Python/WarpXWrappers.cpp diff --git a/Docs/source/usage/python.rst b/Docs/source/usage/python.rst index 90015a85b9a..dd44c4098fd 100644 --- a/Docs/source/usage/python.rst +++ b/Docs/source/usage/python.rst @@ -14,7 +14,7 @@ defining the simulation time, field solver, registered species, etc. .. _usage-picmi-parameters: Classes ----------- +------- Simulation and grid setup ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -222,6 +222,8 @@ where ```` is the number of MPI ranks used, and ```` is the name of the script. +.. _usage-picmi-extend: + Extending a Simulation from Python ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -235,85 +237,101 @@ Places in the WarpX loop where callbacks are available include: ``afterinit``, ``beforecollisions``, ``aftercollisions``, ``beforeEsolve``, ``afterEsolve``, ``beforeInitEsolve``, ``afterInitEsolve``, ``beforedeposition``, ``afterdeposition``, ``beforestep``, ``afterstep``, ``afterdiagnostics``,``afterrestart`` and ``oncheckpointsignal``. -See the examples in *Examples/Tests/ParticleDataPython* for references on how to use +See the examples in ``Examples/Tests/ParticleDataPython`` for references on how to use ``callbacks``. There are several "hooks" available via the ``libwarpx`` shared library to access and manipulate simulation objects (particles, fields and memory buffers) as well as general properties (such as processor number). These "hooks" are accessible through the `Simulation.extension` object. -.. autofunction:: pywarpx.picmi.Simulation.extension.getNProcs +An important object is ``Simulation.extension.warpx``, which is available during simulation run. +This object is the Python equivalent to the central ``WarpX`` simulation class and provides access to +field ``MultiFab`` and ``ParticleContainer`` data. -.. autofunction:: pywarpx.picmi.Simulation.extension.getMyProc +.. function:: pywarpx.picmi.Simulation.extension.warpx.getistep -.. autofunction:: pywarpx.picmi.Simulation.extension.get_nattr +.. function:: pywarpx.picmi.Simulation.extension.warpx.gett_new -.. autofunction:: pywarpx.picmi.Simulation.extension.get_nattr_species +.. function:: pywarpx.picmi.Simulation.extension.warpx.evolve -.. autofunction:: pywarpx.picmi.Simulation.extension.getistep +.. autofunction:: pywarpx.picmi.Simulation.extension.finalize -.. autofunction:: pywarpx.picmi.Simulation.extension.gett_new +These and other classes are provided through `pyAMReX `__. +After the simulation is initialized, pyAMReX can be accessed via -.. autofunction:: pywarpx.picmi.Simulation.extension.evolve +.. code-block:: python -.. autofunction:: pywarpx.picmi.Simulation.extension.finalize + from pywarpx import picmi, libwarpx + + # ... simulation definition ... + + # equivalent to + # import amrex.space3d as amr + # for a 3D simulation + amr = libwarpx.amr # picks the right 1d, 2d or 3d variant + +.. function:: amr.ParallelDescriptor.NProcs() -.. autofunction:: pywarpx.picmi.Simulation.extension.getProbLo +.. function:: amr.ParallelDescriptor.MyProc() -.. autofunction:: pywarpx.picmi.Simulation.extension.getProbHi +.. function:: amr.ParallelDescriptor.IOProcessor() -.. autofunction:: pywarpx.picmi.Simulation.extension.getCellSize +.. function:: amr.ParallelDescriptor.IOProcessorNumber() Particles can be added to the simulation at specific positions and with specific attribute values: -.. autofunction:: pywarpx.picmi.Simulation.extension.add_particles +.. code-block:: python + + from pywarpx import particle_containers, picmi + + # ... + + electron_wrapper = particle_containers.ParticleContainerWrapper("electrons") + +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.add_particles Properties of the particles already in the simulation can be obtained with various functions. -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_count +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.get_particle_count -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_structs +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.get_particle_structs -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_arrays +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.get_particle_arrays The ``get_particle_structs()`` and ``get_particle_arrays()`` functions are called by several utility functions of the form ``get_particle_{comp_name}`` where ``comp_name`` is one of ``x``, ``y``, ``z``, ``r``, ``theta``, ``id``, ``cpu``, ``weight``, ``ux``, ``uy`` or ``uz``. -The index of some specific component of the particle data can be obtained. - -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_comp_index - New components can be added via Python. -.. autofunction:: pywarpx.picmi.Simulation.extension.add_real_comp +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.add_real_comp Various diagnostics are also accessible from Python. This includes getting the deposited or total charge density from a given species as well as accessing the scraped particle buffer. See the example in -*Examples/Tests/ParticleBoudaryScrape* for a reference on how to interact +``Examples/Tests/ParticleBoundaryScrape`` for a reference on how to interact with scraped particle data. -.. autofunction:: pywarpx.picmi.Simulation.extension.get_species_charge_sum +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.get_species_charge_sum -.. autofunction:: pywarpx.picmi.Simulation.extension.depositChargeDensity +.. autofunction:: pywarpx.particle_containers.ParticleContainerWrapper.deposit_charge_density -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_boundary_buffer_size +.. autofunction:: pywarpx.particle_containers.ParticleBoundaryBufferWrapper.get_particle_boundary_buffer_size -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_boundary_buffer_structs +.. autofunction:: pywarpx.particle_containers.ParticleBoundaryBufferWrapper.get_particle_boundary_buffer_structs -.. autofunction:: pywarpx.picmi.Simulation.extension.get_particle_boundary_buffer +.. autofunction:: pywarpx.particle_containers.ParticleBoundaryBufferWrapper.get_particle_boundary_buffer -.. autofunction:: pywarpx.picmi.Simulation.extension.clearParticleBoundaryBuffer +.. autofunction:: pywarpx.particle_containers.ParticleBoundaryBufferWrapper.clearParticleBoundaryBuffer The embedded boundary conditions can be modified when using the electrostatic solver. -.. autofunction:: pywarpx.picmi.Simulation.extension.set_potential_EB +.. function:: pywarpx.picmi.Simulation.extension.warpx.set_potential_on_eb -Using Python input as a preprocessor +Using Python Input as a Preprocessor ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In this case, only the pure Python version needs to be installed, as described :ref:`here `. diff --git a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py index dc9428229a4..0c6f9828543 100644 --- a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py +++ b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_1d.py @@ -11,7 +11,7 @@ from scipy.sparse import csc_matrix from scipy.sparse import linalg as sla -from pywarpx import callbacks, fields, particle_containers, picmi +from pywarpx import callbacks, fields, libwarpx, particle_containers, picmi constants = picmi.constants @@ -108,7 +108,7 @@ def solve(self): calculating phi from rho.""" left_voltage = 0.0 - t = self.sim_ext.gett_new() + t = self.sim.extension.warpx.gett_new(0) right_voltage = eval(self.right_voltage) # Construct b vector @@ -300,6 +300,7 @@ def setup_run(self): warpx_load_balance_intervals=self.max_steps//5000, verbose=self.test ) + self.solver.sim = self.sim self.sim.add_species( self.electrons, @@ -313,7 +314,6 @@ def setup_run(self): n_macroparticle_per_cell=[self.seed_nppc], grid=self.grid ) ) - self.solver.sim_ext = self.sim.extension ####################################################################### # Add diagnostics for the CI test to be happy # @@ -325,6 +325,7 @@ def setup_run(self): file_prefix = 'Python_background_mcc_1d_tridiag_plt' particle_diag = picmi.ParticleDiagnostic( + species=[self.electrons, self.ions], name='diag1', period=0, write_dir='.', @@ -343,7 +344,8 @@ def setup_run(self): def _get_rho_ions(self): # deposit the ion density in rho_fp - self.sim.extension.depositChargeDensity('he_ions', 0) + he_ions_wrapper = particle_containers.ParticleContainerWrapper('he_ions') + he_ions_wrapper.deposit_charge_density(level=0) rho_data = self.rho_wrapper[...] self.ion_density_array += rho_data / constants.q_e / self.diag_steps @@ -357,7 +359,7 @@ def run_sim(self): self.sim.step(self.diag_steps) - if self.sim.extension.getMyProc() == 0: + if libwarpx.amr.ParallelDescriptor.MyProc() == 0: np.save(f'ion_density_case_{self.n+1}.npy', self.ion_density_array) # query the particle z-coordinates if this is run during CI testing diff --git a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py index fd66ba34a04..71e1070ef2f 100755 --- a/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py +++ b/Examples/Physics_applications/capacitive_discharge/PICMI_inputs_2d.py @@ -177,7 +177,7 @@ def solve(self): calculating phi from rho.""" right_voltage = eval( self.right_voltage, - {'t':sim.extension.gett_new(0), 'sin':np.sin, 'pi':np.pi} + {'t': sim.extension.warpx.gett_new(0), 'sin': np.sin, 'pi': np.pi} ) left_voltage = 0.0 diff --git a/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py b/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py index 4da7feeff3a..55fbc87bd9e 100755 --- a/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py +++ b/Examples/Tests/electrostatic_sphere_eb/PICMI_inputs_3d.py @@ -119,6 +119,6 @@ sim.step(1) -sim.extension.set_potential_EB("2.") +sim.extension.warpx.set_potential_on_eb("2.") sim.step(1) diff --git a/Examples/Tests/langmuir/PICMI_inputs_rz.py b/Examples/Tests/langmuir/PICMI_inputs_rz.py index 018303a9611..8328aba5185 100755 --- a/Examples/Tests/langmuir/PICMI_inputs_rz.py +++ b/Examples/Tests/langmuir/PICMI_inputs_rz.py @@ -178,7 +178,7 @@ def calcEz( z, r, k0, w0, wp, t, epsilons) : return( Ez_array ) # Current time of the simulation -t0 = sim.extension.gett_new(0) +t0 = sim.extension.warpx.gett_new(0) # Get the raw field data. Note that these are the real and imaginary # parts of the fields for each azimuthal mode. diff --git a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py index b51d291e478..51dbb53ab43 100644 --- a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py @@ -15,7 +15,7 @@ from mpi4py import MPI as mpi import numpy as np -from pywarpx import callbacks, fields, picmi +from pywarpx import callbacks, fields, libwarpx, picmi constants = picmi.constants @@ -25,8 +25,6 @@ warpx_serialize_initial_conditions=True, verbose=0 ) -# make a shorthand for simulation.extension since we use it a lot -sim_ext = simulation.extension class EMModes(object): @@ -318,7 +316,7 @@ def _record_average_fields(self): similar format as the reduced diagnostic so that the same analysis script can be used regardless of the simulation dimension. """ - step = sim_ext.getistep() - 1 + step = simulation.extension.warpx.getistep(lev=0) - 1 if step % self.diag_steps != 0: return @@ -327,7 +325,7 @@ def _record_average_fields(self): By_warpx = fields.BxWrapper()[...] Ez_warpx = fields.EzWrapper()[...] - if sim_ext.getMyProc() != 0: + if libwarpx.amr.ParallelDescriptor.MyProc() != 0: return t = step * self.dt diff --git a/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py b/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py index ecad73d381c..bd28f46b087 100644 --- a/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_ion_Landau_damping/PICMI_inputs.py @@ -14,7 +14,7 @@ from mpi4py import MPI as mpi import numpy as np -from pywarpx import callbacks, fields, particle_containers, picmi +from pywarpx import callbacks, fields, libwarpx, particle_containers, picmi constants = picmi.constants @@ -22,9 +22,8 @@ simulation = picmi.Simulation( warpx_serialize_initial_conditions=True, - verbose=0) -# make a shorthand for simulation.extension since we use it a lot -sim_ext = simulation.extension + verbose=0 +) class IonLandauDamping(object): @@ -266,7 +265,8 @@ def setup_run(self): def text_diag(self): """Diagnostic function to print out timing data and particle numbers.""" - step = sim_ext.getistep(0) + step = simulation.extension.warpx.getistep(lev=0) - 1 + if step % (self.total_steps // 10) != 0: return @@ -290,7 +290,7 @@ def text_diag(self): "{step_rate:4.2f} steps/s" ) - if sim_ext.getMyProc() == 0: + if libwarpx.amr.ParallelDescriptor.MyProc() == 0: print(diag_string.format(**status_dict)) self.prev_time = time.time() @@ -301,14 +301,14 @@ def _record_average_fields(self): similar format as the reduced diagnostic so that the same analysis script can be used regardless of the simulation dimension. """ - step = sim_ext.getistep() - 1 + step = simulation.extension.warpx.getistep(lev=0) - 1 if step % self.diag_steps != 0: return Ez_warpx = fields.EzWrapper()[...] - if sim_ext.getMyProc() != 0: + if libwarpx.amr.ParallelDescriptor.MyProc() != 0: return t = step * self.dt diff --git a/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py b/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py index 21f55fb1166..79f268f6e3b 100644 --- a/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py @@ -15,7 +15,7 @@ from mpi4py import MPI as mpi import numpy as np -from pywarpx import callbacks, fields, particle_containers, picmi +from pywarpx import callbacks, fields, libwarpx, particle_containers, picmi constants = picmi.constants @@ -23,9 +23,8 @@ simulation = picmi.Simulation( warpx_serialize_initial_conditions=True, - verbose=0) -# make a shorthand for simulation.extension since we use it a lot -sim_ext = simulation.extension + verbose=0 +) class HybridPICBeamInstability(object): @@ -336,13 +335,17 @@ def _create_data_arrays(self): self.start_time = self.prev_time self.prev_step = 0 - if sim_ext.getMyProc() == 0: + if libwarpx.amr.ParallelDescriptor.MyProc() == 0: # allocate arrays for storing energy values self.energy_vals = np.zeros((self.total_steps//self.diag_steps, 4)) def text_diag(self): """Diagnostic function to print out timing data and particle numbers.""" - step = sim_ext.getistep(0) + step = simulation.extension.warpx.getistep(lev=0) - 1 + + if not hasattr(self, "prev_time"): + self._create_data_arrays() + if step % (self.total_steps // 10) != 0: return @@ -368,7 +371,7 @@ def text_diag(self): "{step_rate:4.2f} steps/s" ) - if sim_ext.getMyProc() == 0: + if libwarpx.amr.ParallelDescriptor.MyProc() == 0: print(diag_string.format(**status_dict)) self.prev_time = time.time() @@ -377,8 +380,8 @@ def text_diag(self): def energy_diagnostic(self): """Diangostic to get the total, magnetic and kinetic energies in the simulation.""" + step = simulation.extension.warpx.getistep(lev=0) - 1 - step = sim_ext.getistep(0) if step % self.diag_steps != 1: return @@ -391,7 +394,7 @@ def energy_diagnostic(self): Ec_par, Ec_perp = self._get_kinetic_energy(self.ion_container_wrapper) Eb_par, Eb_perp = self._get_kinetic_energy(self.beam_ion_container_wrapper) - if sim_ext.getMyProc() != 0: + if libwarpx.amr.ParallelDescriptor.MyProc() != 0: return self.energy_vals[idx, 0] = Ec_par @@ -426,14 +429,14 @@ def _record_average_fields(self): similar format as the reduced diagnostic so that the same analysis script can be used regardless of the simulation dimension. """ - step = sim_ext.getistep() - 1 + step = simulation.extension.warpx.getistep(lev=0) - 1 if step % self.diag_steps != 0: return By_warpx = fields.BxWrapper()[...] - if sim_ext.getMyProc() != 0: + if libwarpx.amr.ParallelDescriptor.MyProc() != 0: return t = step * self.dt diff --git a/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py b/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py index bdeef96e0a6..143f1f3e82e 100644 --- a/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_magnetic_reconnection/PICMI_inputs.py @@ -16,7 +16,7 @@ from mpi4py import MPI as mpi import numpy as np -from pywarpx import callbacks, fields, picmi +from pywarpx import callbacks, fields, libwarpx, picmi constants = picmi.constants @@ -24,9 +24,8 @@ simulation = picmi.Simulation( warpx_serialize_initial_conditions=True, - verbose=0) -# make a shorthand for simulation.extension since we use it a lot -sim_ext = simulation.extension + verbose=0 +) class ForceFreeSheetReconnection(object): @@ -306,7 +305,7 @@ def setup_run(self): def check_fields(self): - step = sim_ext.getistep() + step = simulation.extension.warpx.getistep(lev=0) - 1 if not (step == 1 or step%self.diag_steps == 0): return @@ -318,7 +317,7 @@ def check_fields(self): By = fields.ByFPWrapper(include_ghosts=False)[...] / self.B0 Bz = fields.BzFPWrapper(include_ghosts=False)[...] / self.B0 - if sim_ext.getMyProc() != 0: + if libwarpx.amr.ParallelDescriptor.MyProc() != 0: return # save the fields to file diff --git a/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py b/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py index 618b01b1c46..9871bdac655 100755 --- a/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py +++ b/Examples/Tests/particle_boundary_scrape/PICMI_inputs_scrape.py @@ -5,7 +5,7 @@ import numpy as np -from pywarpx import particle_containers, picmi +from pywarpx import libwarpx, particle_containers, picmi ########################## # numerics parameters @@ -125,7 +125,7 @@ from mpi4py import MPI as mpi -my_id = sim.extension.getMyProc() +my_id = libwarpx.amr.ParallelDescriptor.MyProc() particle_buffer = particle_containers.ParticleBoundaryBufferWrapper() diff --git a/Examples/Tests/particle_data_python/PICMI_inputs_2d.py b/Examples/Tests/particle_data_python/PICMI_inputs_2d.py index 877824715cb..a4b7d9e134e 100755 --- a/Examples/Tests/particle_data_python/PICMI_inputs_2d.py +++ b/Examples/Tests/particle_data_python/PICMI_inputs_2d.py @@ -4,7 +4,7 @@ import numpy as np -from pywarpx import callbacks, particle_containers, picmi +from pywarpx import callbacks, libwarpx, particle_containers, picmi # Create the parser and add the argument parser = argparse.ArgumentParser() @@ -119,7 +119,7 @@ elec_wrapper = particle_containers.ParticleContainerWrapper('electrons') elec_wrapper.add_real_comp('newPid') -my_id = sim.extension.getMyProc() +my_id = libwarpx.amr.ParallelDescriptor.MyProc() def add_particles(): diff --git a/Examples/Tests/pass_mpi_communicator/PICMI_inputs_2d.py b/Examples/Tests/pass_mpi_communicator/PICMI_inputs_2d.py index b420b71c3aa..66f259da2ef 100755 --- a/Examples/Tests/pass_mpi_communicator/PICMI_inputs_2d.py +++ b/Examples/Tests/pass_mpi_communicator/PICMI_inputs_2d.py @@ -143,4 +143,4 @@ # verify that amrex proc ranks are offset by -1 from # world comm proc ranks -# assert sim.extension.getMyProc() == rank - 1 +# assert libwarpx.amr.ParallelDescriptor.MyProc() == rank - 1 diff --git a/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py b/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py index 8835e341fc7..d400924a378 100755 --- a/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py +++ b/Examples/Tests/restart/PICMI_inputs_id_cpu_read.py @@ -150,7 +150,7 @@ def add_particles(): # simulation run ########################## -step_number = sim.extension.getistep(0) +step_number = sim.extension.warpx.getistep(lev=0) sim.step(max_steps) ############################################### diff --git a/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py b/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py index deb3c6060f6..706dedb6959 100755 --- a/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py +++ b/Examples/Tests/restart/PICMI_inputs_runtime_component_analyze.py @@ -150,7 +150,7 @@ def add_particles(): # simulation run ########################## -step_number = sim.extension.getistep(0) +step_number = sim.extension.warpx.getistep(lev=0) sim.step(max_steps - 1 - step_number) ########################## diff --git a/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py b/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py index 7ef8e26d1d6..a4727053334 100755 --- a/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py +++ b/Examples/Tests/restart_eb/PICMI_inputs_restart_eb.py @@ -132,5 +132,5 @@ # simulation run ########################## -step_number = sim.extension.getistep(0) +step_number = sim.extension.warpx.getistep(lev=0) sim.step(max_steps - step_number) diff --git a/Python/pywarpx/WarpX.py b/Python/pywarpx/WarpX.py index cfa55641427..3fa59285f26 100644 --- a/Python/pywarpx/WarpX.py +++ b/Python/pywarpx/WarpX.py @@ -98,7 +98,7 @@ def init(self, mpi_comm=None, **kw): libwarpx.initialize(argv, mpi_comm=mpi_comm) def evolve(self, nsteps=-1): - libwarpx.evolve(nsteps) + libwarpx.warpx.evolve(nsteps) def finalize(self, finalize_mpi=1): libwarpx.finalize(finalize_mpi) diff --git a/Python/pywarpx/_libwarpx.py b/Python/pywarpx/_libwarpx.py index bf3ec0ded1a..fa2044d4240 100755 --- a/Python/pywarpx/_libwarpx.py +++ b/Python/pywarpx/_libwarpx.py @@ -14,7 +14,6 @@ import atexit import os -import sys import numpy as np @@ -114,46 +113,6 @@ def load_library(self): self.__version__ = self.libwarpx_so.__version__ - def getNProcs(self): - ''' - - Get the number of processors - - ''' - return self.libwarpx_so.getNProcs() - - def getMyProc(self): - ''' - - Get the number of the processor - - ''' - return self.libwarpx_so.getMyProc() - - def get_nattr(self): - ''' - - Get the number of extra particle attributes. - - ''' - # --- The -3 is because the comps include the velocites - return self.libwarpx_so.warpx_nComps() - 3 - - def get_nattr_species(self, species_name): - ''' - Get the number of real attributes for the given species. - - Parameters - ---------- - - species_name: str - Name of the species - ''' - warpx = self.libwarpx_so.get_instance() - mpc = warpx.multi_particle_container() - pc = mpc.get_particle_container_from_name(species_name) - return pc.num_real_comps() - def amrex_init(self, argv, mpi_comm=None): if mpi_comm is None: # or MPI is None: self.libwarpx_so.amrex_init(argv) @@ -162,9 +121,7 @@ def amrex_init(self, argv, mpi_comm=None): def initialize(self, argv=None, mpi_comm=None): ''' - Initialize WarpX and AMReX. Must be called before doing anything else. - ''' if argv is None: argv = sys.argv @@ -180,9 +137,7 @@ def initialize(self, argv=None, mpi_comm=None): def finalize(self, finalize_mpi=1): ''' - Call finalize for WarpX and AMReX. Registered to run at program exit. - ''' # TODO: simplify, part of pyAMReX already if self.initialized: @@ -194,193 +149,4 @@ def finalize(self, finalize_mpi=1): from pywarpx import callbacks callbacks.clear_all() - def getistep(self, level=0): - ''' - Get the current time step number for the specified level - - Parameter - --------- - - level : int - The refinement level to reference - ''' - - return self.warpx.getistep(level) - - def gett_new(self, level=0): - ''' - - Get the next time for the specified level. - - Parameters - ---------- - - level : int - The refinement level to reference - ''' - - return self.warpx.gett_new(level) - - def evolve(self, num_steps=-1): - ''' - Evolve the simulation for num_steps steps. If num_steps=-1, - the simulation will be run until the end as specified in the - inputs file. - - Parameters - ---------- - - num_steps: int - The number of steps to take - ''' - - self.warpx.evolve(num_steps) - - def getProbLo(self, direction, level=0): - ''' - Get the values of the lower domain boundary. - - Parameters - ---------- - - direction : int - Direction of interest - ''' - - assert 0 <= direction < self.dim, 'Inappropriate direction specified' - return self.warpx.Geom(level).ProbLo(direction) - - def getProbHi(self, direction, level=0): - ''' - Get the values of the upper domain boundary. - - Parameters - ---------- - - direction : int - Direction of interest - ''' - - assert 0 <= direction < self.dim, 'Inappropriate direction specified' - return self.warpx.Geom(level).ProbHi(direction) - - def getCellSize(self, direction, level=0): - ''' - Get the cell size in the given direction and on the given level. - - Parameters - ---------- - - direction : int - Direction of interest - - level : int - The refinement level to reference - ''' - - assert 0 <= direction < 3, 'Inappropriate direction specified' - assert 0 <= level and level <= self.libwarpx_so.warpx_finestLevel(), 'Inappropriate level specified' - return self.libwarpx_so.warpx_getCellSize(direction, level) - - #def get_sigma(self, direction): - # ''' - # - # Return the 'sigma' PML coefficients for the electric field - # in a given direction. - # - # ''' - # - # size = ctypes.c_int(0) - # data = self.libwarpx_so.warpx_getPMLSigma(direction, ctypes.byref(size)) - # arr = np.ctypeslib.as_array(data, (size.value,)) - # arr.setflags(write=1) - # return arr - # - # - #def get_sigma_star(self, direction): - # ''' - # - # Return the 'sigma*' PML coefficients for the magnetic field - # in the given direction. - # - # ''' - # - # size = ctypes.c_int(0) - # data = self.libwarpx_so.warpx_getPMLSigmaStar(direction, ctypes.byref(size)) - # arr = np.ctypeslib.as_array(data, (size.value,)) - # arr.setflags(write=1) - # return arr - # - # - #def compute_pml_factors(self, lev, dt): - # ''' - # - # This recomputes the PML coefficients for a given level, using the - # time step dt. This needs to be called after modifying the coefficients - # from Python. - # - # ''' - # - # self.libwarpx_so.warpx_ComputePMLFactors(lev, dt) - - - - def depositChargeDensity(self, species_name, level, clear_rho=True, sync_rho=True): - ''' - Deposit the specified species' charge density in rho_fp in order to - access that data via pywarpx.fields.RhoFPWrapper(). - - Parameters - ---------- - - species_name : str - The species name that will be deposited. - - level : int - Which AMR level to retrieve scraped particle data from. - - clear_rho : bool - If True, zero out rho_fp before deposition. - - sync_rho : bool - If True, perform MPI exchange and properly set boundary cells for rho_fp. - ''' - rho_fp = self.warpx.multifab(f'rho_fp[level={level}]') - - if rho_fp is None: - raise RuntimeError("Multifab `rho_fp` is not allocated.") - # ablastr::warn_manager::WMRecordWarning( - # "WarpXWrappers", "rho_fp is not allocated", - # ablastr::warn_manager::WarnPriority::low - # ); - # return - - if clear_rho: - rho_fp.set_val(0.0) - - # deposit the charge density from the desired species - mypc = self.warpx.multi_particle_container() - myspc = mypc.get_particle_container_from_name(species_name) - myspc.deposit_charge(rho_fp, level) - - if self.geometry_dim == 'rz': - self.warpx.apply_inverse_volume_scaling_to_charge_density(rho_fp, level) - - if sync_rho: - self.warpx.sync_rho() - - - def set_potential_EB(self, potential): - """ - Set the expression string for the embedded boundary potential - - Parameters - ---------- - - potential : str - The expression string - """ - self.warpx.set_potential_on_eb(potential) - - libwarpx = LibWarpX() diff --git a/Python/pywarpx/particle_containers.py b/Python/pywarpx/particle_containers.py index c74549bcdb7..64c97499a01 100644 --- a/Python/pywarpx/particle_containers.py +++ b/Python/pywarpx/particle_containers.py @@ -29,6 +29,7 @@ def __init__(self, species_name): mypc = libwarpx.warpx.multi_particle_container() self.particle_container = mypc.get_particle_container_from_name(self.name) + def add_particles(self, x=None, y=None, z=None, ux=None, uy=None, uz=None, w=None, unique_particles=True, **kwargs): ''' @@ -149,6 +150,7 @@ def add_particles(self, x=None, y=None, z=None, ux=None, uy=None, nattr, attr, nattr_int, attr_int, unique_particles ) + def get_particle_count(self, local=False): ''' Get the number of particles of this species in the simulation. @@ -169,6 +171,7 @@ def get_particle_count(self, local=False): return self.particle_container.total_number_of_particles(True, local) nps = property(get_particle_count) + def add_real_comp(self, pid_name, comm=True): ''' Add a real component to the particle data array. @@ -184,6 +187,7 @@ def add_real_comp(self, pid_name, comm=True): ''' self.particle_container.add_real_comp(pid_name, comm) + def get_particle_structs(self, level, copy_to_host=False): ''' This returns a list of numpy or cupy arrays containing the particle struct data @@ -233,6 +237,7 @@ def get_particle_structs(self, level, copy_to_host=False): particle_data.append(aos_arr) return particle_data + def get_particle_arrays(self, comp_name, level, copy_to_host=False): ''' This returns a list of numpy or cupy arrays containing the particle array data @@ -278,6 +283,7 @@ def get_particle_arrays(self, comp_name, level, copy_to_host=False): return data_array + def get_particle_id(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle 'id' @@ -302,6 +308,7 @@ def get_particle_id(self, level=0, copy_to_host=False): structs = self.get_particle_structs(level, copy_to_host) return [libwarpx.amr.unpack_ids(struct['cpuid']) for struct in structs] + def get_particle_cpu(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle 'cpu' @@ -326,6 +333,7 @@ def get_particle_cpu(self, level=0, copy_to_host=False): structs = self.get_particle_structs(level, copy_to_host) return [libwarpx.amr.unpack_cpus(struct['cpuid']) for struct in structs] + def get_particle_x(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle 'x' @@ -363,6 +371,7 @@ def get_particle_x(self, level=0, copy_to_host=False): raise Exception('get_particle_x: There is no x coordinate with 1D Cartesian') xp = property(get_particle_x) + def get_particle_y(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle 'y' @@ -400,6 +409,7 @@ def get_particle_y(self, level=0, copy_to_host=False): raise Exception('get_particle_y: There is no y coordinate with 1D or 2D Cartesian') yp = property(get_particle_y) + def get_particle_r(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle 'r' @@ -432,6 +442,7 @@ def get_particle_r(self, level=0, copy_to_host=False): raise Exception('get_particle_r: There is no r coordinate with 1D or 2D Cartesian') rp = property(get_particle_r) + def get_particle_theta(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle @@ -464,6 +475,7 @@ def get_particle_theta(self, level=0, copy_to_host=False): raise Exception('get_particle_theta: There is no theta coordinate with 1D or 2D Cartesian') thetap = property(get_particle_theta) + def get_particle_z(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle 'z' @@ -494,6 +506,7 @@ def get_particle_z(self, level=0, copy_to_host=False): return [struct['x'] for struct in structs] zp = property(get_particle_z) + def get_particle_weight(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle @@ -518,6 +531,7 @@ def get_particle_weight(self, level=0, copy_to_host=False): return self.get_particle_arrays('w', level, copy_to_host=copy_to_host) wp = property(get_particle_weight) + def get_particle_ux(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle @@ -542,6 +556,7 @@ def get_particle_ux(self, level=0, copy_to_host=False): return self.get_particle_arrays('ux', level, copy_to_host=copy_to_host) uxp = property(get_particle_ux) + def get_particle_uy(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle @@ -566,6 +581,7 @@ def get_particle_uy(self, level=0, copy_to_host=False): return self.get_particle_arrays('uy', level, copy_to_host=copy_to_host) uyp = property(get_particle_uy) + def get_particle_uz(self, level=0, copy_to_host=False): ''' Return a list of numpy or cupy arrays containing the particle @@ -591,6 +607,7 @@ def get_particle_uz(self, level=0, copy_to_host=False): return self.get_particle_arrays('uz', level, copy_to_host=copy_to_host) uzp = property(get_particle_uz) + def get_species_charge_sum(self, local=False): ''' Returns the total charge in the simulation due to the given species. @@ -606,31 +623,70 @@ def get_species_charge_sum(self, local=False): ctypes.c_char_p(species_name.encode('utf-8')), local ) + def getex(self): raise NotImplementedError('Particle E fields not supported') ex = property(getex) + def getey(self): raise NotImplementedError('Particle E fields not supported') ey = property(getey) + def getez(self): raise NotImplementedError('Particle E fields not supported') ez = property(getez) + def getbx(self): raise NotImplementedError('Particle B fields not supported') bx = property(getbx) + def getby(self): raise NotImplementedError('Particle B fields not supported') by = property(getby) + def getbz(self): raise NotImplementedError('Particle B fields not supported') bz = property(getbz) + def deposit_charge_density(self, level, clear_rho=True, sync_rho=True): + ''' + Deposit this species' charge density in rho_fp in order to + access that data via pywarpx.fields.RhoFPWrapper(). + Parameters + ---------- + species_name : str + The species name that will be deposited. + level : int + Which AMR level to retrieve scraped particle data from. + clear_rho : bool + If True, zero out rho_fp before deposition. + sync_rho : bool + If True, perform MPI exchange and properly set boundary cells for rho_fp. + ''' + rho_fp = libwarpx.warpx.multifab(f'rho_fp[level={level}]') + + if rho_fp is None: + raise RuntimeError("Multifab `rho_fp` is not allocated.") + + if clear_rho: + rho_fp.set_val(0.0) + + # deposit the charge density from the desired species + self.particle_container.deposit_charge(rho_fp, level) + + if libwarpx.geometry_dim == 'rz': + libwarpx.warpx.apply_inverse_volume_scaling_to_charge_density(rho_fp, level) + + if sync_rho: + libwarpx.warpx.sync_rho() + + class ParticleBoundaryBufferWrapper(object): """Wrapper around particle boundary buffer containers. This provides a convenient way to query data in the particle boundary @@ -640,6 +696,7 @@ class ParticleBoundaryBufferWrapper(object): def __init__(self): self.particle_buffer = libwarpx.warpx.get_particle_boundary_buffer() + def get_particle_boundary_buffer_size(self, species_name, boundary, local=False): ''' This returns the number of particles that have been scraped so far in the simulation @@ -664,6 +721,7 @@ def get_particle_boundary_buffer_size(self, species_name, boundary, local=False) local=local ) + def get_particle_boundary_buffer_structs(self, species_name, boundary, level): ''' This returns a list of numpy or cupy arrays containing the particle struct data @@ -707,6 +765,7 @@ def get_particle_boundary_buffer_structs(self, species_name, boundary, level): _libc.free(data) return particle_data + def get_particle_boundary_buffer(self, species_name, boundary, comp_name, level): ''' This returns a list of numpy or cupy arrays containing the particle array data @@ -755,6 +814,7 @@ def get_particle_boundary_buffer(self, species_name, boundary, comp_name, level) return data_array + def clear_buffer(self): ''' @@ -763,6 +823,7 @@ def clear_buffer(self): ''' self.particle_buffer.clear_particles() + def _get_boundary_number(self, boundary): ''' diff --git a/Source/Python/CMakeLists.txt b/Source/Python/CMakeLists.txt index 4b4be199c31..d976e3d2c05 100644 --- a/Source/Python/CMakeLists.txt +++ b/Source/Python/CMakeLists.txt @@ -8,9 +8,6 @@ foreach(D IN LISTS WarpX_DIMS) PRIVATE # callback hooks WarpX_py.cpp - - # legacy C wrapper APIs - WarpXWrappers.cpp ) if(WarpX_PYTHON) target_sources(pyWarpX_${SD} diff --git a/Source/Python/Make.package b/Source/Python/Make.package index 9f421456b40..f107fbbfaaa 100644 --- a/Source/Python/Make.package +++ b/Source/Python/Make.package @@ -1,4 +1,3 @@ -CEXE_sources += WarpXWrappers.cpp CEXE_sources += WarpX_py.cpp VPATH_LOCATIONS += $(WARPX_HOME)/Source/Python diff --git a/Source/Python/WarpXWrappers.H b/Source/Python/WarpXWrappers.H deleted file mode 100644 index d2d57445880..00000000000 --- a/Source/Python/WarpXWrappers.H +++ /dev/null @@ -1,80 +0,0 @@ -/* Copyright 2019 Andrew Myers, David Grote, Maxence Thevenet - * Remi Lehe, Weiqun Zhang - * - * This file is part of WarpX. - * - * This file is a legacy file and will be removed soon. - * Please do NOT add new bindings here! Please see the other files - * in this directory for the new pybind11-based bindings! - * - * License: BSD-3-Clause-LBNL - */ -#ifndef WARPX_WRAPPERS_H_ -#define WARPX_WRAPPERS_H_ - -#include "Particles/WarpXParticleContainer.H" -#include "Evolve/WarpXDtType.H" -#include -#include - -#ifdef AMREX_USE_MPI -# include -#endif - -#ifdef __cplusplus -extern "C" { -#endif - - int warpx_Real_size(); - int warpx_ParticleReal_size(); - - int warpx_nSpecies(); - - bool warpx_use_fdtd_nci_corr(); - - int warpx_galerkin_interpolation(); - - void amrex_init_with_inited_mpi (int argc, char* argv[], MPI_Comm mpicomm); - - typedef void(*WARPX_CALLBACK_PY_FUNC_0)(); - - void warpx_ConvertLabParamsToBoost(); - - void warpx_ReadBCParams(); - - void warpx_CheckGriddingForRZSpectral(); - - amrex::Real warpx_getCellSize(int dir, int lev); - - amrex::Real warpx_sumParticleCharge(const char* char_species_name, bool local); - - void warpx_ComputeDt (); - void warpx_MoveWindow (int step, bool move_j); - - void warpx_EvolveE (amrex::Real dt); - void warpx_EvolveB (amrex::Real dt, DtType a_dt_type); - void warpx_FillBoundaryE (); - void warpx_FillBoundaryB (); - void warpx_SyncCurrent ( - const amrex::Vector,3>>& J_fp, - const amrex::Vector,3>>& J_cp, - const amrex::Vector,3>>& J_buffer); - void warpx_UpdateAuxilaryData (); - void warpx_PushParticlesandDepose (amrex::Real cur_time); - - void warpx_setistep (int lev, int ii); - void warpx_sett_new (int lev, amrex::Real time); - amrex::Real warpx_getdt (int lev); - - int warpx_maxStep (); - amrex::Real warpx_stopTime (); - - int warpx_finestLevel (); - - void mypc_Redistribute (); - -#ifdef __cplusplus -} -#endif - -#endif diff --git a/Source/Python/WarpXWrappers.cpp b/Source/Python/WarpXWrappers.cpp deleted file mode 100644 index 44820dd1f53..00000000000 --- a/Source/Python/WarpXWrappers.cpp +++ /dev/null @@ -1,172 +0,0 @@ -/* Copyright 2019 Andrew Myers, Axel Huebl, David Grote - * Luca Fedeli, Maxence Thevenet, Remi Lehe - * Weiqun Zhang - * - * This file is part of WarpX. - * - * License: BSD-3-Clause-LBNL - */ -#include "BoundaryConditions/PML.H" -#include "FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H" -#include "Initialization/WarpXAMReXInit.H" -#include "Particles/MultiParticleContainer.H" -#include "Particles/ParticleBoundaryBuffer.H" -#include "Particles/WarpXParticleContainer.H" -#include "Utils/WarpXProfilerWrapper.H" -#include "Utils/WarpXUtil.H" -#include "WarpX.H" -#include "WarpXWrappers.H" -#include "WarpX_py.H" - -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include - - int warpx_Real_size() - { - return (int)sizeof(amrex::Real); - } - - int warpx_ParticleReal_size() - { - return (int)sizeof(amrex::ParticleReal); - } - - int warpx_nSpecies() - { - const auto & mypc = WarpX::GetInstance().GetPartContainer(); - return mypc.nSpecies(); - } - - bool warpx_use_fdtd_nci_corr() - { - return WarpX::use_fdtd_nci_corr; - } - - int warpx_galerkin_interpolation() - { - return WarpX::galerkin_interpolation; - } - - void amrex_init_with_inited_mpi (int argc, char* argv[], MPI_Comm /* mpicomm */) - { - warpx::initialization::amrex_init(argc, argv, true); - } - - void warpx_ConvertLabParamsToBoost() - { - ConvertLabParamsToBoost(); - } - - void warpx_ReadBCParams() - { - ReadBCParams(); - } - - void warpx_CheckGriddingForRZSpectral() - { - CheckGriddingForRZSpectral(); - } - - amrex::Real warpx_getCellSize(int dir, int lev) { - const std::array& dx = WarpX::CellSize(lev); - return dx[dir]; - } - - amrex::Real warpx_sumParticleCharge(const char* char_species_name, const bool local) - { - auto & mypc = WarpX::GetInstance().GetPartContainer(); - const std::string species_name(char_species_name); - auto & myspc = mypc.GetParticleContainerFromName(species_name); - return myspc.sumParticleCharge(local); - } - - void warpx_ComputeDt () { - WarpX& warpx = WarpX::GetInstance(); - warpx.ComputeDt(); - } - void warpx_MoveWindow (int step,bool move_j) { - WarpX& warpx = WarpX::GetInstance(); - warpx.MoveWindow(step, move_j); - } - - void warpx_EvolveE (amrex::Real dt) { - WarpX& warpx = WarpX::GetInstance(); - warpx.EvolveE(dt); - } - void warpx_EvolveB (amrex::Real dt, DtType a_dt_type) { - WarpX& warpx = WarpX::GetInstance(); - warpx.EvolveB(dt, a_dt_type); - } - void warpx_FillBoundaryE () { - WarpX& warpx = WarpX::GetInstance(); - warpx.FillBoundaryE(warpx.getngEB()); - } - void warpx_FillBoundaryB () { - WarpX& warpx = WarpX::GetInstance(); - warpx.FillBoundaryB(warpx.getngEB()); - } - void warpx_SyncCurrent ( - const amrex::Vector,3>>& J_fp, - const amrex::Vector,3>>& J_cp, - const amrex::Vector,3>>& J_buffer) { - WarpX& warpx = WarpX::GetInstance(); - warpx.SyncCurrent(J_fp, J_cp, J_buffer); - } - void warpx_UpdateAuxilaryData () { - WarpX& warpx = WarpX::GetInstance(); - warpx.UpdateAuxilaryData(); - } - void warpx_PushParticlesandDepose (amrex::Real cur_time) { - WarpX& warpx = WarpX::GetInstance(); - warpx.PushParticlesandDepose(cur_time); - } - - void warpx_setistep (int lev, int ii) { - WarpX& warpx = WarpX::GetInstance(); - warpx.setistep(lev, ii); - } - void warpx_sett_new (int lev, amrex::Real time) { - WarpX& warpx = WarpX::GetInstance(); - warpx.sett_new(lev, time); - } - amrex::Real warpx_getdt (int lev) { - const WarpX& warpx = WarpX::GetInstance(); - return warpx.getdt(lev); - } - - int warpx_maxStep () { - const WarpX& warpx = WarpX::GetInstance(); - return warpx.maxStep(); - } - amrex::Real warpx_stopTime () { - const WarpX& warpx = WarpX::GetInstance(); - return warpx.stopTime(); - } - - int warpx_finestLevel () { - const WarpX& warpx = WarpX::GetInstance(); - return warpx.finestLevel(); - } - - void mypc_Redistribute () { - auto & mypc = WarpX::GetInstance().GetPartContainer(); - mypc.Redistribute(); - } From df9fbaa665eb7992f010f3339ae085550bb39a62 Mon Sep 17 00:00:00 2001 From: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Mon, 16 Oct 2023 13:26:04 -0700 Subject: [PATCH 055/110] Restore `get_particle_boundary_buffer_structs` functionality from before pyamrex transition (#4367) --- Python/pywarpx/particle_containers.py | 64 +++++++++++++++++---------- 1 file changed, 41 insertions(+), 23 deletions(-) diff --git a/Python/pywarpx/particle_containers.py b/Python/pywarpx/particle_containers.py index 64c97499a01..c77781f6544 100644 --- a/Python/pywarpx/particle_containers.py +++ b/Python/pywarpx/particle_containers.py @@ -722,15 +722,23 @@ def get_particle_boundary_buffer_size(self, species_name, boundary, local=False) ) - def get_particle_boundary_buffer_structs(self, species_name, boundary, level): + def get_particle_boundary_buffer_structs( + self, species_name, boundary, level, copy_to_host=False + ): ''' This returns a list of numpy or cupy arrays containing the particle struct data for a species that has been scraped by a specific simulation boundary. The particle data is represented as a structured array and contains the particle 'x', 'y', 'z', and 'idcpu'. - The data for the arrays are not copied, but share the underlying - memory buffer with WarpX. The arrays are fully writeable. + Unless copy_to_host is specified, the data for the structs are + not copied, but share the underlying memory buffer with WarpX. The + arrays are fully writeable. + + Note that cupy does not support structs: + https://github.com/cupy/cupy/issues/2031 + and will return arrays of binary blobs for the AoS (DP: "|V24"). If copied + to host via copy_to_host, we correct for the right numpy AoS type. Parameters ---------- @@ -743,26 +751,38 @@ def get_particle_boundary_buffer_structs(self, species_name, boundary, level): form x/y/z_hi/lo or eb. level : int - Which AMR level to retrieve scraped particle data from. - ''' + The refinement level to reference (default=0) + + copy_to_host : bool + For GPU-enabled runs, one can either return the GPU + arrays (the default) or force a device-to-host copy. + + Returns + ------- - particles_per_tile = _LP_c_int() - num_tiles = ctypes.c_int(0) - data = self.libwarpx_so.warpx_getParticleBoundaryBufferStructs( - ctypes.c_char_p(species_name.encode('utf-8')), - self._get_boundary_number(boundary), level, - ctypes.byref(num_tiles), ctypes.byref(particles_per_tile) + List of arrays + The requested particle struct data + ''' + particle_container = self.particle_buffer.get_particle_container( + species_name, self._get_boundary_number(boundary) ) particle_data = [] - for i in range(num_tiles.value): - if particles_per_tile[i] == 0: - continue - arr = self._array1d_from_pointer(data[i], self._p_dtype, particles_per_tile[i]) - particle_data.append(arr) - - _libc.free(particles_per_tile) - _libc.free(data) + for pti in libwarpx.libwarpx_so.BoundaryBufferParIter(particle_container, level): + if copy_to_host: + particle_data.append(pti.aos().to_numpy(copy=True)) + else: + if libwarpx.amr.Config.have_gpu: + libwarpx.amr.Print( + "get_particle_structs: cupy does not yet support structs. " + "https://github.com/cupy/cupy/issues/2031" + "Did you mean copy_to_host=True?" + ) + xp, cupy_status = load_cupy() + if cupy_status is not None: + libwarpx.amr.Print(cupy_status) + aos_arr = xp.array(pti.aos(), copy=False) # void blobs for cupy + particle_data.append(aos_arr) return particle_data @@ -805,13 +825,11 @@ def get_particle_boundary_buffer(self, species_name, boundary, comp_name, level) soa = pti.soa() data_array.append(xp.array(soa.GetIntData(comp_idx), copy=False)) else: - mypc = libwarpx.warpx.multi_particle_container() - sim_part_container_wrapper = mypc.get_particle_container_from_name(species_name) - comp_idx = sim_part_container_wrapper.get_comp_index(comp_name) + container_wrapper = ParticleContainerWrapper(species_name) + comp_idx = container_wrapper.particle_container.get_comp_index(comp_name) for ii, pti in enumerate(libwarpx.libwarpx_so.BoundaryBufferParIter(part_container, level)): soa = pti.soa() data_array.append(xp.array(soa.GetRealData(comp_idx), copy=False)) - return data_array From 363d00e63db46302f7f0eb9f3045968267c4e8a4 Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Mon, 16 Oct 2023 23:10:45 +0200 Subject: [PATCH 056/110] Fix bug in clang-tidy configuration file (#4321) * fix issue in clang-tidy * address most of the issues found with clang-tidy * address issues found with clang-tidy * address issues found with clang-tidy * address issues found with clang-tidy * fix issues found with clang-tidy * fix issues found with clang-tidy * refactor variable name * fix residual issues found with clang-tidy * revert jn -> cyl_bessel change * cleaning * fix bug * fix issues found by clang-tidy in new code * address issues found by clang-tidy in new code * fixed issues * address residual issues found with clang-tidy --- .clang-tidy | 14 +- Source/BoundaryConditions/PML.cpp | 6 +- Source/Diagnostics/BTDiagnostics.cpp | 31 ++-- .../BoundaryScrapingDiagnostics.cpp | 2 +- .../BackTransformParticleFunctor.cpp | 2 +- .../ParticleReductionFunctor.cpp | 2 +- .../FlushFormats/FlushFormatOpenPMD.cpp | 5 +- .../FlushFormats/FlushFormatPlotfile.cpp | 2 +- Source/Diagnostics/FullDiagnostics.cpp | 7 +- Source/Diagnostics/ParticleIO.cpp | 2 +- .../Diagnostics/ReducedDiags/BeamRelevant.cpp | 4 +- .../ReducedDiags/ColliderRelevant.cpp | 4 +- .../Diagnostics/ReducedDiags/FieldProbe.cpp | 2 +- .../FieldProbeParticleContainer.cpp | 2 +- .../ReducedDiags/LoadBalanceCosts.cpp | 12 +- .../ReducedDiags/ParticleHistogram.cpp | 4 +- .../Diagnostics/ReducedDiags/RhoMaximum.cpp | 2 +- Source/Diagnostics/WarpXOpenPMD.cpp | 18 +-- Source/Evolve/WarpXEvolve.cpp | 4 +- .../ApplySilverMuellerBoundary.cpp | 2 +- .../FiniteDifferenceSolver/ComputeDivE.cpp | 10 +- .../FiniteDifferenceSolver/EvolveB.cpp | 24 +-- .../FiniteDifferenceSolver/EvolveBPML.cpp | 6 +- .../FiniteDifferenceSolver/EvolveE.cpp | 20 +-- .../FiniteDifferenceSolver/EvolveECTRho.cpp | 4 +- .../FiniteDifferenceSolver/EvolveEPML.cpp | 6 +- .../FiniteDifferenceSolver/EvolveF.cpp | 10 +- .../FiniteDifferenceSolver/EvolveFPML.cpp | 6 +- .../FiniteDifferenceSolver/EvolveG.cpp | 6 +- .../HybridPICModel/HybridPICModel.H | 2 +- .../HybridPICSolveE.cpp | 24 +-- .../MacroscopicEvolveE.cpp | 6 +- .../SpectralSolver/SpectralBinomialFilter.cpp | 4 +- .../SpectralSolver/SpectralFieldData.cpp | 12 +- .../SpectralSolver/SpectralFieldDataRZ.cpp | 32 ++-- .../SpectralHankelTransform/BesselRoots.cpp | 16 +- .../HankelTransform.cpp | 7 +- .../SpectralSolver/SpectralKSpace.cpp | 10 +- Source/FieldSolver/WarpXPushFieldsEM.cpp | 18 +-- .../FieldSolver/WarpXPushFieldsHybridPIC.cpp | 4 +- Source/FieldSolver/WarpX_FDTD.H | 7 +- Source/Filter/BilinearFilter.cpp | 4 +- Source/Filter/Filter.cpp | 8 +- Source/Fluids/MultiFluidContainer.H | 2 +- Source/Fluids/MusclHancockUtils.H | 149 +++++++++++------- Source/Fluids/WarpXFluidContainer.cpp | 88 +++++------ Source/Initialization/InjectorMomentum.H | 8 +- Source/Initialization/PlasmaInjector.cpp | 4 +- Source/Initialization/WarpXInitData.cpp | 38 ++--- .../LaserProfileFromFile.cpp | 76 +++++---- Source/Parallelization/GuardCellManager.cpp | 2 +- .../BackgroundMCC/BackgroundMCCCollision.cpp | 28 ++-- .../BackgroundMCC/ImpactIonization.H | 6 +- .../Collision/BackgroundMCC/MCCProcess.cpp | 2 +- .../BackgroundStopping/BackgroundStopping.cpp | 4 +- .../BinaryCollision/BinaryCollision.H | 12 +- .../Coulomb/ElasticCollisionPerez.H | 24 +-- .../NuclearFusion/NuclearFusionFunc.H | 31 ++-- .../ProtonBoronFusionInitializeMomentum.H | 42 ++--- .../BinaryCollision/ParticleCreationFunc.H | 10 +- .../BreitWheelerEngineWrapper.cpp | 4 +- .../QEDInternals/QuantumSyncEngineWrapper.cpp | 4 +- Source/Particles/Gather/GetExternalFields.H | 2 +- Source/Particles/Gather/GetExternalFields.cpp | 4 +- Source/Particles/LaserParticleContainer.cpp | 18 ++- Source/Particles/MultiParticleContainer.H | 10 +- Source/Particles/MultiParticleContainer.cpp | 28 ++-- Source/Particles/ParticleBoundaryBuffer.H | 2 +- Source/Particles/ParticleBoundaryBuffer.cpp | 12 +- .../Particles/ParticleCreation/SmartUtils.H | 2 +- .../Particles/PhysicalParticleContainer.cpp | 59 ++++--- Source/Particles/Pusher/CopyParticleAttribs.H | 4 +- Source/Particles/Pusher/GetAndSetPosition.H | 12 +- .../Particles/Resampling/LevelingThinning.cpp | 4 +- Source/Particles/WarpXParticleContainer.H | 2 +- Source/Particles/WarpXParticleContainer.cpp | 24 +-- Source/Utils/Parser/IntervalsParser.cpp | 4 +- Source/Utils/Parser/ParserUtils.H | 30 ++-- Source/Utils/ParticleUtils.H | 4 +- Source/Utils/WarpXMovingWindow.cpp | 7 +- Source/WarpX.cpp | 32 ++-- Source/ablastr/fields/PoissonSolver.H | 12 +- Source/ablastr/fields/VectorPoissonSolver.H | 2 +- Source/ablastr/utils/SignalHandling.cpp | 2 +- Source/ablastr/utils/timer/Timer.H | 10 +- Source/ablastr/utils/timer/Timer.cpp | 4 +- 86 files changed, 633 insertions(+), 564 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index f40dd987add..2d45d750ddc 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -1,10 +1,11 @@ -Checks: '-*, - bugprone-* +Checks: ' + -*, + bugprone-*, -bugprone-easily-swappable-parameters, -bugprone-implicit-widening-of-multiplication-result, -bugprone-misplaced-widening-cast, -bugprone-unchecked-optional-access, - cert-* + cert-*, -cert-err58-cpp, cppcoreguidelines-avoid-goto, cppcoreguidelines-interfaces-global-init, @@ -74,9 +75,12 @@ Checks: '-*, ' CheckOptions: -- key: modernize-pass-by-value.ValuesOnly - value: 'true' +- key: bugprone-narrowing-conversions.WarnOnIntegerToFloatingPointNarrowingConversion + value: "false" - key: misc-definitions-in-headers.HeaderFileExtensions value: "H," +- key: modernize-pass-by-value.ValuesOnly + value: "true" + HeaderFilterRegex: 'Source[a-z_A-Z0-9\/]+\.H$' diff --git a/Source/BoundaryConditions/PML.cpp b/Source/BoundaryConditions/PML.cpp index 811d1aa469d..98f2b9fa546 100644 --- a/Source/BoundaryConditions/PML.cpp +++ b/Source/BoundaryConditions/PML.cpp @@ -877,7 +877,8 @@ PML::MakeBoxArray_single (const amrex::Box& regular_domain, const amrex::BoxArra const amrex::IntVect& do_pml_Hi) { BoxList bl; - for (int i = 0, N = grid_ba.size(); i < N; ++i) { + const auto grid_ba_size = static_cast(grid_ba.size()); + for (int i = 0; i < grid_ba_size; ++i) { Box const& b = grid_ba[i]; for (OrientationIter oit; oit.isValid(); ++oit) { // In 3d, a Box has 6 faces. This iterates over the 6 faces. @@ -926,7 +927,8 @@ PML::MakeBoxArray_multiple (const amrex::Geometry& geom, const amrex::BoxArray& } } BoxList bl; - for (int i = 0, N = grid_ba.size(); i < N; ++i) + const auto grid_ba_size = static_cast(grid_ba.size()); + for (int i = 0; i < grid_ba_size; ++i) { const Box& grid_bx = grid_ba[i]; const IntVect& grid_bx_sz = grid_bx.size(); diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp index 34bd720be96..6f976d9ee11 100644 --- a/Source/Diagnostics/BTDiagnostics.cpp +++ b/Source/Diagnostics/BTDiagnostics.cpp @@ -394,7 +394,7 @@ BTDiagnostics::InitializeBufferData ( int i_buffer , int lev, bool restart) diag_dom.setLo( idim, warpx.Geom(lev).ProbLo(idim) + diag_ba.getCellCenteredBox(0).smallEnd(idim) * warpx.Geom(lev).CellSize(idim)); diag_dom.setHi( idim, warpx.Geom(lev).ProbLo(idim) + - (diag_ba.getCellCenteredBox( diag_ba.size()-1 ).bigEnd(idim) + 1) * warpx.Geom(lev).CellSize(idim)); + (diag_ba.getCellCenteredBox( static_cast(diag_ba.size()-1) ).bigEnd(idim) + 1) * warpx.Geom(lev).CellSize(idim)); } // Define buffer_domain in lab-frame for the i^th snapshot. @@ -561,7 +561,9 @@ BTDiagnostics::InitializeFieldFunctors (int lev) // Fill vector of cell-center functors for all field-components, namely, // Ex, Ey, Ez, Bx, By, Bz, jx, jy, jz, and rho are included in the // cell-center functors for BackTransform Diags - for (int comp=0, n=m_cell_center_functors.at(lev).size(); comp( + m_cell_center_functors.at(lev).size()); + for (int comp=0; comp(warpx.get_pointer_Efield_aux(lev, 0), lev, m_crse_ratio); } else if ( m_cellcenter_varnames[comp] == "Ey" ){ @@ -600,13 +602,14 @@ BTDiagnostics::UpdateVarnamesForRZopenPMD () const bool update_varnames = true; if (update_varnames) { - const int n_rz = ncomp * m_varnames_fields.size(); + const auto n_rz = ncomp * static_cast(m_varnames_fields.size()); m_varnames.clear(); m_varnames.reserve(n_rz); } // AddRZ modes to output names for the back-transformed data if (update_varnames) { - for (int comp=0, n=m_varnames_fields.size(); comp(m_varnames_fields.size()); + for (int comp=0; comp(m_cellcenter_varnames.size()); m_cellcenter_varnames.clear(); m_cellcenter_varnames.reserve(n_rz); - for (int comp=0, n=m_cellcenter_varnames_fields.size(); comp(m_cellcenter_varnames_fields.size()); + for (int comp=0; comp(m_cell_center_functors.at(lev).size()); + for (int comp=0; comp(warpx.get_pointer_Efield_aux(lev, 0), lev, m_crse_ratio, false, ncomp); } else if ( m_cellcenter_varnames_fields[comp] == "Et" ){ @@ -786,7 +791,8 @@ BTDiagnostics::PrepareFieldDataForOutput () // Call m_cell_center_functors->operator for (int lev = 0; lev < nmax_lev; ++lev) { int icomp_dst = 0; - for (int icomp = 0, n=m_cell_center_functors.at(lev).size(); icomp(m_cell_center_functors.at(lev).size()); + for (int icomp = 0; icomp 0; --lev) { ablastr::coarsen::sample::Coarsen(*m_cell_centered_data[lev - 1], *m_cell_centered_data[lev], 0, 0, - m_cellcenter_varnames.size(), 0, WarpX::RefRatio(lev-1) ); + static_cast(m_cellcenter_varnames.size()), 0, WarpX::RefRatio(lev-1) ); } const int num_BT_functors = 1; @@ -901,7 +907,7 @@ BTDiagnostics::DefineFieldBufferMultiFab (const int i_buffer, const int lev) // Unlike FullDiagnostics, "m_format == sensei" option is not included here. const int ngrow = 0; m_mf_output[i_buffer][lev] = amrex::MultiFab( buffer_ba, buffer_dmap, - m_varnames.size(), ngrow ); + static_cast(m_varnames.size()), ngrow ); m_mf_output[i_buffer][lev].setVal(0.); amrex::IntVect ref_ratio = amrex::IntVect(1); @@ -915,7 +921,7 @@ BTDiagnostics::DefineFieldBufferMultiFab (const int i_buffer, const int lev) - m_snapshot_box[i_buffer].smallEnd(idim) ) * cellsize; const amrex::Real buffer_hi = m_snapshot_domain_lab[i_buffer].lo(idim) - + ( buffer_ba.getCellCenteredBox( buffer_ba.size()-1 ).bigEnd(idim) + + ( buffer_ba.getCellCenteredBox( static_cast(buffer_ba.size()-1) ).bigEnd(idim) - m_snapshot_box[i_buffer].smallEnd(idim) + 1 ) * cellsize; m_buffer_domain_lab[i_buffer].setLo(idim, buffer_lo); @@ -1482,7 +1488,8 @@ void BTDiagnostics::UpdateTotalParticlesFlushed(int i_buffer) { for (int isp = 0; isp < m_totalParticles_flushed_already[i_buffer].size(); ++isp) { - m_totalParticles_flushed_already[i_buffer][isp] += m_particles_buffer[i_buffer][isp]->TotalNumberOfParticles(); + m_totalParticles_flushed_already[i_buffer][isp] += static_cast( + m_particles_buffer[i_buffer][isp]->TotalNumberOfParticles()); } } diff --git a/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp b/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp index a9b75481bb7..a3ea097352c 100644 --- a/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp +++ b/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp @@ -105,7 +105,7 @@ BoundaryScrapingDiagnostics::InitializeParticleBuffer () // Initialize total number of particles flushed m_totalParticles_flushed_already.resize(m_num_buffers); for (int i_buffer = 0; i_buffer < m_num_buffers; ++i_buffer) { - int const n_species = m_output_species_names.size(); + int const n_species = static_cast(m_output_species_names.size()); m_totalParticles_flushed_already[i_buffer].resize(n_species); for (int i_species=0; i_species(pc_dst.TotalNumberOfParticles()); } diff --git a/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp b/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp index 203872b1372..60b62181c4b 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp +++ b/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.cpp @@ -135,7 +135,7 @@ ParticleReductionFunctor::operator() (amrex::MultiFab& mf_dst, const int dcomp, const amrex::ParticleReal uy = p.rdata(PIdx::uy) / PhysConst::c; const amrex::ParticleReal uz = p.rdata(PIdx::uz) / PhysConst::c; amrex::Real filter; - if ((do_filter) && (!filter_fn(xw, yw, zw, ux, uy, uz))) filter = 0._rt; + if ((do_filter) && (filter_fn(xw, yw, zw, ux, uy, uz) == 0._rt)) filter = 0._rt; else filter = 1._rt; amrex::Gpu::Atomic::AddNoRet(&out_array(ii, jj, kk, 0), (amrex::Real)(p.rdata(PIdx::w) * filter)); }); diff --git a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp index 3832bf5e9c0..2185d4b4cca 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp @@ -155,10 +155,11 @@ FlushFormatOpenPMD::WriteToFile ( // fields: only dumped for coarse level m_OpenPMDPlotWriter->WriteOpenPMDFieldsAll( - varnames, mf, geom, output_levels, output_iteration, time, isBTD, full_BTD_snapshot); + varnames, mf, geom, output_levels, output_iteration, static_cast(time), isBTD, full_BTD_snapshot); // particles: all (reside only on locally finest level) - m_OpenPMDPlotWriter->WriteOpenPMDParticles(particle_diags, time, use_pinned_pc, isBTD, isLastBTDFlush, totalParticlesFlushedAlready); + m_OpenPMDPlotWriter->WriteOpenPMDParticles( + particle_diags, static_cast(time), use_pinned_pc, isBTD, isLastBTDFlush, totalParticlesFlushedAlready); // signal that no further updates will be written to this iteration m_OpenPMDPlotWriter->CloseStep(isBTD, isLastBTDFlush); diff --git a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp index db5dea10adc..a18bab25308 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp @@ -99,7 +99,7 @@ FlushFormatPlotfile::WriteToFile ( WriteAllRawFields(plot_raw_fields, nlev, filename, plot_raw_fields_guards); - WriteParticles(filename, particle_diags, time, isBTD); + WriteParticles(filename, particle_diags, static_cast(time), isBTD); WriteJobInfo(filename); diff --git a/Source/Diagnostics/FullDiagnostics.cpp b/Source/Diagnostics/FullDiagnostics.cpp index d5109277de2..6848f9e1a18 100644 --- a/Source/Diagnostics/FullDiagnostics.cpp +++ b/Source/Diagnostics/FullDiagnostics.cpp @@ -191,7 +191,7 @@ FullDiagnostics::InitializeFieldFunctorsRZopenPMD (int lev) const bool update_varnames = (lev==0); if (update_varnames) { m_varnames.clear(); - const int n_rz = ncomp * m_varnames.size(); + const auto n_rz = ncomp * static_cast(m_varnames.size()); m_varnames.reserve(n_rz); } @@ -209,7 +209,8 @@ FullDiagnostics::InitializeFieldFunctorsRZopenPMD (int lev) bool deposit_current = !m_solver_deposits_current; // Fill vector of functors for all components except individual cylindrical modes. - for (int comp=0, n=m_varnames_fields.size(); comp(m_varnames_fields.size()); + for (int comp=0; comp(warpx.get_pointer_Efield_aux(lev, 0), lev, m_crse_ratio, false, ncomp); @@ -392,7 +393,7 @@ FullDiagnostics::AddRZModesToDiags (int lev) bool deposit_current = !m_solver_deposits_current; // First index of m_all_field_functors[lev] where RZ modes are stored - int icomp = m_all_field_functors[0].size(); + auto icomp =static_cast(m_all_field_functors[0].size()); const std::array coord {"r", "theta", "z"}; // Er, Etheta, Ez, Br, Btheta, Bz, jr, jtheta, jz diff --git a/Source/Diagnostics/ParticleIO.cpp b/Source/Diagnostics/ParticleIO.cpp index c10e3468a71..7ca5e6541d7 100644 --- a/Source/Diagnostics/ParticleIO.cpp +++ b/Source/Diagnostics/ParticleIO.cpp @@ -95,7 +95,7 @@ RigidInjectedParticleContainer::WriteHeader (std::ostream& os) const PhysicalParticleContainer::WriteHeader( os ); // Write quantities that are specific to the rigid-injected species - const int nlevs = zinject_plane_levels.size(); + const auto nlevs = static_cast(zinject_plane_levels.size()); os << nlevs << "\n"; for (int i = 0; i < nlevs; ++i) { diff --git a/Source/Diagnostics/ReducedDiags/BeamRelevant.cpp b/Source/Diagnostics/ReducedDiags/BeamRelevant.cpp index ac5f3fd6632..2a29748b3e1 100644 --- a/Source/Diagnostics/ReducedDiags/BeamRelevant.cpp +++ b/Source/Diagnostics/ReducedDiags/BeamRelevant.cpp @@ -250,7 +250,7 @@ void BeamRelevant::ComputeDiags (int step) // reduced sum over mpi ranks (allreduce) amrex::ParallelAllReduce::Sum - ( values_per_rank_1st.data(), values_per_rank_1st.size(), ParallelDescriptor::Communicator()); + ( values_per_rank_1st.data(), static_cast(values_per_rank_1st.size()), ParallelDescriptor::Communicator()); const ParticleReal w_sum = values_per_rank_1st.at(0); const ParticleReal x_mean = values_per_rank_1st.at(1) /= w_sum; @@ -345,7 +345,7 @@ void BeamRelevant::ComputeDiags (int step) // reduced sum over mpi ranks (reduce to IO rank) ParallelDescriptor::ReduceRealSum - ( values_per_rank_2nd.data(), values_per_rank_2nd.size(), ParallelDescriptor::IOProcessorNumber()); + ( values_per_rank_2nd.data(), static_cast(values_per_rank_2nd.size()), ParallelDescriptor::IOProcessorNumber()); const ParticleReal x_ms = values_per_rank_2nd.at(0) /= w_sum; const ParticleReal y_ms = values_per_rank_2nd.at(1) /= w_sum; diff --git a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp index 1e2c16ac737..188b5f3f76f 100644 --- a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp +++ b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp @@ -222,7 +222,7 @@ void ColliderRelevant::ComputeDiags (int step) using PType = typename WarpXParticleContainer::SuperParticleType; num_dens[i_s] = myspc.GetChargeDensity(0); - num_dens[i_s]->mult(1./q); + num_dens[i_s]->mult(1._prt/q); #if defined(WARPX_DIM_1D_Z) // w_tot @@ -542,7 +542,7 @@ void ColliderRelevant::ComputeDiags (int step) // compute luminosity amrex::Real const n1_dot_n2 = amrex::MultiFab::Dot(mf_dst1, 0, mf_dst2, 0, 1, 0); - amrex::Real const lumi = 2. * PhysConst::c * n1_dot_n2 * dV; + amrex::Real const lumi = 2._rt * PhysConst::c * n1_dot_n2 * dV; m_data[get_idx("dL_dt")] = lumi; #endif // not RZ } diff --git a/Source/Diagnostics/ReducedDiags/FieldProbe.cpp b/Source/Diagnostics/ReducedDiags/FieldProbe.cpp index 7749d5ecc8d..11975ef0483 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbe.cpp +++ b/Source/Diagnostics/ReducedDiags/FieldProbe.cpp @@ -591,7 +591,7 @@ void FieldProbe::ComputeDiags (int step) if (amrex::ParallelDescriptor::IOProcessor()) { length_vector.resize(mpisize, 0); } - localsize.resize(1, m_data.size()); + localsize.resize(1, static_cast(m_data.size())); // gather size of m_data from each processor amrex::ParallelDescriptor::Gather(localsize.data(), 1, diff --git a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp index d928bd33fb9..5d990b9eb07 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp +++ b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp @@ -76,7 +76,7 @@ FieldProbeParticleContainer::AddNParticles (int lev, AMREX_ALWAYS_ASSERT(x.size() == z.size()); // number of particles to add - int const np = x.size(); + auto const np = static_cast(x.size()); // have to resize here, not in the constructor because grids have not // been built when constructor was called. diff --git a/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.cpp b/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.cpp index 1ce88d043ff..7843527cbf5 100644 --- a/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.cpp +++ b/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.cpp @@ -144,7 +144,7 @@ void LoadBalanceCosts::ComputeDiags (int step) #else m_data[shift_m_data + mfi.index()*m_nDataFields + 5] = 0.; #endif - m_data[shift_m_data + mfi.index()*m_nDataFields + 6] = tbx.d_numPts(); // note: difference to volume + m_data[shift_m_data + mfi.index()*m_nDataFields + 6] = static_cast(tbx.d_numPts()); // note: difference to volume m_data[shift_m_data + mfi.index()*m_nDataFields + 7] = countBoxMacroParticles(mfi, lev); #ifdef AMREX_USE_GPU m_data[shift_m_data + mfi.index()*m_nDataFields + 8] = amrex::Gpu::Device::deviceId(); @@ -158,7 +158,7 @@ void LoadBalanceCosts::ComputeDiags (int step) // parallel reduce to IO proc and get data over all procs ParallelDescriptor::ReduceRealSum(m_data.data(), - m_data.size(), + static_cast(m_data.size()), ParallelDescriptor::IOProcessorNumber()); #ifdef AMREX_USE_MPI @@ -367,7 +367,11 @@ void LoadBalanceCosts::WriteToFile (int step) const ofstmp.close(); // remove the original, rename tmp file - std::remove(fileDataName.c_str()); - std::rename(fileTmpName.c_str(), fileDataName.c_str()); + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + std::remove(fileDataName.c_str()) == EXIT_SUCCESS, + "Failed to remove " + fileDataName); + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + std::rename(fileTmpName.c_str(), fileDataName.c_str()) == EXIT_SUCCESS, + "Failed to rename " + fileTmpName + " into " + fileDataName); } } diff --git a/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp b/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp index 242659236d6..1cbbbaaebff 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp +++ b/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp @@ -219,7 +219,7 @@ void ParticleHistogram::ComputeDiags (int step) // don't count a particle if it is filtered out if (do_parser_filter) - if (!fun_filterparser(t, x, y, z, ux, uy, uz)) + if (fun_filterparser(t, x, y, z, ux, uy, uz) == 0._rt) return; // continue function if particle is not filtered out auto const f = fun_partparser(t, x, y, z, ux, uy, uz); @@ -247,7 +247,7 @@ void ParticleHistogram::ComputeDiags (int step) // reduced sum over mpi ranks ParallelDescriptor::ReduceRealSum - (m_data.data(), m_data.size(), ParallelDescriptor::IOProcessorNumber()); + (m_data.data(), static_cast(m_data.size()), ParallelDescriptor::IOProcessorNumber()); // normalize the maximum value to be one if ( m_norm == NormalizationType::max_to_unity ) diff --git a/Source/Diagnostics/ReducedDiags/RhoMaximum.cpp b/Source/Diagnostics/ReducedDiags/RhoMaximum.cpp index 79698682b97..787a95ce412 100644 --- a/Source/Diagnostics/ReducedDiags/RhoMaximum.cpp +++ b/Source/Diagnostics/ReducedDiags/RhoMaximum.cpp @@ -133,7 +133,7 @@ void RhoMaximum::ComputeDiags (int step) // get number of levels const auto nLevel = warpx.finestLevel() + 1; - const int n_charged_species = m_rho_functors[0].size() - 1; + const auto n_charged_species = static_cast(m_rho_functors[0].size() - 1); // Min and max of total rho + max of |rho| for each species const int noutputs_per_level = 2+n_charged_species; diff --git a/Source/Diagnostics/WarpXOpenPMD.cpp b/Source/Diagnostics/WarpXOpenPMD.cpp index 69163b9c529..458760d3664 100644 --- a/Source/Diagnostics/WarpXOpenPMD.cpp +++ b/Source/Diagnostics/WarpXOpenPMD.cpp @@ -69,7 +69,7 @@ namespace detail snakeToCamel (const std::string& snake_string) { std::string camelString = snake_string; - const int n = camelString.length(); + const auto n = static_cast(camelString.length()); for (int x = 0; x < n; x++) { if (x == 0) @@ -217,10 +217,8 @@ namespace detail getParticlePositionComponentLabels (bool ignore_dims=false) { using vs = std::vector< std::string >; - vs positionComponents; - if (ignore_dims) { - positionComponents = vs{"x", "y", "z"}; - } else { + auto positionComponents = vs{"x", "y", "z"}; + if (!ignore_dims) { #if defined(WARPX_DIM_1D_Z) positionComponents = vs{"z"}; #elif defined(WARPX_DIM_XZ) @@ -783,7 +781,7 @@ WarpXOpenPMDPlot::DumpToFile (ParticleContainer* pc, [](uint64_t const *p) { delete[] p; } ); for (auto i = 0; i < numParticleOnTile; i++) { - ids.get()[i] = ablastr::particles::localIDtoGlobal(aos[i].id(), aos[i].cpu()); + ids.get()[i] = ablastr::particles::localIDtoGlobal(static_cast(aos[i].id()), static_cast(aos[i].cpu())); } auto const scalar = openPMD::RecordComponent::SCALAR; currSpecies["id"][scalar].storeChunk(ids, {offset}, {numParticleOnTile64}); @@ -1148,11 +1146,13 @@ WarpXOpenPMDPlot::SetupFields ( openPMD::Container< openPMD::Mesh >& meshes, fieldBoundary.resize(AMREX_SPACEDIM * 2); particleBoundary.resize(AMREX_SPACEDIM * 2); - for (auto i = 0u; i < fieldBoundary.size() / 2u; ++i) + const auto HalfFieldBoundarySize = static_cast(fieldBoundary.size() / 2u); + + for (auto i = 0; i < HalfFieldBoundarySize; ++i) if (m_fieldPMLdirections.at(i)) fieldBoundary.at(i) = "open"; - for (auto i = 0u; i < fieldBoundary.size() / 2u; ++i) + for (int i = 0; i < HalfFieldBoundarySize; ++i) if (period.isPeriodic(i)) { fieldBoundary.at(2u * i) = "periodic"; fieldBoundary.at(2u * i + 1u) = "periodic"; @@ -1540,7 +1540,7 @@ WarpXParticleCounter::GetParticleOffsetOfProcessor ( amrex::ParallelGather::Gather (numParticles, result.data(), -1, amrex::ParallelDescriptor::Communicator()); sum = 0; - int const num_results = result.size(); + auto const num_results = static_cast(result.size()); for (int i=0; i(amrex::second()); //Check and clear signal flags and asynchronously broadcast them from process 0 SignalHandling::CheckSignals(); @@ -378,7 +378,7 @@ WarpX::Evolve (int numsteps) } // create ending time stamp for calculating elapsed time each iteration - const Real evolve_time_end_step = amrex::second(); + const auto evolve_time_end_step = static_cast(amrex::second()); evolve_time += evolve_time_end_step - evolve_time_beg_step; HandleSignals(); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/ApplySilverMuellerBoundary.cpp b/Source/FieldSolver/FiniteDifferenceSolver/ApplySilverMuellerBoundary.cpp index 4c6748cfbce..b447d4e06a3 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/ApplySilverMuellerBoundary.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/ApplySilverMuellerBoundary.cpp @@ -64,7 +64,7 @@ void FiniteDifferenceSolver::ApplySilverMuellerBoundary ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_h_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_h_stencil_coefs_z.size()); // Extract cylindrical specific parameters Real const dr = m_dr; diff --git a/Source/FieldSolver/FiniteDifferenceSolver/ComputeDivE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/ComputeDivE.cpp index 163792a2737..62ba35faea8 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/ComputeDivE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/ComputeDivE.cpp @@ -94,11 +94,11 @@ void FiniteDifferenceSolver::ComputeDivECartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tdive = mfi.tilebox(divEfield.ixType().toIntVect()); @@ -140,9 +140,9 @@ void FiniteDifferenceSolver::ComputeDivECylindrical ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_r = m_stencil_coefs_r.dataPtr(); - int const n_coefs_r = m_stencil_coefs_r.size(); + auto const n_coefs_r = static_cast(m_stencil_coefs_r.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract cylindrical specific parameters Real const dr = m_dr; diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp index 0cbad684c68..0ccff8bac9a 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp @@ -120,7 +120,7 @@ void FiniteDifferenceSolver::EvolveBCartesian ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile Array4 const& Bx = Bfield[0]->array(mfi); @@ -132,11 +132,11 @@ void FiniteDifferenceSolver::EvolveBCartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tbx = mfi.tilebox(Bfield[0]->ixType().toIntVect()); @@ -195,7 +195,7 @@ void FiniteDifferenceSolver::EvolveBCartesian ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -233,7 +233,7 @@ void FiniteDifferenceSolver::EvolveBCartesianECT ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); for (int idim = 0; idim < AMREX_SPACEDIM; ++idim) { // Extract field data for this grid/tile @@ -356,7 +356,7 @@ void FiniteDifferenceSolver::EvolveBCartesianECT ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -385,7 +385,7 @@ void FiniteDifferenceSolver::EvolveBCylindrical ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile Array4 const& Br = Bfield[0]->array(mfi); @@ -397,9 +397,9 @@ void FiniteDifferenceSolver::EvolveBCylindrical ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_r = m_stencil_coefs_r.dataPtr(); - int const n_coefs_r = m_stencil_coefs_r.size(); + auto const n_coefs_r = static_cast(m_stencil_coefs_r.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract cylindrical specific parameters Real const dr = m_dr; @@ -462,7 +462,7 @@ void FiniteDifferenceSolver::EvolveBCylindrical ( }, [=] AMREX_GPU_DEVICE (int i, int j, int /*k*/){ - Real const r = rmin + (i + 0.5)*dr; // r on a cell-centered grid (Bz is cell-centered in r) + Real const r = rmin + (i + 0.5_rt)*dr; // r on a cell-centered grid (Bz is cell-centered in r) Bz(i, j, 0, 0) += dt*( - T_Algo::UpwardDrr_over_r(Et, r, dr, coefs_r, n_coefs_r, i, j, 0, 0)); for (int m=1 ; m(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp index 9ae67d4f960..35cbb6ede93 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp @@ -98,11 +98,11 @@ void FiniteDifferenceSolver::EvolveBPMLCartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tbx = mfi.tilebox(Bfield[0]->ixType().ixType()); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp index b044986383e..74922650a42 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp @@ -117,7 +117,7 @@ void FiniteDifferenceSolver::EvolveECartesian ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile Array4 const& Ex = Efield[0]->array(mfi); @@ -138,11 +138,11 @@ void FiniteDifferenceSolver::EvolveECartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tex = mfi.tilebox(Efield[0]->ixType().toIntVect()); @@ -221,7 +221,7 @@ void FiniteDifferenceSolver::EvolveECartesian ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -249,7 +249,7 @@ void FiniteDifferenceSolver::EvolveECylindrical ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile Array4 const& Er = Efield[0]->array(mfi); @@ -264,9 +264,9 @@ void FiniteDifferenceSolver::EvolveECylindrical ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_r = m_stencil_coefs_r.dataPtr(); - int const n_coefs_r = m_stencil_coefs_r.size(); + auto const n_coefs_r = static_cast(m_stencil_coefs_r.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract cylindrical specific parameters Real const dr = m_dr; @@ -284,7 +284,7 @@ void FiniteDifferenceSolver::EvolveECylindrical ( amrex::ParallelFor(ter, tet, tez, [=] AMREX_GPU_DEVICE (int i, int j, int /*k*/){ - Real const r = rmin + (i + 0.5)*dr; // r on cell-centered point (Er is cell-centered in r) + Real const r = rmin + (i + 0.5_rt)*dr; // r on cell-centered point (Er is cell-centered in r) Er(i, j, 0, 0) += c2 * dt*( - T_Algo::DownwardDz(Bt, coefs_z, n_coefs_z, i, j, 0, 0) - PhysConst::mu0 * jr(i, j, 0, 0) ); // Mode m=0 @@ -423,7 +423,7 @@ void FiniteDifferenceSolver::EvolveECylindrical ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } // end of loop over grid/tiles diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveECTRho.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveECTRho.cpp index b7a42ad28b7..95f899c98e1 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveECTRho.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveECTRho.cpp @@ -88,7 +88,7 @@ void FiniteDifferenceSolver::EvolveRhoCartesianECT ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile amrex::Array4 const &Ex = Efield[0]->array(mfi); @@ -149,7 +149,7 @@ void FiniteDifferenceSolver::EvolveRhoCartesianECT ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } #ifdef WARPX_DIM_XZ diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp index 2ecc9ee0e04..87145daa79b 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp @@ -117,11 +117,11 @@ void FiniteDifferenceSolver::EvolveEPMLCartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tex = mfi.tilebox(Efield[0]->ixType().ixType()); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp index c43d965b3a4..752adb8e628 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp @@ -103,11 +103,11 @@ void FiniteDifferenceSolver::EvolveFCartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z =static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tf = mfi.tilebox(Ffield->ixType().toIntVect()); @@ -156,9 +156,9 @@ void FiniteDifferenceSolver::EvolveFCylindrical ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_r = m_stencil_coefs_r.dataPtr(); - int const n_coefs_r = m_stencil_coefs_r.size(); + auto const n_coefs_r = static_cast(m_stencil_coefs_r.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract cylindrical specific parameters Real const dr = m_dr; diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp index 5b69737c79c..ee9b7ec368e 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp @@ -92,11 +92,11 @@ void FiniteDifferenceSolver::EvolveFPMLCartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tf = mfi.tilebox(Ffield->ixType().ixType()); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveG.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveG.cpp index 30d19efe008..b6bc8fdca7f 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveG.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveG.cpp @@ -95,9 +95,9 @@ void FiniteDifferenceSolver::EvolveGCartesian ( amrex::Real const* const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); amrex::Real const* const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - const int n_coefs_x = m_stencil_coefs_x.size(); - const int n_coefs_y = m_stencil_coefs_y.size(); - const int n_coefs_z = m_stencil_coefs_z.size(); + const auto n_coefs_x = static_cast(m_stencil_coefs_x.size()); + const auto n_coefs_y = static_cast(m_stencil_coefs_y.size()); + const auto n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tilebox to loop over amrex::Box const& tf = mfi.tilebox(Gfield->ixType().toIntVect()); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H index 51e11aa46f3..452dbc3057a 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H +++ b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H @@ -176,7 +176,7 @@ struct ElectronPressure { amrex::Real const T0, amrex::Real const gamma, amrex::Real const rho) { - return n0 * T0 * pow((rho/PhysConst::q_e)/n0, gamma); + return n0 * T0 * std::pow((rho/PhysConst::q_e)/n0, gamma); } }; diff --git a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp index f4f236e61d1..38ca6c67fff 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp @@ -98,7 +98,7 @@ void FiniteDifferenceSolver::CalculateCurrentAmpereCartesian ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile Array4 const& Jx = Jfield[0]->array(mfi); @@ -116,11 +116,11 @@ void FiniteDifferenceSolver::CalculateCurrentAmpereCartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // Extract tileboxes for which to loop Box const& tjx = mfi.tilebox(Jfield[0]->ixType().toIntVect()); @@ -179,7 +179,7 @@ void FiniteDifferenceSolver::CalculateCurrentAmpereCartesian ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -316,7 +316,7 @@ void FiniteDifferenceSolver::HybridPICSolveECartesian ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); Array4 const& enE_nodal = enE_nodal_mf.array(mfi); Array4 const& Jx = Jfield[0]->const_array(mfi); @@ -365,7 +365,7 @@ void FiniteDifferenceSolver::HybridPICSolveECartesian ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -380,7 +380,7 @@ void FiniteDifferenceSolver::HybridPICSolveECartesian ( { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Extract field data for this grid/tile Array4 const& Ex = Efield[0]->array(mfi); @@ -401,11 +401,11 @@ void FiniteDifferenceSolver::HybridPICSolveECartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); Box const& tex = mfi.tilebox(Efield[0]->ixType().toIntVect()); Box const& tey = mfi.tilebox(Efield[1]->ixType().toIntVect()); @@ -497,7 +497,7 @@ void FiniteDifferenceSolver::HybridPICSolveECartesian ( if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } diff --git a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp index fe9aa979a7e..3c9dc8db1c1 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp @@ -152,11 +152,11 @@ void FiniteDifferenceSolver::MacroscopicEvolveECartesian ( // Extract stencil coefficients Real const * const AMREX_RESTRICT coefs_x = m_stencil_coefs_x.dataPtr(); - int const n_coefs_x = m_stencil_coefs_x.size(); + auto const n_coefs_x = static_cast(m_stencil_coefs_x.size()); Real const * const AMREX_RESTRICT coefs_y = m_stencil_coefs_y.dataPtr(); - int const n_coefs_y = m_stencil_coefs_y.size(); + auto const n_coefs_y = static_cast(m_stencil_coefs_y.size()); Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); - int const n_coefs_z = m_stencil_coefs_z.size(); + auto const n_coefs_z = static_cast(m_stencil_coefs_z.size()); // This functor computes Hx = Bx/mu // Note that mu is cell-centered here and will be interpolated/averaged diff --git a/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.cpp b/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.cpp index 14246b5be13..2fbab2422ea 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.cpp @@ -21,7 +21,7 @@ SpectralBinomialFilter::InitFilterArray (HankelTransform::RealVector const & kve bool const compensation, KFilterArray & filter) { - const int N = kvec.size(); + const auto N = static_cast(kvec.size()); filter.resize(N); amrex::Real* p_filter = filter.data(); amrex::Real const* p_kvec = kvec.data(); @@ -30,7 +30,7 @@ SpectralBinomialFilter::InitFilterArray (HankelTransform::RealVector const & kve { amrex::Real const ss = std::sin(0.5_rt*p_kvec[i]*dels); amrex::Real const ss2 = ss*ss; - amrex::Real filt = std::pow(1._rt - ss2, npasses); + auto filt = static_cast(std::pow(1._rt - ss2, npasses)); if (compensation) { filt *= (1._rt + npasses*ss2); } diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp b/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp index 0cd7a346352..3547a5a55f5 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp @@ -173,7 +173,7 @@ SpectralFieldData::SpectralFieldData( const int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Note: the size of the real-space box and spectral-space box // differ when using real-to-complex FFT. When initializing @@ -193,7 +193,7 @@ SpectralFieldData::SpectralFieldData( const int lev, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -242,7 +242,7 @@ SpectralFieldData::ForwardTransform (const int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Copy the real-space field `mf` to the temporary field `tmpRealField` // This ensures that all fields have the same number of points @@ -309,7 +309,7 @@ SpectralFieldData::ForwardTransform (const int lev, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -367,7 +367,7 @@ SpectralFieldData::BackwardTransform (const int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Copy the spectral-space field `tmpSpectralField` to the appropriate // field (specified by the input argument field_index) @@ -470,7 +470,7 @@ SpectralFieldData::BackwardTransform (const int lev, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp index db189f4287a..cc0e64fd1de 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp @@ -470,7 +470,7 @@ SpectralFieldDataRZ::ForwardTransform (const int lev, { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // Perform the Hankel transform first. // tempHTransformedSplit includes the imaginary component of mode 0. @@ -490,7 +490,7 @@ SpectralFieldDataRZ::ForwardTransform (const int lev, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -527,7 +527,7 @@ SpectralFieldDataRZ::ForwardTransform (const int lev, { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); amrex::Box const& realspace_bx = tempHTransformed[mfi].box(); @@ -556,7 +556,7 @@ SpectralFieldDataRZ::ForwardTransform (const int lev, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -589,7 +589,7 @@ SpectralFieldDataRZ::BackwardTransform (const int lev, { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); amrex::Box realspace_bx = tempHTransformed[mfi].box(); @@ -629,18 +629,18 @@ SpectralFieldDataRZ::BackwardTransform (const int lev, sign = +1._rt; } else { // Odd modes are anti-symmetric - int imode = (icomp + 1)/2; - sign = std::pow(-1._rt, imode); + const auto imode = (icomp + 1)/2; + sign = static_cast(std::pow(-1._rt, imode)); } } - int ic = icomp + i_comp; + const auto ic = icomp + i_comp; field_mf_array(i,j,k,ic) = sign*field_mf_copy_array(ii,j,k,icomp); }); if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -674,7 +674,7 @@ SpectralFieldDataRZ::BackwardTransform (const int lev, { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); amrex::Box realspace_bx = tempHTransformed[mfi].box(); @@ -720,7 +720,7 @@ SpectralFieldDataRZ::BackwardTransform (const int lev, } else { // Even modes are anti-symmetric int imode = (icomp + 1)/2; - sign = std::pow(-1._rt, imode+1); + sign = static_cast(std::pow(-1._rt, imode+1)); } } if (icomp == 0) { @@ -736,7 +736,7 @@ SpectralFieldDataRZ::BackwardTransform (const int lev, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -773,7 +773,7 @@ SpectralFieldDataRZ::ApplyFilter (const int lev, int const field_index) { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto const & filter_r = binomialfilter[mfi].getFilterArrayR(); auto const & filter_z = binomialfilter[mfi].getFilterArrayZ(); @@ -798,7 +798,7 @@ SpectralFieldDataRZ::ApplyFilter (const int lev, int const field_index) if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -818,7 +818,7 @@ SpectralFieldDataRZ::ApplyFilter (const int lev, int const field_index1, { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto const & filter_r = binomialfilter[mfi].getFilterArrayR(); auto const & filter_z = binomialfilter[mfi].getFilterArrayZ(); @@ -847,7 +847,7 @@ SpectralFieldDataRZ::ApplyFilter (const int lev, int const field_index1, if (do_costs) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp index 210a4baffc7..7a74acfadf0 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp @@ -55,8 +55,8 @@ namespace{ p0 = c[ntry]*(*zeroj); p1 = *zeroj; - q0 = jn(n, p0); - q1 = jn(n, p1); + q0 = static_cast(jn(n, p0)); + q1 = static_cast(jn(n, p1)); for (int it=1; it <= nitmx; it++) { if (q1 == q0) break; p = p1 - q1*(p1 - p0)/(q1 - q0); @@ -68,7 +68,7 @@ namespace{ p0 = p1; q0 = q1; p1 = p; - q1 = jn(n, p1); + q1 = static_cast(jn(n, p1)); } } *ier = 3; @@ -84,7 +84,7 @@ void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vec int ierror, ik, k; const amrex::Real tol = 1e-14_rt; - const amrex::Real nitmx = 10; + const int nitmx = 10; const amrex::Real c1 = 1.8557571_rt; const amrex::Real c2 = 1.033150_rt; @@ -113,9 +113,9 @@ void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vec // Include the trivial root ier[0] = 0; roots[0] = 0.; - const amrex::Real f1 = std::pow(n, (1.0_rt/3.0_rt)); - const amrex::Real f2 = f1*f1*n; - const amrex::Real f3 = f1*n*n; + const auto f1 = static_cast(std::pow(n, (1.0_rt/3.0_rt))); + const auto f2 = f1*f1*n; + const auto f3 = f1*n*n; zeroj = n + c1*f1 + (c2/f1) - (c3/n) - (c4/f2) + (c5/f3); ::SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); ier[1] = ierror; @@ -139,7 +139,7 @@ void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vec zeroj = b0 - (t1/b1) - (t3/b3) - (t5/b5) - (t7/b7); - const amrex::Real errj = std::abs(jn(n, zeroj)); + const amrex::Real errj = static_cast(std::abs(jn(n, zeroj))); // improve solution using procedure SecantRootFinder if (errj > tol) ::SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/HankelTransform.cpp b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/HankelTransform.cpp index da62382dffa..aff5ad25fcc 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/HankelTransform.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/HankelTransform.cpp @@ -68,7 +68,7 @@ HankelTransform::HankelTransform (int const hankel_order, amrex::Vector denom(m_nk); for (int ik=0 ; ik < m_nk ; ik++) { - const amrex::Real jna = jn(p_denom, alphas[ik]); + const auto jna = static_cast(jn(p_denom, alphas[ik])); denom[ik] = MathConst::pi*rmax*rmax*jna*jna; } @@ -76,7 +76,7 @@ HankelTransform::HankelTransform (int const hankel_order, for (int ir=0 ; ir < m_nr ; ir++) { for (int ik=0 ; ik < m_nk ; ik++) { int const ii = ik + ir*m_nk; - num[ii] = jn(hankel_order, rmesh[ir]*kr[ik]); + num[ii] = static_cast(jn(hankel_order, rmesh[ir]*kr[ik])); } } @@ -99,7 +99,8 @@ HankelTransform::HankelTransform (int const hankel_order, if (hankel_order == azimuthal_mode-1) { for (int ir=0 ; ir < m_nr ; ir++) { int const ii = ir*m_nk; - invM[ii] = std::pow(rmesh[ir], (azimuthal_mode-1))/(MathConst::pi*std::pow(rmax, (azimuthal_mode+1))); + invM[ii] = static_cast( + std::pow(rmesh[ir], (azimuthal_mode-1))/(MathConst::pi*std::pow(rmax, (azimuthal_mode+1)))); } } else { for (int ir=0 ; ir < m_nr ; ir++) { diff --git a/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp b/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp index b8d81000cda..d882d32d758 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp @@ -154,7 +154,7 @@ SpectralKSpace::getSpectralShiftFactor( const DistributionMapping& dm, Gpu::DeviceVector& shift = shift_factor[mfi]; // Allocate shift coefficients - const int N = k.size(); + const auto N = static_cast(k.size()); shift.resize(N); Real const* pk = k.data(); Complex* pshift = shift.data(); @@ -202,7 +202,7 @@ SpectralKSpace::getModifiedKComponent( const DistributionMapping& dm, Gpu::DeviceVector& modified_k = modified_k_comp[mfi]; // Allocate modified_k to the same size as k - const int N = k.size(); + const auto N = static_cast(k.size());; modified_k.resize(N); // Fill the modified k vector @@ -216,7 +216,7 @@ SpectralKSpace::getModifiedKComponent( const DistributionMapping& dm, Gpu::copyAsync(Gpu::hostToDevice, h_stencil_coef.begin(), h_stencil_coef.end(), d_stencil_coef.begin()); Gpu::synchronize(); - const int nstencil = d_stencil_coef.size(); + const auto nstencil = static_cast(d_stencil_coef.size()); Real const* p_stencil_coef = d_stencil_coef.data(); // Loop over boxes and allocate the corresponding DeviceVector @@ -227,7 +227,7 @@ SpectralKSpace::getModifiedKComponent( const DistributionMapping& dm, Gpu::DeviceVector& modified_k = modified_k_comp[mfi]; // Allocate modified_k to the same size as k - const int N = k.size(); + const auto N = static_cast(k.size());; modified_k.resize(N); Real const* p_k = k.data(); Real * p_modified_k = modified_k.data(); @@ -242,7 +242,7 @@ SpectralKSpace::getModifiedKComponent( const DistributionMapping& dm, std::sin( p_k[i]*(n+1)*delta_x )/( (n+1)*delta_x ); } else { p_modified_k[i] += p_stencil_coef[n]* \ - std::sin( p_k[i]*(n+0.5)*delta_x )/( (n+0.5)*delta_x ); + std::sin( p_k[i]*(n+0.5_rt)*delta_x )/( (n+0.5_rt)*delta_x ); } } diff --git a/Source/FieldSolver/WarpXPushFieldsEM.cpp b/Source/FieldSolver/WarpXPushFieldsEM.cpp index d515462d730..af1c5224df0 100644 --- a/Source/FieldSolver/WarpXPushFieldsEM.cpp +++ b/Source/FieldSolver/WarpXPushFieldsEM.cpp @@ -1291,8 +1291,8 @@ WarpX::ApplyInverseVolumeScalingToCurrentDensity (MultiFab* Jx, MultiFab* Jy, Mu // Wrap the current density deposited in the guard cells around // to the cells above the axis. if (rmin == 0._rt && 1-ishift_r <= i && i < ngJ[0]-ishift_r) { - Jr_arr(i,j,0,2*imode-1) += std::pow(-1, imode+1)*Jr_arr(-ishift_r-i,j,0,2*imode-1); - Jr_arr(i,j,0,2*imode) += std::pow(-1, imode+1)*Jr_arr(-ishift_r-i,j,0,2*imode); + Jr_arr(i,j,0,2*imode-1) += static_cast(std::pow(-1, imode+1)*Jr_arr(-ishift_r-i,j,0,2*imode-1)); + Jr_arr(i,j,0,2*imode) += static_cast(std::pow(-1, imode+1)*Jr_arr(-ishift_r-i,j,0,2*imode)); } // Apply the inverse volume scaling // Jr is forced to zero on axis. @@ -1328,8 +1328,8 @@ WarpX::ApplyInverseVolumeScalingToCurrentDensity (MultiFab* Jx, MultiFab* Jy, Mu // Wrap the current density deposited in the guard cells around // to the cells above the axis. if (rmin == 0._rt && 1-ishift_t <= i && i <= ngJ[0]-ishift_t) { - Jt_arr(i,j,0,2*imode-1) += std::pow(-1, imode+1)*Jt_arr(-ishift_t-i,j,0,2*imode-1); - Jt_arr(i,j,0,2*imode) += std::pow(-1, imode+1)*Jt_arr(-ishift_t-i,j,0,2*imode); + Jt_arr(i,j,0,2*imode-1) += static_cast(std::pow(-1, imode+1)*Jt_arr(-ishift_t-i,j,0,2*imode-1)); + Jt_arr(i,j,0,2*imode) += static_cast(std::pow(-1, imode+1)*Jt_arr(-ishift_t-i,j,0,2*imode)); } // Apply the inverse volume scaling @@ -1365,8 +1365,8 @@ WarpX::ApplyInverseVolumeScalingToCurrentDensity (MultiFab* Jx, MultiFab* Jy, Mu // Wrap the current density deposited in the guard cells around // to the cells above the axis. if (rmin == 0._rt && 1-ishift_z <= i && i <= ngJ[0]-ishift_z) { - Jz_arr(i,j,0,2*imode-1) -= std::pow(-1, imode+1)*Jz_arr(-ishift_z-i,j,0,2*imode-1); - Jz_arr(i,j,0,2*imode) -= std::pow(-1, imode+1)*Jz_arr(-ishift_z-i,j,0,2*imode); + Jz_arr(i,j,0,2*imode-1) -= static_cast(std::pow(-1, imode+1)*Jz_arr(-ishift_z-i,j,0,2*imode-1)); + Jz_arr(i,j,0,2*imode) -= static_cast(std::pow(-1, imode+1)*Jz_arr(-ishift_z-i,j,0,2*imode)); } // Apply the inverse volume scaling @@ -1411,7 +1411,7 @@ WarpX::ApplyInverseVolumeScalingToChargeDensity (MultiFab* Rho, int lev) const std::array& xyzmin = WarpX::LowerCorner(tilebox, lev, 0._rt); const Dim3 lo = lbound(tilebox); const Real rmin = xyzmin[0]; - const Real rminr = xyzmin[0] + (tb.type(0) == NODE ? 0. : 0.5*dx[0]); + const Real rminr = xyzmin[0] + (tb.type(0) == NODE ? 0._rt : 0.5_rt*dx[0]); const int irmin = lo.x; const int ishift = (rminr > rmin ? 1 : 0); @@ -1445,7 +1445,7 @@ WarpX::ApplyInverseVolumeScalingToChargeDensity (MultiFab* Rho, int lev) else { imode = (icomp - ncomp/2 + 1)/2; } - Rho_arr(i,j,0,icomp) -= std::pow(-1, imode+1)*Rho_arr(-ishift-i,j,0,icomp); + Rho_arr(i,j,0,icomp) -= static_cast(std::pow(-1, imode+1)*Rho_arr(-ishift-i,j,0,icomp)); } // Apply the inverse volume scaling @@ -1453,7 +1453,7 @@ WarpX::ApplyInverseVolumeScalingToChargeDensity (MultiFab* Rho, int lev) if (r == 0.) { Rho_arr(i,j,0,icomp) /= (MathConst::pi*dr*axis_volume_factor); } else { - Rho_arr(i,j,0,icomp) /= (2.*MathConst::pi*r); + Rho_arr(i,j,0,icomp) /= (2._rt*MathConst::pi*r); } }); } diff --git a/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp b/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp index 36029e7d3eb..9a5449dfef8 100644 --- a/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp +++ b/Source/FieldSolver/WarpXPushFieldsHybridPIC.cpp @@ -101,7 +101,7 @@ void WarpX::HybridPICEvolveFields () true ); FillBoundaryE(guard_cells.ng_FieldSolver, WarpX::sync_nodal_points); - EvolveB(0.5 / sub_steps * dt[0], DtType::FirstHalf); + EvolveB(0.5_rt / sub_steps * dt[0], DtType::FirstHalf); FillBoundaryB(guard_cells.ng_FieldSolver, WarpX::sync_nodal_points); } @@ -129,7 +129,7 @@ void WarpX::HybridPICEvolveFields () true ); FillBoundaryE(guard_cells.ng_FieldSolver, WarpX::sync_nodal_points); - EvolveB(0.5 / sub_steps * dt[0], DtType::SecondHalf); + EvolveB(0.5_rt / sub_steps * dt[0], DtType::SecondHalf); FillBoundaryB(guard_cells.ng_FieldSolver, WarpX::sync_nodal_points); } diff --git a/Source/FieldSolver/WarpX_FDTD.H b/Source/FieldSolver/WarpX_FDTD.H index 6f1ebad7819..c6f3c16a2c4 100644 --- a/Source/FieldSolver/WarpX_FDTD.H +++ b/Source/FieldSolver/WarpX_FDTD.H @@ -24,6 +24,9 @@ void warpx_computedivb(int i, int j, int k, int dcomp, #endif ) { + + using namespace amrex; + #if defined WARPX_DIM_3D divB(i,j,k,dcomp) = (Bx(i+1,j ,k ) - Bx(i,j,k))*dxinv + (By(i ,j+1,k ) - By(i,j,k))*dyinv @@ -37,8 +40,8 @@ void warpx_computedivb(int i, int j, int k, int dcomp, amrex::ignore_unused(j, Bx, dxinv); amrex::ignore_unused(k, By, dyinv); #elif defined WARPX_DIM_RZ - const amrex::Real ru = 1. + 0.5/(rmin*dxinv + i + 0.5); - const amrex::Real rd = 1. - 0.5/(rmin*dxinv + i + 0.5); + const amrex::Real ru = 1._rt + 0.5_rt/(rmin*dxinv + i + 0.5_rt); + const amrex::Real rd = 1._rt - 0.5_rt/(rmin*dxinv + i + 0.5_rt); divB(i,j,0,dcomp) = (ru*Bx(i+1,j,0) - rd*Bx(i,j,0))*dxinv + (Bz(i,j+1,0) - Bz(i,j,0))*dzinv; amrex::ignore_unused(k, By, dyinv); diff --git a/Source/Filter/BilinearFilter.cpp b/Source/Filter/BilinearFilter.cpp index c8e921aad31..ecfee2cd7c9 100644 --- a/Source/Filter/BilinearFilter.cpp +++ b/Source/Filter/BilinearFilter.cpp @@ -63,8 +63,8 @@ namespace { void BilinearFilter::ComputeStencils(){ WARPX_PROFILE("BilinearFilter::ComputeStencils()"); int i = 0; - for (auto el : npass_each_dir ) - stencil_length_each_dir[i++] = el; + for (const auto& el : npass_each_dir ) + stencil_length_each_dir[i++] = static_cast(el); stencil_length_each_dir += 1.; #if defined(WARPX_DIM_3D) // npass_each_dir = npass_x npass_y npass_z diff --git a/Source/Filter/Filter.cpp b/Source/Filter/Filter.cpp index b6d5c503953..5952ffde59c 100644 --- a/Source/Filter/Filter.cpp +++ b/Source/Filter/Filter.cpp @@ -47,7 +47,7 @@ Filter::ApplyStencil (MultiFab& dstmf, const MultiFab& srcmf, const int lev, int { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); const auto& src = srcmf.array(mfi); const auto& dst = dstmf.array(mfi); @@ -59,7 +59,7 @@ Filter::ApplyStencil (MultiFab& dstmf, const MultiFab& srcmf, const int lev, int if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -218,7 +218,7 @@ Filter::ApplyStencil (amrex::MultiFab& dstmf, const amrex::MultiFab& srcmf, cons { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); const auto& srcfab = srcmf[mfi]; auto& dstfab = dstmf[mfi]; @@ -236,7 +236,7 @@ Filter::ApplyStencil (amrex::MultiFab& dstmf, const amrex::MultiFab& srcmf, cons if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } diff --git a/Source/Fluids/MultiFluidContainer.H b/Source/Fluids/MultiFluidContainer.H index 94d95d179e5..ef6cc6115cb 100644 --- a/Source/Fluids/MultiFluidContainer.H +++ b/Source/Fluids/MultiFluidContainer.H @@ -61,7 +61,7 @@ public: amrex::MultiFab* rho, amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz, amrex::Real cur_time, bool skip_deposition=false); - int nSpecies() const {return species_names.size();} + int nSpecies() const {return static_cast(species_names.size());} void DepositCharge (int lev, amrex::MultiFab &rho); void DepositCurrent (int lev, diff --git a/Source/Fluids/MusclHancockUtils.H b/Source/Fluids/MusclHancockUtils.H index d0c37f9c58f..764143a7220 100644 --- a/Source/Fluids/MusclHancockUtils.H +++ b/Source/Fluids/MusclHancockUtils.H @@ -18,7 +18,8 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real F_r (amrex::Real r, amrex::Real u_r, amrex::Real u_theta, amrex::Real u_z, amrex::Real dt) { - return dt*(-u_theta*u_theta/r)/sqrt(1.0 + u_r*u_r + u_theta*u_theta + u_z*u_z) + u_r; + using namespace amrex::literals; + return dt*(-u_theta*u_theta/r)/std::sqrt(1.0_rt + u_r*u_r + u_theta*u_theta + u_z*u_z) + u_r; } // Euler push for momentum source (theta-direction) @@ -26,26 +27,29 @@ amrex::Real F_r (amrex::Real r, amrex::Real u_r, amrex::Real u_theta, amrex::Rea AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real F_theta (amrex::Real r, amrex::Real u_r, amrex::Real u_theta, amrex::Real u_z, amrex::Real dt) { - return dt*(u_theta*u_r/r)/sqrt(1.0 + u_r*u_r + u_theta*u_theta + u_z*u_z) + u_theta; + using namespace amrex::literals; + return dt*(u_theta*u_r/r)/std::sqrt(1.0_rt + u_r*u_r + u_theta*u_theta + u_z*u_z) + u_theta; } // Velocity at the half step AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real V_calc (const amrex::Array4& U, int i, int j, int k, int comp, amrex::Real c) { + using namespace amrex::literals; // comp -> x, y, z -> 0, 1, 2, return Vx, Vy, or Vz: - amrex::Real gamma = std::sqrt(1.0 + (U(i,j,k,1)*U(i,j,k,1) + U(i,j,k,2)*U(i,j,k,2) + U(i,j,k,3)*U(i,j,k,3))/(c*c)); + amrex::Real gamma = std::sqrt(1.0_rt + (U(i,j,k,1)*U(i,j,k,1) + U(i,j,k,2)*U(i,j,k,2) + U(i,j,k,3)*U(i,j,k,3))/(c*c)); return U(i,j,k,comp+1)/gamma; } // mindmod AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real minmod (amrex::Real a, amrex::Real b) { - if (a > 0.0 && b > 0.0) + using namespace amrex::literals; + if (a > 0.0_rt && b > 0.0_rt) return std::min(a, b); - else if (a < 0.0 && b < 0.0) + else if (a < 0.0_rt && b < 0.0_rt) return std::max(a, b); else - return 0.0; + return 0.0_rt; } // Min of 3 inputs AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE @@ -63,9 +67,10 @@ amrex::Real max3 (amrex::Real a, amrex::Real b, amrex::Real c) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real minmod3 (amrex::Real a, amrex::Real b , amrex::Real c) { - if (a > 0.0 && b > 0.0 && c > 0.0) + using namespace amrex::literals; + if (a > 0.0_rt && b > 0.0_rt && c > 0.0_rt) return min3(a,b,c); - else if (a < 0.0 && b < 0.0 && c < 0.0) + else if (a < 0.0_rt && b < 0.0_rt && c < 0.0_rt) return max3(a,b,c); else return 0.0; @@ -74,89 +79,99 @@ amrex::Real minmod3 (amrex::Real a, amrex::Real b , amrex::Real c) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real maxmod (amrex::Real a, amrex::Real b) { - if (a > 0.0 && b > 0.0) + using namespace amrex::literals; + if (a > 0.0_rt && b > 0.0_rt) return std::max(a, b); - else if (a < 0.0 && b < 0.0) + else if (a < 0.0_rt && b < 0.0_rt) return std::min(a, b); else - return 0.0; + return 0.0_rt; } // Rusanov Flux (density) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real flux_N (const amrex::Array4& Um, const amrex::Array4& Up, int i, int j, int k, amrex::Real Vm, amrex::Real Vp) { + using namespace amrex::literals; amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); - return 0.5*(Vm*Um(i,j,k,0) + Vp*Up(i,j,k,0)) - (0.5*c)*(Up(i,j,k,0) - Um(i,j,k,0)); + return 0.5_rt*(Vm*Um(i,j,k,0) + Vp*Up(i,j,k,0)) - (0.5_rt*c)*(Up(i,j,k,0) - Um(i,j,k,0)); } // Rusanov Flux (Momentum density x) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real flux_NUx (const amrex::Array4& Um, const amrex::Array4& Up, int i, int j, int k, amrex::Real Vm, amrex::Real Vp) { + using namespace amrex::literals; amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); - return 0.5*(Vm*Um(i,j,k,0)*Um(i,j,k,1) + Vp*Up(i,j,k,0)*Up(i,j,k,1)) - - (0.5*c)*(Up(i,j,k,0)*Up(i,j,k,1) - Um(i,j,k,0)*Um(i,j,k,1)); + return 0.5_rt*(Vm*Um(i,j,k,0)*Um(i,j,k,1) + Vp*Up(i,j,k,0)*Up(i,j,k,1)) + - (0.5_rt*c)*(Up(i,j,k,0)*Up(i,j,k,1) - Um(i,j,k,0)*Um(i,j,k,1)); } // Rusanov Flux (Momentum density y) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real flux_NUy (const amrex::Array4& Um, const amrex::Array4& Up, int i, int j, int k, amrex::Real Vm, amrex::Real Vp) { + using namespace amrex::literals; amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); - return 0.5*(Vm*Um(i,j,k,0)*Um(i,j,k,2) + Vp*Up(i,j,k,0)*Up(i,j,k,2)) - - (0.5*c)*(Up(i,j,k,0)*Up(i,j,k,2) - Um(i,j,k,0)*Um(i,j,k,2)); + return 0.5_rt*(Vm*Um(i,j,k,0)*Um(i,j,k,2) + Vp*Up(i,j,k,0)*Up(i,j,k,2)) + - (0.5_rt*c)*(Up(i,j,k,0)*Up(i,j,k,2) - Um(i,j,k,0)*Um(i,j,k,2)); } // Rusanov Flux (Momentum density z) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real flux_NUz (const amrex::Array4& Um, const amrex::Array4& Up, int i, int j, int k, amrex::Real Vm, amrex::Real Vp) { + using namespace amrex::literals; amrex::Real c = std::max( std::abs(Vm) , std::abs(Vp) ); - return 0.5*(Vm*Um(i,j,k,0)*Um(i,j,k,3) + Vp*Up(i,j,k,0)*Up(i,j,k,3)) - - (0.5*c)*(Up(i,j,k,0)*Up(i,j,k,3) - Um(i,j,k,0)*Um(i,j,k,3)); + return 0.5_rt*(Vm*Um(i,j,k,0)*Um(i,j,k,3) + Vp*Up(i,j,k,0)*Up(i,j,k,3)) + - (0.5_rt*c)*(Up(i,j,k,0)*Up(i,j,k,3) - Um(i,j,k,0)*Um(i,j,k,3)); } // ave_minmod high diffusivity, sigma can be between [1,2] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real ave_adjustable_diff (amrex::Real a, amrex::Real b) { - amrex::Real sigma = 2.0*0.732050807568877; - if (a*b > 0.0) - return minmod3( (a+b)/2.0, sigma*a, sigma*b ); + using namespace amrex::literals; + constexpr auto sigma = static_cast(2.0*0.732050807568877); + if (a*b > 0.0_rt) + return minmod3( (a+b)/2.0_rt, sigma*a, sigma*b ); else - return 0.0; + return 0.0_rt; } // ave_minmod Low diffusivity AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real ave (amrex::Real a, amrex::Real b) { - if (a*b > 0.0) - return minmod3( (a+b)/2.0, 2.0*a, 2.0*b ); + using namespace amrex::literals; + if (a*b > 0.0_rt) + return minmod3( (a+b)/2.0_rt, 2.0_rt*a, 2.0_rt*b ); else - return 0.0; + return 0.0_rt; } // ave_superbee AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real ave_superbee (amrex::Real a, amrex::Real b) { - if (a*b > 0.0) - return minmod( maxmod(a,b), minmod(2.0*a,2.0*b)); + using namespace amrex::literals; + if (a*b > 0.0_rt) + return minmod( maxmod(a,b), minmod(2.0_rt*a,2.0_rt*b)); else - return 0.0; + return 0.0_rt; } // stage2 slope limiting AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real ave_stage2 (amrex::Real dQ, amrex::Real a, amrex::Real b, amrex::Real c) { + using namespace amrex::literals; // sigma = sqrt(3) -1 - amrex::Real sigma = 0.732050807568877; - amrex::Real dq_min = 2.0*std::min( b - min3(a,b,c), max3(a,b,c) - b); + constexpr auto sigma = 0.732050807568877_rt; + amrex::Real dq_min = 2.0_rt*std::min( b - min3(a,b,c), max3(a,b,c) - b); return ( std::abs(dQ)/dQ ) * std::min( std::abs(dQ) , sigma*std::abs(dq_min) ); } // Returns the offset indices for the "plus" grid AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE void plus_index_offsets (int i, int j, int k, int& ip, int& jp, int& kp, int comp) { + using namespace amrex::literals; // Find the correct offsets #if defined(WARPX_DIM_3D) if (comp == 0) { //x @@ -184,23 +199,23 @@ void compute_U_edges (const amrex::Array4& Um, const amrex::Array4< amrex::Real U_tilde0, amrex::Real U_tilde1, amrex::Real U_tilde2, amrex::Real U_tilde3, amrex::Real dU0x, amrex::Real dU1x, amrex::Real dU2x, amrex::Real dU3x, int comp) { - + using namespace amrex::literals; // comp -> x, y, z -> 0, 1, 2 int ip, jp, kp; plus_index_offsets(i, j, k, ip, jp, kp, comp); if ( box.contains(i,j,k) ) { - Um(i,j,k,0) = U_tilde0 + dU0x/2.0; - Um(i,j,k,1) = U_tilde1 + dU1x/2.0; - Um(i,j,k,2) = U_tilde2 + dU2x/2.0; - Um(i,j,k,3) = U_tilde3 + dU3x/2.0; + Um(i,j,k,0) = U_tilde0 + dU0x/2.0_rt; + Um(i,j,k,1) = U_tilde1 + dU1x/2.0_rt; + Um(i,j,k,2) = U_tilde2 + dU2x/2.0_rt; + Um(i,j,k,3) = U_tilde3 + dU3x/2.0_rt; } if ( box.contains(ip,jp,kp) ) { - Up(ip,jp,kp,0) = U_tilde0 - dU0x/2.0; - Up(ip,jp,kp,1) = U_tilde1 - dU1x/2.0; - Up(ip,jp,kp,2) = U_tilde2 - dU2x/2.0; - Up(ip,jp,kp,3) = U_tilde3 - dU3x/2.0; + Up(ip,jp,kp,0) = U_tilde0 - dU0x/2.0_rt; + Up(ip,jp,kp,1) = U_tilde1 - dU1x/2.0_rt; + Up(ip,jp,kp,2) = U_tilde2 - dU2x/2.0_rt; + Up(ip,jp,kp,3) = U_tilde3 - dU3x/2.0_rt; } } // Compute the zero edges @@ -208,23 +223,23 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE void set_U_edges_to_zero (const amrex::Array4& Um, const amrex::Array4& Up, int i, int j, int k, amrex::Box box, int comp) { - + using namespace amrex::literals; // comp -> x, y, z -> 0, 1, 2 int ip, jp, kp; plus_index_offsets(i, j, k, ip, jp, kp, comp); if ( box.contains(i,j,k) ) { - Um(i,j,k,0) = 0.0; - Um(i,j,k,1) = 0.0; - Um(i,j,k,2) = 0.0; - Um(i,j,k,3) = 0.0; + Um(i,j,k,0) = 0.0_rt; + Um(i,j,k,1) = 0.0_rt; + Um(i,j,k,2) = 0.0_rt; + Um(i,j,k,3) = 0.0_rt; } if ( box.contains(ip,jp,kp) ) { - Up(ip,jp,kp,0) = 0.0; - Up(ip,jp,kp,1) = 0.0; - Up(ip,jp,kp,2) = 0.0; - Up(ip,jp,kp,3) = 0.0; + Up(ip,jp,kp,0) = 0.0_rt; + Up(ip,jp,kp,1) = 0.0_rt; + Up(ip,jp,kp,2) = 0.0_rt; + Up(ip,jp,kp,3) = 0.0_rt; } } // Positivity Limiter @@ -236,6 +251,7 @@ int i, int j, int k, amrex::Box box, amrex::Real Ux, amrex::Real Uy, amrex::Real int comp) { + using namespace amrex::literals; // comp -> x, y, z -> 0, 1, 2 int ip, jp, kp; plus_index_offsets(i, j, k, ip, jp, kp, comp); @@ -244,7 +260,7 @@ int comp) // set the slope to zero (hence why we have the three cases, the first is when // both points exist, and the second two are are edge cases) if (( box.contains(i,j,k) ) && ( box.contains(ip,jp,kp) )) { - if ((U_edge_minus(i,j,k,0) < 0.0) || (U_edge_plus(ip,jp,kp,0) < 0.0)) { + if ((U_edge_minus(i,j,k,0) < 0.0_rt) || (U_edge_plus(ip,jp,kp,0) < 0.0_rt)) { U_edge_minus(i,j,k,0) = N_arr(i,j,k); U_edge_minus(i,j,k,1) = Ux; U_edge_minus(i,j,k,2) = Uy; @@ -255,14 +271,14 @@ int comp) U_edge_plus(ip,jp,kp,3) = Uz; } } else if (( box.contains(i,j,k) ) && ( box.contains(ip,jp,kp) != 1)) { - if (U_edge_minus(i,j,k,0) < 0.0) { + if (U_edge_minus(i,j,k,0) < 0.0_rt) { U_edge_minus(i,j,k,0) = N_arr(i,j,k); U_edge_minus(i,j,k,1) = Ux; U_edge_minus(i,j,k,2) = Uy; U_edge_minus(i,j,k,3) = Uz; } } else if (( box.contains(i,j,k) != 1 ) && ( box.contains(ip,jp,kp) )) { - if (U_edge_plus(ip,jp,kp,0) < 0.0){ + if (U_edge_plus(ip,jp,kp,0) < 0.0_rt){ U_edge_plus(ip,jp,kp,0) = N_arr(i,j,k); U_edge_plus(ip,jp,kp,1) = Ux; U_edge_plus(ip,jp,kp,2) = Uy; @@ -275,54 +291,59 @@ int comp) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real DownDx_N (const amrex::Array4& N, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) return N(i,j,k) - N(i-1,j,k); #else amrex::ignore_unused(N, i, j, k); - return 0.0; + return 0.0_rt; #endif } // Compute the difference in N (up-x) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real UpDx_N (const amrex::Array4& N, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) return N(i+1,j,k) - N(i,j,k); #else amrex::ignore_unused(N, i, j, k); - return 0.0; + return 0.0_rt; #endif } // Compute the difference in N (down-y) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real DownDy_N (const amrex::Array4& N, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) return N(i,j,k) - N(i,j-1,k); #else amrex::ignore_unused(N, i, j, k); - return 0.0; + return 0.0_rt; #endif } // Compute the difference in N (up-y) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real UpDy_N (const amrex::Array4& N, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) return N(i,j+1,k) - N(i,j,k); #else amrex::ignore_unused(N, i, j, k); - return 0.0; + return 0.0_rt; #endif } // Compute the difference in N (down-z) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real DownDz_N (const amrex::Array4& N, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) return N(i,j,k) - N(i,j,k-1); @@ -336,6 +357,7 @@ amrex::Real DownDz_N (const amrex::Array4& N, int i, int j, int k) AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real UpDz_N (const amrex::Array4& N, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) return N(i,j,k+1) - N(i,j,k); @@ -352,6 +374,7 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real DownDx_U (const amrex::Array4& N, const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) // U is zero if N is zero, Check positivity before dividing @@ -360,7 +383,7 @@ const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) return U - U_m; #else amrex::ignore_unused(N, NU, U, i, j, k); - return 0.0; + return 0.0_rt; #endif } // Compute the difference in U (up-x) @@ -368,6 +391,7 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real UpDx_U (const amrex::Array4& N, const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) || defined(WARPX_DIM_RZ) || defined(WARPX_DIM_XZ) // U is zero if N is zero, Check positivity before dividing @@ -376,7 +400,7 @@ const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) return U_p - U; #else amrex::ignore_unused(N, NU, U, i, j, k); - return 0.0; + return 0.0_rt; #endif } @@ -385,6 +409,7 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real DownDy_U (const amrex::Array4& N, const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) // U is zero if N is zero, Check positivity before dividing @@ -393,7 +418,7 @@ const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) return U - U_m; #else amrex::ignore_unused(N, NU, U, i, j, k); - return 0.0; + return 0.0_rt; #endif } // Compute the difference in U (up-y) @@ -401,6 +426,7 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real UpDy_U (const amrex::Array4& N, const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences #if defined(WARPX_DIM_3D) // U is zero if N is zero, Check positivity before dividing @@ -409,7 +435,7 @@ const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) return U_p - U; #else amrex::ignore_unused(N, NU, U, i, j, k); - return 0.0; + return 0.0_rt; #endif } @@ -418,8 +444,9 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real DownDz_U (const amrex::Array4& N, const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences - amrex::Real U_m = 0; + amrex::Real U_m = 0_rt; // U is zero if N is zero, Check positivity before dividing #if defined(WARPX_DIM_3D) @@ -438,6 +465,7 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real UpDz_U (const amrex::Array4& N, const amrex::Array4& NU, amrex::Real& U, int i, int j, int k) { + using namespace amrex::literals; // Write the correct differences amrex::Real U_p = 0; @@ -460,6 +488,7 @@ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::Real dF (const amrex::Array4& U_minus, const amrex::Array4& U_plus,int i,int j,int k,amrex::Real clight, int comp, int dir) { + using namespace amrex::literals; // dir -> x, y, z -> 0, 1, 2 int ip, jp, kp; plus_index_offsets(i, j, k, ip, jp, kp, dir); diff --git a/Source/Fluids/WarpXFluidContainer.cpp b/Source/Fluids/WarpXFluidContainer.cpp index 758e353208e..48631035397 100644 --- a/Source/Fluids/WarpXFluidContainer.cpp +++ b/Source/Fluids/WarpXFluidContainer.cpp @@ -233,8 +233,8 @@ void WarpXFluidContainer::InitData(int lev, amrex::Box init_box, amrex::Real cur // Lorentz transform n, u (from lab to boosted frame) if (n > 0.0){ if (gamma_boost > 1._rt){ - amrex::Real gamma = sqrt(1.0 + (u.x*u.x + u.y*u.y + u.z*u.z)/(clight*clight)); - amrex::Real n_boosted = gamma_boost*n*( 1.0 - beta_boost*u.z/(gamma*clight) ); + amrex::Real gamma = std::sqrt(1.0_rt + (u.x*u.x + u.y*u.y + u.z*u.z)/(clight*clight)); + amrex::Real n_boosted = gamma_boost*n*( 1.0_rt - beta_boost*u.z/(gamma*clight) ); amrex::Real uz_boosted = gamma_boost*(u.z - beta_boost*clight*gamma); u.z = uz_boosted; n = n_boosted; @@ -417,22 +417,22 @@ void WarpXFluidContainer::AdvectivePush_Muscl (int lev) amrex::Real dt_over_dx = (dt/dx[0]); amrex::Real dt_over_dy = (dt/dx[1]); amrex::Real dt_over_dz = (dt/dx[2]); - amrex::Real dt_over_dx_half = 0.5*(dt/dx[0]); - amrex::Real dt_over_dy_half = 0.5*(dt/dx[1]); - amrex::Real dt_over_dz_half = 0.5*(dt/dx[2]); + amrex::Real dt_over_dx_half = 0.5_rt*(dt/dx[0]); + amrex::Real dt_over_dy_half = 0.5_rt*(dt/dx[1]); + amrex::Real dt_over_dz_half = 0.5_rt*(dt/dx[2]); #elif defined(WARPX_DIM_XZ) - amrex::Real dt_over_dx_half = 0.5*(dt/dx[0]); - amrex::Real dt_over_dz_half = 0.5*(dt/dx[1]); + amrex::Real dt_over_dx_half = 0.5_rt*(dt/dx[0]); + amrex::Real dt_over_dz_half = 0.5_rt*(dt/dx[1]); amrex::Real dt_over_dx = (dt/dx[0]); amrex::Real dt_over_dz = (dt/dx[1]); #elif defined(WARPX_DIM_RZ) const auto problo = geom.ProbLoArray(); - amrex::Real dt_over_dx_half = 0.5*(dt/dx[0]); - amrex::Real dt_over_dz_half = 0.5*(dt/dx[1]); + amrex::Real dt_over_dx_half = 0.5_rt*(dt/dx[0]); + amrex::Real dt_over_dz_half = 0.5_rt*(dt/dx[1]); amrex::Box const& domain = geom.Domain(); #else amrex::Real dt_over_dz = (dt/dx[0]); - amrex::Real dt_over_dz_half = 0.5*(dt/dx[0]); + amrex::Real dt_over_dz_half = 0.5_rt*(dt/dx[0]); #endif amrex::BoxArray ba = N[lev]->boxArray(); @@ -532,8 +532,8 @@ void WarpXFluidContainer::AdvectivePush_Muscl (int lev) // Compute useful quantities for J amrex::Real c_sq = clight*clight; - amrex::Real gamma = sqrt(1.0 + (Ux*Ux + Uy*Uy + Uz*Uz)/(c_sq) ); - amrex::Real inv_c2_gamma3 = 1./(c_sq*gamma*gamma*gamma); + amrex::Real gamma = std::sqrt(1.0_rt + (Ux*Ux + Uy*Uy + Uz*Uz)/(c_sq) ); + amrex::Real inv_c2_gamma3 = 1._rt/(c_sq*gamma*gamma*gamma); // J represents are 4x4 matrices that show up in the advection // equations written as a function of U = {N, Ux, Uy, Uz}: @@ -654,14 +654,14 @@ void WarpXFluidContainer::AdvectivePush_Muscl (int lev) dU0x = ave( -UpDx_N(N_arr,i,j,k) , UpDx_N(N_arr,i,j,k) ); // First term in the ave is: U_{x,y} + U_{x,y}_p, // which can be writen as 2*U_{x,y} + UpDx_U(U_{x,y}) - dU1x = ave( 2.0*Ux + UpDx_U(N_arr,NUx_arr,Ux,i,j,k) , UpDx_U(N_arr,NUx_arr,Ux,i,j,k) ); - dU2x = ave( 2.0*Uy + UpDx_U(N_arr,NUy_arr,Uy,i,j,k) , UpDx_U(N_arr,NUy_arr,Uy,i,j,k) ); + dU1x = ave( 2.0_rt*Ux + UpDx_U(N_arr,NUx_arr,Ux,i,j,k) , UpDx_U(N_arr,NUx_arr,Ux,i,j,k) ); + dU2x = ave( 2.0_rt*Uy + UpDx_U(N_arr,NUy_arr,Uy,i,j,k) , UpDx_U(N_arr,NUy_arr,Uy,i,j,k) ); dU3x = ave( -UpDx_U(N_arr,NUz_arr,Uz,i,j,k) , UpDx_U(N_arr,NUz_arr,Uz,i,j,k) ); } else if (i == domain.bigEnd(0)+1) { - dU0x = ave( DownDx_N(N_arr,i,j,k) , 0.0 ); - dU1x = ave( DownDx_U(N_arr,NUx_arr,Ux,i,j,k) , 0.0 ); - dU2x = ave( DownDx_U(N_arr,NUy_arr,Uy,i,j,k) , 0.0 ); - dU3x = ave( DownDx_U(N_arr,NUz_arr,Uz,i,j,k) , 0.0 ); + dU0x = ave( DownDx_N(N_arr,i,j,k) , 0.0_rt ); + dU1x = ave( DownDx_U(N_arr,NUx_arr,Ux,i,j,k) , 0.0_rt ); + dU2x = ave( DownDx_U(N_arr,NUy_arr,Uy,i,j,k) , 0.0_rt ); + dU3x = ave( DownDx_U(N_arr,NUz_arr,Uz,i,j,k) , 0.0_rt ); } // RZ sources: @@ -679,7 +679,7 @@ void WarpXFluidContainer::AdvectivePush_Muscl (int lev) amrex::Real JdU1z = J11z*dU1z; amrex::Real JdU2z = J22z*dU2z; amrex::Real JdU3z = J33z*dU3z; - amrex::Real U_tilde0 = N_arr(i,j,k) - dt_over_dx_half*JdU0x - dt_over_dz_half*JdU0z - (dt/2.0)*N_source; + amrex::Real U_tilde0 = N_arr(i,j,k) - dt_over_dx_half*JdU0x - dt_over_dz_half*JdU0z - (dt/2.0_rt)*N_source; amrex::Real U_tilde1 = Ux - dt_over_dx_half*JdU1x - dt_over_dz_half*JdU1z; amrex::Real U_tilde2 = Uy - dt_over_dx_half*JdU2x - dt_over_dz_half*JdU2z; amrex::Real U_tilde3 = Uz - dt_over_dx_half*JdU3x - dt_over_dz_half*JdU3z; @@ -804,28 +804,28 @@ void WarpXFluidContainer::AdvectivePush_Muscl (int lev) amrex::Real dr = dx[0]; amrex::Real dz = dx[1]; amrex::Real r = problo[0] + i * dr; - amrex::Real Vij = 0.0; - amrex::Real S_Az = 0.0; + amrex::Real Vij = 0.0_rt; + amrex::Real S_Az = 0.0_rt; // Volume element and z-facing surfaces if (i == domain.smallEnd(0)) { - Vij = 2.0*MathConst::pi*(dr/2.0)*(dr/4.0)*dz; - S_Az = 2.0*MathConst::pi*(dr/4.0)*(dr/2.0); + Vij = 2.0_rt*MathConst::pi*(dr/2.0_rt)*(dr/4.0_rt)*dz; + S_Az = 2.0_rt*MathConst::pi*(dr/4.0_rt)*(dr/2.0_rt); } else if (i == domain.bigEnd(0)+1) { - Vij = 2.0*MathConst::pi*(r - dr/4.0)*(dr/2.0)*dz; - S_Az = 2.0*MathConst::pi*(r - dr/4.0)*(dr/2.0); + Vij = 2.0_rt*MathConst::pi*(r - dr/4.0_rt)*(dr/2.0_rt)*dz; + S_Az = 2.0_rt*MathConst::pi*(r - dr/4.0_rt)*(dr/2.0_rt); } else { - Vij = 2.0*MathConst::pi*r*dr*dz; - S_Az = 2.0*MathConst::pi*(r)*dr; + Vij = 2.0_rt*MathConst::pi*r*dr*dz; + S_Az = 2.0_rt*MathConst::pi*(r)*dr; } // Radial Surfaces - amrex::Real S_Ar_plus = 2.0*MathConst::pi*(r + dr/2.0)*dz; - amrex::Real S_Ar_minus = 2.0*MathConst::pi*(r - dr/2.0)*dz; + amrex::Real S_Ar_plus = 2.0_rt*MathConst::pi*(r + dr/2.0_rt)*dz; + amrex::Real S_Ar_minus = 2.0_rt*MathConst::pi*(r - dr/2.0_rt)*dz; if (i == domain.smallEnd(0)) - S_Ar_minus = 0.0; + S_Ar_minus = 0.0_rt; if (i == domain.bigEnd(0)+1) - S_Ar_plus = 2.0*MathConst::pi*(r)*dz; + S_Ar_plus = 2.0_rt*MathConst::pi*(r)*dz; // Impose "none" boundaries // Condition: Vx(r) = 0 at boundaries @@ -834,9 +834,9 @@ void WarpXFluidContainer::AdvectivePush_Muscl (int lev) // compute the fluxes: // (note that _plus is shifted due to grid location) - amrex::Real Vx_L_minus = 0.0, Vx_I_plus = 0.0; - amrex::Real F0_minusx = 0.0, F1_minusx = 0.0, F2_minusx = 0.0, F3_minusx = 0.0; - amrex::Real F0_plusx = 0.0, F1_plusx = 0.0, F2_plusx = 0.0, F3_plusx = 0.0; + amrex::Real Vx_L_minus = 0.0_rt, Vx_I_plus = 0.0_rt; + amrex::Real F0_minusx = 0.0_rt, F1_minusx = 0.0_rt, F2_minusx = 0.0_rt, F3_minusx = 0.0_rt; + amrex::Real F0_plusx = 0.0_rt, F1_plusx = 0.0_rt, F2_plusx = 0.0_rt, F3_plusx = 0.0_rt; if (i != domain.smallEnd(0)) { Vx_L_minus = V_calc(U_minus_x,i-1,j,k,0,clight); F0_minusx = flux_N( U_minus_x, U_plus_x, i-1, j, k, Vx_L_minus, Vx_L_plus)*S_Ar_minus; @@ -905,7 +905,7 @@ void WarpXFluidContainer::centrifugal_source_rz (int lev) { // Verify density is non-zero - if (N_arr(i,j,k)>0.0) { + if (N_arr(i,j,k)>0.0_rt) { // Compute r amrex::Real r = problo[0] + i * dx[0]; @@ -920,10 +920,10 @@ void WarpXFluidContainer::centrifugal_source_rz (int lev) if (i != domain.smallEnd(0)) { amrex::Real u_r_1 = F_r(r,u_r,u_theta,u_z,dt); amrex::Real u_theta_1 = F_theta(r,u_r,u_theta,u_z,dt); - amrex::Real u_r_2 = (0.75)*(u_r) + (0.25)*F_r(r,u_r_1,u_theta_1,u_z,dt); - amrex::Real u_theta_2 = (0.75)*(u_theta) + (0.25)*F_theta(r,u_r_1,u_theta_1,u_z,dt); - u_r = (1.0/3.0)*(u_r) + (2.0/3.0)*F_r(r,u_r_2,u_theta_2,u_z,dt); - u_theta = (1.0/3.0)*(u_theta) + (2.0/3.0)*F_theta(r,u_r_2,u_theta_2,u_z,dt); + amrex::Real u_r_2 = (0.75_rt)*(u_r) + (0.25_rt)*F_r(r,u_r_1,u_theta_1,u_z,dt); + amrex::Real u_theta_2 = (0.75_rt)*(u_theta) + (0.25_rt)*F_theta(r,u_r_1,u_theta_1,u_z,dt); + u_r = (1.0_rt/3.0_rt)*(u_r) + (2.0_rt/3.0_rt)*F_r(r,u_r_2,u_theta_2,u_z,dt); + u_theta = (1.0_rt/3.0_rt)*(u_theta) + (2.0_rt/3.0_rt)*F_theta(r,u_r_2,u_theta_2,u_z,dt); // Calculate NU, save NUr, NUtheta NUx_arr(i,j,k) = N_arr(i,j,k)*u_r*clight; @@ -931,8 +931,8 @@ void WarpXFluidContainer::centrifugal_source_rz (int lev) // BC r = 0, u_theta = 0, and there is no extra source terms } else { - NUx_arr(i,j,k) = 0.0; - NUy_arr(i,j,k) = 0.0; + NUx_arr(i,j,k) = 0.0_rt; + NUy_arr(i,j,k) = 0.0_rt; } } } @@ -1286,12 +1286,12 @@ void WarpXFluidContainer::DepositCurrent( [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept { // Calculate J from fluid quantities - amrex::Real gamma = 1.0, Ux = 0.0, Uy = 0.0, Uz = 0.0; - if (N_arr(i, j, k)>0.0){ + amrex::Real gamma = 1.0_rt, Ux = 0.0_rt, Uy = 0.0_rt, Uz = 0.0_rt; + if (N_arr(i, j, k)>0.0_rt){ Ux = NUx_arr(i, j, k)/N_arr(i, j, k); Uy = NUy_arr(i, j, k)/N_arr(i, j, k); Uz = NUz_arr(i, j, k)/N_arr(i, j, k); - gamma = std::sqrt(1.0 + ( Ux*Ux + Uy*Uy + Uz*Uz) * inv_clight_sq ) ; + gamma = std::sqrt(1.0_rt + ( Ux*Ux + Uy*Uy + Uz*Uz) * inv_clight_sq ) ; } tmp_jx_fluid_arr(i, j, k) = q * (NUx_arr(i, j, k) / gamma); tmp_jy_fluid_arr(i, j, k) = q * (NUy_arr(i, j, k) / gamma); diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index b0afafa730b..b597e361dfa 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -231,12 +231,14 @@ struct InjectorMomentumUniform : m_ux_min(a_ux_min), m_uy_min(a_uy_min), m_uz_min(a_uz_min), m_ux_max(a_ux_max), m_uy_max(a_uy_max), m_uz_max(a_uz_max) { + using namespace amrex; + m_Dux = m_ux_max - m_ux_min; m_Duy = m_uy_max - m_uy_min; m_Duz = m_uz_max - m_uz_min; - m_ux_h = 0.5 * (m_ux_max + m_ux_min); - m_uy_h = 0.5 * (m_uy_max + m_uy_min); - m_uz_h = 0.5 * (m_uz_max + m_uz_min); + m_ux_h = 0.5_rt * (m_ux_max + m_ux_min); + m_uy_h = 0.5_rt * (m_uy_max + m_uy_min); + m_uz_h = 0.5_rt * (m_uz_max + m_uz_min); } AMREX_GPU_HOST_DEVICE diff --git a/Source/Initialization/PlasmaInjector.cpp b/Source/Initialization/PlasmaInjector.cpp index 56d29a8a1bb..0486f6e9537 100644 --- a/Source/Initialization/PlasmaInjector.cpp +++ b/Source/Initialization/PlasmaInjector.cpp @@ -487,7 +487,7 @@ void PlasmaInjector::setupExternalFile (const amrex::ParmParse& pp_species_name) ps["charge"][openPMD::RecordComponent::SCALAR].loadChunk(); m_openpmd_input_series->flush(); amrex::ParticleReal const p_q = p_q_ptr.get()[0]; - double const charge_unit = ps["charge"][openPMD::RecordComponent::SCALAR].unitSI(); + auto const charge_unit = static_cast(ps["charge"][openPMD::RecordComponent::SCALAR].unitSI()); charge = p_q * charge_unit; } if (mass_is_specified) { @@ -508,7 +508,7 @@ void PlasmaInjector::setupExternalFile (const amrex::ParmParse& pp_species_name) ps["mass"][openPMD::RecordComponent::SCALAR].loadChunk(); m_openpmd_input_series->flush(); amrex::ParticleReal const p_m = p_m_ptr.get()[0]; - double const mass_unit = ps["mass"][openPMD::RecordComponent::SCALAR].unitSI(); + auto const mass_unit = static_cast(ps["mass"][openPMD::RecordComponent::SCALAR].unitSI()); mass = p_m * mass_unit; } } // IOProcessor diff --git a/Source/Initialization/WarpXInitData.cpp b/Source/Initialization/WarpXInitData.cpp index bfac6db544d..4e82ed0dfb5 100644 --- a/Source/Initialization/WarpXInitData.cpp +++ b/Source/Initialization/WarpXInitData.cpp @@ -1415,27 +1415,27 @@ WarpX::ReadExternalFieldFromFile ( #endif const auto offset = F.gridGlobalOffset(); - const amrex::Real offset0 = offset[0]; - const amrex::Real offset1 = offset[1]; + const auto offset0 = static_cast(offset[0]); + const auto offset1 = static_cast(offset[1]); #if defined(WARPX_DIM_3D) - const amrex::Real offset2 = offset[2]; + const auto offset2 = static_cast(offset[2]); #endif const auto d = F.gridSpacing(); #if defined(WARPX_DIM_RZ) - const amrex::Real file_dr = d[0]; - const amrex::Real file_dz = d[1]; + const auto file_dr = static_cast(d[0]); + const auto file_dz = static_cast(d[1]); #elif defined(WARPX_DIM_3D) - const amrex::Real file_dx = d[0]; - const amrex::Real file_dy = d[1]; - const amrex::Real file_dz = d[2]; + const auto file_dx = static_cast(d[0]); + const auto file_dy = static_cast(d[1]); + const auto file_dz = static_cast(d[2]); #endif auto FC = F[F_component]; const auto extent = FC.getExtent(); - const int extent0 = extent[0]; - const int extent1 = extent[1]; - const int extent2 = extent[2]; + const auto extent0 = static_cast(extent[0]); + const auto extent1 = static_cast(extent[1]); + const auto extent2 = static_cast(extent[2]); // Determine the chunk data that will be loaded. // Now, the full range of data is loaded. @@ -1482,11 +1482,11 @@ WarpX::ReadExternalFieldFromFile ( // 0,1 denote r,z in 2D rz. amrex::Real x0, x1; if ( box.type(0)==amrex::IndexType::CellIndex::NODE ) - { x0 = real_box.lo(0) + ii*dx[0]; } - else { x0 = real_box.lo(0) + ii*dx[0] + 0.5*dx[0]; } + { x0 = static_cast(real_box.lo(0)) + ii*dx[0]; } + else { x0 = static_cast(real_box.lo(0)) + ii*dx[0] + 0.5_rt*dx[0]; } if ( box.type(1)==amrex::IndexType::CellIndex::NODE ) { x1 = real_box.lo(1) + j*dx[1]; } - else { x1 = real_box.lo(1) + j*dx[1] + 0.5*dx[1]; } + else { x1 = real_box.lo(1) + j*dx[1] + 0.5_rt*dx[1]; } #if defined(WARPX_DIM_RZ) // Get index of the external field array @@ -1501,7 +1501,7 @@ WarpX::ReadExternalFieldFromFile ( amrex::Real x2; if ( box.type(2)==amrex::IndexType::CellIndex::NODE ) { x2 = real_box.lo(2) + k*dx[2]; } - else { x2 = real_box.lo(2) + k*dx[2] + 0.5*dx[2]; } + else { x2 = real_box.lo(2) + k*dx[2] + 0.5_rt*dx[2]; } // Get index of the external field array int const ix = floor( (x0-offset0)/file_dx ); @@ -1521,10 +1521,10 @@ WarpX::ReadExternalFieldFromFile ( f01 = fc_array(0, iz , ir+1), f10 = fc_array(0, iz+1, ir ), f11 = fc_array(0, iz+1, ir+1); - mffab(i,j,k) = utils::algorithms::bilinear_interp + mffab(i,j,k) = static_cast(utils::algorithms::bilinear_interp (xx0, xx0+file_dr, xx1, xx1+file_dz, f00, f01, f10, f11, - x0, x1); + x0, x1)); #elif defined(WARPX_DIM_3D) const amrex::Array4 fc_array(FC_data, {0,0,0}, {extent2, extent1, extent0}, 1); const double @@ -1536,10 +1536,10 @@ WarpX::ReadExternalFieldFromFile ( f101 = fc_array(iz+1, iy , ix+1), f110 = fc_array(iz , iy+1, ix+1), f111 = fc_array(iz+1, iy+1, ix+1); - mffab(i,j,k) = utils::algorithms::trilinear_interp + mffab(i,j,k) = static_cast(utils::algorithms::trilinear_interp (xx0, xx0+file_dx, xx1, xx1+file_dy, xx2, xx2+file_dz, f000, f001, f010, f011, f100, f101, f110, f111, - x0, x1, x2); + x0, x1, x2)); #endif } diff --git a/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp b/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp index fbac9d9707f..714e4bd52e1 100644 --- a/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp +++ b/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp @@ -183,33 +183,33 @@ WarpXLaserProfiles::FromFileLaserProfile::parse_lasy_file(std::string lasy_file_ //Dimensions of lasy file data: {m,t,r} amrex::Print() << Utils::TextMsg::Info( "Found lasy file in RZ geometry" ); m_params.file_in_cartesian_geom = 0; - m_params.n_rz_azimuthal_components = extent[0]; - m_params.nt = extent[1]; - m_params.nr = extent[2]; + m_params.n_rz_azimuthal_components = static_cast(extent[0]); + m_params.nt = static_cast(extent[1]); + m_params.nr = static_cast(extent[2]); if(m_params.nt <= 1) WARPX_ABORT_WITH_MESSAGE("nt in lasy file must be >=2"); if(m_params.nr <= 1) WARPX_ABORT_WITH_MESSAGE("nr in lasy file must be >=2"); // Calculate the min and max of the grid - m_params.t_min = offset[0] + position[0]*spacing[0]; - m_params.t_max = m_params.t_min + (m_params.nt-1)*spacing[0]; - m_params.r_min = offset[1] + position[1]*spacing[1]; - m_params.r_max = m_params.r_min + (m_params.nr-1)*spacing[1]; + m_params.t_min = static_cast(offset[0] + position[0]*spacing[0]); + m_params.t_max = static_cast(m_params.t_min + (m_params.nt-1)*spacing[0]); + m_params.r_min = static_cast(offset[1] + position[1]*spacing[1]); + m_params.r_max = static_cast(m_params.r_min + (m_params.nr-1)*spacing[1]); } else if (fileGeom=="cartesian"){ //Dimensions of lasy file data: {t,y,x} amrex::Print() << Utils::TextMsg::Info( "Found lasy file in 3D cartesian geometry"); m_params.file_in_cartesian_geom = 1; - m_params.nt = extent[0]; - m_params.ny = extent[1]; - m_params.nx = extent[2]; + m_params.nt = static_cast(extent[0]); + m_params.ny = static_cast(extent[1]); + m_params.nx = static_cast(extent[2]); WARPX_ALWAYS_ASSERT_WITH_MESSAGE(m_params.nt > 1, "nt in lasy file must be >=2"); WARPX_ALWAYS_ASSERT_WITH_MESSAGE(m_params.nx > 1, "nx in lasy file must be >=2"); WARPX_ALWAYS_ASSERT_WITH_MESSAGE(m_params.ny > 1, "ny in lasy file must be >=2 in 3D"); // Calculate the min and max of the grid - m_params.t_min = offset[0] + position[0]*spacing[0]; - m_params.t_max = m_params.t_min + (m_params.nt-1)*spacing[0]; - m_params.y_min = offset[1] + position[1]*spacing[1]; - m_params.y_max = m_params.y_min + (m_params.ny-1)*spacing[1]; - m_params.x_min = offset[2] + position[2]*spacing[2]; - m_params.x_max = m_params.x_min + (m_params.nx-1)*spacing[2]; + m_params.t_min = static_cast(offset[0] + position[0]*spacing[0]); + m_params.t_max = static_cast(m_params.t_min + (m_params.nt-1)*spacing[0]); + m_params.y_min = static_cast(offset[1] + position[1]*spacing[1]); + m_params.y_max = static_cast(m_params.y_min + (m_params.ny-1)*spacing[1]); + m_params.x_min = static_cast(offset[2] + position[2]*spacing[2]); + m_params.x_max = static_cast(m_params.x_min + (m_params.nx-1)*spacing[2]); } else{ WARPX_ABORT_WITH_MESSAGE("The lasy file's geometry has to be in either RZ or 3D cartesian coordinates"); } @@ -269,11 +269,11 @@ WarpXLaserProfiles::FromFileLaserProfile::parse_binary_file (std::string binary_ dbuf_y.resize(1); #endif inp.read(reinterpret_cast(dbuf_t.dataPtr()), - dbuf_t.size()*sizeof(double)); + static_cast(dbuf_t.size()*sizeof(double))); inp.read(reinterpret_cast(dbuf_x.dataPtr()), - dbuf_x.size()*sizeof(double)); + static_cast(dbuf_x.size()*sizeof(double))); inp.read(reinterpret_cast(dbuf_y.dataPtr()), - dbuf_y.size()*sizeof(double)); + static_cast(dbuf_y.size()*sizeof(double))); if(!inp) WARPX_ABORT_WITH_MESSAGE("Failed to read coords from binary file"); m_params.t_min = static_cast(dbuf_t[0]); @@ -313,17 +313,15 @@ WarpXLaserProfiles::FromFileLaserProfile::read_data_t_chunk (int t_begin, int t_ { #ifdef WARPX_USE_OPENPMD //Indices of the first and last timestep to read - std::uint64_t const i_first = max(0, t_begin); - std::uint64_t const i_last = min(t_end-1, m_params.nt-1); + auto const i_first = static_cast(max(0, t_begin)); + auto const i_last = static_cast(min(t_end-1, m_params.nt-1)); amrex::Print() << Utils::TextMsg::Info( "Reading [" + std::to_string(i_first) + ", " + std::to_string(i_last) + "] data chunk from " + m_params.lasy_file_name); - int data_size; - if (m_params.file_in_cartesian_geom==0) { - data_size = m_params.n_rz_azimuthal_components*(i_last-i_first+1)*m_params.nr; - } else { - data_size = (i_last-i_first+1)*m_params.nx*m_params.ny; - } + const auto data_size = + (m_params.file_in_cartesian_geom==0)? + (m_params.n_rz_azimuthal_components*(i_last-i_first+1)*m_params.nr) : + (i_last-i_first+1)*m_params.nx*m_params.ny; m_params.E_lasy_data.resize(data_size); Vector h_E_lasy_data(m_params.E_lasy_data.size()); if(ParallelDescriptor::IOProcessor()){ @@ -335,10 +333,10 @@ WarpXLaserProfiles::FromFileLaserProfile::read_data_t_chunk (int t_begin, int t_ if (m_params.file_in_cartesian_geom==0) { const openPMD::Extent read_extent = { full_extent[0], (i_last - i_first + 1), full_extent[2]}; auto r_data = E_laser.loadChunk< std::complex >(io::Offset{ 0, i_first, 0}, read_extent); - const int read_size = (i_last - i_first + 1)*m_params.nr; + const auto read_size = (i_last - i_first + 1)*m_params.nr; series.flush(); for (int m=0; m(r_data.get()[j+m*read_size].real()), static_cast(r_data.get()[j+m*read_size].imag())}; @@ -347,9 +345,9 @@ WarpXLaserProfiles::FromFileLaserProfile::read_data_t_chunk (int t_begin, int t_ } else{ const openPMD::Extent read_extent = {(i_last - i_first + 1), full_extent[1], full_extent[2]}; auto x_data = E_laser.loadChunk< std::complex >(io::Offset{i_first, 0, 0}, read_extent); - const int read_size = (i_last - i_first + 1)*m_params.nx*m_params.ny; + const auto read_size = (i_last - i_first + 1)*m_params.nx*m_params.ny; series.flush(); - for (int j=0; j(x_data.get()[j].real()), static_cast(x_data.get()[j].imag())}; @@ -362,8 +360,8 @@ WarpXLaserProfiles::FromFileLaserProfile::read_data_t_chunk (int t_begin, int t_ Gpu::copyAsync(Gpu::hostToDevice,h_E_lasy_data.begin(),h_E_lasy_data.end(),m_params.E_lasy_data.begin()); Gpu::synchronize(); //Update first and last indices - m_params.first_time_index = i_first; - m_params.last_time_index = i_last; + m_params.first_time_index = static_cast(i_first); + m_params.last_time_index = static_cast(i_last); #else amrex::ignore_unused(t_begin, t_end); #endif @@ -402,12 +400,12 @@ WarpXLaserProfiles::FromFileLaserProfile::read_binary_data_t_chunk (int t_begin, 1*sizeof(double) + sizeof(double)*t_begin*m_params.nx*m_params.ny; #endif - inp.seekg(skip_amount); + inp.seekg(static_cast(skip_amount)); if(!inp) WARPX_ABORT_WITH_MESSAGE("Failed to read field data from binary file"); const int read_size = (i_last - i_first + 1)* m_params.nx*m_params.ny; Vector buf_e(read_size); - inp.read(reinterpret_cast(buf_e.dataPtr()), read_size*sizeof(double)); + inp.read(reinterpret_cast(buf_e.dataPtr()), static_cast(read_size*sizeof(double))); if(!inp) WARPX_ABORT_WITH_MESSAGE("Failed to read field data from binary file"); std::transform(buf_e.begin(), buf_e.end(), h_E_binary_data.begin(), [](auto x) {return static_cast(x);} ); @@ -421,8 +419,8 @@ WarpXLaserProfiles::FromFileLaserProfile::read_binary_data_t_chunk (int t_begin, Gpu::synchronize(); //Update first and last indices - m_params.first_time_index = i_first; - m_params.last_time_index = i_last; + m_params.first_time_index = static_cast(i_first); + m_params.last_time_index = static_cast(i_last); } void @@ -434,7 +432,7 @@ WarpXLaserProfiles::FromFileLaserProfile::internal_fill_amplitude_uniform_cartes { // Copy member variables to tmp copies // and get pointers to underlying data for GPU. - const amrex::Real omega_t = 2.*MathConst::pi*PhysConst::c*t/m_common_params.wavelength; + const amrex::Real omega_t = 2._rt*MathConst::pi*PhysConst::c*t/m_common_params.wavelength; const Complex exp_omega_t = Complex{ std::cos(-omega_t), std::sin(-omega_t) }; const auto tmp_x_min = m_params.x_min; const auto tmp_x_max = m_params.x_max; @@ -519,7 +517,7 @@ WarpXLaserProfiles::FromFileLaserProfile::internal_fill_amplitude_uniform_cylind { // Copy member variables to tmp copies // and get pointers to underlying data for GPU. - const amrex::Real omega_t = 2.*MathConst::pi*PhysConst::c*t/m_common_params.wavelength; + const amrex::Real omega_t = 2._rt*MathConst::pi*PhysConst::c*t/m_common_params.wavelength; const Complex exp_omega_t = Complex{ std::cos(-omega_t), std::sin(-omega_t) }; const auto tmp_r_min = m_params.r_min; const auto tmp_r_max = m_params.r_max; diff --git a/Source/Parallelization/GuardCellManager.cpp b/Source/Parallelization/GuardCellManager.cpp index a4547c25c6d..4d31cad9a1c 100644 --- a/Source/Parallelization/GuardCellManager.cpp +++ b/Source/Parallelization/GuardCellManager.cpp @@ -103,7 +103,7 @@ guardCellManager::Init ( if (do_moving_window) { WARPX_ALWAYS_ASSERT_WITH_MESSAGE(ref_ratios.size() <= 1, "The number of grow cells for the moving window currently assumes 2 levels max."); - const int nlevs = ref_ratios.size()+1; + const auto nlevs = static_cast(ref_ratios.size()+1); const int max_r = (nlevs > 1) ? ref_ratios[0][moving_window_dir] : 2; ngx = std::max(ngx,max_r); diff --git a/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp b/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp index 3531afb7240..ce7fed952a7 100644 --- a/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp +++ b/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp @@ -181,7 +181,8 @@ BackgroundMCCCollision::get_nu_max(amrex::Vector const& mcc_processe E_step = (energy_step < E_step) ? energy_step : E_step; } - for (amrex::ParticleReal E = E_start; E < E_end; E+=E_step) { + amrex::ParticleReal E = E_start; + while(E < E_end){ amrex::ParticleReal sigma_E = 0.0; // loop through all collision pathways @@ -199,6 +200,8 @@ BackgroundMCCCollision::get_nu_max(amrex::Vector const& mcc_processe if (nu > nu_max) { nu_max = nu; } + + E+=E_step; } return nu_max; } @@ -285,14 +288,14 @@ BackgroundMCCCollision::doCollisions (amrex::Real cur_time, amrex::Real dt, Mult { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); doBackgroundCollisionsWithinTile(pti, cur_time); if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[pti.index()], wt); } } @@ -322,7 +325,7 @@ void BackgroundMCCCollision::doBackgroundCollisionsWithinTile // get collision parameters auto scattering_processes = m_scattering_processes_exe.data(); - int const process_count = m_scattering_processes_exe.size(); + auto const process_count = static_cast(m_scattering_processes_exe.size()); auto const total_collision_prob = m_total_collision_prob; auto const nu_max = m_nu_max; @@ -379,7 +382,7 @@ void BackgroundMCCCollision::doBackgroundCollisionsWithinTile vy = uy[ip] - ua_y; vz = uz[ip] - ua_z; v_coll2 = (vx*vx + vy*vy + vz*vz); - v_coll = sqrt(v_coll2); + v_coll = std::sqrt(v_coll2); // calculate the collision energy in eV ParticleUtils::getCollisionEnergy(v_coll2, m, M, gamma, E_coll); @@ -389,7 +392,7 @@ void BackgroundMCCCollision::doBackgroundCollisionsWithinTile auto const& scattering_process = *(scattering_processes + i); // get collision cross-section - sigma_E = scattering_process.getCrossSection(E_coll); + sigma_E = scattering_process.getCrossSection(static_cast(E_coll)); // calculate normalized collision frequency nu_i += n_a * sigma_E * v_coll / nu_max; @@ -411,16 +414,17 @@ void BackgroundMCCCollision::doBackgroundCollisionsWithinTile // At this point the given particle has been chosen for a collision // and so we perform the needed calculations to transform to the // COM frame. - uCOM_x = m * vx / (gamma * m + M); - uCOM_y = m * vy / (gamma * m + M); - uCOM_z = m * vz / (gamma * m + M); + uCOM_x = static_cast(m * vx / (gamma * m + M)); + uCOM_y = static_cast(m * vy / (gamma * m + M)); + uCOM_z = static_cast(m * vz / (gamma * m + M)); // subtract any energy penalty of the collision from the // projectile energy if (scattering_process.m_energy_penalty > 0.0_prt) { ParticleUtils::getEnergy(v_coll2, m, E_coll); E_coll = (E_coll - scattering_process.m_energy_penalty) * PhysConst::q_e; - auto scale_fac = sqrt(E_coll * (E_coll + 2.0_prt*mc2) / c2) / m / v_coll; + const auto scale_fac = static_cast( + std::sqrt(E_coll * (E_coll + 2.0_prt*mc2) / c2) / m / v_coll); vx *= scale_fac; vy *= scale_fac; vz *= scale_fac; @@ -484,7 +488,7 @@ void BackgroundMCCCollision::doBackgroundIonization { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto& elec_tile = species1.ParticlesAt(lev, pti); auto& ion_tile = species2.ParticlesAt(lev, pti); @@ -508,7 +512,7 @@ void BackgroundMCCCollision::doBackgroundIonization if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[pti.index()], wt); } } diff --git a/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H b/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H index 6e9827d8ab3..88ec7693b4a 100644 --- a/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H +++ b/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H @@ -98,7 +98,7 @@ public: ParticleUtils::getEnergy(u_coll2, m_mass, E_coll); // get collision cross-section - const ParticleReal sigma_E = m_mcc_process.getCrossSection(E_coll); + const ParticleReal sigma_E = m_mcc_process.getCrossSection(static_cast(E_coll)); // calculate normalized collision frequency const ParticleReal nu_i = n_a * sigma_E * sqrt(u_coll2) / m_nu_max; @@ -182,7 +182,7 @@ public: // calculate standard deviation in neutral velocity distribution using // the local temperature - const ParticleReal ion_vel_std = m_sqrt_kb_m * sqrt(m_T_a_func(x, y, z, m_t)); + const ParticleReal ion_vel_std = m_sqrt_kb_m * std::sqrt(m_T_a_func(x, y, z, m_t)); // get references to the original particle's velocity auto& ux = src.m_rdata[PIdx::ux][i_src]; @@ -202,7 +202,7 @@ public: ParticleUtils::getEnergy(u_coll2, m_mass1, E_coll); // each electron gets half the energy (could change this later) - const amrex::ParticleReal E_out = (E_coll - m_energy_cost) / 2.0_prt * PhysConst::q_e; + const amrex::ParticleReal E_out = static_cast((E_coll - m_energy_cost) / 2.0_prt * PhysConst::q_e); // precalculate often used value constexpr auto c2 = PhysConst::c * PhysConst::c; diff --git a/Source/Particles/Collision/BackgroundMCC/MCCProcess.cpp b/Source/Particles/Collision/BackgroundMCC/MCCProcess.cpp index 0451f62be60..bf2138bc78b 100644 --- a/Source/Particles/Collision/BackgroundMCC/MCCProcess.cpp +++ b/Source/Particles/Collision/BackgroundMCC/MCCProcess.cpp @@ -40,7 +40,7 @@ MCCProcess::init (const std::string& scattering_process, const amrex::ParticleRe m_exe_h.m_sigmas_data = m_sigmas_h.data(); // save energy grid parameters for easy use - m_grid_size = m_energies.size(); + m_grid_size = static_cast(m_energies.size()); m_exe_h.m_energy_lo = m_energies[0]; m_exe_h.m_energy_hi = m_energies[m_grid_size-1]; m_exe_h.m_sigma_lo = m_sigmas_h[0]; diff --git a/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp b/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp index e832020683d..878ef8a1f64 100644 --- a/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp +++ b/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp @@ -115,7 +115,7 @@ BackgroundStopping::doCollisions (amrex::Real cur_time, amrex::Real dt, MultiPar { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); if (background_type == BackgroundStoppingType::ELECTRONS) { doBackgroundStoppingOnElectronsWithinTile(pti, dt, cur_time, species_mass, species_charge); @@ -126,7 +126,7 @@ BackgroundStopping::doCollisions (amrex::Real cur_time, amrex::Real dt, MultiPar if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add(&(*cost)[pti.index()], wt); } } diff --git a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H index 9cf2f91b5aa..6728e4cef53 100644 --- a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H +++ b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H @@ -178,7 +178,7 @@ public: { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); doCollisionsWithinTile( dt, lev, mfi, species1, species2, product_species_vector, copy_species1_data, copy_species2_data); @@ -186,7 +186,7 @@ public: if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -247,7 +247,7 @@ public: // Loop over cells, and collide the particles in each cell // Extract low-level data - int const n_cells = bins_1.numBins(); + auto const n_cells = static_cast(bins_1.numBins()); // - Species 1 const auto soa_1 = ptile_1.getParticleTileData(); index_type* AMREX_RESTRICT indices_1 = bins_1.permutationPtr(); @@ -371,7 +371,7 @@ public: for (int i = 0; i < n_product_species; i++) { - setNewParticleIDs(*(tile_products_data[i]), products_np[i], num_added[i]); + setNewParticleIDs(*(tile_products_data[i]), static_cast(products_np[i]), num_added[i]); } } else // species_1 != species_2 @@ -387,7 +387,7 @@ public: // Loop over cells, and collide the particles in each cell // Extract low-level data - int const n_cells = bins_1.numBins(); + auto const n_cells = static_cast(bins_1.numBins()); // - Species 1 const auto soa_1 = ptile_1.getParticleTileData(); index_type* AMREX_RESTRICT indices_1 = bins_1.permutationPtr(); @@ -532,7 +532,7 @@ public: for (int i = 0; i < n_product_species; i++) { - setNewParticleIDs(*(tile_products_data[i]), products_np[i], num_added[i]); + setNewParticleIDs(*(tile_products_data[i]), static_cast(products_np[i]), num_added[i]); } } // end if ( m_isSameSpecies) diff --git a/Source/Particles/Collision/BinaryCollision/Coulomb/ElasticCollisionPerez.H b/Source/Particles/Collision/BinaryCollision/Coulomb/ElasticCollisionPerez.H index 239f63c7f23..b54404d46ba 100644 --- a/Source/Particles/Collision/BinaryCollision/Coulomb/ElasticCollisionPerez.H +++ b/Source/Particles/Collision/BinaryCollision/Coulomb/ElasticCollisionPerez.H @@ -54,8 +54,8 @@ void ElasticCollisionPerez ( amrex::RandomEngine const& engine, bool const isSameSpecies) { - const int NI1 = I1e - I1s; - const int NI2 = I2e - I2s; + const T_index NI1 = I1e - I1s; + const T_index NI2 = I2e - I2s; T_PR * const AMREX_RESTRICT w1 = soa_1.m_rdata[PIdx::w]; T_PR * const AMREX_RESTRICT u1x = soa_1.m_rdata[PIdx::ux]; @@ -84,8 +84,8 @@ void ElasticCollisionPerez ( T_PR n1 = T_PR(0.0); T_PR n2 = T_PR(0.0); T_PR n12 = T_PR(0.0); - for (int i1=I1s; i1(I1e); ++i1) { n1 += w1[ I1[i1] ]; } - for (int i2=I2s; i2(I2e); ++i2) { n2 += w2[ I2[i2] ]; } + for (T_index i1=I1s; i1(I1e) ) { i1 = I1s; } - ++i2; if ( i2 == static_cast(I2e) ) { i2 = I2s; } + ++i1; if ( i1 == I1e ) { i1 = I1s; } + ++i2; if ( i2 == I2e ) { i2 = I2s; } } n12 = n12 / dV; } @@ -126,8 +126,8 @@ void ElasticCollisionPerez ( // call UpdateMomentumPerezElastic() { - int i1 = I1s; int i2 = I2s; - for (int k = 0; k < amrex::max(NI1,NI2); ++k) + T_index i1 = I1s; T_index i2 = I2s; + for (T_index k = 0; k < amrex::max(NI1,NI2); ++k) { #if (defined WARPX_DIM_RZ) @@ -159,8 +159,8 @@ void ElasticCollisionPerez ( u1y[I1[i1]] = u1xbuf_new*std::sin(-theta) + u1y[I1[i1]]*std::cos(-theta); #endif - ++i1; if ( i1 == static_cast(I1e) ) { i1 = I1s; } - ++i2; if ( i2 == static_cast(I2e) ) { i2 = I2s; } + ++i1; if ( i1 == I1e ) { i1 = I1s; } + ++i2; if ( i2 == I2e ) { i2 = I2s; } } } diff --git a/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H b/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H index 23939bbc2ab..db8dacce1d4 100644 --- a/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H +++ b/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H @@ -144,36 +144,37 @@ public: amrex::ParticleReal * const AMREX_RESTRICT u2z = soa_2.m_rdata[PIdx::uz]; // Number of macroparticles of each species - const int NI1 = I1e - I1s; - const int NI2 = I2e - I2s; - const int max_N = amrex::max(NI1,NI2); + const index_type NI1 = I1e - I1s; + const index_type NI2 = I2e - I2s; + const index_type max_N = amrex::max(NI1,NI2); - int i1 = I1s; - int i2 = I2s; - int pair_index = cell_start_pair; + index_type i1 = I1s; + index_type i2 = I2s; + index_type pair_index = cell_start_pair; // Because the number of particles of each species is not always equal (NI1 != NI2 // in general), some macroparticles will be paired with multiple macroparticles of the // other species and we need to decrease their weight accordingly. // c1 corresponds to the minimum number of times a particle of species 1 will be paired // with a particle of species 2. Same for c2. - const int c1 = amrex::max(NI2/NI1,1); - const int c2 = amrex::max(NI1/NI2,1); + const index_type c1 = amrex::max(NI2/NI1,1u); + const index_type c2 = amrex::max(NI1/NI2,1u); // multiplier ratio to take into account unsampled pairs - const int multiplier_ratio = (m_isSameSpecies)?(2*max_N - 1):(max_N); + const auto multiplier_ratio = static_cast( + (m_isSameSpecies)?(2u*max_N - 1):(max_N)); #if (defined WARPX_DIM_RZ) amrex::ParticleReal * const AMREX_RESTRICT theta1 = soa_1.m_rdata[PIdx::theta]; amrex::ParticleReal * const AMREX_RESTRICT theta2 = soa_2.m_rdata[PIdx::theta]; #endif - for (int k = 0; k < max_N; ++k) + for (index_type k = 0; k < max_N; ++k) { // c1k : how many times the current particle of species 1 is paired with a particle // of species 2. Same for c2k. - const int c1k = (k%NI1 < max_N%NI1) ? c1 + 1: c1; - const int c2k = (k%NI2 < max_N%NI2) ? c2 + 1: c2; + const index_type c1k = (k%NI1 < max_N%NI1) ? c1 + 1: c1; + const index_type c2k = (k%NI2 < max_N%NI2) ? c2 + 1: c2; #if (defined WARPX_DIM_RZ) /* In RZ geometry, macroparticles can collide with other macroparticles @@ -194,7 +195,7 @@ public: u1x[ I1[i1] ], u1y[ I1[i1] ], u1z[ I1[i1] ], u2x[ I2[i2] ], u2y[ I2[i2] ], u2z[ I2[i2] ], m1, m2, w1[ I1[i1] ]/c1k, w2[ I2[i2] ]/c2k, - dt, dV, pair_index, p_mask, p_pair_reaction_weight, + dt, dV, static_cast(pair_index), p_mask, p_pair_reaction_weight, m_fusion_multiplier, multiplier_ratio, m_probability_threshold, m_probability_target_value, @@ -208,8 +209,8 @@ public: p_pair_indices_1[pair_index] = I1[i1]; p_pair_indices_2[pair_index] = I2[i2]; - ++i1; if ( i1 == static_cast(I1e) ) { i1 = I1s; } - ++i2; if ( i2 == static_cast(I2e) ) { i2 = I2s; } + ++i1; if ( i1 == I1e ) { i1 = I1s; } + ++i2; if ( i2 == I2e ) { i2 = I2s; } ++pair_index; } diff --git a/Source/Particles/Collision/BinaryCollision/NuclearFusion/ProtonBoronFusionInitializeMomentum.H b/Source/Particles/Collision/BinaryCollision/NuclearFusion/ProtonBoronFusionInitializeMomentum.H index 085fc597f58..0e92bda5beb 100644 --- a/Source/Particles/Collision/BinaryCollision/NuclearFusion/ProtonBoronFusionInitializeMomentum.H +++ b/Source/Particles/Collision/BinaryCollision/NuclearFusion/ProtonBoronFusionInitializeMomentum.H @@ -97,20 +97,20 @@ namespace { soa_2.m_rdata[PIdx::ux][idx_2], soa_2.m_rdata[PIdx::uy][idx_2], soa_2.m_rdata[PIdx::uz][idx_2], m2, - ux_alpha1, uy_alpha1, uz_alpha1, m_alpha, - ux_Be, uy_Be, uz_Be, m_beryllium, + ux_alpha1, uy_alpha1, uz_alpha1, static_cast(m_alpha), + ux_Be, uy_Be, uz_Be, static_cast(m_beryllium), E_fusion, engine); // Compute momentum of beryllium in lab frame - const amrex::ParticleReal px_Be = m_beryllium * ux_Be; - const amrex::ParticleReal py_Be = m_beryllium * uy_Be; - const amrex::ParticleReal pz_Be = m_beryllium * uz_Be; + const amrex::ParticleReal px_Be = static_cast(m_beryllium) * ux_Be; + const amrex::ParticleReal py_Be = static_cast(m_beryllium) * uy_Be; + const amrex::ParticleReal pz_Be = static_cast(m_beryllium) * uz_Be; // Compute momentum norm of second and third alphas in Beryllium rest frame // Factor 0.5 is here because each alpha only gets half of the decay energy constexpr amrex::ParticleReal gamma_Bestar = (1._prt + 0.5_prt*E_decay/(m_alpha*c_sq)); constexpr amrex::ParticleReal gamma_Bestar_sq_minus_one = gamma_Bestar*gamma_Bestar - 1._prt; - const amrex::ParticleReal p_Bestar = m_alpha*PhysConst::c*std::sqrt(gamma_Bestar_sq_minus_one); + const amrex::ParticleReal p_Bestar = static_cast(m_alpha)*PhysConst::c*std::sqrt(gamma_Bestar_sq_minus_one); // Compute momentum of second alpha in Beryllium rest frame, assuming isotropic distribution amrex::ParticleReal px_Bestar, py_Bestar, pz_Bestar; @@ -120,8 +120,8 @@ namespace { amrex::ParticleReal px_alpha2, py_alpha2, pz_alpha2; // Preliminary calculation: compute Beryllium velocity v_Be const amrex::ParticleReal p_Be_sq = px_Be*px_Be + py_Be*py_Be + pz_Be*pz_Be; - const amrex::ParticleReal g_Be = std::sqrt(1._prt + p_Be_sq / (mBe_sq*c_sq)); - const amrex::ParticleReal mg_Be = m_beryllium*g_Be; + const amrex::ParticleReal g_Be = std::sqrt(1._prt + p_Be_sq / (static_cast(mBe_sq)*c_sq)); + const amrex::ParticleReal mg_Be = static_cast(m_beryllium)*g_Be; const amrex::ParticleReal v_Bex = px_Be / mg_Be; const amrex::ParticleReal v_Bey = py_Be / mg_Be; const amrex::ParticleReal v_Bez = pz_Be / mg_Be; @@ -133,7 +133,7 @@ namespace { { const amrex::ParticleReal vcDps = v_Bex*px_Bestar + v_Bey*py_Bestar + v_Bez*pz_Bestar; const amrex::ParticleReal factor0 = (g_Be-1._prt)/v_Be_sq; - const amrex::ParticleReal factor = factor0*vcDps + m_alpha*gamma_Bestar*g_Be; + const amrex::ParticleReal factor = factor0*vcDps + static_cast(m_alpha)*gamma_Bestar*g_Be; px_alpha2 = px_Bestar + v_Bex * factor; py_alpha2 = py_Bestar + v_Bey * factor; pz_alpha2 = pz_Bestar + v_Bez * factor; @@ -159,18 +159,18 @@ namespace { soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 1] = ux_alpha1; soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 1] = uy_alpha1; soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 1] = uz_alpha1; - soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 2] = px_alpha2/m_alpha; - soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 2] = py_alpha2/m_alpha; - soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 2] = pz_alpha2/m_alpha; - soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 3] = px_alpha2/m_alpha; - soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 3] = py_alpha2/m_alpha; - soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 3] = pz_alpha2/m_alpha; - soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 4] = px_alpha3/m_alpha; - soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 4] = py_alpha3/m_alpha; - soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 4] = pz_alpha3/m_alpha; - soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 5] = px_alpha3/m_alpha; - soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 5] = py_alpha3/m_alpha; - soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 5] = pz_alpha3/m_alpha; + soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 2] = px_alpha2/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 2] = py_alpha2/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 2] = pz_alpha2/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 3] = px_alpha2/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 3] = py_alpha2/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 3] = pz_alpha2/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 4] = px_alpha3/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 4] = py_alpha3/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 4] = pz_alpha3/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::ux][idx_alpha_start + 5] = px_alpha3/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uy][idx_alpha_start + 5] = py_alpha3/static_cast(m_alpha); + soa_alpha.m_rdata[PIdx::uz][idx_alpha_start + 5] = pz_alpha3/static_cast(m_alpha); } } diff --git a/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H b/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H index 7fdf04de6ca..8d85dab11d7 100644 --- a/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H +++ b/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H @@ -130,7 +130,7 @@ public: // one electron, we create two electrons, one at the position of each particle that // collided. This allows for exact charge conservation. const index_type num_added = total * m_num_products_host[i] * 2; - num_added_vec[i] = num_added; + num_added_vec[i] = static_cast(num_added); tile_products[i]->resize(products_np[i] + num_added); } @@ -184,11 +184,11 @@ public: const auto product_index = products_np_data[j] + 2*(p_offsets[i]*p_num_products_device[j] + k); // Create product particle at position of particle 1 - copy_species1[j](soa_products_data[j], soa_1, p_pair_indices_1[i], - product_index, engine); + copy_species1[j](soa_products_data[j], soa_1, static_cast(p_pair_indices_1[i]), + static_cast(product_index), engine); // Create another product particle at position of particle 2 - copy_species2[j](soa_products_data[j], soa_2, p_pair_indices_2[i], - product_index + 1, engine); + copy_species2[j](soa_products_data[j], soa_2, static_cast(p_pair_indices_2[i]), + static_cast(product_index + 1), engine); // Set the weight of the new particles to p_pair_reaction_weight[i]/2 soa_products_data[j].m_rdata[PIdx::w][product_index] = diff --git a/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp b/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp index 12f479d32ed..911a5c64ea0 100644 --- a/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp +++ b/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp @@ -72,10 +72,10 @@ BreitWheelerEngine::init_lookup_tables_from_raw_data ( if(size_first <= 0 || size_first >= raw_data.size() ) return false; const auto raw_dndt_table = vector{ - raw_iter, raw_iter+size_first}; + raw_iter, raw_iter+static_cast(size_first)}; const auto raw_pair_prod_table = vector{ - raw_iter+size_first, raw_data.end()}; + raw_iter+static_cast(size_first), raw_data.end()}; m_dndt_table = BW_dndt_table{raw_dndt_table}; m_pair_prod_table = BW_pair_prod_table{raw_pair_prod_table}; diff --git a/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp b/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp index c9bede89167..180bd5babce 100644 --- a/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp +++ b/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp @@ -71,10 +71,10 @@ QuantumSynchrotronEngine::init_lookup_tables_from_raw_data ( if(size_first <= 0 || size_first >= raw_data.size() ) return false; const auto raw_dndt_table = vector{ - raw_iter, raw_iter+size_first}; + raw_iter, raw_iter+static_cast(size_first)}; const auto raw_phot_em_table = vector{ - raw_iter+size_first, raw_data.end()}; + raw_iter+static_cast(size_first), raw_data.end()}; m_dndt_table = QS_dndt_table{raw_dndt_table}; m_phot_em_table = QS_phot_em_table{raw_phot_em_table}; diff --git a/Source/Particles/Gather/GetExternalFields.H b/Source/Particles/Gather/GetExternalFields.H index 5fd0acb8eb2..2f388363e17 100644 --- a/Source/Particles/Gather/GetExternalFields.H +++ b/Source/Particles/Gather/GetExternalFields.H @@ -27,7 +27,7 @@ struct GetExternalEBField GetExternalEBField () = default; - GetExternalEBField (const WarpXParIter& a_pti, int a_offset = 0) noexcept; + GetExternalEBField (const WarpXParIter& a_pti, long a_offset = 0) noexcept; ExternalFieldInitType m_Etype; ExternalFieldInitType m_Btype; diff --git a/Source/Particles/Gather/GetExternalFields.cpp b/Source/Particles/Gather/GetExternalFields.cpp index c6d94686c06..057ac51d597 100644 --- a/Source/Particles/Gather/GetExternalFields.cpp +++ b/Source/Particles/Gather/GetExternalFields.cpp @@ -13,7 +13,7 @@ using namespace amrex::literals; -GetExternalEBField::GetExternalEBField (const WarpXParIter& a_pti, int a_offset) noexcept +GetExternalEBField::GetExternalEBField (const WarpXParIter& a_pti, long a_offset) noexcept { auto& warpx = WarpX::GetInstance(); auto& mypc = warpx.GetPartContainer(); @@ -22,7 +22,7 @@ GetExternalEBField::GetExternalEBField (const WarpXParIter& a_pti, int a_offset) AcceleratorLattice const & accelerator_lattice = warpx.get_accelerator_lattice(lev); if (accelerator_lattice.m_lattice_defined) { - d_lattice_element_finder = accelerator_lattice.GetFinderDeviceInstance(a_pti, a_offset); + d_lattice_element_finder = accelerator_lattice.GetFinderDeviceInstance(a_pti, static_cast(a_offset)); } m_gamma_boost = WarpX::gamma_boost; diff --git a/Source/Particles/LaserParticleContainer.cpp b/Source/Particles/LaserParticleContainer.cpp index ff11d1a30e9..098dba0997b 100644 --- a/Source/Particles/LaserParticleContainer.cpp +++ b/Source/Particles/LaserParticleContainer.cpp @@ -487,7 +487,8 @@ LaserParticleContainer::InitData (int lev) const DistributionMapping plane_dm {plane_ba, nprocs}; const Vector& procmap = plane_dm.ProcessorMap(); - for (int i = 0, n = plane_ba.size(); i < n; ++i) + const auto plane_ba_size = static_cast(plane_ba.size()); + for (int i = 0; i < plane_ba_size; ++i) { if (procmap[i] == myproc) { @@ -518,15 +519,16 @@ LaserParticleContainer::InitData (int lev) particle_w.push_back(-m_weight); #else // Particles are laid out in radial spokes - const int n_spokes = (WarpX::n_rz_azimuthal_modes - 1)*m_min_particles_per_mode; + const auto n_spokes = + static_cast((WarpX::n_rz_azimuthal_modes - 1)*m_min_particles_per_mode); for (int spoke = 0 ; spoke < n_spokes ; spoke++) { - const Real phase = 2.*MathConst::pi*spoke/n_spokes; + const Real phase = 2._rt*MathConst::pi*spoke/n_spokes; for (int k = 0; k<2; ++k) { particle_x.push_back(pos[0]*std::cos(phase)); particle_y.push_back(pos[0]*std::sin(phase)); particle_z.push_back(pos[2]); } - const Real r_weight = m_weight*2.*MathConst::pi*pos[0]/n_spokes; + const Real r_weight = m_weight*2._rt*MathConst::pi*pos[0]/n_spokes; particle_w.push_back( r_weight); particle_w.push_back(-r_weight); } @@ -535,7 +537,7 @@ LaserParticleContainer::InitData (int lev) } } } - const int np = particle_z.size(); + const auto np = static_cast(particle_z.size()); amrex::Vector particle_ux(np, 0.0); amrex::Vector particle_uy(np, 0.0); amrex::Vector particle_uz(np, 0.0); @@ -637,18 +639,18 @@ LaserParticleContainer::Evolve (int lev, // WARPX_PROFILE_VAR_START(blp_pp); // Find the coordinates of the particles in the emission plane - calculate_laser_plane_coordinates(pti, np, + calculate_laser_plane_coordinates(pti, static_cast(np), plane_Xp.dataPtr(), plane_Yp.dataPtr()); // Calculate the laser amplitude to be emitted, // at the position of the emission plane m_up_laser_profile->fill_amplitude( - np, plane_Xp.dataPtr(), plane_Yp.dataPtr(), + static_cast(np), plane_Xp.dataPtr(), plane_Yp.dataPtr(), t_lab, amplitude_E.dataPtr()); // Calculate the corresponding momentum and position for the particles - update_laser_particle(pti, np, uxp.dataPtr(), uyp.dataPtr(), + update_laser_particle(pti, static_cast(np), uxp.dataPtr(), uyp.dataPtr(), uzp.dataPtr(), wp.dataPtr(), amplitude_E.dataPtr(), dt); WARPX_PROFILE_VAR_STOP(blp_pp); diff --git a/Source/Particles/MultiParticleContainer.H b/Source/Particles/MultiParticleContainer.H index af826068806..16817cbec07 100644 --- a/Source/Particles/MultiParticleContainer.H +++ b/Source/Particles/MultiParticleContainer.H @@ -229,9 +229,9 @@ public: void SetParticleBoxArray (int lev, amrex::BoxArray& new_ba); void SetParticleDistributionMap (int lev, amrex::DistributionMapping& new_dm); - int nSpecies () const {return species_names.size();} - int nLasers () const {return lasers_names.size();} - int nContainers () const {return allcontainers.size();} + int nSpecies () const {return static_cast(species_names.size());} + int nLasers () const {return static_cast(lasers_names.size());} + int nContainers () const {return static_cast(allcontainers.size());} /** Whether back-transformed diagnostics need to be performed for any plasma species. * @@ -248,13 +248,13 @@ public: int nSpeciesDepositOnMainGrid () const { bool const onMainGrid = true; auto const & v = m_deposit_on_main_grid; - return std::count( v.begin(), v.end(), onMainGrid ); + return static_cast(std::count( v.begin(), v.end(), onMainGrid )); } int nSpeciesGatherFromMainGrid() const { bool const fromMainGrid = true; auto const & v = m_gather_from_main_grid; - return std::count( v.begin(), v.end(), fromMainGrid ); + return static_cast(std::count( v.begin(), v.end(), fromMainGrid )); } // Inject particles during the simulation (for particles entering the diff --git a/Source/Particles/MultiParticleContainer.cpp b/Source/Particles/MultiParticleContainer.cpp index b9802fa448b..67dfda1dbe0 100644 --- a/Source/Particles/MultiParticleContainer.cpp +++ b/Source/Particles/MultiParticleContainer.cpp @@ -294,7 +294,7 @@ MultiParticleContainer::ReadParameters () it != species_names.end(), "species '" + name + "' in particles.deposit_on_main_grid must be part of particles.species_names"); - const int i = std::distance(species_names.begin(), it); + const auto i = static_cast(std::distance(species_names.begin(), it)); m_deposit_on_main_grid[i] = true; } @@ -307,7 +307,7 @@ MultiParticleContainer::ReadParameters () it != species_names.end(), "species '" + name + "' in particles.gather_from_main_grid must be part of particles.species_names"); - const int i = std::distance(species_names.begin(), it); + const auto i = static_cast(std::distance(species_names.begin(), it)); m_gather_from_main_grid.at(i) = true; } @@ -323,7 +323,7 @@ MultiParticleContainer::ReadParameters () it != species_names.end(), "species '" + name + "' in particles.rigid_injected_species must be part of particles.species_names"); - const int i = std::distance(species_names.begin(), it); + const auto i = static_cast(std::distance(species_names.begin(), it)); species_types[i] = PCTypes::RigidInjected; } } @@ -337,7 +337,7 @@ MultiParticleContainer::ReadParameters () it != species_names.end(), "species '" + name + "' in particles.photon_species must be part of particles.species_names"); - const int i = std::distance(species_names.begin(), it); + const auto i = static_cast(std::distance(species_names.begin(), it)); species_types[i] = PCTypes::Photon; } } @@ -366,7 +366,7 @@ MultiParticleContainer::ReadParameters () it != lasers_names.end(), "laser '" + name + "' in lasers.deposit_on_main_grid must be part of lasers.lasers_names"); - const int i = std::distance(lasers_names.begin(), it); + const auto i = static_cast(std::distance(lasers_names.begin(), it)); m_laser_deposit_on_main_grid[i] = true; } @@ -413,7 +413,7 @@ MultiParticleContainer::GetParticleContainerFromName (const std::string& name) c WARPX_ALWAYS_ASSERT_WITH_MESSAGE( it != species_names.end(), "unknown species name"); - const int i = std::distance(species_names.begin(), it); + const auto i = static_cast(std::distance(species_names.begin(), it)); return *allcontainers[i]; } @@ -671,7 +671,7 @@ Vector MultiParticleContainer::GetZeroParticlesInGrid (const int lev) const { const WarpX& warpx = WarpX::GetInstance(); - const int num_boxes = warpx.boxArray(lev).size(); + const auto num_boxes = static_cast(warpx.boxArray(lev).size()); Vector r(num_boxes, 0); return r; } @@ -694,7 +694,7 @@ MultiParticleContainer::NumberOfParticlesInGrid (int lev) const r[j] += ri[j]; } } - ParallelDescriptor::ReduceLongSum(r.data(),r.size()); + ParallelDescriptor::ReduceLongSum(r.data(),static_cast(r.size())); return r; } } @@ -910,7 +910,7 @@ MultiParticleContainer::doFieldIonization (int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto& src_tile = pc_source ->ParticlesAt(lev, pti); auto& dst_tile = pc_product->ParticlesAt(lev, pti); @@ -928,7 +928,7 @@ MultiParticleContainer::doFieldIonization (int lev, if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[pti.index()], wt); } } @@ -1542,7 +1542,7 @@ void MultiParticleContainer::doQedBreitWheeler (int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto Transform = PairGenerationTransformFunc(pair_gen_functor, pti, lev, Ex.nGrowVect(), @@ -1566,7 +1566,7 @@ void MultiParticleContainer::doQedBreitWheeler (int lev, if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[pti.index()], wt); } } @@ -1617,7 +1617,7 @@ void MultiParticleContainer::doQedQuantumSync (int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto Transform = PhotonEmissionTransformFunc( m_shr_p_qs_engine->build_optical_depth_functor(), @@ -1645,7 +1645,7 @@ void MultiParticleContainer::doQedQuantumSync (int lev, if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[pti.index()], wt); } } diff --git a/Source/Particles/ParticleBoundaryBuffer.H b/Source/Particles/ParticleBoundaryBuffer.H index 7099ef4d691..cbc8667c8d4 100644 --- a/Source/Particles/ParticleBoundaryBuffer.H +++ b/Source/Particles/ParticleBoundaryBuffer.H @@ -35,7 +35,7 @@ public: /** Move operator for NamedComponentParticleContainer */ ParticleBoundaryBuffer& operator= ( ParticleBoundaryBuffer && ) = default; - int numSpecies() const { return getSpeciesNames().size(); } + int numSpecies() const { return static_cast(getSpeciesNames().size()); } const std::vector& getSpeciesNames() const; diff --git a/Source/Particles/ParticleBoundaryBuffer.cpp b/Source/Particles/ParticleBoundaryBuffer.cpp index 70c9459df5e..8e6decb1f3f 100644 --- a/Source/Particles/ParticleBoundaryBuffer.cpp +++ b/Source/Particles/ParticleBoundaryBuffer.cpp @@ -149,7 +149,7 @@ void ParticleBoundaryBuffer::printNumParticles () const { auto& buffer = m_particle_containers[2*idim+iside]; for (int i = 0; i < numSpecies(); ++i) { - const int np = buffer[i].isDefined() ? buffer[i].TotalNumberOfParticles(false) : 0; + const auto np = buffer[i].isDefined() ? buffer[i].TotalNumberOfParticles(false) : 0; amrex::Print() << Utils::TextMsg::Info( "Species " + getSpeciesNames()[i] + " has " + std::to_string(np) + " particles in the boundary buffer " @@ -162,7 +162,7 @@ void ParticleBoundaryBuffer::printNumParticles () const { auto& buffer = m_particle_containers[2*AMREX_SPACEDIM]; for (int i = 0; i < numSpecies(); ++i) { - const int np = buffer[i].isDefined() ? buffer[i].TotalNumberOfParticles(false) : 0; + const auto np = buffer[i].isDefined() ? buffer[i].TotalNumberOfParticles(false) : 0; amrex::Print() << Utils::TextMsg::Info( "Species " + getSpeciesNames()[i] + " has " + std::to_string(np) + " particles in the EB boundary buffer" @@ -370,8 +370,12 @@ int ParticleBoundaryBuffer::getNumParticlesInContainer( auto& buffer = m_particle_containers[boundary]; auto index = WarpX::GetInstance().GetPartContainer().getSpeciesID(species_name); - if (buffer[index].isDefined()) return buffer[index].TotalNumberOfParticles(false, local); - else return 0; + if (buffer[index].isDefined()){ + return static_cast(buffer[index].TotalNumberOfParticles(false, local)); + } + else{ + return 0; + } } PinnedMemoryParticleContainer & diff --git a/Source/Particles/ParticleCreation/SmartUtils.H b/Source/Particles/ParticleCreation/SmartUtils.H index ee1dc321cef..4604ef59680 100644 --- a/Source/Particles/ParticleCreation/SmartUtils.H +++ b/Source/Particles/ParticleCreation/SmartUtils.H @@ -31,7 +31,7 @@ struct SmartCopyTag amrex::Gpu::DeviceVector src_comps; amrex::Gpu::DeviceVector dst_comps; - int size () const noexcept { return common_names.size(); } + int size () const noexcept { return static_cast(common_names.size()); } }; PolicyVec getPolicies (const NameMap& names) noexcept; diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp index 1358011b9b0..f515b750a3e 100644 --- a/Source/Particles/PhysicalParticleContainer.cpp +++ b/Source/Particles/PhysicalParticleContainer.cpp @@ -304,7 +304,7 @@ PhysicalParticleContainer::PhysicalParticleContainer (AmrCore* amr_core, int isp // User-defined integer attributes pp_species_name.queryarr("addIntegerAttributes", m_user_int_attribs); - const int n_user_int_attribs = m_user_int_attribs.size(); + const auto n_user_int_attribs = static_cast(m_user_int_attribs.size()); std::vector< std::string > str_int_attrib_function; str_int_attrib_function.resize(n_user_int_attribs); m_user_int_attrib_parser.resize(n_user_int_attribs); @@ -319,7 +319,7 @@ PhysicalParticleContainer::PhysicalParticleContainer (AmrCore* amr_core, int isp // User-defined real attributes pp_species_name.queryarr("addRealAttributes", m_user_real_attribs); - const int n_user_real_attribs = m_user_real_attribs.size(); + const auto n_user_real_attribs = static_cast(m_user_real_attribs.size()); std::vector< std::string > str_real_attrib_function; str_real_attrib_function.resize(n_user_real_attribs); m_user_real_attrib_parser.resize(n_user_real_attribs); @@ -455,7 +455,6 @@ PhysicalParticleContainer::AddGaussianBeam ( Gpu::HostVector particle_uy; Gpu::HostVector particle_uz; Gpu::HostVector particle_w; - int np = 0; if (ParallelDescriptor::IOProcessor()) { // If do_symmetrize, create either 4x or 8x fewer particles, and @@ -551,7 +550,7 @@ PhysicalParticleContainer::AddGaussianBeam ( } } // Add the temporary CPU vectors to the particle structure - np = particle_z.size(); + auto const np = static_cast(particle_z.size()); amrex::Vector xp(particle_x.data(), particle_x.data() + np); amrex::Vector yp(particle_y.data(), particle_y.data() + np); amrex::Vector zp(particle_z.data(), particle_z.data() + np); @@ -606,30 +605,30 @@ PhysicalParticleContainer::AddPlasmaFromFile(ParticleReal q_tot, #if !defined(WARPX_DIM_1D_Z) // 2D, 3D, and RZ const std::shared_ptr ptr_x = ps["position"]["x"].loadChunk(); const std::shared_ptr ptr_offset_x = ps["positionOffset"]["x"].loadChunk(); - double const position_unit_x = ps["position"]["x"].unitSI(); - double const position_offset_unit_x = ps["positionOffset"]["x"].unitSI(); + auto const position_unit_x = static_cast(ps["position"]["x"].unitSI()); + auto const position_offset_unit_x = static_cast(ps["positionOffset"]["x"].unitSI()); #endif #if !(defined(WARPX_DIM_XZ) || defined(WARPX_DIM_1D_Z)) const std::shared_ptr ptr_y = ps["position"]["y"].loadChunk(); const std::shared_ptr ptr_offset_y = ps["positionOffset"]["y"].loadChunk(); - double const position_unit_y = ps["position"]["y"].unitSI(); - double const position_offset_unit_y = ps["positionOffset"]["y"].unitSI(); + auto const position_unit_y = static_cast(ps["position"]["y"].unitSI()); + auto const position_offset_unit_y = static_cast(ps["positionOffset"]["y"].unitSI()); #endif const std::shared_ptr ptr_z = ps["position"]["z"].loadChunk(); const std::shared_ptr ptr_offset_z = ps["positionOffset"]["z"].loadChunk(); - double const position_unit_z = ps["position"]["z"].unitSI(); - double const position_offset_unit_z = ps["positionOffset"]["z"].unitSI(); + auto const position_unit_z = static_cast(ps["position"]["z"].unitSI()); + auto const position_offset_unit_z = static_cast(ps["positionOffset"]["z"].unitSI()); const std::shared_ptr ptr_ux = ps["momentum"]["x"].loadChunk(); - double const momentum_unit_x = ps["momentum"]["x"].unitSI(); + auto const momentum_unit_x = static_cast(ps["momentum"]["x"].unitSI()); const std::shared_ptr ptr_uz = ps["momentum"]["z"].loadChunk(); - double const momentum_unit_z = ps["momentum"]["z"].unitSI(); + auto const momentum_unit_z = static_cast(ps["momentum"]["z"].unitSI()); const std::shared_ptr ptr_w = ps["weighting"][openPMD::RecordComponent::SCALAR].loadChunk(); - double const w_unit = ps["weighting"][openPMD::RecordComponent::SCALAR].unitSI(); + auto const w_unit = static_cast(ps["weighting"][openPMD::RecordComponent::SCALAR].unitSI()); std::shared_ptr ptr_uy = nullptr; - double momentum_unit_y = 1.0; + auto momentum_unit_y = 1.0_prt; if (ps["momentum"].contains("y")) { ptr_uy = ps["momentum"]["y"].loadChunk(); - momentum_unit_y = ps["momentum"]["y"].unitSI(); + momentum_unit_y = static_cast(ps["momentum"]["y"].unitSI()); } series->flush(); // shared_ptr data can be read now @@ -666,7 +665,7 @@ PhysicalParticleContainer::AddPlasmaFromFile(ParticleReal q_tot, CheckAndAddParticle(x, y, z, ux, uy, uz, weight, particle_x, particle_y, particle_z, particle_ux, particle_uy, particle_uz, - particle_w, t_lab); + particle_w, static_cast(t_lab)); } } auto const np = particle_z.size(); @@ -676,7 +675,7 @@ PhysicalParticleContainer::AddPlasmaFromFile(ParticleReal q_tot, ablastr::warn_manager::WarnPriority::high); } } // IO Processor - auto const np = particle_z.size(); + auto const np = static_cast(particle_z.size()); amrex::Vector xp(particle_x.data(), particle_x.data() + np); amrex::Vector yp(particle_y.data(), particle_y.data() + np); amrex::Vector zp(particle_z.data(), particle_z.data() + np); @@ -711,8 +710,8 @@ PhysicalParticleContainer::DefaultInitializeRuntimeAttributes ( const int np = pinned_tile.numParticles(); // Preparing data needed for user defined attributes - const int n_user_real_attribs = m_user_real_attribs.size(); - const int n_user_int_attribs = m_user_int_attribs.size(); + const auto n_user_real_attribs = static_cast(m_user_real_attribs.size()); + const auto n_user_int_attribs = static_cast(m_user_int_attribs.size()); const auto get_position = GetParticlePosition(pinned_tile); const auto soa = pinned_tile.getParticleTileData(); const amrex::ParticleReal* AMREX_RESTRICT ux = soa.m_rdata[PIdx::ux]; @@ -877,7 +876,7 @@ PhysicalParticleContainer::AddParticles (int lev) amrex::Vector> attr; attr.push_back(plasma_injector->multiple_particles_weight); amrex::Vector> attr_int; - AddNParticles(lev, plasma_injector->multiple_particles_pos_x.size(), + AddNParticles(lev, static_cast(plasma_injector->multiple_particles_pos_x.size()), plasma_injector->multiple_particles_pos_x, plasma_injector->multiple_particles_pos_y, plasma_injector->multiple_particles_pos_z, @@ -984,7 +983,7 @@ PhysicalParticleContainer::AddPlasma (int lev, RealBox part_realbox) { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); const Box& tile_box = mfi.tilebox(); const RealBox tile_realbox = WarpX::getRealBox(tile_box, lev); @@ -1120,8 +1119,8 @@ PhysicalParticleContainer::AddPlasma (int lev, RealBox part_realbox) pa[ia] = soa.GetRealData(ia).data() + old_size; } // user-defined integer and real attributes - const int n_user_int_attribs = m_user_int_attribs.size(); - const int n_user_real_attribs = m_user_real_attribs.size(); + const auto n_user_int_attribs = static_cast(m_user_int_attribs.size()); + const auto n_user_real_attribs = static_cast(m_user_real_attribs.size()); amrex::Gpu::PinnedVector pa_user_int_pinned(n_user_int_attribs); amrex::Gpu::PinnedVector pa_user_real_pinned(n_user_real_attribs); amrex::Gpu::PinnedVector< amrex::ParserExecutor<7> > user_int_attrib_parserexec_pinned(n_user_int_attribs); @@ -1432,7 +1431,7 @@ PhysicalParticleContainer::AddPlasma (int lev, RealBox part_realbox) if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -1525,7 +1524,7 @@ PhysicalParticleContainer::AddPlasmaFlux (amrex::Real dt) { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); const Box& tile_box = mfi.tilebox(); const RealBox tile_realbox = WarpX::getRealBox(tile_box, 0); @@ -1670,8 +1669,8 @@ PhysicalParticleContainer::AddPlasmaFlux (amrex::Real dt) } // user-defined integer and real attributes - const int n_user_int_attribs = m_user_int_attribs.size(); - const int n_user_real_attribs = m_user_real_attribs.size(); + const auto n_user_int_attribs = static_cast(m_user_int_attribs.size()); + const auto n_user_real_attribs = static_cast(m_user_real_attribs.size()); amrex::Gpu::PinnedVector pa_user_int_pinned(n_user_int_attribs); amrex::Gpu::PinnedVector pa_user_real_pinned(n_user_real_attribs); amrex::Gpu::PinnedVector< amrex::ParserExecutor<7> > user_int_attrib_parserexec_pinned(n_user_int_attribs); @@ -1923,7 +1922,7 @@ PhysicalParticleContainer::AddPlasmaFlux (amrex::Real dt) if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -2011,7 +2010,7 @@ PhysicalParticleContainer::Evolve (int lev, { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); const Box& box = pti.validbox(); @@ -2180,7 +2179,7 @@ PhysicalParticleContainer::Evolve (int lev, if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[pti.index()], wt); } } diff --git a/Source/Particles/Pusher/CopyParticleAttribs.H b/Source/Particles/Pusher/CopyParticleAttribs.H index 697fe1e0b38..29e5017a3a7 100644 --- a/Source/Particles/Pusher/CopyParticleAttribs.H +++ b/Source/Particles/Pusher/CopyParticleAttribs.H @@ -47,7 +47,7 @@ struct CopyParticleAttribs * always start at the particle with index 0. */ CopyParticleAttribs (const WarpXParIter& a_pti, TmpParticles& tmp_particle_data, - int a_offset = 0) noexcept + long a_offset = 0) noexcept { if (tmp_particle_data.empty()) return; @@ -73,7 +73,7 @@ struct CopyParticleAttribs * temporary data holder */ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - void operator() (const int i) const noexcept + void operator() (const long i) const noexcept { AMREX_ASSERT(uxp != nullptr); AMREX_ASSERT(uyp != nullptr); diff --git a/Source/Particles/Pusher/GetAndSetPosition.H b/Source/Particles/Pusher/GetAndSetPosition.H index d8cce61756d..45b9c39fe9d 100644 --- a/Source/Particles/Pusher/GetAndSetPosition.H +++ b/Source/Particles/Pusher/GetAndSetPosition.H @@ -74,7 +74,7 @@ struct GetParticlePosition * \param a_offset offset to apply to the particle indices */ template - GetParticlePosition (const ptiType& a_pti, int a_offset = 0) noexcept + GetParticlePosition (const ptiType& a_pti, long a_offset = 0) noexcept { const auto& aos = a_pti.GetArrayOfStructs(); m_structs = aos().dataPtr() + a_offset; @@ -88,7 +88,7 @@ struct GetParticlePosition * located at index `i + a_offset` and store them in the variables * `x`, `y`, `z` */ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - void operator() (const int i, RType& x, RType& y, RType& z) const noexcept + void operator() (const long i, RType& x, RType& y, RType& z) const noexcept { const PType& p = m_structs[i]; #ifdef WARPX_DIM_RZ @@ -117,7 +117,7 @@ struct GetParticlePosition * This is only different for RZ since this returns (r, theta, z) * in that case. */ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - void AsStored (const int i, RType& x, RType& y, RType& z) const noexcept + void AsStored (const long i, RType& x, RType& y, RType& z) const noexcept { const PType& p = m_structs[i]; #ifdef WARPX_DIM_RZ @@ -157,7 +157,7 @@ struct SetParticlePosition #endif template - SetParticlePosition (const ptiType& a_pti, int a_offset = 0) noexcept + SetParticlePosition (const ptiType& a_pti, long a_offset = 0) noexcept { auto& aos = a_pti.GetArrayOfStructs(); m_structs = aos().dataPtr() + a_offset; @@ -170,7 +170,7 @@ struct SetParticlePosition /** \brief Set the position of the particle at index `i + a_offset` * to `x`, `y`, `z` */ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - void operator() (const int i, RType x, RType y, RType z) const noexcept + void operator() (const long i, RType x, RType y, RType z) const noexcept { #if defined(WARPX_DIM_XZ) amrex::ignore_unused(y); @@ -199,7 +199,7 @@ struct SetParticlePosition * This is only different for RZ since the input should * be (r, theta, z) in that case. */ AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - void AsStored (const int i, RType x, RType y, RType z) const noexcept + void AsStored (const long i, RType x, RType y, RType z) const noexcept { #if defined(WARPX_DIM_XZ) amrex::ignore_unused(y); diff --git a/Source/Particles/Resampling/LevelingThinning.cpp b/Source/Particles/Resampling/LevelingThinning.cpp index 6877529053c..e3910286c91 100644 --- a/Source/Particles/Resampling/LevelingThinning.cpp +++ b/Source/Particles/Resampling/LevelingThinning.cpp @@ -70,7 +70,7 @@ void LevelingThinning::operator() (WarpXParIter& pti, const int lev, // algorithm. auto bins = ParticleUtils::findParticlesInEachCell(lev, pti, ptile); - const int n_cells = bins.numBins(); + const auto n_cells = static_cast(bins.numBins()); const auto indices = bins.permutationPtr(); const auto cell_offsets = bins.offsetsPtr(); @@ -83,7 +83,7 @@ void LevelingThinning::operator() (WarpXParIter& pti, const int lev, { // The particles that are in the cell `i_cell` are // given by the `indices[cell_start:cell_stop]` - const auto cell_start = cell_offsets[i_cell]; + const auto cell_start = static_cast(cell_offsets[i_cell]); const auto cell_stop = static_cast(cell_offsets[i_cell+1]); const int cell_numparts = cell_stop - cell_start; diff --git a/Source/Particles/WarpXParticleContainer.H b/Source/Particles/WarpXParticleContainer.H index c6da328264b..2d3ba230bf1 100644 --- a/Source/Particles/WarpXParticleContainer.H +++ b/Source/Particles/WarpXParticleContainer.H @@ -297,7 +297,7 @@ public: * @param[in] id if different than -1, this id will be assigned to the particles (used for * particle tagging in some routines, e.g. SplitParticle) */ - void AddNParticles (int lev, int n, + void AddNParticles (int lev, long n, amrex::Vector const & x, amrex::Vector const & y, amrex::Vector const & z, diff --git a/Source/Particles/WarpXParticleContainer.cpp b/Source/Particles/WarpXParticleContainer.cpp index d2da12888ca..a7405bb5741 100644 --- a/Source/Particles/WarpXParticleContainer.cpp +++ b/Source/Particles/WarpXParticleContainer.cpp @@ -144,7 +144,7 @@ WarpXParticleContainer::AllocData () } void -WarpXParticleContainer::AddNParticles (int /*lev*/, int n, +WarpXParticleContainer::AddNParticles (int /*lev*/, long n, amrex::Vector const & x, amrex::Vector const & y, amrex::Vector const & z, @@ -164,13 +164,13 @@ WarpXParticleContainer::AddNParticles (int /*lev*/, int n, WARPX_ALWAYS_ASSERT_WITH_MESSAGE(nattr_int <= NumIntComps(), "Too many integer attributes specified"); - int ibegin = 0; - int iend = n; + long ibegin = 0; + long iend = n; if (!uniqueparticles) { const int myproc = amrex::ParallelDescriptor::MyProc(); const int nprocs = amrex::ParallelDescriptor::NProcs(); - const int navg = n/nprocs; - const int nleft = n - navg * nprocs; + const auto navg = n/nprocs; + const auto nleft = n - navg * nprocs; if (myproc < nleft) { ibegin = myproc*(navg+1); iend = ibegin + navg+1; @@ -196,7 +196,7 @@ WarpXParticleContainer::AddNParticles (int /*lev*/, int n, amrex::Vector theta(np); #endif - for (int i = ibegin; i < iend; ++i) + for (auto i = ibegin; i < iend; ++i) { ParticleType p; if (id==-1) @@ -615,7 +615,7 @@ WarpXParticleContainer::DepositCurrent ( const amrex::Real dt, const amrex::Real relative_time) { // Loop over the refinement levels - int const finest_level = J.size() - 1; + auto const finest_level = static_cast(J.size() - 1); for (int lev = 0; lev <= finest_level; ++lev) { // Loop over particle tiles and deposit current on each level @@ -824,8 +824,8 @@ WarpXParticleContainer::DepositCharge (WarpXParIter& pti, RealVector const& wp, auto permutation = bins.permutationPtr(); amrex::ParallelFor(bins.numBins(), [=] AMREX_GPU_DEVICE (int ibin) { - const int bin_start = offsets_ptr[ibin]; - const int bin_stop = offsets_ptr[ibin+1]; + const auto bin_start = offsets_ptr[ibin]; + const auto bin_stop = offsets_ptr[ibin+1]; if (bin_start < bin_stop) { auto p = pstruct_ptr[permutation[bin_start]]; Box tbx; @@ -964,7 +964,7 @@ WarpXParticleContainer::DepositCharge (amrex::Vector(rho.size() - 1); for (int lev = 0; lev <= finest_level; ++lev) { DepositCharge ( @@ -1252,7 +1252,7 @@ WarpXParticleContainer::PushX (int lev, amrex::Real dt) { amrex::Gpu::synchronize(); } - Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); // // Particle Push @@ -1280,7 +1280,7 @@ WarpXParticleContainer::PushX (int lev, amrex::Real dt) if (costs && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*costs)[pti.index()], wt); } } diff --git a/Source/Utils/Parser/IntervalsParser.cpp b/Source/Utils/Parser/IntervalsParser.cpp index 35d13143ddb..16595f6f49c 100644 --- a/Source/Utils/Parser/IntervalsParser.cpp +++ b/Source/Utils/Parser/IntervalsParser.cpp @@ -205,7 +205,7 @@ utils::parser::BTDIntervalsParser::BTDIntervalsParser ( } else { - btd_iter_ind = m_btd_iterations.size() - 1; + btd_iter_ind = static_cast(m_btd_iterations.size() - 1); while (start < m_btd_iterations.at(btd_iter_ind) and btd_iter_ind>0) { btd_iter_ind--; @@ -256,7 +256,7 @@ utils::parser::BTDIntervalsParser::BTDIntervalsParser ( int utils::parser::BTDIntervalsParser::NumSnapshots () const { - return m_btd_iterations.size(); + return static_cast(m_btd_iterations.size()); } diff --git a/Source/Utils/Parser/ParserUtils.H b/Source/Utils/Parser/ParserUtils.H index 870c84e241b..fb86996bcd9 100644 --- a/Source/Utils/Parser/ParserUtils.H +++ b/Source/Utils/Parser/ParserUtils.H @@ -128,9 +128,10 @@ namespace utils::parser auto parser = makeParser(str_val, {}); - if (std::is_same::value) { + if constexpr (std::is_same::value) { - val = safeCastToInt(std::round(parser.compileHost<0>()()), str); + val = safeCastToInt( + static_cast(std::round(parser.compileHost<0>()())), str); } else { val = static_cast(parser.compileHost<0>()()); @@ -154,8 +155,9 @@ namespace utils::parser val.resize(n); for (int i=0 ; i < n ; i++) { auto parser = makeParser(tmp_str_arr[i], {}); - if (std::is_same::value) { - val[i] = safeCastToInt(std::round(parser.compileHost<0>()()), str); + if constexpr (std::is_same::value) { + val[i] = safeCastToInt( + static_cast(std::round(parser.compileHost<0>()())), str); } else { val[i] = static_cast(parser.compileHost<0>()()); @@ -196,8 +198,9 @@ namespace utils::parser val.resize(n); for (int i=0 ; i < n ; i++) { auto parser = makeParser(tmp_str_arr[i], {}); - if (std::is_same::value) { - val[i] = safeCastToInt(std::round(parser.compileHost<0>()()), str); + if constexpr (std::is_same::value) { + val[i] = safeCastToInt( + static_cast(std::round(parser.compileHost<0>()())), str); } else { val[i] = static_cast(parser.compileHost<0>()()); @@ -228,8 +231,9 @@ namespace utils::parser Store_parserString(a_pp, str, str_val); auto parser = makeParser(str_val, {}); - if (std::is_same::value) { - val = safeCastToInt(std::round(parser.compileHost<0>()()), str); + if constexpr (std::is_same::value) { + val = safeCastToInt( + static_cast(std::round(parser.compileHost<0>()())), str); } else { val = static_cast(parser.compileHost<0>()()); @@ -247,8 +251,9 @@ namespace utils::parser val.resize(n); for (int i=0 ; i < n ; i++) { auto parser = makeParser(tmp_str_arr[i], {}); - if (std::is_same::value) { - val[i] = safeCastToInt(std::round(parser.compileHost<0>()()), str); + if constexpr (std::is_same::value) { + val[i] = safeCastToInt( + static_cast(std::round(parser.compileHost<0>()())), str); } else { val[i] = static_cast(parser.compileHost<0>()()); @@ -284,8 +289,9 @@ namespace utils::parser val.resize(n); for (int i=0 ; i < n ; i++) { auto parser = makeParser(tmp_str_arr[i], {}); - if (std::is_same::value) { - val[i] = safeCastToInt(std::round(parser.compileHost<0>()()), str); + if constexpr (std::is_same::value) { + val[i] = safeCastToInt( + static_cast(std::round(parser.compileHost<0>()())), str); } else { val[i] = static_cast(parser.compileHost<0>()()); diff --git a/Source/Utils/ParticleUtils.H b/Source/Utils/ParticleUtils.H index 50dc12a722c..ecd6b7c2739 100644 --- a/Source/Utils/ParticleUtils.H +++ b/Source/Utils/ParticleUtils.H @@ -99,8 +99,8 @@ namespace ParticleUtils { // precompute repeatedly used quantities constexpr auto c2 = PhysConst::c * PhysConst::c; const auto V2 = (Vx*Vx + Vy*Vy + Vz*Vz); - const auto gamma_V = 1.0_prt / sqrt(1.0_prt - V2 / c2); - const auto gamma_u = sqrt(1.0_prt + (ux*ux + uy*uy + uz*uz) / c2); + const auto gamma_V = 1.0_prt / std::sqrt(1.0_prt - V2 / c2); + const auto gamma_u = std::sqrt(1.0_prt + (ux*ux + uy*uy + uz*uz) / c2); // copy velocity vector values const auto vx = ux; diff --git a/Source/Utils/WarpXMovingWindow.cpp b/Source/Utils/WarpXMovingWindow.cpp index 62fabef2542..3fd993ef9da 100644 --- a/Source/Utils/WarpXMovingWindow.cpp +++ b/Source/Utils/WarpXMovingWindow.cpp @@ -524,7 +524,7 @@ WarpX::shiftMF (amrex::MultiFab& mf, const amrex::Geometry& geom, { amrex::Gpu::synchronize(); } - amrex::Real wt = amrex::second(); + auto wt = static_cast(amrex::second()); auto const& dstfab = mf.array(mfi); auto const& srcfab = tmpmf.array(mfi); @@ -589,7 +589,7 @@ WarpX::shiftMF (amrex::MultiFab& mf, const amrex::Geometry& geom, WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) { amrex::Gpu::synchronize(); - wt = amrex::second() - wt; + wt = static_cast(amrex::second()) - wt; amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); } } @@ -607,7 +607,8 @@ WarpX::shiftMF (amrex::MultiFab& mf, const amrex::Geometry& geom, // The temporary MultiFab is setup to refer to the data of the original Multifab (this can // be done since the shape of the data is all the same, just the indexing is different). amrex::BoxList bl; - for (int i = 0, N=ba.size(); i < N; ++i) { + const auto ba_size = static_cast(ba.size()); + for (int i = 0; i < ba_size; ++i) { bl.push_back(amrex::grow(ba[i], 0, mf.nGrowVect()[0])); } const amrex::BoxArray rba(std::move(bl)); diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 2df53dc6d2c..306c4ba049e 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -529,7 +529,7 @@ WarpX::ReadParameters () { const ParmParse pp_algo("algo"); - electromagnetic_solver_id = GetAlgorithmInteger(pp_algo, "maxwell_solver"); + electromagnetic_solver_id = static_cast(GetAlgorithmInteger(pp_algo, "maxwell_solver")); } { @@ -1014,7 +1014,7 @@ WarpX::ReadParameters () // Integer that corresponds to the type of grid used in the simulation // (collocated, staggered, hybrid) - grid_type = GetAlgorithmInteger(pp_warpx, "grid_type"); + grid_type = static_cast(GetAlgorithmInteger(pp_warpx, "grid_type")); // Use same shape factors in all directions, for gathering if (grid_type == GridType::Collocated) galerkin_interpolation = false; @@ -1031,7 +1031,9 @@ WarpX::ReadParameters () const ParmParse pp_fluids("fluids"); std::vector fluid_species_names = {}; pp_fluids.queryarr("species_names", fluid_species_names); + if (!fluid_species_names.empty()) do_fluid_species = 1; + if (do_fluid_species) { WARPX_ALWAYS_ASSERT_WITH_MESSAGE(max_level <= 1, "Fluid species cannot currently be used with mesh refinement."); @@ -1133,9 +1135,9 @@ WarpX::ReadParameters () // note: current_deposition must be set after maxwell_solver (electromagnetic_solver_id) or // do_electrostatic (electrostatic_solver_id) are already determined, // because its default depends on the solver selection - current_deposition_algo = GetAlgorithmInteger(pp_algo, "current_deposition"); - charge_deposition_algo = GetAlgorithmInteger(pp_algo, "charge_deposition"); - particle_pusher_algo = GetAlgorithmInteger(pp_algo, "particle_pusher"); + current_deposition_algo = static_cast(GetAlgorithmInteger(pp_algo, "current_deposition")); + charge_deposition_algo = static_cast(GetAlgorithmInteger(pp_algo, "charge_deposition")); + particle_pusher_algo = static_cast(GetAlgorithmInteger(pp_algo, "particle_pusher")); WARPX_ALWAYS_ASSERT_WITH_MESSAGE( current_deposition_algo != CurrentDepositionAlgo::Esirkepov || @@ -1167,7 +1169,7 @@ WarpX::ReadParameters () // Query algo.field_gathering from input, set field_gathering_algo to // "default" if not found (default defined in Utils/WarpXAlgorithmSelection.cpp) - field_gathering_algo = GetAlgorithmInteger(pp_algo, "field_gathering"); + field_gathering_algo = static_cast(GetAlgorithmInteger(pp_algo, "field_gathering")); // Set default field gathering algorithm for hybrid grids (momentum-conserving) std::string tmp_algo; @@ -1227,7 +1229,7 @@ WarpX::ReadParameters () pp_algo.query("load_balance_knapsack_factor", load_balance_knapsack_factor); utils::parser::queryWithParser(pp_algo, "load_balance_efficiency_ratio_threshold", load_balance_efficiency_ratio_threshold); - load_balance_costs_update_algo = GetAlgorithmInteger(pp_algo, "load_balance_costs_update"); + load_balance_costs_update_algo = static_cast(GetAlgorithmInteger(pp_algo, "load_balance_costs_update")); if (WarpX::load_balance_costs_update_algo==LoadBalanceCostsUpdateAlgo::Heuristic) { utils::parser::queryWithParser( pp_algo, "costs_heuristic_cells_wt", costs_heuristic_cells_wt); @@ -1407,12 +1409,12 @@ WarpX::ReadParameters () // Integer that corresponds to the order of the PSATD solution // (whether the PSATD equations are derived from first-order or // second-order solution) - psatd_solution_type = GetAlgorithmInteger(pp_psatd, "solution_type"); + psatd_solution_type = static_cast(GetAlgorithmInteger(pp_psatd, "solution_type")); // Integers that correspond to the time dependency of J (constant, linear) // and rho (linear, quadratic) for the PSATD algorithm - J_in_time = GetAlgorithmInteger(pp_psatd, "J_in_time"); - rho_in_time = GetAlgorithmInteger(pp_psatd, "rho_in_time"); + J_in_time = static_cast(GetAlgorithmInteger(pp_psatd, "J_in_time")); + rho_in_time = static_cast(GetAlgorithmInteger(pp_psatd, "rho_in_time")); if (psatd_solution_type != PSATDSolutionType::FirstOrder || !do_multi_J) { @@ -3051,11 +3053,11 @@ amrex::Vector WarpX::getFornbergStencilCoefficients(const int n_ord if (a_grid_type == GridType::Collocated) { // First coefficient - coeffs.at(0) = m * 2. / (m+1); + coeffs.at(0) = m * 2._rt / (m+1); // Other coefficients by recurrence for (int n = 1; n < m; n++) { - coeffs.at(n) = - (m-n) * 1. / (m+n+1) * coeffs.at(n-1); + coeffs.at(n) = - (m-n) * 1._rt / (m+n+1) * coeffs.at(n-1); } } // Coefficients for staggered finite-difference approximation @@ -3064,14 +3066,14 @@ amrex::Vector WarpX::getFornbergStencilCoefficients(const int n_ord Real prod = 1.; for (int k = 1; k < m+1; k++) { - prod *= (m + k) / (4. * k); + prod *= (m + k) / (4._rt * k); } // First coefficient - coeffs.at(0) = 4 * m * prod * prod; + coeffs.at(0) = 4_rt * m * prod * prod; // Other coefficients by recurrence for (int n = 1; n < m; n++) { - coeffs.at(n) = - ((2*n-1) * (m-n)) * 1. / ((2*n+1) * (m+n)) * coeffs.at(n-1); + coeffs.at(n) = - ((2_rt*n-1) * (m-n)) * 1._rt / ((2_rt*n+1) * (m+n)) * coeffs.at(n-1); } } diff --git a/Source/ablastr/fields/PoissonSolver.H b/Source/ablastr/fields/PoissonSolver.H index c5824be1774..953c67ab54e 100644 --- a/Source/ablastr/fields/PoissonSolver.H +++ b/Source/ablastr/fields/PoissonSolver.H @@ -81,7 +81,7 @@ namespace details AMREX_GPU_DEVICE AMREX_FORCE_INLINE void - operator() (long i, long j, long k) const noexcept + operator() (int i, int j, int k) const noexcept { amrex::mf_nodebilin_interp(i, j, k, 0, m_phi_fp_arr, 0, m_phi_cp_arr, 0, m_refratio); @@ -155,7 +155,7 @@ computePhi (amrex::Vector const & rho, rel_ref_ratio = amrex::Vector{{amrex::IntVect(AMREX_D_DECL(1, 1, 1))}}; } - int const finest_level = rho.size() - 1u; + auto const finest_level = static_cast(rho.size() - 1); // scale rho appropriately; also determine if rho is zero everywhere amrex::Real max_norm_b = 0.0; @@ -196,10 +196,10 @@ computePhi (amrex::Vector const & rho, geom[lev].CellSize(2)/std::sqrt(1._rt-beta_solver[2]*beta_solver[2]))}; int max_semicoarsening_level = 0; int semicoarsening_direction = -1; - const int min_dir = std::distance(dx_scaled.begin(), - std::min_element(dx_scaled.begin(),dx_scaled.end())); - const int max_dir = std::distance(dx_scaled.begin(), - std::max_element(dx_scaled.begin(),dx_scaled.end())); + const auto min_dir = static_cast(std::distance(dx_scaled.begin(), + std::min_element(dx_scaled.begin(),dx_scaled.end()))); + const auto max_dir = static_cast(std::distance(dx_scaled.begin(), + std::max_element(dx_scaled.begin(),dx_scaled.end()))); if (dx_scaled[max_dir] > dx_scaled[min_dir]) { semicoarsening_direction = max_dir; max_semicoarsening_level = static_cast diff --git a/Source/ablastr/fields/VectorPoissonSolver.H b/Source/ablastr/fields/VectorPoissonSolver.H index 3284b49c8fe..75957974fdb 100644 --- a/Source/ablastr/fields/VectorPoissonSolver.H +++ b/Source/ablastr/fields/VectorPoissonSolver.H @@ -115,7 +115,7 @@ computeVectorPotential ( amrex::Vector > co rel_ref_ratio = amrex::Vector{{amrex::IntVect(AMREX_D_DECL(1, 1, 1))}}; } - int const finest_level = curr.size() - 1u; + auto const finest_level = static_cast(curr.size()) - 1; // scale J appropriately; also determine if current is zero everywhere amrex::Real max_comp_J = 0.0; diff --git a/Source/ablastr/utils/SignalHandling.cpp b/Source/ablastr/utils/SignalHandling.cpp index feaf69d002f..18d43409ec3 100644 --- a/Source/ablastr/utils/SignalHandling.cpp +++ b/Source/ablastr/utils/SignalHandling.cpp @@ -90,7 +90,7 @@ SignalHandling::parseSignalNameToNumber (const std::string &str) std::string name_upper = sp.abbrev; std::string name_lower = name_upper; for (char &c : name_lower) { - c = std::tolower(c); + c = static_cast(std::tolower(static_cast(c))); } signals_parser.setConstant(name_upper, sp.value); diff --git a/Source/ablastr/utils/timer/Timer.H b/Source/ablastr/utils/timer/Timer.H index 6743607a0d6..e7787f36b75 100644 --- a/Source/ablastr/utils/timer/Timer.H +++ b/Source/ablastr/utils/timer/Timer.H @@ -8,8 +8,6 @@ #ifndef ABLASTR_TIMER_H_ #define ABLASTR_TIMER_H_ -#include - namespace ablastr::utils::timer { @@ -46,7 +44,7 @@ namespace ablastr::utils::timer * * @return the duration */ - amrex::Real get_duration () noexcept; + double get_duration () noexcept; /** @@ -55,12 +53,12 @@ namespace ablastr::utils::timer * * @return the maximum duration across all the MPI ranks */ - amrex::Real get_global_duration (); + double get_global_duration (); private: - amrex::Real m_start_time /*! The start time*/; - amrex::Real m_stop_time /*! The stop time*/; + double m_start_time /*! The start time*/; + double m_stop_time /*! The stop time*/; }; } diff --git a/Source/ablastr/utils/timer/Timer.cpp b/Source/ablastr/utils/timer/Timer.cpp index 6ce44eab312..90f6fcb99c5 100644 --- a/Source/ablastr/utils/timer/Timer.cpp +++ b/Source/ablastr/utils/timer/Timer.cpp @@ -25,13 +25,13 @@ Timer::record_stop_time() noexcept m_stop_time = amrex::ParallelDescriptor::second(); } -amrex::Real +double Timer::get_duration () noexcept { return m_stop_time - m_start_time; } -amrex::Real +double Timer::get_global_duration () { auto duration = this->get_duration(); From 38f95017cb85540bd633793a57c639f3e50b81ee Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 16 Oct 2023 16:26:01 -0700 Subject: [PATCH 057/110] Fix: CMake `Python_EXECUTABLE` (#4379) Replace the hard-coded `python3` with the found/set Python executable of the current build environment. --- CMakeLists.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 5d00093e893..d7b68f8cfee 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -646,7 +646,7 @@ if(WarpX_PYTHON) ${CMAKE_COMMAND} -E rm -f -r warpx-whl COMMAND ${CMAKE_COMMAND} -E env PYWARPX_LIB_DIR=$ - python3 -m pip wheel -v --no-build-isolation --no-deps --wheel-dir=warpx-whl ${WarpX_SOURCE_DIR} + ${Python_EXECUTABLE} -m pip wheel -v --no-build-isolation --no-deps --wheel-dir=warpx-whl ${WarpX_SOURCE_DIR} WORKING_DIRECTORY ${WarpX_BINARY_DIR} DEPENDS @@ -660,7 +660,7 @@ if(WarpX_PYTHON) set(pyWarpX_REQUIREMENT_FILE "requirements.txt") endif() add_custom_target(${WarpX_CUSTOM_TARGET_PREFIX}pip_install_requirements - python3 -m pip install ${PYINSTALLOPTIONS} -r "${WarpX_SOURCE_DIR}/${pyWarpX_REQUIREMENT_FILE}" + ${Python_EXECUTABLE} -m pip install ${PYINSTALLOPTIONS} -r "${WarpX_SOURCE_DIR}/${pyWarpX_REQUIREMENT_FILE}" WORKING_DIRECTORY ${WarpX_BINARY_DIR} ) @@ -677,7 +677,7 @@ if(WarpX_PYTHON) # because otherwise pip would also force reinstall all dependencies. add_custom_target(${WarpX_CUSTOM_TARGET_PREFIX}pip_install ${CMAKE_COMMAND} -E env WARPX_MPI=${WarpX_MPI} - python3 -m pip install --force-reinstall --no-index --no-deps ${PYINSTALLOPTIONS} --find-links=warpx-whl pywarpx + ${Python_EXECUTABLE} -m pip install --force-reinstall --no-index --no-deps ${PYINSTALLOPTIONS} --find-links=warpx-whl pywarpx WORKING_DIRECTORY ${WarpX_BINARY_DIR} DEPENDS From 8ca2b937f232ef95d7240f6f0aec279d64fabda2 Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Tue, 17 Oct 2023 02:04:47 +0200 Subject: [PATCH 058/110] Reorder WarpX.H so that public, protected, and private only appear once (#4340) * Reorder WarpX.H so that public, protected, and private only appear once Co-authored-by: Axel Huebl --- Source/WarpX.H | 210 ++++++++++++++++++++++++------------------------- 1 file changed, 104 insertions(+), 106 deletions(-) diff --git a/Source/WarpX.H b/Source/WarpX.H index 1bd3f5bf46c..e56d2e647c1 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -1042,6 +1042,109 @@ public: // Return the accelerator lattice instance defined at the given refinement level const AcceleratorLattice& get_accelerator_lattice (int lev) {return *(m_accelerator_lattice[lev]);} + // for cuda + void BuildBufferMasksInBox ( amrex::Box tbx, amrex::IArrayBox &buffer_mask, + const amrex::IArrayBox &guard_mask, int ng ); +#ifdef AMREX_USE_EB + amrex::EBFArrayBoxFactory const& fieldEBFactory (int lev) const noexcept { + return static_cast(*m_field_factory[lev]); + } +#endif + + void InitEB (); + +#ifdef AMREX_USE_EB + /** + * \brief Compute the length of the mesh edges. Here the length is a value in [0, 1]. + * An edge of length 0 is fully covered. + */ + static void ComputeEdgeLengths (std::array< std::unique_ptr, 3 >& edge_lengths, + const amrex::EBFArrayBoxFactory& eb_fact); + /** + * \brief Compute the area of the mesh faces. Here the area is a value in [0, 1]. + * An edge of area 0 is fully covered. + */ + static void ComputeFaceAreas (std::array< std::unique_ptr, 3 >& face_areas, + const amrex::EBFArrayBoxFactory& eb_fact); + + /** + * \brief Scale the edges lengths by the mesh width to obtain the real lengths. + */ + static void ScaleEdges (std::array< std::unique_ptr, 3 >& edge_lengths, + const std::array& cell_size); + /** + * \brief Scale the edges areas by the mesh width to obtain the real areas. + */ + static void ScaleAreas (std::array< std::unique_ptr, 3 >& face_areas, + const std::array& cell_size); + /** + * \brief Initialize information for cell extensions. + * The flags convention for m_flag_info_face is as follows + * - 0 for unstable cells + * - 1 for stable cells which have not been intruded + * - 2 for stable cells which have been intruded + * Here we cannot know if a cell is intruded or not so we initialize all stable cells with 1 + */ + void MarkCells(); +#endif + + /** + * \brief Compute the level set function used for particle-boundary interaction. + */ + void ComputeDistanceToEB (); + /** + * \brief Auxiliary function to count the amount of faces which still need to be extended + */ + amrex::Array1D CountExtFaces(); + /** + * \brief Main function computing the cell extension. Where possible it computes one-way + * extensions and, when this is not possible, it does eight-ways extensions. + */ + void ComputeFaceExtensions(); + /** + * \brief Initialize the memory for the FaceInfoBoxes + */ + void InitBorrowing(); + /** + * \brief Shrink the vectors in the FaceInfoBoxes + */ + void ShrinkBorrowing(); + /** + * \brief Do the one-way extension + */ + void ComputeOneWayExtensions(); + /** + * \brief Do the eight-ways extension + */ + void ComputeEightWaysExtensions(); + /** + * \brief Whenever an unstable cell cannot be extended we increase its area to be the minimal for stability. + * This is the method Benkler-Chavannes-Kuster method and it is less accurate than the regular ECT but it + * still works better than staircasing. (see https://ieeexplore.ieee.org/document/1638381) + * + * @param idim Integer indicating the dimension (x->0, y->1, z->2) for which the BCK correction is done + * + */ + void ApplyBCKCorrection(int idim); + + /** + * \brief Subtract the average of the cumulative sums of the preliminary current D + * from the current J (computed from D according to the Vay deposition scheme) + */ + void PSATDSubtractCurrentPartialSumsAvg (); + +#ifdef WARPX_USE_PSATD + +# ifdef WARPX_DIM_RZ + SpectralSolverRZ& +# else + SpectralSolver& +# endif + get_spectral_solver_fp (int lev) {return *spectral_solver_fp[lev];} +#endif + + FiniteDifferenceSolver * get_pointer_fdtd_solver_fp (int lev) { return m_fdtd_solver_fp[lev].get(); } + protected: /** @@ -1241,10 +1344,7 @@ private: void PerformanceHints (); void BuildBufferMasks (); -public: // for cuda - void BuildBufferMasksInBox ( amrex::Box tbx, amrex::IArrayBox &buffer_mask, - const amrex::IArrayBox &guard_mask, int ng ); -private: + const amrex::iMultiFab* getCurrentBufferMasks (int lev) const { return current_buffer_masks[lev].get(); } @@ -1565,97 +1665,7 @@ private: amrex::FabFactory const& fieldFactory (int lev) const noexcept { return *m_field_factory[lev]; } -#ifdef AMREX_USE_EB -public: - amrex::EBFArrayBoxFactory const& fieldEBFactory (int lev) const noexcept { - return static_cast(*m_field_factory[lev]); - } -#endif - -public: - void InitEB (); - /** - * \brief Compute the length of the mesh edges. Here the length is a value in [0, 1]. - * An edge of length 0 is fully covered. - */ - -public: -#ifdef AMREX_USE_EB - static void ComputeEdgeLengths (std::array< std::unique_ptr, 3 >& edge_lengths, - const amrex::EBFArrayBoxFactory& eb_fact); - /** - * \brief Compute the area of the mesh faces. Here the area is a value in [0, 1]. - * An edge of area 0 is fully covered. - */ - static void ComputeFaceAreas (std::array< std::unique_ptr, 3 >& face_areas, - const amrex::EBFArrayBoxFactory& eb_fact); - - /** - * \brief Scale the edges lengths by the mesh width to obtain the real lengths. - */ - static void ScaleEdges (std::array< std::unique_ptr, 3 >& edge_lengths, - const std::array& cell_size); - /** - * \brief Scale the edges areas by the mesh width to obtain the real areas. - */ - static void ScaleAreas (std::array< std::unique_ptr, 3 >& face_areas, - const std::array& cell_size); - /** - * \brief Initialize information for cell extensions. - * The flags convention for m_flag_info_face is as follows - * - 0 for unstable cells - * - 1 for stable cells which have not been intruded - * - 2 for stable cells which have been intruded - * Here we cannot know if a cell is intruded or not so we initialize all stable cells with 1 - */ - void MarkCells(); - /** - * \brief Compute the level set function used for particle-boundary interaction. - */ -#endif - void ComputeDistanceToEB (); - /** - * \brief Auxiliary function to count the amount of faces which still need to be extended - */ - amrex::Array1D CountExtFaces(); - /** - * \brief Main function computing the cell extension. Where possible it computes one-way - * extensions and, when this is not possible, it does eight-ways extensions. - */ - void ComputeFaceExtensions(); - /** - * \brief Initialize the memory for the FaceInfoBoxes - */ - void InitBorrowing(); - /** - * \brief Shrink the vectors in the FaceInfoBoxes - */ - void ShrinkBorrowing(); - /** - * \brief Do the one-way extension - */ - void ComputeOneWayExtensions(); - /** - * \brief Do the eight-ways extension - */ - void ComputeEightWaysExtensions(); - /** - * \brief Whenever an unstable cell cannot be extended we increase its area to be the minimal for stability. - * This is the method Benkler-Chavannes-Kuster method and it is less accurate than the regular ECT but it - * still works better than staircasing. (see https://ieeexplore.ieee.org/document/1638381) - * - * @param idim Integer indicating the dimension (x->0, y->1, z->2) for which the BCK correction is done - * - */ - void ApplyBCKCorrection(int idim); - - /** - * \brief Subtract the average of the cumulative sums of the preliminary current D - * from the current J (computed from D according to the Vay deposition scheme) - */ - void PSATDSubtractCurrentPartialSumsAvg (); -private: void ScrapeParticles (); void PushPSATD (); @@ -1826,20 +1836,8 @@ private: amrex::Vector> spectral_solver_cp; # endif -public: - -# ifdef WARPX_DIM_RZ - SpectralSolverRZ& -# else - SpectralSolver& -# endif - get_spectral_solver_fp (int lev) {return *spectral_solver_fp[lev];} #endif -public: - FiniteDifferenceSolver * get_pointer_fdtd_solver_fp (int lev) { return m_fdtd_solver_fp[lev].get(); } - -private: amrex::Vector> m_fdtd_solver_fp; amrex::Vector> m_fdtd_solver_cp; }; From b600807246b3a601128341044c715710e3d49d7e Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 16 Oct 2023 21:57:17 -0700 Subject: [PATCH 059/110] CMake: `pip_install_nodeps` Target (#4361) Add a target that does not search of check any dependencies with pip. Useful in package managers. --- CMakeLists.txt | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CMakeLists.txt b/CMakeLists.txt index d7b68f8cfee..4dc886be52d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -686,6 +686,17 @@ if(WarpX_PYTHON) ${WarpX_CUSTOM_TARGET_PREFIX}pip_install_requirements ${_EXTRA_INSTALL_DEPENDS} ) + + # this is for package managers only + add_custom_target(${WarpX_CUSTOM_TARGET_PREFIX}pip_install_nodeps + ${CMAKE_COMMAND} -E env WARPX_MPI=${WarpX_MPI} + ${Python_EXECUTABLE} -m pip install --force-reinstall --no-index --no-deps ${PYINSTALLOPTIONS} --find-links=warpx-whl pywarpx + WORKING_DIRECTORY + ${WarpX_BINARY_DIR} + DEPENDS + pyWarpX_${WarpX_DIMS_LAST} + ${WarpX_CUSTOM_TARGET_PREFIX}pip_wheel + ) endif() From 138bb3e3fd5588aa7ee98b90bce0c95cd2d38c03 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 16 Oct 2023 21:59:28 -0700 Subject: [PATCH 060/110] [pre-commit.ci] pre-commit autoupdate (#4378) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/hadialqattan/pycln: v2.2.2 → v2.3.0](https://github.com/hadialqattan/pycln/compare/v2.2.2...v2.3.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index bb8b6c8b33d..b514758caba 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -68,7 +68,7 @@ repos: # Autoremoves unused Python imports - repo: https://github.com/hadialqattan/pycln - rev: v2.2.2 + rev: v2.3.0 hooks: - id: pycln name: pycln (python) From 1b8071289eaeaac5b52c91c78861b18eb0e8e379 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 16 Oct 2023 22:22:59 -0700 Subject: [PATCH 061/110] Fix: Fails in Python Callbacks Abort (#4380) * Fix: Fails in Python Callbacks Abort Make sure that errors in Python callbacks abort the simulation. Before, MPI-parallel simulations would hang. * Include * Include --- Source/Python/WarpX_py.H | 1 + Source/Python/WarpX_py.cpp | 22 ++++++++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/Source/Python/WarpX_py.H b/Source/Python/WarpX_py.H index a18f9a7f1ce..590f26a804b 100644 --- a/Source/Python/WarpX_py.H +++ b/Source/Python/WarpX_py.H @@ -12,6 +12,7 @@ #include "Utils/export.H" #include "Utils/WarpXProfilerWrapper.H" +#include #include #include diff --git a/Source/Python/WarpX_py.cpp b/Source/Python/WarpX_py.cpp index 49328d0a4a7..afc734d780e 100644 --- a/Source/Python/WarpX_py.cpp +++ b/Source/Python/WarpX_py.cpp @@ -8,6 +8,11 @@ */ #include "WarpX_py.H" +#include +#include +#include + + std::map< std::string, std::function > warpx_callback_py_map; void InstallPythonCallback ( std::string name, std::function callback ) @@ -24,8 +29,21 @@ bool IsPythonCallbackInstalled ( std::string name ) void ExecutePythonCallback ( std::string name ) { if ( IsPythonCallbackInstalled(name) ) { - WARPX_PROFILE("warpx_py_"+name); - warpx_callback_py_map[name](); + WARPX_PROFILE("warpx_py_" + name); + try { + warpx_callback_py_map[name](); + } catch (std::exception &e) { + std::cerr << "Python callback '" << name << "' failed!" << std::endl; + std::cerr << e.what() << std::endl; + std::exit(3); // note: NOT amrex::Abort(), to avoid hangs with MPI + + // future note: + // if we want to rethrow/raise exceptions from Python callbacks through here (C++) and + // back the managing Python interpreter, we first need to discard and clear + // out the Python error in py::error_already_set. Otherwise, MPI-runs will hang + // (and Python will be in continued error state). + // https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html#handling-unraisable-exceptions + } } } From 2770d3dd104c0d450face979f806df1dbc1c0435 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 17 Oct 2023 15:03:20 -0700 Subject: [PATCH 062/110] AMReX/PICSAR: Weekly Update (#4363) * AMReX: Weekly Update * AMReX: Weekly Update --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- run_test.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 1754549ab2d..0425842e7cc 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 23.10 && cd - + cd ../amrex && git checkout --detach 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index 1a1c1fc612c..aa0167565ea 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 23.10 +branch = 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 3eb8101addb..71c510b81a3 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 23.10 +branch = 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index a4d1f0fed5c..0313f4a2128 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -257,7 +257,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "23.10" +set(WarpX_amrex_branch "7ee29121ed70d7e255ad98a8b1690d345cb4fb33" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/run_test.sh b/run_test.sh index 1c948e48713..e362a0ef332 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 23.10 && cd - +cd amrex && git checkout --detach 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 0442f3ffd5ccaeb361bf29855c760ea7ed54c740 Mon Sep 17 00:00:00 2001 From: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Tue, 17 Oct 2023 20:50:40 -0700 Subject: [PATCH 063/110] Add RZ support for Ohm's law solver (#4161) * add RZ support for `FiniteDifferenceSolver::CalculateCurrentAmpere` * add RZ support in `HybridPICSolveECylindrical` * allow an initial Bz field to be set in RZ * apply PEC boundary to r_max boundary in RZ * WIP normal modes example / CI test * fix typo in Cartesian EM modes example * code cleanup and addition of CI test * some progress on CI test * fix compile issue when `USE_EB=True`; update CI test and analysis script * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix CI test and update docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reset benchmark values for failing tests * update reference amplitude values based on previous values from Azure * reset hybrid-PIC RZ checksum values - not sure why this is needed * add `-DWarpX_PYTHON=ON` cmake option to hybrid-PIC RZ test configuration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * apply changes requested during code review * fix clang-tidy narrowing conversion warnings --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- Docs/source/refs.bib | 13 + Docs/source/usage/examples.rst | 14 +- .../Tests/ohm_solver_EM_modes/PICMI_inputs.py | 6 +- .../ohm_solver_EM_modes/PICMI_inputs_rz.py | 248 ++++++++++ .../Tests/ohm_solver_EM_modes/analysis_rz.py | 169 +++++++ .../LaserInjectionFromLASYFile_RZ.json | 16 +- .../LaserInjectionFromRZLASYFile.json | 14 +- .../Python_ohms_law_solver_EM_modes_rz.json | 12 + Regression/WarpX-tests.ini | 19 + Source/BoundaryConditions/WarpX_PEC.H | 4 +- Source/BoundaryConditions/WarpX_PEC.cpp | 30 -- .../HybridPICModel/HybridPICModel.cpp | 6 + .../HybridPICSolveE.cpp | 450 +++++++++++++++++- Source/Initialization/WarpXInitData.cpp | 41 +- 14 files changed, 959 insertions(+), 83 deletions(-) create mode 100644 Examples/Tests/ohm_solver_EM_modes/PICMI_inputs_rz.py create mode 100755 Examples/Tests/ohm_solver_EM_modes/analysis_rz.py create mode 100644 Regression/Checksum/benchmarks_json/Python_ohms_law_solver_EM_modes_rz.json diff --git a/Docs/source/refs.bib b/Docs/source/refs.bib index 72d54456a89..fdbbe57cf72 100644 --- a/Docs/source/refs.bib +++ b/Docs/source/refs.bib @@ -483,3 +483,16 @@ @article{Stanier2020 author = {A. Stanier and L. Chacón and A. Le}, keywords = {Hybrid, Particle-in-cell, Plasma, Asymptotic-preserving, Cancellation problem, Space weather}, } + +@book{Stix1992, + author = {Stix, T.H.}, + date-added = {2023-06-29 13:51:16 -0700}, + date-modified = {2023-06-29 13:51:16 -0700}, + isbn = {978-0-88318-859-0}, + lccn = {lc91033341}, + publisher = {American Inst. of Physics}, + title = {Waves in {Plasmas}}, + url = {https://books.google.com/books?id=OsOWJ8iHpmMC}, + year = {1992}, + bdsk-url-1 = {https://books.google.com/books?id=OsOWJ8iHpmMC} +} diff --git a/Docs/source/usage/examples.rst b/Docs/source/usage/examples.rst index e733938a93c..48d37db98c4 100644 --- a/Docs/source/usage/examples.rst +++ b/Docs/source/usage/examples.rst @@ -164,6 +164,18 @@ ion-Bernstein modes as indicated below. python3 PICMI_inputs.py -dim {1/2/3} --bdir {x/y/z} +A RZ-geometry example case for normal modes propagating along an applied magnetic field in a cylinder is also available. +The analytical solution for these modes are described in :cite:t:`ex-Stix1992` Chapter 6, Sec. 2. + +.. figure:: https://user-images.githubusercontent.com/40245517/259251824-33e78375-81d8-410d-a147-3fa0498c66be.png + :alt: Normal EM modes in a metallic cylinder + :width: 90% + +The input file for this example and corresponding analysis can be found at: + +* :download:`Cylinderical modes input <../../../Examples/Tests/ohm_solver_EM_modes/PICMI_inputs_rz.py>` +* :download:`Analysis script <../../../Examples/Tests/ohm_solver_EM_modes/analysis_rz.py>` + Ion beam R instability ^^^^^^^^^^^^^^^^^^^^^^ @@ -243,7 +255,7 @@ The input file for this example and corresponding analysis can be found at: * :download:`Analysis script <../../../Examples/Tests/ohm_solver_magnetic_reconnection/analysis.py>` Many Further Examples, Demos and Tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------------- WarpX runs over 200 integration tests on a variety of modeling cases, which validate and demonstrate its functionality. Please see the `Examples/Tests/ `__ directory for many more examples. diff --git a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py index 51dbb53ab43..9363bcebd18 100644 --- a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py +++ b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs.py @@ -4,8 +4,8 @@ # --- treated as kinetic particles and electrons as an isothermal, inertialess # --- background fluid. The script is set up to produce either parallel or # --- perpendicular (Bernstein) EM modes and can be run in 1d, 2d or 3d -# --- Cartesian geometries. See Section 4.2 and 4.3 of Munoz et al. (2018) As a -# --- CI test only a small number of steps are taken using the 1d version. +# --- Cartesian geometries. See Section 4.2 and 4.3 of Munoz et al. (2018). +# --- As a CI test only a small number of steps are taken using the 1d version. import argparse import os @@ -322,7 +322,7 @@ def _record_average_fields(self): return Bx_warpx = fields.BxWrapper()[...] - By_warpx = fields.BxWrapper()[...] + By_warpx = fields.ByWrapper()[...] Ez_warpx = fields.EzWrapper()[...] if libwarpx.amr.ParallelDescriptor.MyProc() != 0: diff --git a/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs_rz.py b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs_rz.py new file mode 100644 index 00000000000..21d8cafe750 --- /dev/null +++ b/Examples/Tests/ohm_solver_EM_modes/PICMI_inputs_rz.py @@ -0,0 +1,248 @@ +#!/usr/bin/env python3 +# +# --- Test script for the kinetic-fluid hybrid model in WarpX wherein ions are +# --- treated as kinetic particles and electrons as an isothermal, inertialess +# --- background fluid. The script is set up to produce parallel normal EM modes +# --- in a metallic cylinder and is run in RZ geometry. +# --- As a CI test only a small number of steps are taken. + +import argparse +import sys + +import dill +from mpi4py import MPI as mpi +import numpy as np + +from pywarpx import picmi + +constants = picmi.constants + +comm = mpi.COMM_WORLD + +simulation = picmi.Simulation(verbose=0) + + +class CylindricalNormalModes(object): + '''The following runs a simulation of an uniform plasma at a set ion + temperature (and Te = 0) with an external magnetic field applied in the + z-direction (parallel to domain). + The analysis script (in this same directory) analyzes the output field + data for EM modes. + ''' + # Applied field parameters + B0 = 0.5 # Initial magnetic field strength (T) + beta = 0.01 # Plasma beta, used to calculate temperature + + # Plasma species parameters + m_ion = 400.0 # Ion mass (electron masses) + vA_over_c = 5e-3 # ratio of Alfven speed and the speed of light + + # Spatial domain + Nz = 512 # number of cells in z direction + Nr = 128 # number of cells in r direction + + # Temporal domain (if not run as a CI test) + LT = 800.0 # Simulation temporal length (ion cyclotron periods) + + # Numerical parameters + NPPC = 8000 # Seed number of particles per cell + DZ = 0.4 # Cell size (ion skin depths) + DR = 0.4 # Cell size (ion skin depths) + DT = 0.02 # Time step (ion cyclotron periods) + + # Plasma resistivity - used to dampen the mode excitation + eta = 5e-4 + # Number of substeps used to update B + substeps = 250 + + def __init__(self, test, verbose): + """Get input parameters for the specific case desired.""" + self.test = test + self.verbose = verbose or self.test + + # calculate various plasma parameters based on the simulation input + self.get_plasma_quantities() + + if not self.test: + self.total_steps = int(self.LT / self.DT) + else: + # if this is a test case run for only a small number of steps + self.total_steps = 100 + # and make the grid and particle count smaller + self.Nz = 128 + self.Nr = 64 + self.NPPC = 200 + # output diagnostics 5 times per cyclotron period + self.diag_steps = max(10, int(1.0 / 5 / self.DT)) + + self.Lz = self.Nz * self.DZ * self.l_i + self.Lr = self.Nr * self.DR * self.l_i + + self.dt = self.DT * self.t_ci + + # dump all the current attributes to a dill pickle file + if comm.rank == 0: + with open(f'sim_parameters.dpkl', 'wb') as f: + dill.dump(self, f) + + # print out plasma parameters + if comm.rank == 0: + print( + f"Initializing simulation with input parameters:\n" + f"\tT = {self.T_plasma:.3f} eV\n" + f"\tn = {self.n_plasma:.1e} m^-3\n" + f"\tB0 = {self.B0:.2f} T\n" + f"\tM/m = {self.m_ion:.0f}\n" + ) + print( + f"Plasma parameters:\n" + f"\tl_i = {self.l_i:.1e} m\n" + f"\tt_ci = {self.t_ci:.1e} s\n" + f"\tv_ti = {self.v_ti:.1e} m/s\n" + f"\tvA = {self.vA:.1e} m/s\n" + ) + print( + f"Numerical parameters:\n" + f"\tdt = {self.dt:.1e} s\n" + f"\tdiag steps = {self.diag_steps:d}\n" + f"\ttotal steps = {self.total_steps:d}\n", + flush=True + ) + self.setup_run() + + def get_plasma_quantities(self): + """Calculate various plasma parameters based on the simulation input.""" + # Ion mass (kg) + self.M = self.m_ion * constants.m_e + + # Cyclotron angular frequency (rad/s) and period (s) + self.w_ci = constants.q_e * abs(self.B0) / self.M + self.t_ci = 2.0 * np.pi / self.w_ci + + # Alfven speed (m/s): vA = B / sqrt(mu0 * n * (M + m)) = c * omega_ci / w_pi + self.vA = self.vA_over_c * constants.c + self.n_plasma = ( + (self.B0 / self.vA)**2 / (constants.mu0 * (self.M + constants.m_e)) + ) + + # Ion plasma frequency (Hz) + self.w_pi = np.sqrt( + constants.q_e**2 * self.n_plasma / (self.M * constants.ep0) + ) + + # Skin depth (m) + self.l_i = constants.c / self.w_pi + + # Ion thermal velocity (m/s) from beta = 2 * (v_ti / vA)**2 + self.v_ti = np.sqrt(self.beta / 2.0) * self.vA + + # Temperature (eV) from thermal speed: v_ti = sqrt(kT / M) + self.T_plasma = self.v_ti**2 * self.M / constants.q_e # eV + + # Larmor radius (m) + self.rho_i = self.v_ti / self.w_ci + + def setup_run(self): + """Setup simulation components.""" + + ####################################################################### + # Set geometry and boundary conditions # + ####################################################################### + + self.grid = picmi.CylindricalGrid( + number_of_cells=[self.Nr, self.Nz], + warpx_max_grid_size=self.Nz, + lower_bound=[0, -self.Lz/2.0], + upper_bound=[self.Lr, self.Lz/2.0], + lower_boundary_conditions = ['none', 'periodic'], + upper_boundary_conditions = ['dirichlet', 'periodic'], + lower_boundary_conditions_particles = ['absorbing', 'periodic'], + upper_boundary_conditions_particles = ['reflecting', 'periodic'] + ) + simulation.time_step_size = self.dt + simulation.max_steps = self.total_steps + simulation.current_deposition_algo = 'direct' + simulation.particle_shape = 1 + simulation.verbose = self.verbose + + ####################################################################### + # Field solver and external field # + ####################################################################### + + self.solver = picmi.HybridPICSolver( + grid=self.grid, + Te=0.0, n0=self.n_plasma, plasma_resistivity=self.eta, + substeps=self.substeps, + n_floor=self.n_plasma*0.05 + ) + simulation.solver = self.solver + + B_ext = picmi.AnalyticInitialField( + Bz_expression=self.B0 + ) + simulation.add_applied_field(B_ext) + + ####################################################################### + # Particle types setup # + ####################################################################### + + self.ions = picmi.Species( + name='ions', charge='q_e', mass=self.M, + initial_distribution=picmi.UniformDistribution( + density=self.n_plasma, + rms_velocity=[self.v_ti]*3, + ) + ) + simulation.add_species( + self.ions, + layout=picmi.PseudoRandomLayout( + grid=self.grid, n_macroparticles_per_cell=self.NPPC + ) + ) + + ####################################################################### + # Add diagnostics # + ####################################################################### + + field_diag = picmi.FieldDiagnostic( + name='field_diag', + grid=self.grid, + period=self.diag_steps, + data_list=['B', 'E'], + write_dir='diags', + warpx_file_prefix='field_diags', + warpx_format='openpmd', + warpx_openpmd_backend='h5', + ) + simulation.add_diagnostic(field_diag) + + # add particle diagnostic for checksum + if self.test: + part_diag = picmi.ParticleDiagnostic( + name='diag1', + period=self.total_steps, + species=[self.ions], + data_list=['ux', 'uy', 'uz', 'weighting'], + write_dir='.', + warpx_file_prefix='Python_ohms_law_solver_EM_modes_rz_plt' + ) + simulation.add_diagnostic(part_diag) + + +########################## +# parse input parameters +########################## + +parser = argparse.ArgumentParser() +parser.add_argument( + '-t', '--test', help='toggle whether this script is run as a short CI test', + action='store_true', +) +parser.add_argument( + '-v', '--verbose', help='Verbose output', action='store_true', +) +args, left = parser.parse_known_args() +sys.argv = sys.argv[:1]+left + +run = CylindricalNormalModes(test=args.test, verbose=args.verbose) +simulation.step() diff --git a/Examples/Tests/ohm_solver_EM_modes/analysis_rz.py b/Examples/Tests/ohm_solver_EM_modes/analysis_rz.py new file mode 100755 index 00000000000..9eb747dd6e8 --- /dev/null +++ b/Examples/Tests/ohm_solver_EM_modes/analysis_rz.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +# +# --- Analysis script for the hybrid-PIC example producing EM modes. + +import dill +from matplotlib import colors +import matplotlib.pyplot as plt +import numpy as np +from openpmd_viewer import OpenPMDTimeSeries +import scipy.fft as fft +from scipy.interpolate import RegularGridInterpolator +from scipy.special import j1, jn, jn_zeros + +from pywarpx import picmi + +constants = picmi.constants + +# load simulation parameters +with open(f'sim_parameters.dpkl', 'rb') as f: + sim = dill.load(f) + +diag_dir = "diags/field_diags" + +ts = OpenPMDTimeSeries(diag_dir, check_all_files=True) + +def transform_spatially(data_for_transform): + # interpolate from regular r-grid to special r-grid + interp = RegularGridInterpolator( + (info.z, info.r), data_for_transform, + method='linear' + ) + data_interp = interp((zg, rg)) + + # Applying manual hankel in r + # Fmz = np.sum(proj*data_for_transform, axis=(2,3)) + Fmz = np.einsum('ijkl,kl->ij', proj, data_interp) + # Standard fourier in z + Fmn = fft.fftshift(fft.fft(Fmz, axis=1), axes=1) + return Fmn + +def process(it): + print(f"Processing iteration {it}", flush=True) + field, info = ts.get_field('E', 'y', iteration=it) + F_k = transform_spatially(field) + return F_k + +# grab the first iteration to get the grids +Bz, info = ts.get_field('B', 'z', iteration=0) + +nr = len(info.r) +nz = len(info.z) + +nkr = 12 # number of radial modes to solve for + +r_max = np.max(info.r) + +# create r-grid with points spaced out according to zeros of the Bessel function +r_grid = jn_zeros(1, nr) / jn_zeros(1, nr)[-1] * r_max + +zg, rg = np.meshgrid(info.z, r_grid) + +# Setup Hankel Transform +j_1M = jn_zeros(1, nr)[-1] +r_modes = np.arange(nkr) + +A = ( + 4.0 * np.pi * r_max**2 / j_1M**2 + * j1(np.outer(jn_zeros(1, max(r_modes)+1)[r_modes], jn_zeros(1, nr)) / j_1M) + / jn(2 ,jn_zeros(1, nr))**2 +) + +# No transformation for z +B = np.identity(nz) + +# combine projection arrays +proj = np.einsum('ab,cd->acbd', A, B) + +results = np.zeros((len(ts.t), nkr, nz), dtype=complex) +for ii, it in enumerate(ts.iterations): + results[ii] = process(it) + +# now Fourier transform in time +F_kw = fft.fftshift(fft.fft(results, axis=0), axes=0) + +dz = info.z[1] - info.z[0] +kz = 2*np.pi*fft.fftshift(fft.fftfreq(F_kw[0].shape[1], dz)) +dt = ts.iterations[1] - ts.iterations[0] +omega = 2*np.pi*fft.fftshift(fft.fftfreq(F_kw.shape[0], sim.dt*dt)) + +# Save data for future plotting purposes +np.savez( + "diags/spectrograms.npz", + F_kw=F_kw, dz=dz, kz=kz, dt=dt, omega=omega +) + +# plot the resulting dispersions +k = np.linspace(0, 250, 500) +kappa = k * sim.l_i + +fig, axes = plt.subplots(2, 2, sharex=True, sharey=True, figsize=(6.75, 5)) + +vmin = [2e-3, 1.5e-3, 7.5e-4, 5e-4] +vmax = 1.0 + +# plot m = 1 +for ii, m in enumerate([1, 3, 6, 8]): + ax = axes.flatten()[ii] + ax.set_title(f"m = {m}", fontsize=11) + m -= 1 + pm1 = ax.pcolormesh( + kz*sim.l_i, omega/sim.w_ci, + abs(F_kw[:, m, :])/np.max(abs(F_kw[:, m, :])), + norm=colors.LogNorm(vmin=vmin[ii], vmax=vmax), + cmap='inferno' + ) + cb = fig.colorbar(pm1, ax=ax) + cb.set_label(r'Normalized $E_\theta(k_z, m, \omega)$') + + # Get dispersion relation - see for example + # T. Stix, Waves in Plasmas (American Inst. of Physics, 1992), Chap 6, Sec 2 + nu_m = jn_zeros(1, m+1)[-1] / sim.Lr + R2 = 0.5 * (nu_m**2 * (1.0 + kappa**2) + k**2 * (kappa**2 + 2.0)) + P4 = k**2 * (nu_m**2 + k**2) + omega_fast = sim.vA * np.sqrt(R2 + np.sqrt(R2**2 - P4)) + omega_slow = sim.vA * np.sqrt(R2 - np.sqrt(R2**2 - P4)) + # Upper right corner + ax.plot(k*sim.l_i, omega_fast/sim.w_ci, 'w--', label = f"$\omega_{{fast}}$") + ax.plot(k*sim.l_i, omega_slow/sim.w_ci, color='white', linestyle='--', label = f"$\omega_{{slow}}$") + # Thermal resonance + thermal_res = sim.w_ci + 3*sim.v_ti*k + ax.plot(k*sim.l_i, thermal_res/sim.w_ci, color='magenta', linestyle='--', label = "$\omega = \Omega_i + 3v_{th,i}k$") + ax.plot(-k*sim.l_i, thermal_res/sim.w_ci, color='magenta', linestyle='--', label = "") + thermal_res = sim.w_ci - 3*sim.v_ti*k + ax.plot(k*sim.l_i, thermal_res/sim.w_ci, color='magenta', linestyle='--', label = "$\omega = \Omega_i + 3v_{th,i}k$") + ax.plot(-k*sim.l_i, thermal_res/sim.w_ci, color='magenta', linestyle='--', label = "") + + +for ax in axes.flatten(): + ax.set_xlim(-1.75, 1.75) + ax.set_ylim(0, 1.6) + +axes[0, 0].set_ylabel('$\omega/\Omega_{ci}$') +axes[1, 0].set_ylabel('$\omega/\Omega_{ci}$') +axes[1, 0].set_xlabel('$k_zl_i$') +axes[1, 1].set_xlabel('$k_zl_i$') + +plt.savefig('normal_modes_disp.png', dpi=600) +if not sim.test: + plt.show() +else: + plt.close() + + # check if power spectrum sampling match earlier results + amps = np.abs(F_kw[2, 1, len(kz)//2-2:len(kz)//2+2]) + print("Amplitude sample: ", amps) + assert np.allclose( + amps, np.array([61.41633903, 19.39353485, 101.08693342, 11.09248295]) + ) + +if sim.test: + import os + import sys + sys.path.insert(1, '../../../../warpx/Regression/Checksum/') + import checksumAPI + + # this will be the name of the plot file + fn = sys.argv[1] + test_name = os.path.split(os.getcwd())[1] + checksumAPI.evaluate_checksum(test_name, fn, rtol=1e-6) diff --git a/Regression/Checksum/benchmarks_json/LaserInjectionFromLASYFile_RZ.json b/Regression/Checksum/benchmarks_json/LaserInjectionFromLASYFile_RZ.json index 0d7afc38099..d98aabd84e9 100644 --- a/Regression/Checksum/benchmarks_json/LaserInjectionFromLASYFile_RZ.json +++ b/Regression/Checksum/benchmarks_json/LaserInjectionFromLASYFile_RZ.json @@ -1,12 +1,12 @@ { "lev=0": { - "Br": 100278934.99628444, - "Bz": 2509040.7710191337, - "Er": 3.258814166960793, - "Et": 3.0045384450930932e+16, - "Ez": 0.24012250267165877, - "jr": 50.13336345238631, - "jt": 4.3064420556391546e+17, + "Br": 100278645.72225758, + "Bz": 2509008.1146699777, + "Er": 3.258880159474263, + "Et": 3.0045297157982412e+16, + "Ez": 0.24010979707502123, + "jr": 50.13668665903749, + "jt": 4.305139762894391e+17, "jz": 0.0 } -} \ No newline at end of file +} diff --git a/Regression/Checksum/benchmarks_json/LaserInjectionFromRZLASYFile.json b/Regression/Checksum/benchmarks_json/LaserInjectionFromRZLASYFile.json index 008080d9f5e..8d58f5c9551 100644 --- a/Regression/Checksum/benchmarks_json/LaserInjectionFromRZLASYFile.json +++ b/Regression/Checksum/benchmarks_json/LaserInjectionFromRZLASYFile.json @@ -1,12 +1,12 @@ { "lev=0": { - "Br": 193619956.9879392, - "Bz": 4691018.480978214, - "Er": 197265445300168.38, - "Et": 5.972016823892645e+16, - "Ez": 1383831581199501.0, - "jr": 18581252675.28125, - "jt": 1.194996054262225e+18, + "Br": 193609031.89343664, + "Bz": 4689634.787639907, + "Er": 197280940689075.1, + "Et": 5.9716806429474664e+16, + "Ez": 1383812184076636.0, + "jr": 18582087468.75, + "jt": 1.193738117372394e+18, "jz": 0.0 } } diff --git a/Regression/Checksum/benchmarks_json/Python_ohms_law_solver_EM_modes_rz.json b/Regression/Checksum/benchmarks_json/Python_ohms_law_solver_EM_modes_rz.json new file mode 100644 index 00000000000..870e4a82180 --- /dev/null +++ b/Regression/Checksum/benchmarks_json/Python_ohms_law_solver_EM_modes_rz.json @@ -0,0 +1,12 @@ +{ + "lev=0": {}, + "ions": { + "particle_momentum_x": 5.043873837579827e-17, + "particle_momentum_y": 5.04445113937769e-17, + "particle_momentum_z": 5.051930416996439e-17, + "particle_position_x": 143164.4268324243, + "particle_position_y": 143166.5184643828, + "particle_theta": 2573261.7450408577, + "particle_weight": 8.12868064536689e+18 + } +} diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 71c510b81a3..6e0efc9ecba 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -3431,6 +3431,25 @@ doVis = 0 compareParticles = 1 analysisRoutine = Examples/Tests/ohm_solver_EM_modes/analysis.py +[Python_ohms_law_solver_EM_modes_rz] +buildDir = . +inputFile = Examples/Tests/ohm_solver_EM_modes/PICMI_inputs_rz.py +runtime_params = warpx.abort_on_warning_threshold = medium +customRunCmd = python3 PICMI_inputs_rz.py --test +dim = 1 +addToCompileString = USE_PYTHON_MAIN=TRUE USE_OPENPMD=TRUE QED=FALSE USE_RZ=TRUE +cmakeSetupOpts = -DWarpX_DIMS=RZ -DWarpX_APP=OFF -DWarpX_QED=OFF -DWarpX_PYTHON=ON +target = pip_install +restartTest = 0 +useMPI = 1 +numprocs = 2 +useOMP = 1 +numthreads = 1 +compileTest = 0 +doVis = 0 +compareParticles = 1 +analysisRoutine = Examples/Tests/ohm_solver_EM_modes/analysis_rz.py + [Python_ohms_law_solver_ion_beam_1d] buildDir = . inputFile = Examples/Tests/ohm_solver_ion_beam_instability/PICMI_inputs.py diff --git a/Source/BoundaryConditions/WarpX_PEC.H b/Source/BoundaryConditions/WarpX_PEC.H index 78da9349d0d..3d3e4729d45 100644 --- a/Source/BoundaryConditions/WarpX_PEC.H +++ b/Source/BoundaryConditions/WarpX_PEC.H @@ -379,7 +379,7 @@ using namespace amrex; amrex::IntVect iv_mirror = ijk_vec; iv_mirror[idim] = mirrorfac[idim][iside] - ijk_vec[idim]; - // On the PEC boundary the current density is set to 0 + // On the PEC boundary the charge/current density is set to 0 if (ijk_vec == iv_mirror) field(ijk_vec, n) = 0._rt; // otherwise update the internal cell if the mirror guard cell exists else if (fabbox.contains(iv_mirror)) @@ -389,7 +389,7 @@ using namespace amrex; } } // 2) The guard cells are updated with the appropriate image - // charge current based on the current in the valid cells + // charge based on the charge/current in the valid cells for (int idim = 0; idim < AMREX_SPACEDIM; ++idim) { for (int iside = 0; iside < 2; ++iside) diff --git a/Source/BoundaryConditions/WarpX_PEC.cpp b/Source/BoundaryConditions/WarpX_PEC.cpp index b3986da0645..0b8d1c0fcc8 100644 --- a/Source/BoundaryConditions/WarpX_PEC.cpp +++ b/Source/BoundaryConditions/WarpX_PEC.cpp @@ -266,16 +266,6 @@ PEC::ApplyPECtoRhofield (amrex::MultiFab* rho, const int lev, PatchType patch_ty } const int nComp = rho->nComp(); -#ifdef WARPX_DIM_RZ - if (is_pec[0][1]) { - ablastr::warn_manager::WMRecordWarning( - "PEC", - "PEC boundary handling is not yet properly implemented for r_max so it is skipped in PEC::ApplyPECtoRhofield", - ablastr::warn_manager::WarnPriority::medium); - is_pec[0][1] = false; - } -#endif - #ifdef AMREX_USE_OMP #pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) #endif @@ -396,16 +386,6 @@ PEC::ApplyPECtoJfield(amrex::MultiFab* Jx, amrex::MultiFab* Jy, mirrorfac[2][idim][1] = 2*domain_hi[idim] - (1 - Jz_nodal[idim]); } -#ifdef WARPX_DIM_RZ - if (is_pec[0][1]) { - ablastr::warn_manager::WMRecordWarning( - "PEC", - "PEC boundary handling is not yet properly implemented for r_max so it is skipped in PEC::ApplyPECtoJfield", - ablastr::warn_manager::WarnPriority::medium); - is_pec[0][1] = false; - } -#endif - // Each current component is handled separately below, starting with Jx. #ifdef AMREX_USE_OMP #pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) @@ -548,16 +528,6 @@ PEC::ApplyPECtoElectronPressure (amrex::MultiFab* Pefield, const int lev, } const int nComp = Pefield->nComp(); -#ifdef WARPX_DIM_RZ - if (is_pec[0][1]) { - ablastr::warn_manager::WMRecordWarning( - "PEC", - "PEC boundary handling is not yet properly implemented for r_max so it is skipped in PEC::ApplyPECtoPefield", - ablastr::warn_manager::WarnPriority::medium); - is_pec[0][1] = false; - } -#endif - #ifdef AMREX_USE_OMP #pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) #endif diff --git a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.cpp b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.cpp index 7213e599b67..a94593ded83 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.cpp @@ -86,6 +86,12 @@ void HybridPICModel::AllocateLevelMFs (int lev, const BoxArray& ba, const Distri dm, ncomps, ngJ, lev, "current_fp_ampere[y]", 0.0_rt); WarpX::AllocInitMultiFab(current_fp_ampere[lev][2], amrex::convert(ba, jz_nodal_flag), dm, ncomps, ngJ, lev, "current_fp_ampere[z]", 0.0_rt); + +#ifdef WARPX_DIM_RZ + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + (ncomps == 1), + "Ohm's law solver only support m = 0 azimuthal mode at present."); +#endif } void HybridPICModel::ClearLevel (int lev) diff --git a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp index 38ca6c67fff..d58c2889c62 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp @@ -49,10 +49,10 @@ void FiniteDifferenceSolver::CalculateCurrentAmpere ( } // /** -// * \brief Calculate electron current from Ampere's law without displacement -// * current and the kinetically tracked ion currents i.e. J_e = curl B. +// * \brief Calculate total current from Ampere's law without displacement +// * current i.e. J = 1/mu_0 curl x B. // * -// * \param[out] Jfield vector of electron current MultiFabs at a given level +// * \param[out] Jfield vector of total current MultiFabs at a given level // * \param[in] Bfield vector of magnetic field MultiFabs at a given level // */ #ifdef WARPX_DIM_RZ @@ -64,11 +64,189 @@ void FiniteDifferenceSolver::CalculateCurrentAmpereCylindrical ( int lev ) { - amrex::ignore_unused(Jfield, Bfield, edge_lengths, lev); - amrex::Abort(Utils::TextMsg::Err( - "currently hybrid E-solve does not work for RZ")); + // for the profiler + amrex::LayoutData* cost = WarpX::getCosts(lev); + +#ifndef AMREX_USE_EB + amrex::ignore_unused(edge_lengths); +#endif + + // reset Jfield + Jfield[0]->setVal(0); + Jfield[1]->setVal(0); + Jfield[2]->setVal(0); + + // Loop through the grids, and over the tiles within each grid +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for ( MFIter mfi(*Jfield[0], TilingIfNotGPU()); mfi.isValid(); ++mfi ) { + if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) + { + amrex::Gpu::synchronize(); + } + Real wt = static_cast(amrex::second()); + + // Extract field data for this grid/tile + Array4 const& Jr = Jfield[0]->array(mfi); + Array4 const& Jt = Jfield[1]->array(mfi); + Array4 const& Jz = Jfield[2]->array(mfi); + Array4 const& Br = Bfield[0]->array(mfi); + Array4 const& Bt = Bfield[1]->array(mfi); + Array4 const& Bz = Bfield[2]->array(mfi); + +#ifdef AMREX_USE_EB + amrex::Array4 const& lr = edge_lengths[0]->array(mfi); + amrex::Array4 const& lt = edge_lengths[1]->array(mfi); + amrex::Array4 const& lz = edge_lengths[2]->array(mfi); +#endif + + // Extract stencil coefficients + Real const * const AMREX_RESTRICT coefs_r = m_stencil_coefs_r.dataPtr(); + int const n_coefs_r = static_cast(m_stencil_coefs_r.size()); + Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); + int const n_coefs_z = static_cast(m_stencil_coefs_z.size()); + + // Extract cylindrical specific parameters + Real const dr = m_dr; + int const nmodes = m_nmodes; + Real const rmin = m_rmin; + + // Extract tileboxes for which to loop + Box const& tjr = mfi.tilebox(Jfield[0]->ixType().toIntVect()); + Box const& tjt = mfi.tilebox(Jfield[1]->ixType().toIntVect()); + Box const& tjz = mfi.tilebox(Jfield[2]->ixType().toIntVect()); + + Real const one_over_mu0 = 1._rt / PhysConst::mu0; + + // Calculate the total current, using Ampere's law, on the same grid + // as the E-field + amrex::ParallelFor(tjr, tjt, tjz, + + // Jr calculation + [=] AMREX_GPU_DEVICE (int i, int j, int /*k*/){ +#ifdef AMREX_USE_EB + // Skip if this cell is fully covered by embedded boundaries + if (lr(i, j, 0) <= 0) return; +#endif + // Mode m=0 + Jr(i, j, 0, 0) = one_over_mu0 * ( + - T_Algo::DownwardDz(Bt, coefs_z, n_coefs_z, i, j, 0, 0) + ); + + // Higher-order modes + // r on cell-centered point (Jr is cell-centered in r) + Real const r = rmin + (i + 0.5_rt)*dr; + for (int m=1; m 0.5_rt*dr) { + // Mode m=0 + Jt(i, j, 0, 0) = one_over_mu0 * ( + - T_Algo::DownwardDr(Bz, coefs_r, n_coefs_r, i, j, 0, 0) + + T_Algo::DownwardDz(Br, coefs_z, n_coefs_z, i, j, 0, 0) + ); + + // Higher-order modes + for (int m=1 ; m 0.5_rt*dr) { + // Mode m=0 + Jz(i, j, 0, 0) = one_over_mu0 * ( + T_Algo::DownwardDrr_over_r(Bt, r, dr, coefs_r, n_coefs_r, i, j, 0, 0) + ); + // Higher-order modes + for (int m=1 ; m(amrex::second()) - wt; + amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); + } + } } + #else + template void FiniteDifferenceSolver::CalculateCurrentAmpereCartesian ( std::array< std::unique_ptr, 3 >& Jfield, @@ -129,8 +307,8 @@ void FiniteDifferenceSolver::CalculateCurrentAmpereCartesian ( Real const one_over_mu0 = 1._rt / PhysConst::mu0; - // First calculate the total current using Ampere's law on the - // same grid as the E-field + // Calculate the total current, using Ampere's law, on the same grid + // as the E-field amrex::ParallelFor(tjx, tjy, tjz, // Jx calculation @@ -238,12 +416,255 @@ void FiniteDifferenceSolver::HybridPICSolveECylindrical ( #ifndef AMREX_USE_EB amrex::ignore_unused(edge_lengths); #endif - amrex::ignore_unused( - Efield, Jfield, Jifield, Bfield, rhofield, Pefield, edge_lengths, - lev, hybrid_model, include_resistivity_term - ); - amrex::Abort(Utils::TextMsg::Err( - "currently hybrid E-solve does not work for RZ")); + + // Both steps below do not currently support m > 0 and should be + // modified if such support wants to be added + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( + (m_nmodes == 1), + "Ohm's law solver only support m = 0 azimuthal mode at present."); + + // for the profiler + amrex::LayoutData* cost = WarpX::getCosts(lev); + + using namespace ablastr::coarsen::sample; + + // get hybrid model parameters + const auto eta = hybrid_model->m_eta; + const auto rho_floor = hybrid_model->m_n_floor * PhysConst::q_e; + + // Index type required for interpolating fields from their respective + // staggering to the Ex, Ey, Ez locations + amrex::GpuArray const& Er_stag = hybrid_model->Ex_IndexType; + amrex::GpuArray const& Et_stag = hybrid_model->Ey_IndexType; + amrex::GpuArray const& Ez_stag = hybrid_model->Ez_IndexType; + amrex::GpuArray const& Jr_stag = hybrid_model->Jx_IndexType; + amrex::GpuArray const& Jt_stag = hybrid_model->Jy_IndexType; + amrex::GpuArray const& Jz_stag = hybrid_model->Jz_IndexType; + amrex::GpuArray const& Br_stag = hybrid_model->Bx_IndexType; + amrex::GpuArray const& Bt_stag = hybrid_model->By_IndexType; + amrex::GpuArray const& Bz_stag = hybrid_model->Bz_IndexType; + + // Parameters for `interp` that maps from Yee to nodal mesh and back + amrex::GpuArray const& nodal = {1, 1, 1}; + // The "coarsening is just 1 i.e. no coarsening" + amrex::GpuArray const& coarsen = {1, 1, 1}; + + // The E-field calculation is done in 2 steps: + // 1) The J x B term is calculated on a nodal mesh in order to ensure + // energy conservation. + // 2) The nodal E-field values are averaged onto the Yee grid and the + // electron pressure & resistivity terms are added (these terms are + // naturally located on the Yee grid). + + // Create a temporary multifab to hold the nodal E-field values + // Note the multifab has 3 values for Ex, Ey and Ez which we can do here + // since all three components will be calculated on the same grid. + // Also note that enE_nodal_mf does not need to have any guard cells since + // these values will be interpolated to the Yee mesh which is contained + // by the nodal mesh. + auto const& ba = convert(rhofield->boxArray(), IntVect::TheNodeVector()); + MultiFab enE_nodal_mf(ba, rhofield->DistributionMap(), 3, IntVect::TheZeroVector()); + + // Loop through the grids, and over the tiles within each grid for the + // initial, nodal calculation of E +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for ( MFIter mfi(enE_nodal_mf, TilingIfNotGPU()); mfi.isValid(); ++mfi ) { + if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) + { + amrex::Gpu::synchronize(); + } + Real wt = static_cast(amrex::second()); + + Array4 const& enE_nodal = enE_nodal_mf.array(mfi); + Array4 const& Jr = Jfield[0]->const_array(mfi); + Array4 const& Jt = Jfield[1]->const_array(mfi); + Array4 const& Jz = Jfield[2]->const_array(mfi); + Array4 const& Jir = Jifield[0]->const_array(mfi); + Array4 const& Jit = Jifield[1]->const_array(mfi); + Array4 const& Jiz = Jifield[2]->const_array(mfi); + Array4 const& Br = Bfield[0]->const_array(mfi); + Array4 const& Bt = Bfield[1]->const_array(mfi); + Array4 const& Bz = Bfield[2]->const_array(mfi); + + // Loop over the cells and update the nodal E field + amrex::ParallelFor(mfi.tilebox(), [=] AMREX_GPU_DEVICE (int i, int j, int /*k*/){ + + // interpolate the total current to a nodal grid + auto const jr_interp = Interp(Jr, Jr_stag, nodal, coarsen, i, j, 0, 0); + auto const jt_interp = Interp(Jt, Jt_stag, nodal, coarsen, i, j, 0, 0); + auto const jz_interp = Interp(Jz, Jz_stag, nodal, coarsen, i, j, 0, 0); + + // interpolate the ion current to a nodal grid + auto const jir_interp = Interp(Jir, Jr_stag, nodal, coarsen, i, j, 0, 0); + auto const jit_interp = Interp(Jit, Jt_stag, nodal, coarsen, i, j, 0, 0); + auto const jiz_interp = Interp(Jiz, Jz_stag, nodal, coarsen, i, j, 0, 0); + + // interpolate the B field to a nodal grid + auto const Br_interp = Interp(Br, Br_stag, nodal, coarsen, i, j, 0, 0); + auto const Bt_interp = Interp(Bt, Bt_stag, nodal, coarsen, i, j, 0, 0); + auto const Bz_interp = Interp(Bz, Bz_stag, nodal, coarsen, i, j, 0, 0); + + // calculate enE = (J - Ji) x B + enE_nodal(i, j, 0, 0) = ( + (jt_interp - jit_interp) * Bz_interp + - (jz_interp - jiz_interp) * Bt_interp + ); + enE_nodal(i, j, 0, 1) = ( + (jz_interp - jiz_interp) * Br_interp + - (jr_interp - jir_interp) * Bz_interp + ); + enE_nodal(i, j, 0, 2) = ( + (jr_interp - jir_interp) * Bt_interp + - (jt_interp - jit_interp) * Br_interp + ); + }); + + if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) + { + amrex::Gpu::synchronize(); + wt = static_cast(amrex::second()) - wt; + amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); + } + } + + // Loop through the grids, and over the tiles within each grid again + // for the Yee grid calculation of the E field +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif + for ( MFIter mfi(*Efield[0], TilingIfNotGPU()); mfi.isValid(); ++mfi ) { + if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) + { + amrex::Gpu::synchronize(); + } + Real wt = static_cast(amrex::second()); + + // Extract field data for this grid/tile + Array4 const& Er = Efield[0]->array(mfi); + Array4 const& Et = Efield[1]->array(mfi); + Array4 const& Ez = Efield[2]->array(mfi); + Array4 const& Jr = Jfield[0]->const_array(mfi); + Array4 const& Jt = Jfield[1]->const_array(mfi); + Array4 const& Jz = Jfield[2]->const_array(mfi); + Array4 const& enE = enE_nodal_mf.const_array(mfi); + Array4 const& rho = rhofield->const_array(mfi); + Array4 const& Pe = Pefield->array(mfi); + +#ifdef AMREX_USE_EB + amrex::Array4 const& lr = edge_lengths[0]->array(mfi); + amrex::Array4 const& lt = edge_lengths[1]->array(mfi); + amrex::Array4 const& lz = edge_lengths[2]->array(mfi); +#endif + + // Extract stencil coefficients + Real const * const AMREX_RESTRICT coefs_r = m_stencil_coefs_r.dataPtr(); + int const n_coefs_r = static_cast(m_stencil_coefs_r.size()); + Real const * const AMREX_RESTRICT coefs_z = m_stencil_coefs_z.dataPtr(); + int const n_coefs_z = static_cast(m_stencil_coefs_z.size()); + + // Extract cylindrical specific parameters + Real const dr = m_dr; + Real const rmin = m_rmin; + + Box const& ter = mfi.tilebox(Efield[0]->ixType().toIntVect()); + Box const& tet = mfi.tilebox(Efield[1]->ixType().toIntVect()); + Box const& tez = mfi.tilebox(Efield[2]->ixType().toIntVect()); + + // Loop over the cells and update the E field + amrex::ParallelFor(ter, tet, tez, + + // Er calculation + [=] AMREX_GPU_DEVICE (int i, int j, int /*k*/){ +#ifdef AMREX_USE_EB + // Skip if this cell is fully covered by embedded boundaries + if (lr(i, j, 0) <= 0) return; +#endif + // Interpolate to get the appropriate charge density in space + Real rho_val = Interp(rho, nodal, Er_stag, coarsen, i, j, 0, 0); + + // safety condition since we divide by rho_val later + if (rho_val < rho_floor) rho_val = rho_floor; + + // Get the gradient of the electron pressure + auto grad_Pe = T_Algo::UpwardDr(Pe, coefs_r, n_coefs_r, i, j, 0, 0); + + // interpolate the nodal neE values to the Yee grid + auto enE_r = Interp(enE, nodal, Er_stag, coarsen, i, j, 0, 0); + + Er(i, j, 0) = (enE_r - grad_Pe) / rho_val; + + // Add resistivity only if E field value is used to update B + if (include_resistivity_term) Er(i, j, 0) += eta(rho_val) * Jr(i, j, 0); + }, + + // Et calculation + [=] AMREX_GPU_DEVICE (int i, int j, int /*k*/){ +#ifdef AMREX_USE_EB + // In RZ Et is associated with a mesh node, so we need to check if the mesh node is covered + amrex::ignore_unused(lt); + if (lr(i, j, 0)<=0 || lr(i-1, j, 0)<=0 || lz(i, j-1, 0)<=0 || lz(i, j, 0)<=0) return; +#endif + // r on a nodal grid (Et is nodal in r) + Real const r = rmin + i*dr; + // Mode m=0: // Ensure that Et remains 0 on axis + if (r < 0.5_rt*dr) { + Et(i, j, 0, 0) = 0.; + return; + } + + // Interpolate to get the appropriate charge density in space + Real rho_val = Interp(rho, nodal, Er_stag, coarsen, i, j, 0, 0); + + // safety condition since we divide by rho_val later + if (rho_val < rho_floor) rho_val = rho_floor; + + // Get the gradient of the electron pressure + // -> d/dt = 0 for m = 0 + auto grad_Pe = 0.0_rt; + + // interpolate the nodal neE values to the Yee grid + auto enE_t = Interp(enE, nodal, Et_stag, coarsen, i, j, 0, 1); + + Et(i, j, 0) = (enE_t - grad_Pe) / rho_val; + + // Add resistivity only if E field value is used to update B + if (include_resistivity_term) Et(i, j, 0) += eta(rho_val) * Jt(i, j, 0); + }, + + // Ez calculation + [=] AMREX_GPU_DEVICE (int i, int j, int k){ +#ifdef AMREX_USE_EB + // Skip field solve if this cell is fully covered by embedded boundaries + if (lz(i,j,0) <= 0) return; +#endif + // Interpolate to get the appropriate charge density in space + Real rho_val = Interp(rho, nodal, Ez_stag, coarsen, i, j, k, 0); + + // safety condition since we divide by rho_val later + if (rho_val < rho_floor) rho_val = rho_floor; + + // Get the gradient of the electron pressure + auto grad_Pe = T_Algo::UpwardDz(Pe, coefs_z, n_coefs_z, i, j, k, 0); + + // interpolate the nodal neE values to the Yee grid + auto enE_z = Interp(enE, nodal, Ez_stag, coarsen, i, j, k, 2); + + Ez(i, j, k) = (enE_z - grad_Pe) / rho_val; + + // Add resistivity only if E field value is used to update B + if (include_resistivity_term) Ez(i, j, k) += eta(rho_val) * Jz(i, j, k); + } + ); + + if (cost && WarpX::load_balance_costs_update_algo == LoadBalanceCostsUpdateAlgo::Timers) + { + amrex::Gpu::synchronize(); + wt = static_cast(amrex::second()) - wt; + amrex::HostDevice::Atomic::Add( &(*cost)[mfi.index()], wt); + } + } } #else @@ -470,7 +891,6 @@ void FiniteDifferenceSolver::HybridPICSolveECartesian ( // Ez calculation [=] AMREX_GPU_DEVICE (int i, int j, int k){ - #ifdef AMREX_USE_EB // Skip field solve if this cell is fully covered by embedded boundaries if (lz(i,j,k) <= 0) return; diff --git a/Source/Initialization/WarpXInitData.cpp b/Source/Initialization/WarpXInitData.cpp index 4e82ed0dfb5..e16e97dda17 100644 --- a/Source/Initialization/WarpXInitData.cpp +++ b/Source/Initialization/WarpXInitData.cpp @@ -803,28 +803,35 @@ WarpX::InitLevelData (int lev, Real /*time*/) // provided in the input file. if (B_ext_grid_s == "parse_b_ext_grid_function") { -#ifdef WARPX_DIM_RZ - WARPX_ABORT_WITH_MESSAGE( - "E and B parser for external fields does not work with RZ -- TO DO"); -#endif - - //! Strings storing parser function to initialize the components of the magnetic field on the grid - std::string str_Bx_ext_grid_function; - std::string str_By_ext_grid_function; - std::string str_Bz_ext_grid_function; + //! Strings storing parser function to initialize the components of the magnetic field on the grid + std::string str_Bx_ext_grid_function; + std::string str_By_ext_grid_function; + std::string str_Bz_ext_grid_function; +#ifdef WARPX_DIM_RZ + std::stringstream warnMsg; + warnMsg << "Parser for external B (r and theta) fields does not work with RZ\n" + << "The initial Br and Bt fields are currently hardcoded to 0.\n" + << "The initial Bz field should only be a function of z.\n"; + ablastr::warn_manager::WMRecordWarning( + "Inputs", warnMsg.str(), ablastr::warn_manager::WarnPriority::high); + str_Bx_ext_grid_function = "0"; + str_By_ext_grid_function = "0"; +#else utils::parser::Store_parserString(pp_warpx, "Bx_external_grid_function(x,y,z)", str_Bx_ext_grid_function); utils::parser::Store_parserString(pp_warpx, "By_external_grid_function(x,y,z)", str_By_ext_grid_function); - utils::parser::Store_parserString(pp_warpx, "Bz_external_grid_function(x,y,z)", - str_Bz_ext_grid_function); - Bxfield_parser = std::make_unique( - utils::parser::makeParser(str_Bx_ext_grid_function,{"x","y","z"})); - Byfield_parser = std::make_unique( - utils::parser::makeParser(str_By_ext_grid_function,{"x","y","z"})); - Bzfield_parser = std::make_unique( - utils::parser::makeParser(str_Bz_ext_grid_function,{"x","y","z"})); +#endif + utils::parser::Store_parserString(pp_warpx, "Bz_external_grid_function(x,y,z)", + str_Bz_ext_grid_function); + + Bxfield_parser = std::make_unique( + utils::parser::makeParser(str_Bx_ext_grid_function,{"x","y","z"})); + Byfield_parser = std::make_unique( + utils::parser::makeParser(str_By_ext_grid_function,{"x","y","z"})); + Bzfield_parser = std::make_unique( + utils::parser::makeParser(str_Bz_ext_grid_function,{"x","y","z"})); // Initialize Bfield_fp with external function InitializeExternalFieldsOnGridUsingParser(Bfield_fp[lev][0].get(), From 985992b6b0c68be024aa472910029d23fe0f67f6 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 17 Oct 2023 21:40:04 -0700 Subject: [PATCH 064/110] Python: Rename Callback Files (#4381) Renamed after thinning out everything else. --- Source/Evolve/WarpXEvolve.cpp | 2 +- Source/FieldSolver/ElectrostaticSolver.cpp | 2 +- .../MagnetostaticSolver/MagnetostaticSolver.cpp | 2 +- Source/Initialization/WarpXInitData.cpp | 2 +- Source/Python/CMakeLists.txt | 2 +- Source/Python/Make.package | 2 +- Source/Python/{WarpX_py.H => callbacks.H} | 8 ++++---- Source/Python/{WarpX_py.cpp => callbacks.cpp} | 4 ++-- Source/Python/pyWarpX.cpp | 2 +- 9 files changed, 13 insertions(+), 13 deletions(-) rename Source/Python/{WarpX_py.H => callbacks.H} (91%) rename Source/Python/{WarpX_py.cpp => callbacks.cpp} (97%) diff --git a/Source/Evolve/WarpXEvolve.cpp b/Source/Evolve/WarpXEvolve.cpp index d5f01300b06..1515a1b3ff0 100644 --- a/Source/Evolve/WarpXEvolve.cpp +++ b/Source/Evolve/WarpXEvolve.cpp @@ -27,7 +27,7 @@ #include "Fluids/MultiFluidContainer.H" #include "Fluids/WarpXFluidContainer.H" #include "Particles/ParticleBoundaryBuffer.H" -#include "Python/WarpX_py.H" +#include "Python/callbacks.H" #include "Utils/TextMsg.H" #include "Utils/WarpXAlgorithmSelection.H" #include "Utils/WarpXUtil.H" diff --git a/Source/FieldSolver/ElectrostaticSolver.cpp b/Source/FieldSolver/ElectrostaticSolver.cpp index 216e80f7494..73dac03508b 100644 --- a/Source/FieldSolver/ElectrostaticSolver.cpp +++ b/Source/FieldSolver/ElectrostaticSolver.cpp @@ -12,7 +12,7 @@ #include "Parallelization/GuardCellManager.H" #include "Particles/MultiParticleContainer.H" #include "Particles/WarpXParticleContainer.H" -#include "Python/WarpX_py.H" +#include "Python/callbacks.H" #include "Utils/Parser/ParserUtils.H" #include "Utils/WarpXAlgorithmSelection.H" #include "Utils/WarpXConst.H" diff --git a/Source/FieldSolver/MagnetostaticSolver/MagnetostaticSolver.cpp b/Source/FieldSolver/MagnetostaticSolver/MagnetostaticSolver.cpp index 7e9c3e83969..6b77a559b8c 100644 --- a/Source/FieldSolver/MagnetostaticSolver/MagnetostaticSolver.cpp +++ b/Source/FieldSolver/MagnetostaticSolver/MagnetostaticSolver.cpp @@ -10,7 +10,7 @@ #include "Parallelization/GuardCellManager.H" #include "Particles/MultiParticleContainer.H" #include "Particles/WarpXParticleContainer.H" -#include "Python/WarpX_py.H" +#include "Python/callbacks.H" #include "Utils/WarpXAlgorithmSelection.H" #include "Utils/WarpXConst.H" #include "Utils/TextMsg.H" diff --git a/Source/Initialization/WarpXInitData.cpp b/Source/Initialization/WarpXInitData.cpp index e16e97dda17..fbf173992c2 100644 --- a/Source/Initialization/WarpXInitData.cpp +++ b/Source/Initialization/WarpXInitData.cpp @@ -29,7 +29,7 @@ #include "Utils/WarpXConst.H" #include "Utils/WarpXProfilerWrapper.H" #include "Utils/WarpXUtil.H" -#include "Python/WarpX_py.H" +#include "Python/callbacks.H" #include #include diff --git a/Source/Python/CMakeLists.txt b/Source/Python/CMakeLists.txt index d976e3d2c05..17a75301306 100644 --- a/Source/Python/CMakeLists.txt +++ b/Source/Python/CMakeLists.txt @@ -7,7 +7,7 @@ foreach(D IN LISTS WarpX_DIMS) target_sources(lib_${SD} PRIVATE # callback hooks - WarpX_py.cpp + callbacks.cpp ) if(WarpX_PYTHON) target_sources(pyWarpX_${SD} diff --git a/Source/Python/Make.package b/Source/Python/Make.package index f107fbbfaaa..bf7121e3c77 100644 --- a/Source/Python/Make.package +++ b/Source/Python/Make.package @@ -1,3 +1,3 @@ -CEXE_sources += WarpX_py.cpp +CEXE_sources += callbacks.cpp VPATH_LOCATIONS += $(WARPX_HOME)/Source/Python diff --git a/Source/Python/WarpX_py.H b/Source/Python/callbacks.H similarity index 91% rename from Source/Python/WarpX_py.H rename to Source/Python/callbacks.H index 590f26a804b..f0c98caf636 100644 --- a/Source/Python/WarpX_py.H +++ b/Source/Python/callbacks.H @@ -2,12 +2,12 @@ * * This file is part of WarpX. * - * Authors: David Grote, Maxence Thevenet, Weiqun Zhang, Roelof Groenewald + * Authors: David Grote, Maxence Thevenet, Weiqun Zhang, Roelof Groenewald, Axel Huebl * * License: BSD-3-Clause-LBNL */ -#ifndef WARPX_PY_H_ -#define WARPX_PY_H_ +#ifndef WARPX_PY_CALLBACKS_H_ +#define WARPX_PY_CALLBACKS_H_ #include "Utils/export.H" #include "Utils/WarpXProfilerWrapper.H" @@ -46,4 +46,4 @@ void ExecutePythonCallback ( std::string name ); */ void ClearPythonCallback ( std::string name ); -#endif +#endif // WARPX_PY_CALLBACKS_H_ diff --git a/Source/Python/WarpX_py.cpp b/Source/Python/callbacks.cpp similarity index 97% rename from Source/Python/WarpX_py.cpp rename to Source/Python/callbacks.cpp index afc734d780e..930c88e65d1 100644 --- a/Source/Python/WarpX_py.cpp +++ b/Source/Python/callbacks.cpp @@ -2,11 +2,11 @@ * * This file is part of WarpX. * - * Authors: David Grote, Maxence Thevenet, Weiqun Zhang, Roelof Groenewald + * Authors: David Grote, Maxence Thevenet, Weiqun Zhang, Roelof Groenewald, Axel Huebl * * License: BSD-3-Clause-LBNL */ -#include "WarpX_py.H" +#include "callbacks.H" #include #include diff --git a/Source/Python/pyWarpX.cpp b/Source/Python/pyWarpX.cpp index 64db77153bd..26f4c77502d 100644 --- a/Source/Python/pyWarpX.cpp +++ b/Source/Python/pyWarpX.cpp @@ -4,7 +4,7 @@ * License: BSD-3-Clause-LBNL */ #include "pyWarpX.H" -#include "WarpX_py.H" +#include "callbacks.H" #include // todo: move this out to Python/WarpX.cpp #include // todo: move to its own Python/Utils.cpp From 8459109a725bc50cfe7df02616e2119bba2cd1c5 Mon Sep 17 00:00:00 2001 From: Marta Galbiati <47305118+MartaGalbi@users.noreply.github.com> Date: Thu, 19 Oct 2023 05:57:57 +0200 Subject: [PATCH 065/110] add instructions for leonardo hpc (#4353) * add instructions for leonardo hpc * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix wrong name for build directory in leonardo.rst * Update Tools/machines/leonardo-cineca/install_gpu_dependencies.sh add copyright and license notice Co-authored-by: Luca Fedeli * fix formatting in Docs/source/install/hpc/leonardo.rst add space Co-authored-by: Axel Huebl * Fix formatting in Docs/source/install/hpc/leonardo.rst Add equals Co-authored-by: Axel Huebl * Fix formatting in Docs/source/install/hpc/leonardo.rst change quotation marks Co-authored-by: Axel Huebl * Fix formatting in Docs/source/install/hpc/leonardo.rst Co-authored-by: Axel Huebl * add separate instructions to install WarpX as a Python module Co-authored-by: Axel Huebl * add cleaning of both standard and python builds Co-authored-by: Axel Huebl * removed source of profile file in job.sh --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Luca Fedeli Co-authored-by: Axel Huebl --- Docs/source/install/hpc.rst | 1 + Docs/source/install/hpc/leonardo.rst | 172 ++++++++++++++++++ .../install_gpu_dependencies.sh | 103 +++++++++++ Tools/machines/leonardo-cineca/job.sh | 20 ++ .../leonardo_gpu_warpx.profile.example | 43 +++++ 5 files changed, 339 insertions(+) create mode 100644 Docs/source/install/hpc/leonardo.rst create mode 100644 Tools/machines/leonardo-cineca/install_gpu_dependencies.sh create mode 100644 Tools/machines/leonardo-cineca/job.sh create mode 100644 Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example diff --git a/Docs/source/install/hpc.rst b/Docs/source/install/hpc.rst index 67468cc2f02..d6e901276d7 100644 --- a/Docs/source/install/hpc.rst +++ b/Docs/source/install/hpc.rst @@ -42,6 +42,7 @@ This section documents quick-start guides for a selection of supercomputers that hpc/karolina hpc/lassen hpc/lawrencium + hpc/leonardo hpc/lumi hpc/lxplus hpc/ookami diff --git a/Docs/source/install/hpc/leonardo.rst b/Docs/source/install/hpc/leonardo.rst new file mode 100644 index 00000000000..3de7f4755cb --- /dev/null +++ b/Docs/source/install/hpc/leonardo.rst @@ -0,0 +1,172 @@ +.. _building-leonardo: + +Leonardo (CINECA) +================= + +The `Leonardo cluster `_ is hosted at `CINECA `_. + +On Leonardo, each one of the 3456 compute nodes features a custom Atos Bull Sequana XH21355 "Da Vinci" blade, composed of: + +* 1 x CPU Intel Ice Lake Xeon 8358 32 cores 2.60 GHz +* 512 (8 x 64) GB RAM DDR4 3200 MHz +* 4 x NVidia custom Ampere A100 GPU 64GB HBM2 +* 2 x NVidia HDR 2×100 GB/s cards + +Introduction +------------ + +If you are new to this system, **please see the following resources**: + +* `Leonardo website `_ +* `Leonardo user guide `_ + +Storage organization: + +* ``$HOME``: permanent, backed up, user specific (50 GB quota) +* ``$CINECA_SCRATCH``: temporary, user specific, no backup, a large disk for the storage of run time data and files, automatic cleaning procedure of data older than 40 days +* ``$PUBLIC``: permanent, no backup (50 GB quota) +* ``$WORK``: permanent, project specific, no backup + +.. _building-leonardo-preparation: + +Preparation +----------- + +Use the following commands to download the WarpX source code: + +.. code-block:: bash + + git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx + +We use system software modules, add environment hints and further dependencies via the file ``$HOME/leonardo_gpu_warpx.profile``. +Create it now: + +.. code-block:: bash + + cp $HOME/src/warpx/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example $HOME/leonardo_gpu_warpx.profile + +.. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example + :language: bash + +.. important:: + + Now, and as the first step on future logins to Leonardo, activate these environment settings: + + .. code-block:: bash + + source $HOME/leonardo_gpu_warpx.profile + +Finally, since Leonardo does not yet provide software modules for some of our dependencies, install them once: + +.. code-block:: bash + + bash $HOME/src/warpx/Tools/machines/leonardo_cineca/install_gpu_dependencies.sh + source $HOME/sw/venvs/warpx/bin/activate + +.. dropdown:: Script Details + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../../Tools/machines/leonardo-cineca/install_gpu_dependencies.sh + :language: bash + + +.. _building-leonardo-compilation: + +Compilation +----------- + +Use the following :ref:`cmake commands ` to compile the application executable: + +.. code-block:: bash + + cd $HOME/src/warpx + rm -rf build_gpu + + cmake -S . -B build_gpu -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_gpu -j 16 + +The WarpX application executables are now in ``$HOME/src/warpx/build_gpu/bin/``. +Additionally, the following commands will install WarpX as a Python module: + +.. code-block:: bash + + cd $HOME/src/warpx + rm -rf build_gpu_py + + cmake -S . -B build_gpu_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_PYTHON=ON -DWarpX_APP=OFF -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_gpu_py -j 16 --target pip_install + +Now, you can :ref:`submit Leonardo compute jobs ` for WarpX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). +Or, you can use the WarpX executables to submit Leonardo jobs (:ref:`example inputs `). +For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$CINECA_SCRATCH``. + +.. _building-leonardo-update: + +Update WarpX & Dependencies +--------------------------- + +If you already installed WarpX in the past and want to update it, start by getting the latest source code: + +.. code-block:: bash + + cd $HOME/src/warpx + + # read the output of this command - does it look ok? + git status + + # get the latest WarpX source code + git fetch + git pull + + # read the output of these commands - do they look ok? + git status + git log # press q to exit + +And, if needed, + +- :ref:`update the leonardo_gpu_warpx.profile file `, +- log out and into the system, activate the now updated environment profile as usual, +- :ref:`execute the dependency install scripts `. + +As a last step, clean the build directories ``rm -rf $HOME/src/warpx/build_gpu*`` and rebuild WarpX. + + +.. _running-leonardo: + +Running +------- +The batch script below can be used to run a WarpX simulation on multiple nodes on Leonardo. +Replace descriptions between chevrons ``<>`` by relevant values. +Note that we run one MPI rank per GPU. + +.. literalinclude:: ../../../../Tools/machines/leonardo-cineca/job.sh + :language: bash + :caption: You can copy this file from ``$HOME/src/warpx/Tools/machines/leonardo-cineca/job.sh``. + +To run a simulation, copy the lines above to a file ``job.sh`` and run + +.. code-block:: bash + + sbatch job.sh + +to submit the job. + +.. _post-processing-leonardo: + +Post-Processing +--------------- + +For post-processing, activate the environment settings: + +.. code-block:: bash + + source $HOME/leonardo_gpu_warpx.profile + +and run python scripts. diff --git a/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh b/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh new file mode 100644 index 00000000000..5b6453b3968 --- /dev/null +++ b/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh @@ -0,0 +1,103 @@ +#!/bin/bash +# +# Copyright 2023 The WarpX Community +# +# This file is part of WarpX. +# +# Author: Axel Huebl, Marta Galbiati +# License: BSD-3-Clause-LBNL + +set -eu -o pipefail + + +# Check: ###################################################################### +# +# Was leonardo_gpu_warpx.profile sourced and configured correctly? +# + + +# Remove old dependencies ##################################################### +# +SW_DIR="$HOME/sw" +rm -rf ${SW_DIR} +mkdir -p ${SW_DIR} + + +# General extra dependencies ################################################## +# + +# ADIOS2 +if [ -d $HOME/src/adios2 ] +then + cd $HOME/src/adios2 + git fetch + git checkout master + git pull + cd - +else + git clone https://github.com/ornladios/ADIOS2.git $HOME/src/adios2 +fi +rm -rf $HOME/src/adios2-gpu-build +cmake -S $HOME/src/adios2 -B $HOME/src/adios2-gpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-master +cmake --build $HOME/src/adios2-gpu-build --target install -j 16 +rm -rf $HOME/src/adios2-gpu-build + + +# BLAS++ (for PSATD+RZ) +if [ -d $HOME/src/blaspp ] +then + cd $HOME/src/blaspp + git fetch + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp +fi +rm -rf $HOME/src/blaspp-gpu-build +CXX=$(which g++) cmake -S $HOME/src/blaspp -B $HOME/src/blaspp-gpu-build -Duse_openmp=OFF -Dgpu_backend=cuda -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master +cmake --build $HOME/src/blaspp-gpu-build --target install --parallel 16 +rm -rf $HOME/src/blaspp-gpu-build + + +# LAPACK++ (for PSATD+RZ) +if [ -d $HOME/src/lapackpp ] +then + cd $HOME/src/lapackpp + git fetch + git checkout master + git pull + cd - +else + git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp +fi +rm -rf $HOME/src/lapackpp-gpu-build +CXX=$(which CC) CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S $HOME/src/lapackpp -B $HOME/src/lapackpp-gpu-build -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master +cmake --build $HOME/src/lapackpp-gpu-build --target install --parallel 16 +rm -rf $HOME/src/lapackpp-gpu-build + + +# Python ###################################################################### +# +rm -rf ${SW_DIR}/venvs/warpx +python3 -m venv ${SW_DIR}/venvs/warpx +source ${SW_DIR}/venvs/warpx/bin/activate +python3 -m ensurepip --upgrade +python3 -m pip cache purge +python3 -m pip install --upgrade pip +python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade cython +python3 -m pip install --upgrade numpy +python3 -m pip install --upgrade pandas +python3 -m pip install --upgrade scipy +MPICC="gcc -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py +python3 -m pip install --upgrade openpmd-api +python3 -m pip install --upgrade matplotlib +python3 -m pip install --upgrade yt +# install or update WarpX dependencies such as picmistandard +python3 -m pip install --upgrade -r $HOME/src/warpx/requirements.txt +# optional: for libEnsemble +python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt +# optional: for optimas (based on libEnsemble & ax->botorch->gpytorch->pytorch) +python3 -m pip install --upgrade torch # CUDA 11.8 compatible wheel +python3 -m pip install -r $HOME/src/warpx/Tools/optimas/requirements.txt diff --git a/Tools/machines/leonardo-cineca/job.sh b/Tools/machines/leonardo-cineca/job.sh new file mode 100644 index 00000000000..a50c6e1995f --- /dev/null +++ b/Tools/machines/leonardo-cineca/job.sh @@ -0,0 +1,20 @@ +#!/usr/bin/bash +#SBATCH --time=02:00:00 +#SBATCH --nodes=2 +#SBATCH --ntasks-per-node=4 +#SBATCH --ntasks-per-socket=4 +#SBATCH --cpus-per-task=8 +#SBATCH --gpus-per-node=4 +#SBATCH --gpus-per-task=1 +#SBATCH --mem=494000 +#SBATCH --partition=boost_usr_prod +#SBATCH --job-name= +#SBATCH --gres=gpu:4 +#SBATCH --err=job.err +#SBATCH --out=job.out +#SBATCH --account= +#SBATCH --mail-type=ALL +#SBATCH --mail-user= + +cd /leonardo_scratch/large/userexternal// +srun /leonardo/home/userexternal//src/warpx/build_gpu/bin/warpx.2d > output.txt diff --git a/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example b/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example new file mode 100644 index 00000000000..af0cdbd41c3 --- /dev/null +++ b/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example @@ -0,0 +1,43 @@ +# required dependencies +module load profile/base +module load cmake/3.24.3 +module load gmp/6.2.1 +module load mpfr/4.1.0 +module load mpc/1.2.1 +module load gcc/11.3.0 +module load cuda/11.8 +module load zlib/1.2.13--gcc--11.3.0 +module load openmpi/4.1.4--gcc--11.3.0-cuda-11.8 + +# optional: for QED support with detailed tables +module load boost/1.80.0--openmpi--4.1.4--gcc--11.3.0 + +# optional: for openPMD and PSATD+RZ support +module load openblas/0.3.21--gcc--11.3.0 +export CMAKE_PREFIX_PATH=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-11.3.0/c-blosc-1.21.1-aifmix6v5lwxgt7rigwoebalrgbcnv26:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=$HOME/sw/adios2-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=$HOME/sw/blaspp-master:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=$HOME/sw/lapackpp-master:$CMAKE_PREFIX_PATH + +export LD_LIBRARY_PATH=/leonardo/prod/spack/03/install/0.19/linux-rhel8-icelake/gcc-11.3.0/c-blosc-1.21.1-aifmix6v5lwxgt7rigwoebalrgbcnv26/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$HOME/sw/adios2-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$HOME/sw/blaspp-master/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$HOME/sw/lapackpp-master/lib64:$LD_LIBRARY_PATH + +# optional: for Python bindings or libEnsemble +module load python/3.10.8--gcc--11.3.0 + +if [ -d "$HOME/sw/venvs/warpx" ] +then + source $HOME/sw/venvs/warpx/bin/activate +fi + +# optimize CUDA compilation for A100 +export AMREX_CUDA_ARCH=8.0 + +# compiler environment hints +export CXX=$(which g++) +export CC=$(which gcc) +export FC=$(which gfortran) +export CUDACXX=$(which nvcc) +export CUDAHOSTCXX=${CXX} From c5628f8ea456b923c718fcc4e0848f2bc6fa76a0 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 20 Oct 2023 17:54:43 -0700 Subject: [PATCH 066/110] GetAndSetPosition: General Particle PIdx (#4386) * GetAndSetPosition: General Particle PIdx Generalize the `SetParticlePosition` & `GetParticlePosition` classes to work with arbitrary particle index enumerators. That way, we can reuse them between different particle containers. * FieldProbe: Remove Hack * GetAndSetPosition: Usage w/ Template --- .../AcceleratorLattice/LatticeElementFinder.H | 2 +- .../LatticeElementFinder.cpp | 2 +- .../BackTransformParticleFunctor.H | 4 ++-- .../BackTransformParticleFunctor.cpp | 4 ++-- .../ReducedDiags/ColliderRelevant.cpp | 2 +- Source/Diagnostics/ReducedDiags/FieldProbe.cpp | 4 ++-- .../ReducedDiags/FieldProbeParticleContainer.H | 8 ++------ .../ReducedDiags/ParticleExtrema.cpp | 2 +- .../ReducedDiags/ParticleHistogram.cpp | 2 +- .../ReducedDiags/ParticleHistogram2D.cpp | 2 +- Source/EmbeddedBoundary/ParticleScraper.H | 2 +- .../BackgroundMCC/BackgroundMCCCollision.cpp | 2 +- .../BackgroundStopping/BackgroundStopping.cpp | 4 ++-- .../BinaryCollision/BinaryCollision.H | 10 +++++----- .../Coulomb/PairWiseCoulombCollisionFunc.H | 2 +- .../NuclearFusion/NuclearFusionFunc.H | 2 +- Source/Particles/Deposition/ChargeDeposition.H | 4 ++-- .../Particles/Deposition/CurrentDeposition.H | 8 ++++---- .../Deposition/SharedDepositionUtils.H | 2 +- .../Particles/ElementaryProcess/Ionization.H | 2 +- .../Particles/ElementaryProcess/Ionization.cpp | 2 +- .../ElementaryProcess/QEDPairGeneration.H | 2 +- .../ElementaryProcess/QEDPairGeneration.cpp | 2 +- .../ElementaryProcess/QEDPhotonEmission.H | 2 +- .../ElementaryProcess/QEDPhotonEmission.cpp | 2 +- Source/Particles/Gather/FieldGather.H | 2 +- Source/Particles/Gather/GetExternalFields.H | 2 +- Source/Particles/Gather/GetExternalFields.cpp | 2 +- Source/Particles/LaserParticleContainer.cpp | 6 +++--- Source/Particles/ParticleBoundaryBuffer.cpp | 2 +- Source/Particles/PhotonParticleContainer.cpp | 4 ++-- Source/Particles/PhysicalParticleContainer.cpp | 10 +++++----- Source/Particles/Pusher/CopyParticleAttribs.H | 4 ++-- Source/Particles/Pusher/GetAndSetPosition.H | 18 ++++++++++++++---- Source/Particles/Pusher/PushSelector.H | 4 ++-- .../RigidInjectedParticleContainer.cpp | 10 +++++----- Source/Particles/WarpXParticleContainer.cpp | 12 ++++++------ Source/ablastr/particles/DepositCharge.H | 2 +- 38 files changed, 82 insertions(+), 76 deletions(-) diff --git a/Source/AcceleratorLattice/LatticeElementFinder.H b/Source/AcceleratorLattice/LatticeElementFinder.H index 43e97980774..24cdb691bb0 100644 --- a/Source/AcceleratorLattice/LatticeElementFinder.H +++ b/Source/AcceleratorLattice/LatticeElementFinder.H @@ -167,7 +167,7 @@ struct LatticeElementFinderDevice amrex::ParticleReal m_uz_boost; amrex::Real m_time; - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; const amrex::ParticleReal* AMREX_RESTRICT m_ux = nullptr; const amrex::ParticleReal* AMREX_RESTRICT m_uy = nullptr; const amrex::ParticleReal* AMREX_RESTRICT m_uz = nullptr; diff --git a/Source/AcceleratorLattice/LatticeElementFinder.cpp b/Source/AcceleratorLattice/LatticeElementFinder.cpp index 7e9825c28a6..62ebdcc1f4f 100644 --- a/Source/AcceleratorLattice/LatticeElementFinder.cpp +++ b/Source/AcceleratorLattice/LatticeElementFinder.cpp @@ -96,7 +96,7 @@ LatticeElementFinderDevice::InitLatticeElementFinderDevice (WarpXParIter const& int const lev = a_pti.GetLevel(); - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); auto& attribs = a_pti.GetAttribs(); m_ux = attribs[PIdx::ux].dataPtr() + a_offset; m_uy = attribs[PIdx::uy].dataPtr() + a_offset; diff --git a/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.H index 4bb1b0656aa..afc3e6e8511 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.H @@ -62,7 +62,7 @@ struct SelectParticles } /** Object to extract the positions of the macroparticles inside a ParallelFor kernel */ - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; /** Z coordinate in boosted frame that corresponds to a give snapshot*/ amrex::Real m_current_z_boost; /** Previous Z coordinate in boosted frame that corresponds to a give snapshot*/ @@ -166,7 +166,7 @@ struct LorentzTransformParticles dst.m_rdata[PIdx::uz][i_dst] = uzp; } - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; amrex::ParticleReal* AMREX_RESTRICT m_xpold = nullptr; amrex::ParticleReal* AMREX_RESTRICT m_ypold = nullptr; diff --git a/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.cpp b/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.cpp index 41445c4bede..ce463a11ca3 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.cpp +++ b/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.cpp @@ -20,7 +20,7 @@ SelectParticles::SelectParticles (const WarpXParIter& a_pti, TmpParticles& tmp_p int a_offset) : m_current_z_boost(current_z_boost), m_old_z_boost(old_z_boost) { - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); const auto lev = a_pti.GetLevel(); const auto index = a_pti.GetPairIndex(); @@ -38,7 +38,7 @@ LorentzTransformParticles::LorentzTransformParticles ( const WarpXParIter& a_pti using namespace amrex::literals; if (tmp_particle_data.empty()) return; - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); auto& attribs = a_pti.GetAttribs(); m_wpnew = attribs[PIdx::w].dataPtr(); diff --git a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp index 188b5f3f76f..c4eaf78745d 100644 --- a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp +++ b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp @@ -456,7 +456,7 @@ void ColliderRelevant::ComputeDiags (int step) // Loop over boxes for (WarpXParIter pti(myspc, lev); pti.isValid(); ++pti) { - const auto GetPosition = GetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); // get particle arrays amrex::ParticleReal* const AMREX_RESTRICT ux = pti.GetAttribs()[PIdx::ux].dataPtr(); amrex::ParticleReal* const AMREX_RESTRICT uy = pti.GetAttribs()[PIdx::uy].dataPtr(); diff --git a/Source/Diagnostics/ReducedDiags/FieldProbe.cpp b/Source/Diagnostics/ReducedDiags/FieldProbe.cpp index 11975ef0483..98cba72cfae 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbe.cpp +++ b/Source/Diagnostics/ReducedDiags/FieldProbe.cpp @@ -429,8 +429,8 @@ void FieldProbe::ComputeDiags (int step) for (MyParIter pti(m_probe, lev); pti.isValid(); ++pti) { - const auto getPosition = GetParticlePosition(pti); - auto setPosition = SetParticlePosition(pti); + const auto getPosition = GetParticlePosition(pti); + auto setPosition = SetParticlePosition(pti); const auto& aos = pti.GetArrayOfStructs(); const auto* AMREX_RESTRICT m_structs = aos().dataPtr(); diff --git a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H index d658f209c8f..b3277b875d9 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H +++ b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H @@ -19,21 +19,17 @@ * This enumerated struct is used to index the field probe particle * values that are being stored as SoA data. Nattribs * is enumerated to give the number of attributes stored. - * The strange insertion of `theta` below is due to the use of - * GetParticlePosition for the field probe locations, which reads the probe - * theta value from PIdx::theta = 4. */ struct FieldProbePIdx { enum { Ex = 0, Ey, Ez, - Bx, + Bx, By, Bz, + S, //!< the Poynting vector #ifdef WARPX_DIM_RZ theta, ///< RZ needs all three position components #endif - By, Bz, - S, //!< the Poynting vector nattribs }; }; diff --git a/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp b/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp index e125f44e08f..78f8be8914e 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp +++ b/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp @@ -388,7 +388,7 @@ void ParticleExtrema::ComputeDiags (int step) // Loop over boxes for (WarpXParIter pti(myspc, lev); pti.isValid(); ++pti) { - const auto GetPosition = GetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); // get particle arrays amrex::ParticleReal* const AMREX_RESTRICT ux = pti.GetAttribs()[PIdx::ux].dataPtr(); amrex::ParticleReal* const AMREX_RESTRICT uy = pti.GetAttribs()[PIdx::uy].dataPtr(); diff --git a/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp b/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp index 1cbbbaaebff..9da001ba7ed 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp +++ b/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp @@ -196,7 +196,7 @@ void ParticleHistogram::ComputeDiags (int step) { for (WarpXParIter pti(myspc, lev); pti.isValid(); ++pti) { - auto const GetPosition = GetParticlePosition(pti); + auto const GetPosition = GetParticlePosition(pti); auto & attribs = pti.GetAttribs(); ParticleReal* const AMREX_RESTRICT d_w = attribs[PIdx::w].dataPtr(); diff --git a/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.cpp b/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.cpp index 0567f1cfc29..dfb53b7e2c3 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.cpp +++ b/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.cpp @@ -193,7 +193,7 @@ void ParticleHistogram2D::ComputeDiags (int step) { for (WarpXParIter pti(myspc, lev); pti.isValid(); ++pti) { - auto const GetPosition = GetParticlePosition(pti); + auto const GetPosition = GetParticlePosition(pti); auto & attribs = pti.GetAttribs(); ParticleReal* const AMREX_RESTRICT d_w = attribs[PIdx::w].dataPtr(); diff --git a/Source/EmbeddedBoundary/ParticleScraper.H b/Source/EmbeddedBoundary/ParticleScraper.H index b723fd2ce5e..f1aaf544b32 100644 --- a/Source/EmbeddedBoundary/ParticleScraper.H +++ b/Source/EmbeddedBoundary/ParticleScraper.H @@ -163,7 +163,7 @@ scrapeParticles (PC& pc, const amrex::Vector& distance_t const auto dxi = pc.Geom(lev).InvCellSizeArray(); for(WarpXParIter pti(pc, lev); pti.isValid(); ++pti) { - const auto getPosition = GetParticlePosition(pti); + const auto getPosition = GetParticlePosition(pti); auto& tile = pti.GetParticleTile(); auto ptd = tile.getParticleTileData(); const auto np = tile.numParticles(); diff --git a/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp b/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp index ce7fed952a7..ebf8cb47172 100644 --- a/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp +++ b/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.cpp @@ -340,7 +340,7 @@ void BackgroundMCCCollision::doBackgroundCollisionsWithinTile // we need particle positions in order to calculate the local density // and temperature - auto GetPosition = GetParticlePosition(pti); + auto GetPosition = GetParticlePosition(pti); // get Struct-Of-Array particle data, also called attribs auto& attribs = pti.GetAttribs(); diff --git a/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp b/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp index 878ef8a1f64..83f74840478 100644 --- a/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp +++ b/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.cpp @@ -159,7 +159,7 @@ void BackgroundStopping::doBackgroundStoppingOnElectronsWithinTile (WarpXParIter amrex::ParticleReal* const AMREX_RESTRICT uz = attribs[PIdx::uz].dataPtr(); // May be needed to evaluate the density and/or temperature functions - auto const GetPosition = GetParticlePosition(pti); + auto const GetPosition = GetParticlePosition(pti); amrex::ParallelFor(np, [=] AMREX_GPU_HOST_DEVICE (long ip) @@ -234,7 +234,7 @@ void BackgroundStopping::doBackgroundStoppingOnIonsWithinTile (WarpXParIter& pti amrex::ParticleReal* const AMREX_RESTRICT uz = attribs[PIdx::uz].dataPtr(); // May be needed to evaluate the density function - auto const GetPosition = GetParticlePosition(pti); + auto const GetPosition = GetParticlePosition(pti); amrex::ParallelFor(np, [=] AMREX_GPU_HOST_DEVICE (long ip) diff --git a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H index 6728e4cef53..2f9ddbbcb3f 100644 --- a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H +++ b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H @@ -221,7 +221,7 @@ public: // Store product species data in vectors const int n_product_species = m_product_species.size(); amrex::Vector tile_products; - amrex::Vector get_position_products; + amrex::Vector> get_position_products; amrex::Vector products_np; amrex::Vector products_mass; constexpr int getpos_offset = 0; @@ -229,7 +229,7 @@ public: { ParticleTileType& ptile_product = product_species_vector[i]->ParticlesAt(lev, mfi); tile_products.push_back(&ptile_product); - get_position_products.push_back(GetParticlePosition(ptile_product, + get_position_products.push_back(GetParticlePosition(ptile_product, getpos_offset)); products_np.push_back(ptile_product.numParticles()); products_mass.push_back(product_species_vector[i]->getMass()); @@ -254,7 +254,7 @@ public: index_type const* AMREX_RESTRICT cell_offsets_1 = bins_1.offsetsPtr(); const amrex::ParticleReal q1 = species_1.getCharge(); const amrex::ParticleReal m1 = species_1.getMass(); - auto get_position_1 = GetParticlePosition(ptile_1, getpos_offset); + auto get_position_1 = GetParticlePosition(ptile_1, getpos_offset); // Needed to access the particle id ParticleType * AMREX_RESTRICT particle_ptr_1 = ptile_1.GetArrayOfStructs()().data(); @@ -394,7 +394,7 @@ public: index_type const* AMREX_RESTRICT cell_offsets_1 = bins_1.offsetsPtr(); const amrex::ParticleReal q1 = species_1.getCharge(); const amrex::ParticleReal m1 = species_1.getMass(); - auto get_position_1 = GetParticlePosition(ptile_1, getpos_offset); + auto get_position_1 = GetParticlePosition(ptile_1, getpos_offset); // Needed to access the particle id ParticleType * AMREX_RESTRICT particle_ptr_1 = ptile_1.GetArrayOfStructs()().data(); @@ -404,7 +404,7 @@ public: index_type const* AMREX_RESTRICT cell_offsets_2 = bins_2.offsetsPtr(); const amrex::ParticleReal q2 = species_2.getCharge(); const amrex::ParticleReal m2 = species_2.getMass(); - auto get_position_2 = GetParticlePosition(ptile_2, getpos_offset); + auto get_position_2 = GetParticlePosition(ptile_2, getpos_offset); // Needed to access the particle id ParticleType * AMREX_RESTRICT particle_ptr_2 = ptile_2.GetArrayOfStructs()().data(); diff --git a/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H b/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H index 3c5f3abd841..d8fdaee8bd0 100644 --- a/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H +++ b/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H @@ -77,7 +77,7 @@ public: index_type const* AMREX_RESTRICT I1, index_type const* AMREX_RESTRICT I2, SoaData_type soa_1, SoaData_type soa_2, - GetParticlePosition /*get_position_1*/, GetParticlePosition /*get_position_2*/, + GetParticlePosition /*get_position_1*/, GetParticlePosition /*get_position_2*/, amrex::ParticleReal const q1, amrex::ParticleReal const q2, amrex::ParticleReal const m1, amrex::ParticleReal const m2, amrex::Real const dt, amrex::Real const dV, diff --git a/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H b/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H index db8dacce1d4..83952357ec5 100644 --- a/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H +++ b/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H @@ -123,7 +123,7 @@ public: index_type const* AMREX_RESTRICT I1, index_type const* AMREX_RESTRICT I2, SoaData_type soa_1, SoaData_type soa_2, - GetParticlePosition /*get_position_1*/, GetParticlePosition /*get_position_2*/, + GetParticlePosition /*get_position_1*/, GetParticlePosition /*get_position_2*/, amrex::ParticleReal const /*q1*/, amrex::ParticleReal const /*q2*/, amrex::ParticleReal const m1, amrex::ParticleReal const m2, amrex::Real const dt, amrex::Real const dV, diff --git a/Source/Particles/Deposition/ChargeDeposition.H b/Source/Particles/Deposition/ChargeDeposition.H index 20ec4c2e34d..df7097f7600 100644 --- a/Source/Particles/Deposition/ChargeDeposition.H +++ b/Source/Particles/Deposition/ChargeDeposition.H @@ -37,7 +37,7 @@ * \param load_balance_costs_update_algo Selected method for updating load balance costs. */ template -void doChargeDepositionShapeN (const GetParticlePosition& GetPosition, +void doChargeDepositionShapeN (const GetParticlePosition& GetPosition, const amrex::ParticleReal * const wp, const int* ion_lev, amrex::FArrayBox& rho_fab, @@ -239,7 +239,7 @@ void doChargeDepositionShapeN (const GetParticlePosition& GetPosition, * \param bin_size tile size to use for shared current deposition operations */ template -void doChargeDepositionSharedShapeN (const GetParticlePosition& GetPosition, +void doChargeDepositionSharedShapeN (const GetParticlePosition& GetPosition, const amrex::ParticleReal * const wp, const int* ion_lev, amrex::FArrayBox& rho_fab, diff --git a/Source/Particles/Deposition/CurrentDeposition.H b/Source/Particles/Deposition/CurrentDeposition.H index 49a7c46a3e8..2c4e4bd448d 100644 --- a/Source/Particles/Deposition/CurrentDeposition.H +++ b/Source/Particles/Deposition/CurrentDeposition.H @@ -51,7 +51,7 @@ * \param load_balance_costs_update_algo Selected method for updating load balance costs. */ template -void doDepositionShapeN (const GetParticlePosition& GetPosition, +void doDepositionShapeN (const GetParticlePosition& GetPosition, const amrex::ParticleReal * const wp, const amrex::ParticleReal * const uxp, const amrex::ParticleReal * const uyp, @@ -365,7 +365,7 @@ void doDepositionShapeN (const GetParticlePosition& GetPosition, * \param load_balance_costs_update_algo Selected method for updating load balance costs. */ template -void doDepositionSharedShapeN (const GetParticlePosition& GetPosition, +void doDepositionSharedShapeN (const GetParticlePosition& GetPosition, const amrex::ParticleReal * const wp, const amrex::ParticleReal * const uxp, const amrex::ParticleReal * const uyp, @@ -577,7 +577,7 @@ void doDepositionSharedShapeN (const GetParticlePosition& GetPosition, * \param load_balance_costs_update_algo Selected method for updating load balance costs. */ template -void doEsirkepovDepositionShapeN (const GetParticlePosition& GetPosition, +void doEsirkepovDepositionShapeN (const GetParticlePosition& GetPosition, const amrex::ParticleReal * const wp, const amrex::ParticleReal * const uxp, const amrex::ParticleReal * const uyp, @@ -957,7 +957,7 @@ void doEsirkepovDepositionShapeN (const GetParticlePosition& GetPosition, * \param[in] load_balance_costs_update_algo Selected method for updating load balance costs */ template -void doVayDepositionShapeN (const GetParticlePosition& GetPosition, +void doVayDepositionShapeN (const GetParticlePosition& GetPosition, const amrex::ParticleReal* const wp, const amrex::ParticleReal* const uxp, const amrex::ParticleReal* const uyp, diff --git a/Source/Particles/Deposition/SharedDepositionUtils.H b/Source/Particles/Deposition/SharedDepositionUtils.H index 40407b609c0..646a6add811 100644 --- a/Source/Particles/Deposition/SharedDepositionUtils.H +++ b/Source/Particles/Deposition/SharedDepositionUtils.H @@ -65,7 +65,7 @@ void addLocalToGlobal (const amrex::Box& bx, #if defined(AMREX_USE_HIP) || defined(AMREX_USE_CUDA) template AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE -void depositComponent (const GetParticlePosition& GetPosition, +void depositComponent (const GetParticlePosition& GetPosition, const amrex::ParticleReal * const wp, const amrex::ParticleReal * const uxp, const amrex::ParticleReal * const uyp, diff --git a/Source/Particles/ElementaryProcess/Ionization.H b/Source/Particles/ElementaryProcess/Ionization.H index b6b74644825..742d8127da9 100644 --- a/Source/Particles/ElementaryProcess/Ionization.H +++ b/Source/Particles/ElementaryProcess/Ionization.H @@ -37,7 +37,7 @@ struct IonizationFilterFunc int comp; int m_atomic_number; - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; GetExternalEBField m_get_externalEB; amrex::Array4 m_ex_arr; diff --git a/Source/Particles/ElementaryProcess/Ionization.cpp b/Source/Particles/ElementaryProcess/Ionization.cpp index a4aac9d5e0f..8fcb953e9ff 100644 --- a/Source/Particles/ElementaryProcess/Ionization.cpp +++ b/Source/Particles/ElementaryProcess/Ionization.cpp @@ -42,7 +42,7 @@ IonizationFilterFunc::IonizationFilterFunc (const WarpXParIter& a_pti, int lev, comp = a_comp; m_atomic_number = a_atomic_number; - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); m_get_externalEB = GetExternalEBField(a_pti, a_offset); m_ex_arr = exfab.array(); diff --git a/Source/Particles/ElementaryProcess/QEDPairGeneration.H b/Source/Particles/ElementaryProcess/QEDPairGeneration.H index dbdbd0e6d53..5a3efe483d7 100644 --- a/Source/Particles/ElementaryProcess/QEDPairGeneration.H +++ b/Source/Particles/ElementaryProcess/QEDPairGeneration.H @@ -168,7 +168,7 @@ private: const BreitWheelerGeneratePairs m_generate_functor; /*!< A copy of the functor to generate pairs. It contains only pointers to the lookup tables.*/ - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; GetExternalEBField m_get_externalEB; amrex::Array4 m_ex_arr; diff --git a/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp b/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp index 68475aa144b..31aafbb1379 100644 --- a/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp +++ b/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp @@ -32,7 +32,7 @@ PairGenerationTransformFunc (BreitWheelerGeneratePairs const generate_functor, using namespace amrex::literals; - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); m_get_externalEB = GetExternalEBField(a_pti, a_offset); m_ex_arr = exfab.array(); diff --git a/Source/Particles/ElementaryProcess/QEDPhotonEmission.H b/Source/Particles/ElementaryProcess/QEDPhotonEmission.H index 080555e6fe8..0a889077036 100644 --- a/Source/Particles/ElementaryProcess/QEDPhotonEmission.H +++ b/Source/Particles/ElementaryProcess/QEDPhotonEmission.H @@ -178,7 +178,7 @@ private: const QuantumSynchrotronPhotonEmission m_emission_functor; /*!< A copy of the functor to generate photons. It contains only pointers to the lookup tables.*/ - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; GetExternalEBField m_get_externalEB; amrex::Array4 m_ex_arr; diff --git a/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp b/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp index 2c060a30f01..a3b3c4103f7 100644 --- a/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp +++ b/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp @@ -35,7 +35,7 @@ PhotonEmissionTransformFunc (QuantumSynchrotronGetOpticalDepth opt_depth_functor using namespace amrex::literals; - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); m_get_externalEB = GetExternalEBField(a_pti, a_offset); m_ex_arr = exfab.array(); diff --git a/Source/Particles/Gather/FieldGather.H b/Source/Particles/Gather/FieldGather.H index 4f5064a0e9b..0b2aae52ecc 100644 --- a/Source/Particles/Gather/FieldGather.H +++ b/Source/Particles/Gather/FieldGather.H @@ -432,7 +432,7 @@ void doGatherShapeN (const amrex::ParticleReal xp, * \param n_rz_azimuthal_modes Number of azimuthal modes when using RZ geometry */ template -void doGatherShapeN(const GetParticlePosition& getPosition, +void doGatherShapeN(const GetParticlePosition& getPosition, const GetExternalEBField& getExternalEB, amrex::ParticleReal * const Exp, amrex::ParticleReal * const Eyp, amrex::ParticleReal * const Ezp, amrex::ParticleReal * const Bxp, diff --git a/Source/Particles/Gather/GetExternalFields.H b/Source/Particles/Gather/GetExternalFields.H index 2f388363e17..8db7565a485 100644 --- a/Source/Particles/Gather/GetExternalFields.H +++ b/Source/Particles/Gather/GetExternalFields.H @@ -45,7 +45,7 @@ struct GetExternalEBField amrex::ParserExecutor<4> m_Byfield_partparser; amrex::ParserExecutor<4> m_Bzfield_partparser; - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; amrex::Real m_time; amrex::ParticleReal m_repeated_plasma_lens_period; diff --git a/Source/Particles/Gather/GetExternalFields.cpp b/Source/Particles/Gather/GetExternalFields.cpp index 057ac51d597..af00e092080 100644 --- a/Source/Particles/Gather/GetExternalFields.cpp +++ b/Source/Particles/Gather/GetExternalFields.cpp @@ -56,7 +56,7 @@ GetExternalEBField::GetExternalEBField (const WarpXParIter& a_pti, long a_offset mypc.m_B_ext_particle_s == "repeated_plasma_lens") { m_time = warpx.gett_new(a_pti.GetLevel()); - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); } if (mypc.m_E_ext_particle_s == "parse_e_ext_particle_function") diff --git a/Source/Particles/LaserParticleContainer.cpp b/Source/Particles/LaserParticleContainer.cpp index 098dba0997b..782e630e805 100644 --- a/Source/Particles/LaserParticleContainer.cpp +++ b/Source/Particles/LaserParticleContainer.cpp @@ -790,7 +790,7 @@ LaserParticleContainer::calculate_laser_plane_coordinates (const WarpXParIter& p Real * AMREX_RESTRICT const pplane_Xp, Real * AMREX_RESTRICT const pplane_Yp) { - const auto GetPosition = GetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); #if (AMREX_SPACEDIM >= 2) const Real tmp_u_X_0 = m_u_X[0]; @@ -852,8 +852,8 @@ LaserParticleContainer::update_laser_particle (WarpXParIter& pti, Real const * AMREX_RESTRICT const amplitude, const Real dt) { - const auto GetPosition = GetParticlePosition(pti); - auto SetPosition = SetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); + auto SetPosition = SetParticlePosition(pti); const Real tmp_p_X_0 = m_p_X[0]; const Real tmp_p_X_1 = m_p_X[1]; diff --git a/Source/Particles/ParticleBoundaryBuffer.cpp b/Source/Particles/ParticleBoundaryBuffer.cpp index 8e6decb1f3f..b2dfb2f03ef 100644 --- a/Source/Particles/ParticleBoundaryBuffer.cpp +++ b/Source/Particles/ParticleBoundaryBuffer.cpp @@ -313,7 +313,7 @@ void ParticleBoundaryBuffer::gatherParticles (MultiParticleContainer& mypc, auto index = std::make_pair(pti.index(), pti.LocalTileIndex()); if(plevel.find(index) == plevel.end()) continue; - const auto getPosition = GetParticlePosition(pti); + const auto getPosition = GetParticlePosition(pti); auto& ptile_buffer = species_buffer.DefineAndReturnParticleTile(lev, pti.index(), pti.LocalTileIndex()); const auto& ptile = plevel.at(index); diff --git a/Source/Particles/PhotonParticleContainer.cpp b/Source/Particles/PhotonParticleContainer.cpp index 9b7da18bc5e..7609bd67b13 100644 --- a/Source/Particles/PhotonParticleContainer.cpp +++ b/Source/Particles/PhotonParticleContainer.cpp @@ -129,8 +129,8 @@ PhotonParticleContainer::PushPX (WarpXParIter& pti, auto copyAttribs = CopyParticleAttribs(pti, tmp_particle_data, offset); const int do_copy = (m_do_back_transformed_particles && (a_dt_type!=DtType::SecondHalf) ); - const auto GetPosition = GetParticlePosition(pti, offset); - auto SetPosition = SetParticlePosition(pti, offset); + const auto GetPosition = GetParticlePosition(pti, offset); + auto SetPosition = SetParticlePosition(pti, offset); const auto getExternalEB = GetExternalEBField(pti, offset); diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp index f515b750a3e..632e3fca85b 100644 --- a/Source/Particles/PhysicalParticleContainer.cpp +++ b/Source/Particles/PhysicalParticleContainer.cpp @@ -712,7 +712,7 @@ PhysicalParticleContainer::DefaultInitializeRuntimeAttributes ( // Preparing data needed for user defined attributes const auto n_user_real_attribs = static_cast(m_user_real_attribs.size()); const auto n_user_int_attribs = static_cast(m_user_int_attribs.size()); - const auto get_position = GetParticlePosition(pinned_tile); + const auto get_position = GetParticlePosition(pinned_tile); const auto soa = pinned_tile.getParticleTileData(); const amrex::ParticleReal* AMREX_RESTRICT ux = soa.m_rdata[PIdx::ux]; const amrex::ParticleReal* AMREX_RESTRICT uy = soa.m_rdata[PIdx::uy]; @@ -2298,7 +2298,7 @@ PhysicalParticleContainer::SplitParticles (int lev) // Loop over particle interator for (WarpXParIter pti(*this, lev); pti.isValid(); ++pti) { - const auto GetPosition = GetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); const amrex::Vector ppc_nd = plasma_injector->num_particles_per_cell_each_dim; const std::array& dx = WarpX::CellSize(lev); @@ -2494,7 +2494,7 @@ PhysicalParticleContainer::PushP (int lev, Real dt, const FArrayBox& byfab = By[pti]; const FArrayBox& bzfab = Bz[pti]; - const auto getPosition = GetParticlePosition(pti); + const auto getPosition = GetParticlePosition(pti); const auto getExternalEB = GetExternalEBField(pti); @@ -2670,8 +2670,8 @@ PhysicalParticleContainer::PushPX (WarpXParIter& pti, // Add guard cells to the box. box.grow(ngEB); - const auto getPosition = GetParticlePosition(pti, offset); - auto setPosition = SetParticlePosition(pti, offset); + const auto getPosition = GetParticlePosition(pti, offset); + auto setPosition = SetParticlePosition(pti, offset); const auto getExternalEB = GetExternalEBField(pti, offset); diff --git a/Source/Particles/Pusher/CopyParticleAttribs.H b/Source/Particles/Pusher/CopyParticleAttribs.H index 29e5017a3a7..1f0cee67421 100644 --- a/Source/Particles/Pusher/CopyParticleAttribs.H +++ b/Source/Particles/Pusher/CopyParticleAttribs.H @@ -22,7 +22,7 @@ struct CopyParticleAttribs { using TmpParticles = WarpXParticleContainer::TmpParticles; - GetParticlePosition m_get_position; + GetParticlePosition m_get_position; const amrex::ParticleReal* AMREX_RESTRICT uxp = nullptr; const amrex::ParticleReal* AMREX_RESTRICT uyp = nullptr; @@ -66,7 +66,7 @@ struct CopyParticleAttribs uypold = tmp_particle_data[lev].at(index)[TmpIdx::uyold].dataPtr() + a_offset; uzpold = tmp_particle_data[lev].at(index)[TmpIdx::uzold].dataPtr() + a_offset; - m_get_position = GetParticlePosition(a_pti, a_offset); + m_get_position = GetParticlePosition(a_pti, a_offset); } /** \brief copy the position and momentum of particle i to the diff --git a/Source/Particles/Pusher/GetAndSetPosition.H b/Source/Particles/Pusher/GetAndSetPosition.H index 45b9c39fe9d..e4477a2a60d 100644 --- a/Source/Particles/Pusher/GetAndSetPosition.H +++ b/Source/Particles/Pusher/GetAndSetPosition.H @@ -9,6 +9,7 @@ #define WARPX_PARTICLES_PUSHER_GETANDSETPOSITION_H_ #include "Particles/WarpXParticleContainer.H" +#include "Particles/NamedComponentParticleContainer.H" #include #include @@ -18,7 +19,11 @@ /** \brief Extract the cartesian position coordinates of the particle * p and store them in the variables `x`, `y`, `z` - * This version operates on a SuperParticle */ + * This version operates on a SuperParticle + * + * \tparam T_PIdx particle index enumerator + */ +template AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE void get_particle_position (const WarpXParticleContainer::SuperParticleType& p, amrex::ParticleReal& x, @@ -26,7 +31,7 @@ void get_particle_position (const WarpXParticleContainer::SuperParticleType& p, amrex::ParticleReal& z) noexcept { #ifdef WARPX_DIM_RZ - const amrex::ParticleReal theta = p.rdata(PIdx::theta); + const amrex::ParticleReal theta = p.rdata(T_PIdx::theta); const amrex::ParticleReal r = p.pos(0); x = r*std::cos(theta); y = r*std::sin(theta); @@ -48,7 +53,10 @@ void get_particle_position (const WarpXParticleContainer::SuperParticleType& p, /** \brief Functor that can be used to extract the positions of the macroparticles * inside a ParallelFor kernel + * + * \tparam T_PIdx particle index enumerator */ +template struct GetParticlePosition { using PType = WarpXParticleContainer::ParticleType; @@ -80,7 +88,7 @@ struct GetParticlePosition m_structs = aos().dataPtr() + a_offset; #if defined(WARPX_DIM_RZ) const auto& soa = a_pti.GetStructOfArrays(); - m_theta = soa.GetRealData(PIdx::theta).dataPtr() + a_offset; + m_theta = soa.GetRealData(T_PIdx::theta).dataPtr() + a_offset; #endif } @@ -143,9 +151,11 @@ struct GetParticlePosition /** \brief Functor that can be used to modify the positions of the macroparticles, * inside a ParallelFor kernel. * + * \tparam T_PIdx particle index enumerator * \param a_pti iterator to the tile being modified * \param a_offset offset to apply to the particle indices */ +template struct SetParticlePosition { using PType = WarpXParticleContainer::ParticleType; @@ -163,7 +173,7 @@ struct SetParticlePosition m_structs = aos().dataPtr() + a_offset; #if defined(WARPX_DIM_RZ) auto& soa = a_pti.GetStructOfArrays(); - m_theta = soa.GetRealData(PIdx::theta).dataPtr() + a_offset; + m_theta = soa.GetRealData(T_PIdx::theta).dataPtr() + a_offset; #endif } diff --git a/Source/Particles/Pusher/PushSelector.H b/Source/Particles/Pusher/PushSelector.H index b1e59029958..f6fa3ba164f 100644 --- a/Source/Particles/Pusher/PushSelector.H +++ b/Source/Particles/Pusher/PushSelector.H @@ -43,8 +43,8 @@ template AMREX_GPU_DEVICE AMREX_FORCE_INLINE -void doParticlePush(const GetParticlePosition& GetPosition, - const SetParticlePosition& SetPosition, +void doParticlePush(const GetParticlePosition& GetPosition, + const SetParticlePosition& SetPosition, const CopyParticleAttribs& copyAttribs, const long i, amrex::ParticleReal& ux, diff --git a/Source/Particles/RigidInjectedParticleContainer.cpp b/Source/Particles/RigidInjectedParticleContainer.cpp index 28364cb6ef7..e013d41268a 100644 --- a/Source/Particles/RigidInjectedParticleContainer.cpp +++ b/Source/Particles/RigidInjectedParticleContainer.cpp @@ -121,8 +121,8 @@ RigidInjectedParticleContainer::RemapParticles() const auto uyp = attribs[PIdx::uy].dataPtr(); const auto uzp = attribs[PIdx::uz].dataPtr(); - const auto GetPosition = GetParticlePosition(pti); - auto SetPosition = SetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); + auto SetPosition = SetParticlePosition(pti); // Loop over particles const long np = pti.numParticles(); @@ -181,8 +181,8 @@ RigidInjectedParticleContainer::PushPX (WarpXParIter& pti, amrex::Gpu::DeviceVector optical_depth_save; #endif - const auto GetPosition = GetParticlePosition(pti, offset); - auto SetPosition = SetParticlePosition(pti, offset); + const auto GetPosition = GetParticlePosition(pti, offset); + auto SetPosition = SetParticlePosition(pti, offset); amrex::ParticleReal* const AMREX_RESTRICT ux = uxp.dataPtr() + offset; amrex::ParticleReal* const AMREX_RESTRICT uy = uyp.dataPtr() + offset; @@ -357,7 +357,7 @@ RigidInjectedParticleContainer::PushP (int lev, Real dt, const FArrayBox& byfab = By[pti]; const FArrayBox& bzfab = Bz[pti]; - const auto getPosition = GetParticlePosition(pti); + const auto getPosition = GetParticlePosition(pti); const auto getExternalEB = GetExternalEBField(pti); diff --git a/Source/Particles/WarpXParticleContainer.cpp b/Source/Particles/WarpXParticleContainer.cpp index a7405bb5741..df3938ec909 100644 --- a/Source/Particles/WarpXParticleContainer.cpp +++ b/Source/Particles/WarpXParticleContainer.cpp @@ -424,7 +424,7 @@ WarpXParticleContainer::DepositCurrent (WarpXParIter& pti, Array4 const& jz_arr = local_jz[thread_num].array(); #endif - const auto GetPosition = GetParticlePosition(pti, offset); + const auto GetPosition = GetParticlePosition(pti, offset); // Lower corner of tile box physical domain // Note that this includes guard cells since it is after tilebox.ngrow @@ -870,7 +870,7 @@ WarpXParticleContainer::DepositCharge (WarpXParIter& pti, RealVector const& wp, amrex::LayoutData* costs = WarpX::getCosts(lev); amrex::Real* cost = costs ? &((*costs)[pti.index()]) : nullptr; - const auto GetPosition = GetParticlePosition(pti, offset); + const auto GetPosition = GetParticlePosition(pti, offset); const Geometry& geom = Geom(lev); Box box = pti.validbox(); box.grow(ng_rho); @@ -1258,8 +1258,8 @@ WarpXParticleContainer::PushX (int lev, amrex::Real dt) // Particle Push // - const auto GetPosition = GetParticlePosition(pti); - auto SetPosition = SetParticlePosition(pti); + const auto GetPosition = GetParticlePosition(pti); + auto SetPosition = SetParticlePosition(pti); // - momenta are stored as a struct of array, in `attribs` auto& attribs = pti.GetAttribs(); @@ -1345,8 +1345,8 @@ WarpXParticleContainer::ApplyBoundaryConditions (){ #endif for (WarpXParIter pti(*this, lev); pti.isValid(); ++pti) { - auto GetPosition = GetParticlePosition(pti); - auto SetPosition = SetParticlePosition(pti); + auto GetPosition = GetParticlePosition(pti); + auto SetPosition = SetParticlePosition(pti); #ifndef WARPX_DIM_1D_Z const Real xmin = Geom(lev).ProbLo(0); const Real xmax = Geom(lev).ProbHi(0); diff --git a/Source/ablastr/particles/DepositCharge.H b/Source/ablastr/particles/DepositCharge.H index 2fcdcf3b768..eecc56f9acc 100644 --- a/Source/ablastr/particles/DepositCharge.H +++ b/Source/ablastr/particles/DepositCharge.H @@ -170,7 +170,7 @@ deposit_charge (typename T_PC::ParIterType& pti, auto & rho_fab = local_rho; #endif - const auto GetPosition = GetParticlePosition(pti, offset); + const auto GetPosition = GetParticlePosition(pti, offset); // Indices of the lower bound const amrex::Dim3 lo = lbound(tilebox); From d2afb3e7aaee849b7db9a3588dbeae36f0aa6e67 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 23 Oct 2023 16:06:18 -0700 Subject: [PATCH 067/110] AMReX: Weekly Update (#4387) --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- run_test.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 0425842e7cc..d3610179695 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 && cd - + cd ../amrex && git checkout --detach da79aff8053058371a78d4bf85488384242368ee && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index aa0167565ea..17d98e4b76d 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 +branch = da79aff8053058371a78d4bf85488384242368ee [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 6e0efc9ecba..6f08f34bc2c 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 +branch = da79aff8053058371a78d4bf85488384242368ee [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 0313f4a2128..fbdd39e4f43 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -257,7 +257,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "7ee29121ed70d7e255ad98a8b1690d345cb4fb33" +set(WarpX_amrex_branch "da79aff8053058371a78d4bf85488384242368ee" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/run_test.sh b/run_test.sh index e362a0ef332..2e86618f2f2 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 && cd - +cd amrex && git checkout --detach da79aff8053058371a78d4bf85488384242368ee && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 52561f93f715579a63b71fea06a4397afa8128e2 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Wed, 25 Oct 2023 00:47:32 -0700 Subject: [PATCH 068/110] Lassen (LLNL): `h5py` (#4388) * Lassen (LLNL): `h5py` Install `h5py` on Lassen using our pre-compiled HDF5 package. * All Machines: Ensure Setuptools in Venv --- Docs/source/install/hpc/lawrencium.rst | 1 + Tools/machines/adastra-cines/install_dependencies.sh | 1 + Tools/machines/crusher-olcf/install_dependencies.sh | 1 + Tools/machines/frontier-olcf/install_dependencies.sh | 1 + Tools/machines/hpc3-uci/install_gpu_dependencies.sh | 1 + Tools/machines/karolina-it4i/install_cpu_dependencies.sh | 1 + Tools/machines/karolina-it4i/install_gpu_dependencies.sh | 1 + Tools/machines/lassen-llnl/install_v100_dependencies.sh | 9 ++++++++- .../lassen-llnl/install_v100_dependencies_toss3.sh | 9 ++++++++- .../machines/leonardo-cineca/install_gpu_dependencies.sh | 1 + Tools/machines/lumi-csc/install_dependencies.sh | 1 + .../perlmutter-nersc/install_cpu_dependencies.sh | 1 + .../perlmutter-nersc/install_gpu_dependencies.sh | 1 + Tools/machines/quartz-llnl/install_dependencies.sh | 1 + Tools/machines/summit-olcf/install_gpu_dependencies.sh | 1 + 15 files changed, 29 insertions(+), 2 deletions(-) diff --git a/Docs/source/install/hpc/lawrencium.rst b/Docs/source/install/hpc/lawrencium.rst index 09eb67b2ca5..36039e5eda4 100644 --- a/Docs/source/install/hpc/lawrencium.rst +++ b/Docs/source/install/hpc/lawrencium.rst @@ -81,6 +81,7 @@ Optionally, download and install Python packages for :ref:`PICMI ` source $HOME/sw/v100/venvs/warpx/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel + python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/adastra-cines/install_dependencies.sh b/Tools/machines/adastra-cines/install_dependencies.sh index e532ea33924..b94d5224833 100755 --- a/Tools/machines/adastra-cines/install_dependencies.sh +++ b/Tools/machines/adastra-cines/install_dependencies.sh @@ -102,6 +102,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-adastra source ${SW_DIR}/venvs/warpx-adastra/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/crusher-olcf/install_dependencies.sh b/Tools/machines/crusher-olcf/install_dependencies.sh index e1f196a519b..2defd795786 100755 --- a/Tools/machines/crusher-olcf/install_dependencies.sh +++ b/Tools/machines/crusher-olcf/install_dependencies.sh @@ -87,6 +87,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-crusher source ${SW_DIR}/venvs/warpx-crusher/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/frontier-olcf/install_dependencies.sh b/Tools/machines/frontier-olcf/install_dependencies.sh index 9ef077c69ea..690941b2078 100755 --- a/Tools/machines/frontier-olcf/install_dependencies.sh +++ b/Tools/machines/frontier-olcf/install_dependencies.sh @@ -87,6 +87,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-frontier source ${SW_DIR}/venvs/warpx-frontier/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/hpc3-uci/install_gpu_dependencies.sh b/Tools/machines/hpc3-uci/install_gpu_dependencies.sh index 0f8b5c9b880..e7a5f9f086f 100755 --- a/Tools/machines/hpc3-uci/install_gpu_dependencies.sh +++ b/Tools/machines/hpc3-uci/install_gpu_dependencies.sh @@ -117,6 +117,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-gpu source ${SW_DIR}/venvs/warpx-gpu/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/karolina-it4i/install_cpu_dependencies.sh b/Tools/machines/karolina-it4i/install_cpu_dependencies.sh index c6e858ba8a6..f5067e03447 100755 --- a/Tools/machines/karolina-it4i/install_cpu_dependencies.sh +++ b/Tools/machines/karolina-it4i/install_cpu_dependencies.sh @@ -121,6 +121,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-cpu source ${SW_DIR}/venvs/warpx-cpu/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/karolina-it4i/install_gpu_dependencies.sh b/Tools/machines/karolina-it4i/install_gpu_dependencies.sh index a12092c6e01..4cd1ce5851a 100755 --- a/Tools/machines/karolina-it4i/install_gpu_dependencies.sh +++ b/Tools/machines/karolina-it4i/install_gpu_dependencies.sh @@ -121,6 +121,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-gpu source ${SW_DIR}/venvs/warpx-gpu/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index 223b41d1967..dfbf139f27d 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -108,6 +108,9 @@ cmake --build ${build_dir}/lapackpp-lassen-build --target install --parallel 10 # Python ###################################################################### # +# sometimes, the Lassen PIP Index is down +export PIP_EXTRA_INDEX_URL="https://pypi.org/simple" + python3 -m pip install --upgrade --user virtualenv rm -rf ${SW_DIR}/venvs/warpx-lassen python3 -m venv ${SW_DIR}/venvs/warpx-lassen @@ -115,12 +118,16 @@ source ${SW_DIR}/venvs/warpx-lassen/bin/activate python3 -m pip install --upgrade pip python3 -m pip cache purge python3 -m pip install --upgrade wheel -python3 -m pip install --upgrade cython +python3 -m pip install --upgrade setuptools +# Older version for h4py +# https://github.com/h5py/h5py/issues/2268 +python3 -m pip install --upgrade "cython<3" python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas python3 -m pip install --upgrade -Ccompile-args="-j10" scipy python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py python3 -m pip install --upgrade openpmd-api +CC=mpicc H5PY_SETUP_REQUIRES=0 HDF5_DIR=${SW_DIR}/hdf5-1.14.1.2 HDF5_MPI=ON python3 -m pip install --upgrade h5py --no-cache-dir --no-build-isolation --no-binary h5py MPLLOCALFREETYPE=1 python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself echo "matplotlib==3.2.2" > ${build_dir}/constraints.txt python3 -m pip install --upgrade -c ${build_dir}/constraints.txt yt diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh b/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh index f62c6018cd5..b8273a69c87 100644 --- a/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh @@ -108,6 +108,9 @@ cmake --build ${build_dir}/lapackpp-lassen-build --target install --parallel 10 # Python ###################################################################### # +# sometimes, the Lassen PIP Index is down +export PIP_EXTRA_INDEX_URL="https://pypi.org/simple" + python3 -m pip install --upgrade --user virtualenv rm -rf ${SW_DIR}/venvs/warpx-lassen-toss3 python3 -m venv ${SW_DIR}/venvs/warpx-lassen-toss3 @@ -115,12 +118,16 @@ source ${SW_DIR}/venvs/warpx-lassen-toss3/bin/activate python3 -m pip install --upgrade pip python3 -m pip cache purge python3 -m pip install --upgrade wheel -python3 -m pip install --upgrade cython +python3 -m pip install --upgrade setuptools +# Older version for h4py +# https://github.com/h5py/h5py/issues/2268 +python3 -m pip install --upgrade "cython<3" python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas CMAKE_PREFIX_PATH=/usr/lib64:${CMAKE_PREFIX_PATH} python3 -m pip install --upgrade -Ccompile-args="-j10" -Csetup-args=-Dblas=BLAS -Csetup-args=-Dlapack=BLAS scipy python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py python3 -m pip install --upgrade openpmd-api +CC=mpicc H5PY_SETUP_REQUIRES=0 HDF5_DIR=${SW_DIR}/hdf5-1.14.1.2 HDF5_MPI=ON python3 -m pip install --upgrade h5py --no-cache-dir --no-build-isolation --no-binary h5py MPLLOCALFREETYPE=1 python3 -m pip install --upgrade matplotlib==3.2.2 # does not try to build freetype itself echo "matplotlib==3.2.2" > ${build_dir}/constraints.txt python3 -m pip install --upgrade -c ${build_dir}/constraints.txt yt diff --git a/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh b/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh index 5b6453b3968..3a332f97420 100644 --- a/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh +++ b/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh @@ -86,6 +86,7 @@ python3 -m ensurepip --upgrade python3 -m pip cache purge python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/lumi-csc/install_dependencies.sh b/Tools/machines/lumi-csc/install_dependencies.sh index 1cb1afb4fc4..281ca10c449 100755 --- a/Tools/machines/lumi-csc/install_dependencies.sh +++ b/Tools/machines/lumi-csc/install_dependencies.sh @@ -102,6 +102,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-lumi source ${SW_DIR}/venvs/warpx-lumi/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh b/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh index 6bfa47bc380..fa4e77cb309 100755 --- a/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh +++ b/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh @@ -117,6 +117,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx source ${SW_DIR}/venvs/warpx/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh b/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh index 7bcc7053974..98e14f09ede 100755 --- a/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh +++ b/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh @@ -117,6 +117,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx source ${SW_DIR}/venvs/warpx/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/quartz-llnl/install_dependencies.sh b/Tools/machines/quartz-llnl/install_dependencies.sh index c162ac5d390..114bef3d3c3 100755 --- a/Tools/machines/quartz-llnl/install_dependencies.sh +++ b/Tools/machines/quartz-llnl/install_dependencies.sh @@ -100,6 +100,7 @@ source ${SW_DIR}/venvs/warpx-quartz/bin/activate python3 -m pip install --upgrade pip python3 -m pip cache purge python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas diff --git a/Tools/machines/summit-olcf/install_gpu_dependencies.sh b/Tools/machines/summit-olcf/install_gpu_dependencies.sh index 6bdcda92378..7679b999ea3 100755 --- a/Tools/machines/summit-olcf/install_gpu_dependencies.sh +++ b/Tools/machines/summit-olcf/install_gpu_dependencies.sh @@ -100,6 +100,7 @@ python3 -m venv ${SW_DIR}/venvs/warpx-summit source ${SW_DIR}/venvs/warpx-summit/bin/activate python3 -m pip install --upgrade pip python3 -m pip install --upgrade wheel +python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython python3 -m pip install --upgrade numpy python3 -m pip install --upgrade pandas From ba217db3e9e0322c09b4bd2704c1a02018badeef Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Wed, 25 Oct 2023 00:48:25 -0700 Subject: [PATCH 069/110] Machines: HDF5 & ADIOS2 Tools in PATH (#4389) Ensure tools such as `bpls`, `bpdump` and `h5ls` are available to users if we self-compile them. --- Tools/machines/adastra-cines/adastra_warpx.profile.example | 2 ++ Tools/machines/hpc3-uci/hpc3_gpu_warpx.profile.example | 1 + .../machines/karolina-it4i/karolina_cpu_warpx.profile.example | 3 +++ .../machines/karolina-it4i/karolina_gpu_warpx.profile.example | 3 +++ Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example | 2 ++ .../lassen-llnl/lassen_v100_warpx_toss3.profile.example | 2 ++ .../machines/lawrencium-lbnl/lawrencium_warpx.profile.example | 2 ++ .../leonardo-cineca/leonardo_gpu_warpx.profile.example | 2 ++ Tools/machines/lumi-csc/lumi_warpx.profile.example | 1 + .../perlmutter-nersc/perlmutter_cpu_warpx.profile.example | 2 ++ .../perlmutter-nersc/perlmutter_gpu_warpx.profile.example | 2 ++ Tools/machines/quartz-llnl/quartz_warpx.profile.example | 1 + 12 files changed, 23 insertions(+) diff --git a/Tools/machines/adastra-cines/adastra_warpx.profile.example b/Tools/machines/adastra-cines/adastra_warpx.profile.example index 49f17d395e4..23441638893 100644 --- a/Tools/machines/adastra-cines/adastra_warpx.profile.example +++ b/Tools/machines/adastra-cines/adastra_warpx.profile.example @@ -21,6 +21,8 @@ module load cray-hdf5-parallel export CMAKE_PREFIX_PATH=${HOME}/sw/adastra/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${HOME}/sw/adastra/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH +export PATH=${HOME}/sw/adastra/gpu/adios2-2.8.3/bin:${PATH} + # optional: for Python bindings or libEnsemble module load cray-python/3.9.13.1 diff --git a/Tools/machines/hpc3-uci/hpc3_gpu_warpx.profile.example b/Tools/machines/hpc3-uci/hpc3_gpu_warpx.profile.example index 62733f8e70f..9f41cc7e337 100644 --- a/Tools/machines/hpc3-uci/hpc3_gpu_warpx.profile.example +++ b/Tools/machines/hpc3-uci/hpc3_gpu_warpx.profile.example @@ -27,6 +27,7 @@ export LD_LIBRARY_PATH=${HOME}/sw/hpc3/gpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${HOME}/sw/hpc3/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${HOME}/sw/hpc3/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH +export PATH=${HOME}/sw/hpc3/gpu/adios2-2.8.3/bin:${PATH} # optional: CCache #module load ccache # missing diff --git a/Tools/machines/karolina-it4i/karolina_cpu_warpx.profile.example b/Tools/machines/karolina-it4i/karolina_cpu_warpx.profile.example index 66b2f67a8be..c0a3ed53ee3 100644 --- a/Tools/machines/karolina-it4i/karolina_cpu_warpx.profile.example +++ b/Tools/machines/karolina-it4i/karolina_cpu_warpx.profile.example @@ -27,6 +27,9 @@ export LD_LIBRARY_PATH=${HOME}/sw/karolina/cpu/adios2-2.8.3/lib64:$LD_LIBRARY_PA export LD_LIBRARY_PATH=${HOME}/sw/karolina/cpu/blaspp-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${HOME}/sw/karolina/cpu/lapackpp-master/lib64:$LD_LIBRARY_PATH +export PATH=${HOME}/sw/karolina/cpu/hdf5-1.14.1.2/bin:${PATH} +export PATH=${HOME}/sw/karolina/cpu/adios2-2.8.3/bin:${PATH} + # optional: CCache (not found) #module load ccache diff --git a/Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example b/Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example index f657916dfcd..174598acaac 100644 --- a/Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example +++ b/Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example @@ -29,6 +29,9 @@ export LD_LIBRARY_PATH=${HOME}/sw/karolina/gpu/adios2-2.8.3/lib64:$LD_LIBRARY_PA export LD_LIBRARY_PATH=${HOME}/sw/karolina/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${HOME}/sw/karolina/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH +export PATH=${HOME}/sw/karolina/gpu/hdf5-1.14.1.2/bin:${PATH} +export PATH=${HOME}/sw/karolina/gpu/adios2-2.8.3/bin:${PATH} + # optional: CCache (not found) #ml ccache diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example index 652af2a2822..4d4efbf917d 100644 --- a/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example @@ -18,6 +18,8 @@ export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH export LD_LIBRARY_PATH=${SW_DIR}/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${SW_DIR}/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${SW_DIR}/adios2-2.8.3/lib64:$LD_LIBRARY_PATH +export PATH=${SW_DIR}/hdf5-1.14.1.2/bin:${PATH} +export PATH=${SW_DIR}/adios2-2.8.3/bin:$PATH # optional: for PSATD in RZ geometry support export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-master:$CMAKE_PREFIX_PATH diff --git a/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example b/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example index 979c27989fa..19c74348a99 100644 --- a/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example +++ b/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example @@ -18,6 +18,8 @@ export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH export LD_LIBRARY_PATH=${SW_DIR}/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${SW_DIR}/hdf5-1.14.1.2/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${SW_DIR}/adios2-2.8.3/lib64:$LD_LIBRARY_PATH +export PATH=${SW_DIR}/hdf5-1.14.1.2/bin:${PATH} +export PATH=${SW_DIR}/adios2-2.8.3/bin:${PATH} # optional: for PSATD in RZ geometry support export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-master:$CMAKE_PREFIX_PATH diff --git a/Tools/machines/lawrencium-lbnl/lawrencium_warpx.profile.example b/Tools/machines/lawrencium-lbnl/lawrencium_warpx.profile.example index 9b9bf709ecc..472d0785bb2 100644 --- a/Tools/machines/lawrencium-lbnl/lawrencium_warpx.profile.example +++ b/Tools/machines/lawrencium-lbnl/lawrencium_warpx.profile.example @@ -21,6 +21,8 @@ export CMAKE_PREFIX_PATH=$HOME/sw/v100/adios2-2.8.3:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=$HOME/sw/v100/blaspp-master:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=$HOME/sw/v100/lapackpp-master:$CMAKE_PREFIX_PATH +export PATH=$HOME/sw/v100/adios2-2.8.3/bin:$PATH + # optional: CCache #module load ccache # missing diff --git a/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example b/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example index af0cdbd41c3..cffe565f9d7 100644 --- a/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example +++ b/Tools/machines/leonardo-cineca/leonardo_gpu_warpx.profile.example @@ -24,6 +24,8 @@ export LD_LIBRARY_PATH=$HOME/sw/adios2-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$HOME/sw/blaspp-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$HOME/sw/lapackpp-master/lib64:$LD_LIBRARY_PATH +export PATH=$HOME/sw/adios2-master/bin:$PATH + # optional: for Python bindings or libEnsemble module load python/3.10.8--gcc--11.3.0 diff --git a/Tools/machines/lumi-csc/lumi_warpx.profile.example b/Tools/machines/lumi-csc/lumi_warpx.profile.example index 31c133ed47a..74b8aa8df17 100644 --- a/Tools/machines/lumi-csc/lumi_warpx.profile.example +++ b/Tools/machines/lumi-csc/lumi_warpx.profile.example @@ -22,6 +22,7 @@ module load Boost/1.81.0-cpeCray-23.03 module load cray-hdf5/1.12.2.3 export CMAKE_PREFIX_PATH=${HOME}/sw/lumi/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${HOME}/sw/lumi/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH +export PATH=${HOME}/sw/lumi/gpu/adios2-2.8.3/bin:${PATH} # optional: for Python bindings or libEnsemble module load cray-python/3.9.13.1 diff --git a/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example b/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example index d25b4825dde..0150ded4839 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example +++ b/Tools/machines/perlmutter-nersc/perlmutter_cpu_warpx.profile.example @@ -25,6 +25,8 @@ export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3/lib export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/lapackpp-master/lib64:$LD_LIBRARY_PATH +export PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3/bin:${PATH} + # optional: CCache export PATH=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8-eqk2d3bipbpkgwxq7ujlp6mckwal4dwz/bin:$PATH diff --git a/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example b/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example index af3e69c1190..13ab2ead605 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example +++ b/Tools/machines/perlmutter-nersc/perlmutter_gpu_warpx.profile.example @@ -23,6 +23,8 @@ export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3/ export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH +export PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3/bin:${PATH} + # optional: CCache export PATH=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8-eqk2d3bipbpkgwxq7ujlp6mckwal4dwz/bin:$PATH diff --git a/Tools/machines/quartz-llnl/quartz_warpx.profile.example b/Tools/machines/quartz-llnl/quartz_warpx.profile.example index 810005bafb7..a3646bfd557 100644 --- a/Tools/machines/quartz-llnl/quartz_warpx.profile.example +++ b/Tools/machines/quartz-llnl/quartz_warpx.profile.example @@ -18,6 +18,7 @@ module load hdf5-parallel/1.14.0 SW_DIR="/usr/workspace/${USER}/quartz" export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH +export PATH=${SW_DIR}/adios2-2.8.3/bin:${PATH} # optional: for PSATD in RZ geometry support export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-master:$CMAKE_PREFIX_PATH From eb9d79eaafed115931f8521a94850c42da40756a Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Thu, 26 Oct 2023 15:25:26 +0200 Subject: [PATCH 070/110] Clang tidy CI test: add some cppcoreguidelines-* checks (#4149) * clang-tidy CI test: add some cppcoreguidelines-* checks. Initial work * addressing issues found with clang-tidy * continue addressing issues found with clang-tidy * fix bug * fixed issue * address issues found with clang-tidy * addressing issues found with clang-tidy * fix initialization order * fix initialization order --- .clang-tidy | 9 ++- .../LatticeElements/Drift.H | 2 - .../LatticeElements/HardEdgedPlasmaLens.H | 2 - .../LatticeElements/HardEdgedQuadrupole.H | 2 - .../LatticeElements/LatticeElementBase.H | 2 - .../LatticeElements/LatticeElementBase.cpp | 6 +- Source/BoundaryConditions/PML.H | 10 ++- Source/BoundaryConditions/PML_RZ.H | 2 - Source/Diagnostics/BTD_Plotfile_Header_Impl.H | 9 +-- .../ComputeDiagFunctors/CellCenterFunctor.H | 2 +- .../ComputeDiagFunctors/ComputeDiagFunctor.H | 7 ++ .../ComputeParticleDiagFunctor.H | 6 ++ .../ComputeDiagFunctors/DivBFunctor.H | 2 +- .../ComputeDiagFunctors/DivEFunctor.H | 2 +- .../ComputeDiagFunctors/JFunctor.H | 2 +- .../ComputeDiagFunctors/PartPerCellFunctor.H | 2 +- .../ComputeDiagFunctors/PartPerGridFunctor.H | 2 +- .../ParticleReductionFunctor.H | 2 +- .../ComputeDiagFunctors/RhoFunctor.H | 2 +- Source/Diagnostics/Diagnostics.H | 5 ++ Source/Diagnostics/FlushFormats/FlushFormat.H | 10 ++- .../FlushFormats/FlushFormatAscent.H | 12 +++- .../FlushFormats/FlushFormatAscent.cpp | 2 +- .../FlushFormats/FlushFormatCheckpoint.H | 6 +- .../FlushFormats/FlushFormatCheckpoint.cpp | 2 +- .../FlushFormats/FlushFormatOpenPMD.H | 7 +- .../FlushFormats/FlushFormatOpenPMD.cpp | 2 +- .../FlushFormats/FlushFormatPlotfile.H | 12 +++- .../FlushFormats/FlushFormatPlotfile.cpp | 2 +- .../FlushFormats/FlushFormatSensei.H | 12 ++-- .../FlushFormats/FlushFormatSensei.cpp | 2 +- Source/Diagnostics/FullDiagnostics.cpp | 12 ++-- .../Diagnostics/ReducedDiags/BeamRelevant.H | 2 +- Source/Diagnostics/ReducedDiags/ChargeOnEB.H | 2 +- .../ReducedDiags/ColliderRelevant.H | 2 +- Source/Diagnostics/ReducedDiags/FieldEnergy.H | 2 +- .../Diagnostics/ReducedDiags/FieldMaximum.H | 2 +- .../Diagnostics/ReducedDiags/FieldMomentum.H | 2 +- Source/Diagnostics/ReducedDiags/FieldProbe.H | 8 +-- .../FieldProbeParticleContainer.H | 7 +- .../Diagnostics/ReducedDiags/FieldReduction.H | 2 +- .../ReducedDiags/LoadBalanceCosts.H | 4 +- .../ReducedDiags/LoadBalanceEfficiency.H | 2 +- .../Diagnostics/ReducedDiags/ParticleEnergy.H | 2 +- .../ReducedDiags/ParticleExtrema.H | 2 +- .../ReducedDiags/ParticleHistogram.H | 2 +- .../ReducedDiags/ParticleHistogram.cpp | 17 +++-- .../ReducedDiags/ParticleHistogram2D.H | 4 +- .../ReducedDiags/ParticleMomentum.H | 2 +- .../Diagnostics/ReducedDiags/ParticleNumber.H | 2 +- .../Diagnostics/ReducedDiags/ReducedDiags.H | 7 ++ Source/Diagnostics/ReducedDiags/RhoMaximum.H | 2 +- Source/Diagnostics/WarpXOpenPMD.H | 5 ++ Source/Diagnostics/WarpXOpenPMD.cpp | 13 ++-- .../FiniteDifferenceSolver.cpp | 9 ++- .../PsatdAlgorithmComoving.H | 6 +- .../PsatdAlgorithmFirstOrder.H | 6 +- .../PsatdAlgorithmGalileanRZ.H | 10 +-- .../PsatdAlgorithmGalileanRZ.cpp | 13 ++-- .../PsatdAlgorithmJConstantInTime.H | 6 +- .../PsatdAlgorithmJConstantInTime.cpp | 6 +- .../PsatdAlgorithmJLinearInTime.H | 6 +- .../SpectralAlgorithms/PsatdAlgorithmPml.H | 6 +- .../SpectralAlgorithms/PsatdAlgorithmPml.cpp | 5 +- .../SpectralAlgorithms/PsatdAlgorithmPmlRZ.H | 8 +-- .../PsatdAlgorithmPmlRZ.cpp | 11 ++-- .../SpectralAlgorithms/PsatdAlgorithmRZ.H | 6 +- .../SpectralAlgorithms/PsatdAlgorithmRZ.cpp | 20 +++--- .../SpectralBaseAlgorithm.H | 8 ++- .../SpectralBaseAlgorithmRZ.H | 21 ++++++ .../SpectralSolver/SpectralFieldData.H | 27 +++++++- .../SpectralSolver/SpectralFieldData.cpp | 5 +- .../SpectralSolver/SpectralFieldDataRZ.H | 11 +++- Source/Filter/NCIGodfreyFilter.H | 4 ++ Source/Filter/NCIGodfreyFilter.cpp | 66 +++++++++++-------- Source/Fluids/MultiFluidContainer.H | 5 ++ Source/Fluids/WarpXFluidContainer.H | 5 ++ Source/Fluids/WarpXFluidContainer.cpp | 7 +- Source/Initialization/GetTemperature.cpp | 5 +- Source/Initialization/GetVelocity.cpp | 7 +- Source/Initialization/InjectorDensity.H | 3 + Source/Initialization/InjectorFlux.H | 3 + Source/Initialization/InjectorMomentum.H | 24 +++---- Source/Initialization/InjectorPosition.H | 8 ++- Source/Initialization/PlasmaInjector.H | 7 ++ Source/Initialization/VelocityProperties.cpp | 5 +- Source/Laser/LaserProfiles.H | 24 ++++--- .../BackgroundMCC/BackgroundMCCCollision.H | 7 +- .../Collision/BackgroundMCC/MCCProcess.H | 9 +-- .../BackgroundStopping/BackgroundStopping.H | 7 +- .../BinaryCollision/BinaryCollision.H | 7 +- .../Coulomb/PairWiseCoulombCollisionFunc.H | 9 +-- .../NuclearFusion/NuclearFusionFunc.H | 13 ++-- Source/Particles/Collision/CollisionBase.H | 1 + Source/Particles/Collision/CollisionBase.cpp | 6 +- .../ElementaryProcess/Ionization.cpp | 23 +++---- .../ElementaryProcess/QEDPairGeneration.cpp | 11 ++-- .../ElementaryProcess/QEDPhotonEmission.cpp | 15 +++-- Source/Particles/LaserParticleContainer.H | 24 ++++--- Source/Particles/MultiParticleContainer.H | 5 ++ .../NamedComponentParticleContainer.H | 2 +- Source/Particles/ParticleCreation/SmartCopy.H | 20 +++--- .../Particles/ParticleCreation/SmartCreate.H | 12 ++-- Source/Particles/PhotonParticleContainer.H | 19 ++++-- Source/Particles/PhysicalParticleContainer.H | 29 ++++---- .../Particles/PhysicalParticleContainer.cpp | 15 ++++- .../Particles/Resampling/LevelingThinning.H | 2 +- Source/Particles/Resampling/Resampling.H | 9 +++ .../RigidInjectedParticleContainer.H | 21 ++++-- Source/Particles/Sorting/SortingUtils.H | 42 ++++++------ Source/Particles/WarpXParticleContainer.H | 11 +++- Source/Utils/Parser/IntervalsParser.cpp | 4 +- Source/WarpX.H | 15 ++--- Source/ablastr/parallelization/KernelTimer.H | 5 ++ Source/ablastr/profiler/ProfilerWrapper.H | 6 ++ Source/ablastr/warn_manager/WarnManager.H | 19 +++++- 116 files changed, 604 insertions(+), 379 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index 2d45d750ddc..c7fb58ae117 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -7,14 +7,17 @@ Checks: ' -bugprone-unchecked-optional-access, cert-*, -cert-err58-cpp, - cppcoreguidelines-avoid-goto, - cppcoreguidelines-interfaces-global-init, + cppcoreguidelines-*, + -cppcoreguidelines-avoid-c-arrays, + -cppcoreguidelines-avoid-magic-numbers, -cppcoreguidelines-avoid-non-const-global-variables, -cppcoreguidelines-init-variables, -cppcoreguidelines-macro-usage, - -cppcoreguidelines-narrowing-conversions, -cppcoreguidelines-no-malloc, + -cppcoreguidelines-narrowing-conversions, + -cppcoreguidelines-non-private-member-variables-in-classes, -cppcoreguidelines-owning-memory, + -cppcoreguidelines-pro-*, google-build-explicit-make-pair, google-build-namespaces, google-global-names-in-headers, diff --git a/Source/AcceleratorLattice/LatticeElements/Drift.H b/Source/AcceleratorLattice/LatticeElements/Drift.H index a9067a09caa..7ad36df8dba 100644 --- a/Source/AcceleratorLattice/LatticeElements/Drift.H +++ b/Source/AcceleratorLattice/LatticeElements/Drift.H @@ -17,8 +17,6 @@ struct Drift Drift (); - ~Drift () = default; - /** * \brief Read in an element and add it to the lists * diff --git a/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H b/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H index 43009c0c4ed..c30a91867e6 100644 --- a/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H +++ b/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H @@ -27,8 +27,6 @@ struct HardEdgedPlasmaLens HardEdgedPlasmaLens (); - ~HardEdgedPlasmaLens () = default; - /** * \brief Read in an element and add it to the lists * diff --git a/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H b/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H index 82f1e860a40..0cdefea769e 100644 --- a/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H +++ b/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H @@ -27,8 +27,6 @@ struct HardEdgedQuadrupole HardEdgedQuadrupole (); - ~HardEdgedQuadrupole () = default; - /** * \brief Read in an element and add it to the lists * diff --git a/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.H b/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.H index ad72eeef910..15cb72435bd 100644 --- a/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.H +++ b/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.H @@ -25,8 +25,6 @@ struct LatticeElementBase */ LatticeElementBase (std::string const& element_name); - ~LatticeElementBase () = default; - /** * \brief Read in an element base data and add it to the lists * diff --git a/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.cpp b/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.cpp index 248db59aaf6..19afd8cd617 100644 --- a/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.cpp +++ b/Source/AcceleratorLattice/LatticeElements/LatticeElementBase.cpp @@ -12,10 +12,8 @@ #include -LatticeElementBase::LatticeElementBase (std::string const& element_name) -{ - m_element_name = element_name; -} +LatticeElementBase::LatticeElementBase (std::string const& element_name): + m_element_name{element_name}{} void LatticeElementBase::AddElementBase (amrex::ParmParse & pp_element, amrex::ParticleReal & z_location) diff --git a/Source/BoundaryConditions/PML.H b/Source/BoundaryConditions/PML.H index 4be1be146cc..649d4a6b40b 100644 --- a/Source/BoundaryConditions/PML.H +++ b/Source/BoundaryConditions/PML.H @@ -81,7 +81,7 @@ public: const amrex::IntVect& ncell, const amrex::IntVect& delta, const amrex::Box& regular_domain, const amrex::Real v_sigma_sb) : m_grids(grid_ba), m_dx(dx), m_ncell(ncell), m_delta(delta), m_regdomain(regular_domain), m_v_sigma_sb(v_sigma_sb) {} - virtual ~SigmaBoxFactory () = default; + ~SigmaBoxFactory () override = default; SigmaBoxFactory (const SigmaBoxFactory&) = default; SigmaBoxFactory (SigmaBoxFactory&&) noexcept = default; @@ -90,13 +90,13 @@ public: SigmaBoxFactory& operator= (const SigmaBoxFactory&) = delete; SigmaBoxFactory& operator= (SigmaBoxFactory&&) = delete; - virtual SigmaBox* create (const amrex::Box& box, int /*ncomps*/, + SigmaBox* create (const amrex::Box& box, int /*ncomps*/, const amrex::FabInfo& /*info*/, int /*box_index*/) const final { return new SigmaBox(box, m_grids, m_dx, m_ncell, m_delta, m_regdomain, m_v_sigma_sb); } - virtual void destroy (SigmaBox* fab) const final { + void destroy (SigmaBox* fab) const final { delete fab; } - virtual SigmaBoxFactory* clone () const final { + SigmaBoxFactory* clone () const final { return new SigmaBoxFactory(*this); } private: @@ -200,8 +200,6 @@ public: static void Exchange (amrex::MultiFab& pml, amrex::MultiFab& reg, const amrex::Geometry& geom, int do_pml_in_domain); - ~PML () = default; - private: bool m_ok; diff --git a/Source/BoundaryConditions/PML_RZ.H b/Source/BoundaryConditions/PML_RZ.H index ac4b6ff2c4d..3a5a51770f8 100644 --- a/Source/BoundaryConditions/PML_RZ.H +++ b/Source/BoundaryConditions/PML_RZ.H @@ -52,8 +52,6 @@ public: void CheckPoint (const std::string& dir) const; void Restart (const std::string& dir); - ~PML_RZ () = default; - private: const int m_ncell; diff --git a/Source/Diagnostics/BTD_Plotfile_Header_Impl.H b/Source/Diagnostics/BTD_Plotfile_Header_Impl.H index d8038a48f3d..2becaba85b3 100644 --- a/Source/Diagnostics/BTD_Plotfile_Header_Impl.H +++ b/Source/Diagnostics/BTD_Plotfile_Header_Impl.H @@ -31,8 +31,6 @@ public: * \param[in] Headerfile_path string containing path of Headerfile */ BTDPlotfileHeaderImpl (std::string const& Headerfile_path); - /** Destructor */ - ~BTDPlotfileHeaderImpl () = default; /** Returns the Header file version for plotfile */ std::string fileVersion () const noexcept {return m_file_version; } @@ -166,7 +164,7 @@ class BTDMultiFabHeaderImpl * \param[in] Headerfile_path string containing path of Headerfile */ BTDMultiFabHeaderImpl (std::string const& Headerfile_path); - ~BTDMultiFabHeaderImpl () = default; + /** Reads the Multifab Header file and stores its data. */ void ReadMultiFabHeader (); /** Writes the meta-data of the Multifab in a header file, with path, m_Header_path. */ @@ -249,7 +247,7 @@ public: * \param[in] species_name string containing species name */ BTDSpeciesHeaderImpl (std::string const& Headerfile_path, std::string const& species_name); - ~BTDSpeciesHeaderImpl () = default; + /** Reads the Header file for BTD species*/ void ReadHeader (); /** Writes the meta-data of species Header file, with path, m_Header_path*/ @@ -314,8 +312,7 @@ public: * \param[in] Headerfile_path containing path of Headerfile */ BTDParticleDataHeaderImpl (std::string const& Headerfile_path); - /** Destructor */ - ~BTDParticleDataHeaderImpl () = default; + /** Reads the particle header file at m_Header_path and stores its data*/ void ReadHeader (); /** Writes the meta-data of particle box array in header file, with path, m_Header_path*/ diff --git a/Source/Diagnostics/ComputeDiagFunctors/CellCenterFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/CellCenterFunctor.H index 84ea079c63c..58823d7bb17 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/CellCenterFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/CellCenterFunctor.H @@ -34,7 +34,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; private: /** pointer to source multifab (can be multi-component) */ amrex::MultiFab const * const m_mf_src = nullptr; diff --git a/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H index b4c413b3aa1..9e21160242f 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H @@ -21,6 +21,13 @@ public: m_ncomp(ncomp), m_crse_ratio(crse_ratio) {} //** Virtual Destructor to handle clean destruction of derived classes */ virtual ~ComputeDiagFunctor() = default; + + // Default move and copy operations + ComputeDiagFunctor(const ComputeDiagFunctor&) = default; + ComputeDiagFunctor& operator=(const ComputeDiagFunctor&) = default; + ComputeDiagFunctor(ComputeDiagFunctor&&) = default; + ComputeDiagFunctor& operator=(ComputeDiagFunctor&&) = default; + /** Compute a field and store the result in mf_dst * * \param[out] mf_dst output MultiFab where the result is written diff --git a/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H index e14e777e066..4a5e8976943 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H @@ -24,6 +24,12 @@ public: /** Virtual Destructor to handle clean destruction of derived classes */ virtual ~ComputeParticleDiagFunctor() = default; + /** Default assignment and copy operations*/ + ComputeParticleDiagFunctor(const ComputeParticleDiagFunctor&) = default; + ComputeParticleDiagFunctor& operator=(const ComputeParticleDiagFunctor&) = default; + ComputeParticleDiagFunctor(ComputeParticleDiagFunctor&&) = default; + ComputeParticleDiagFunctor& operator=(ComputeParticleDiagFunctor&&) = default; + /** \brief Prepare data required to back-transform particle attribtutes for * lab-frame snapshot, with index i_buffer. * Note that this function has parameters that are specific to diff --git a/Source/Diagnostics/ComputeDiagFunctors/DivBFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/DivBFunctor.H index f6a297741b9..ebf890f6545 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/DivBFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/DivBFunctor.H @@ -32,7 +32,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer*/) const override; private: /** Vector of pointer to source multifab Bx, By, Bz */ std::array m_arr_mf_src; diff --git a/Source/Diagnostics/ComputeDiagFunctors/DivEFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/DivEFunctor.H index c7d3badcf0d..76e5270b972 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/DivEFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/DivEFunctor.H @@ -31,7 +31,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; private: /** Vector of pointer to source multifab Bx, By, Bz */ std::array m_arr_mf_src; diff --git a/Source/Diagnostics/ComputeDiagFunctors/JFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/JFunctor.H index db0e0331148..c42f63f1369 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/JFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/JFunctor.H @@ -35,7 +35,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; private: /** direction of the current density to save */ const int m_dir; diff --git a/Source/Diagnostics/ComputeDiagFunctors/PartPerCellFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/PartPerCellFunctor.H index 7e435b3675e..af218d2667b 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/PartPerCellFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/PartPerCellFunctor.H @@ -29,7 +29,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; private: int const m_lev; /**< level on which mf_src is defined */ }; diff --git a/Source/Diagnostics/ComputeDiagFunctors/PartPerGridFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/PartPerGridFunctor.H index d5e0c45b956..b4d64117dd7 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/PartPerGridFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/PartPerGridFunctor.H @@ -29,7 +29,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; private: int const m_lev; /**< level on which mf_src is defined */ }; diff --git a/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.H index 09b54726211..12a17ff13af 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/ParticleReductionFunctor.H @@ -42,7 +42,7 @@ public: * \param[in] dcomp first component of mf_dst in which cell-centered * data is stored */ - virtual void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; + void operator()(amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/) const override; private: int const m_lev; /**< level on which mf_src is defined */ int const m_ispec; /**< index of species to average over */ diff --git a/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.H index e359bb2f6a2..46f8a3be251 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.H @@ -38,7 +38,7 @@ public: * \param[out] mf_dst output MultiFab where the result is written * \param[in] dcomp first component of mf_dst in which cell-centered data are stored */ - virtual void operator() ( amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/ ) const override; + void operator() ( amrex::MultiFab& mf_dst, int dcomp, int /*i_buffer=0*/ ) const override; private: diff --git a/Source/Diagnostics/Diagnostics.H b/Source/Diagnostics/Diagnostics.H index 75441d6d9e8..2b5c2362ec9 100644 --- a/Source/Diagnostics/Diagnostics.H +++ b/Source/Diagnostics/Diagnostics.H @@ -40,6 +40,11 @@ public: /** Virtual Destructor to handle clean destruction of derived classes */ virtual ~Diagnostics (); + Diagnostics (Diagnostics const &) = delete; + Diagnostics& operator= (Diagnostics const & ) = delete; + Diagnostics(Diagnostics&& ) = default; + Diagnostics& operator=(Diagnostics&& ) = default; + /** Pack (stack) all fields in the cell-centered output MultiFab m_mf_output. * * Fields are computed (e.g., cell-centered or back-transformed) diff --git a/Source/Diagnostics/FlushFormats/FlushFormat.H b/Source/Diagnostics/FlushFormats/FlushFormat.H index d8820d35a12..f5a693a5fa4 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormat.H +++ b/Source/Diagnostics/FlushFormats/FlushFormat.H @@ -12,7 +12,7 @@ class FlushFormat public: /** Flush fields and particles to file */ virtual void WriteToFile ( - amrex::Vector varnames, + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, amrex::Vector iteration, double time, @@ -27,7 +27,13 @@ public: bool isLastBTDFlush = false, const amrex::Vector& totalParticlesFlushedAlready = amrex::Vector() ) const = 0; - virtual ~FlushFormat() {} + FlushFormat () = default; + virtual ~FlushFormat() {} + + FlushFormat ( FlushFormat const &) = default; + FlushFormat& operator= ( FlushFormat const & ) = default; + FlushFormat ( FlushFormat&& ) = default; + FlushFormat& operator= ( FlushFormat&& ) = default; }; #endif // WARPX_FLUSHFORMAT_H_ diff --git a/Source/Diagnostics/FlushFormats/FlushFormatAscent.H b/Source/Diagnostics/FlushFormats/FlushFormatAscent.H index b15b7432dd5..228e4bc5cf6 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatAscent.H +++ b/Source/Diagnostics/FlushFormats/FlushFormatAscent.H @@ -28,8 +28,8 @@ class FlushFormatAscent : public FlushFormat { public: /** Do in-situ visualization for field and particle data */ - virtual void WriteToFile ( - amrex::Vector varnames, + void WriteToFile ( + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, amrex::Vector iteration, double time, @@ -53,7 +53,13 @@ public: void WriteParticles(const amrex::Vector& particle_diags, conduit::Node& a_bp_mesh) const; #endif - ~FlushFormatAscent() {} + FlushFormatAscent () = default; + ~FlushFormatAscent() override = default; + + FlushFormatAscent ( FlushFormatAscent const &) = default; + FlushFormatAscent& operator= ( FlushFormatAscent const & ) = default; + FlushFormatAscent ( FlushFormatAscent&& ) = default; + FlushFormatAscent& operator= ( FlushFormatAscent&& ) = default; }; #endif // WARPX_FLUSHFORMATASCENT_H_ diff --git a/Source/Diagnostics/FlushFormats/FlushFormatAscent.cpp b/Source/Diagnostics/FlushFormats/FlushFormatAscent.cpp index 8fe25bef32b..980047e3b46 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatAscent.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatAscent.cpp @@ -11,7 +11,7 @@ using namespace amrex; void FlushFormatAscent::WriteToFile ( - const amrex::Vector varnames, + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, const amrex::Vector iteration, const double time, diff --git a/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.H b/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.H index 5131d6f4234..f6aad226d75 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.H +++ b/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.H @@ -15,8 +15,8 @@ class FlushFormatCheckpoint final : public FlushFormatPlotfile { /** Flush fields and particles to plotfile */ - virtual void WriteToFile ( - amrex::Vector varnames, + void WriteToFile ( + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, amrex::Vector iteration, double time, @@ -29,7 +29,7 @@ class FlushFormatCheckpoint final : public FlushFormatPlotfile int bufferID = 1, int numBuffers = 1, const amrex::Geometry& full_BTD_snapshot = amrex::Geometry(), bool isLastBTDFlush = false, - const amrex::Vector& totalParticlesFlushedAlready = amrex::Vector() ) const override final; + const amrex::Vector& totalParticlesFlushedAlready = amrex::Vector() ) const final; void CheckpointParticles (const std::string& dir, const amrex::Vector& particle_diags) const; diff --git a/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.cpp b/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.cpp index 6608123bb33..2247a7eaadf 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatCheckpoint.cpp @@ -27,7 +27,7 @@ namespace void FlushFormatCheckpoint::WriteToFile ( - const amrex::Vector /*varnames*/, + const amrex::Vector& /*varnames*/, const amrex::Vector& /*mf*/, amrex::Vector& geom, const amrex::Vector iteration, const double /*time*/, diff --git a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.H b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.H index 4da45c00bbf..88380407f5e 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.H +++ b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.H @@ -28,7 +28,7 @@ public: /** Flush fields and particles to plotfile */ void WriteToFile ( - amrex::Vector varnames, + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, amrex::Vector iteration, double time, @@ -45,6 +45,11 @@ public: ~FlushFormatOpenPMD () override = default; + FlushFormatOpenPMD ( FlushFormatOpenPMD const &) = delete; + FlushFormatOpenPMD& operator= ( FlushFormatOpenPMD const & ) = delete; + FlushFormatOpenPMD ( FlushFormatOpenPMD&& ) = default; + FlushFormatOpenPMD& operator= ( FlushFormatOpenPMD&& ) = default; + private: /** This is responsible for dumping to file */ std::unique_ptr< WarpXOpenPMDPlot > m_OpenPMDPlotWriter; diff --git a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp index 2185d4b4cca..3e43d18d7fa 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp @@ -112,7 +112,7 @@ FlushFormatOpenPMD::FlushFormatOpenPMD (const std::string& diag_name) void FlushFormatOpenPMD::WriteToFile ( - const amrex::Vector varnames, + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, const amrex::Vector iteration, const double time, diff --git a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H index b8461e51f23..62c7311804e 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H +++ b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H @@ -22,8 +22,8 @@ class FlushFormatPlotfile : public FlushFormat { public: /** Flush fields and particles to plotfile */ - virtual void WriteToFile ( - amrex::Vector varnames, + void WriteToFile ( + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, amrex::Vector iteration, double time, @@ -56,7 +56,13 @@ public: amrex::Real time, bool isBTD = false) const; - ~FlushFormatPlotfile() {} + FlushFormatPlotfile () = default; + ~FlushFormatPlotfile() override {} + + FlushFormatPlotfile ( FlushFormatPlotfile const &) = default; + FlushFormatPlotfile& operator= ( FlushFormatPlotfile const & ) = default; + FlushFormatPlotfile ( FlushFormatPlotfile&& ) = default; + FlushFormatPlotfile& operator= ( FlushFormatPlotfile&& ) = default; }; #endif // WARPX_FLUSHFORMATPLOTFILE_H_ diff --git a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp index a18bab25308..c2e372b1fb8 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.cpp @@ -55,7 +55,7 @@ namespace void FlushFormatPlotfile::WriteToFile ( - const amrex::Vector varnames, + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, const amrex::Vector iteration, const double time, diff --git a/Source/Diagnostics/FlushFormats/FlushFormatSensei.H b/Source/Diagnostics/FlushFormats/FlushFormatSensei.H index 28785c1c905..54eb7099ba4 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatSensei.H +++ b/Source/Diagnostics/FlushFormats/FlushFormatSensei.H @@ -33,10 +33,12 @@ class FlushFormatSensei : public FlushFormat { public: FlushFormatSensei(); - ~FlushFormatSensei(); + ~FlushFormatSensei() override; - FlushFormatSensei(const FlushFormatSensei &) = delete; - void operator=(const FlushFormatSensei &) = delete; + FlushFormatSensei(const FlushFormatSensei &) = delete; + FlushFormatSensei& operator=(const FlushFormatSensei &) = delete; + FlushFormatSensei(FlushFormatSensei&&) = default; + FlushFormatSensei& operator=(FlushFormatSensei&& ) = default; /** Construct. * @@ -46,8 +48,8 @@ public: FlushFormatSensei (amrex::AmrMesh *amr_mesh, std::string diag_name); /** Do in-situ visualization for field and particle data */ - virtual void WriteToFile ( - amrex::Vector varnames, + void WriteToFile ( + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, amrex::Vector iteration, double time, diff --git a/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp b/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp index 060c0e4a5e9..7d047913988 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp @@ -50,7 +50,7 @@ FlushFormatSensei::~FlushFormatSensei () = default; void FlushFormatSensei::WriteToFile ( - const amrex::Vector varnames, + const amrex::Vector& varnames, const amrex::Vector& mf, amrex::Vector& geom, const amrex::Vector iteration, const double time, diff --git a/Source/Diagnostics/FullDiagnostics.cpp b/Source/Diagnostics/FullDiagnostics.cpp index 6848f9e1a18..c1c8f270bca 100644 --- a/Source/Diagnostics/FullDiagnostics.cpp +++ b/Source/Diagnostics/FullDiagnostics.cpp @@ -41,16 +41,14 @@ using namespace amrex::literals; -FullDiagnostics::FullDiagnostics (int i, std::string name) - : Diagnostics(i, name) +FullDiagnostics::FullDiagnostics (int i, std::string name): + Diagnostics{i, name}, + m_solver_deposits_current{ + !(WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::None && + WarpX::electrostatic_solver_id != ElectrostaticSolverAlgo::LabFrameElectroMagnetostatic)} { ReadParameters(); BackwardCompatibility(); - - m_solver_deposits_current = !( - WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::None && - WarpX::electrostatic_solver_id != ElectrostaticSolverAlgo::LabFrameElectroMagnetostatic - ); } void diff --git a/Source/Diagnostics/ReducedDiags/BeamRelevant.H b/Source/Diagnostics/ReducedDiags/BeamRelevant.H index 4e0d0077dad..f5037e37bc0 100644 --- a/Source/Diagnostics/ReducedDiags/BeamRelevant.H +++ b/Source/Diagnostics/ReducedDiags/BeamRelevant.H @@ -33,7 +33,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; diff --git a/Source/Diagnostics/ReducedDiags/ChargeOnEB.H b/Source/Diagnostics/ReducedDiags/ChargeOnEB.H index c7aa58f06e2..ed69ce01767 100644 --- a/Source/Diagnostics/ReducedDiags/ChargeOnEB.H +++ b/Source/Diagnostics/ReducedDiags/ChargeOnEB.H @@ -42,7 +42,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags (int step) override final; + void ComputeDiags (int step) final; private: /// Optional parser to add weight inside the integral diff --git a/Source/Diagnostics/ReducedDiags/ColliderRelevant.H b/Source/Diagnostics/ReducedDiags/ColliderRelevant.H index 1998c56812d..fe00dd2e7e3 100644 --- a/Source/Diagnostics/ReducedDiags/ColliderRelevant.H +++ b/Source/Diagnostics/ReducedDiags/ColliderRelevant.H @@ -44,7 +44,7 @@ public: * [14]thetay_min, [15]thetay_ave, [16]thetay_max, [17]thetay_std * same for second species follows. */ - void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; private: /// auxiliary structure to store headers and indices of the reduced diagnostics diff --git a/Source/Diagnostics/ReducedDiags/FieldEnergy.H b/Source/Diagnostics/ReducedDiags/FieldEnergy.H index 73ebad34ba0..4f5d1656841 100644 --- a/Source/Diagnostics/ReducedDiags/FieldEnergy.H +++ b/Source/Diagnostics/ReducedDiags/FieldEnergy.H @@ -37,7 +37,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; /** * \brief Calculate the integral of the field squared in RZ diff --git a/Source/Diagnostics/ReducedDiags/FieldMaximum.H b/Source/Diagnostics/ReducedDiags/FieldMaximum.H index ea40c0e128c..95ae6ff16da 100644 --- a/Source/Diagnostics/ReducedDiags/FieldMaximum.H +++ b/Source/Diagnostics/ReducedDiags/FieldMaximum.H @@ -31,7 +31,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; diff --git a/Source/Diagnostics/ReducedDiags/FieldMomentum.H b/Source/Diagnostics/ReducedDiags/FieldMomentum.H index 2dadb6a8e86..0f04b6413d1 100644 --- a/Source/Diagnostics/ReducedDiags/FieldMomentum.H +++ b/Source/Diagnostics/ReducedDiags/FieldMomentum.H @@ -32,7 +32,7 @@ public: * * \param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; #endif diff --git a/Source/Diagnostics/ReducedDiags/FieldProbe.H b/Source/Diagnostics/ReducedDiags/FieldProbe.H index 2b232264329..9f318c4f8a0 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbe.H +++ b/Source/Diagnostics/ReducedDiags/FieldProbe.H @@ -46,18 +46,18 @@ public: /** * This function assins test/data particles to constructed environemnt */ - void InitData () override final; + void InitData () final; /** Redistribute parallel data structures during load balance */ - void LoadBalance () override final; + void LoadBalance () final; /** * This function computes the value of Ex, Ey, Ez, Bx, By, Bz and at a given point * * @param[in] step current time step */ - void ComputeDiags (int step) override final; + void ComputeDiags (int step) final; /* * Define constants used throughout FieldProbe @@ -113,7 +113,7 @@ private: /** * Built-in function in ReducedDiags to write out test data */ - virtual void WriteToFile (int step) const override; + void WriteToFile (int step) const override; /** Check if the probe is in the simulation domain boundary */ diff --git a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H index b3277b875d9..c85bf8fd541 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H +++ b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.H @@ -44,7 +44,12 @@ class FieldProbeParticleContainer { public: FieldProbeParticleContainer (amrex::AmrCore* amr_core); - virtual ~FieldProbeParticleContainer() {} + ~FieldProbeParticleContainer() override = default; + + FieldProbeParticleContainer ( FieldProbeParticleContainer const &) = delete; + FieldProbeParticleContainer& operator= ( FieldProbeParticleContainer const & ) = delete; + FieldProbeParticleContainer ( FieldProbeParticleContainer&& ) = default; + FieldProbeParticleContainer& operator= ( FieldProbeParticleContainer&& ) = default; //! amrex iterator for our number of attributes using iterator = amrex::ParIter<0, 0, FieldProbePIdx::nattribs, 0>; diff --git a/Source/Diagnostics/ReducedDiags/FieldReduction.H b/Source/Diagnostics/ReducedDiags/FieldReduction.H index c4c81c0161a..e99c87ad14f 100644 --- a/Source/Diagnostics/ReducedDiags/FieldReduction.H +++ b/Source/Diagnostics/ReducedDiags/FieldReduction.H @@ -58,7 +58,7 @@ public: * * @param[in] step the timestep */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; /** * This function queries deprecated input parameters and aborts diff --git a/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.H b/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.H index f2bfc12d7e1..0ecb74d4bf5 100644 --- a/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.H +++ b/Source/Diagnostics/ReducedDiags/LoadBalanceCosts.H @@ -62,7 +62,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; /** * write to file function for costs; this differs from the base class @@ -71,7 +71,7 @@ public: * * @param[in] step current time step */ - virtual void WriteToFile(int step) const override final; + void WriteToFile(int step) const final; }; diff --git a/Source/Diagnostics/ReducedDiags/LoadBalanceEfficiency.H b/Source/Diagnostics/ReducedDiags/LoadBalanceEfficiency.H index d0c823fb949..a43868ccc5f 100644 --- a/Source/Diagnostics/ReducedDiags/LoadBalanceEfficiency.H +++ b/Source/Diagnostics/ReducedDiags/LoadBalanceEfficiency.H @@ -31,7 +31,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; #endif diff --git a/Source/Diagnostics/ReducedDiags/ParticleEnergy.H b/Source/Diagnostics/ReducedDiags/ParticleEnergy.H index b143cc93f04..b0df66709f3 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleEnergy.H +++ b/Source/Diagnostics/ReducedDiags/ParticleEnergy.H @@ -35,7 +35,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; diff --git a/Source/Diagnostics/ReducedDiags/ParticleExtrema.H b/Source/Diagnostics/ReducedDiags/ParticleExtrema.H index b6bfb7c5e22..51bc1a600f9 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleExtrema.H +++ b/Source/Diagnostics/ReducedDiags/ParticleExtrema.H @@ -35,7 +35,7 @@ public: * * @param[in] step current time step */ - void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; private: /// auxiliary structure to store headers and indices of the reduced diagnostics diff --git a/Source/Diagnostics/ReducedDiags/ParticleHistogram.H b/Source/Diagnostics/ReducedDiags/ParticleHistogram.H index 07a2b9599fb..758f4a399a9 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleHistogram.H +++ b/Source/Diagnostics/ReducedDiags/ParticleHistogram.H @@ -62,7 +62,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; diff --git a/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp b/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp index 9da001ba7ed..ecefd08eab7 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp +++ b/Source/Diagnostics/ReducedDiags/ParticleHistogram.cpp @@ -50,8 +50,8 @@ struct NormalizationType { }; // constructor -ParticleHistogram::ParticleHistogram (std::string rd_name) -: ReducedDiags{rd_name} +ParticleHistogram::ParticleHistogram (std::string rd_name): + ReducedDiags{rd_name} { const ParmParse pp_rd_name(rd_name); @@ -60,10 +60,15 @@ ParticleHistogram::ParticleHistogram (std::string rd_name) pp_rd_name.get("species",selected_species_name); // read bin parameters - utils::parser::getWithParser(pp_rd_name, "bin_number",m_bin_num); - utils::parser::getWithParser(pp_rd_name, "bin_max", m_bin_max); - utils::parser::getWithParser(pp_rd_name, "bin_min", m_bin_min); - m_bin_size = (m_bin_max - m_bin_min) / m_bin_num; + int bin_num = 0; + amrex::Real bin_max = 0.0_rt, bin_min = 0.0_rt; + utils::parser::getWithParser(pp_rd_name, "bin_number", bin_num); + utils::parser::getWithParser(pp_rd_name, "bin_max", bin_max); + utils::parser::getWithParser(pp_rd_name, "bin_min", bin_min); + m_bin_num = bin_num; + m_bin_max = bin_max; + m_bin_min = bin_min; + m_bin_size = (bin_max - bin_min) / bin_num; // read histogram function std::string function_string; diff --git a/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.H b/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.H index 6b82514567d..98697380a4a 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.H +++ b/Source/Diagnostics/ReducedDiags/ParticleHistogram2D.H @@ -84,14 +84,14 @@ public: * * @param[in] step current time step */ - void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; /** * write to file function * * @param[in] step current time step */ - void WriteToFile (int step) const override final; + void WriteToFile (int step) const final; }; diff --git a/Source/Diagnostics/ReducedDiags/ParticleMomentum.H b/Source/Diagnostics/ReducedDiags/ParticleMomentum.H index ec16d8184c5..feb8957e516 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleMomentum.H +++ b/Source/Diagnostics/ReducedDiags/ParticleMomentum.H @@ -34,7 +34,7 @@ public: * * \param [in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; #endif diff --git a/Source/Diagnostics/ReducedDiags/ParticleNumber.H b/Source/Diagnostics/ReducedDiags/ParticleNumber.H index ad6c886e269..3143a387a74 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleNumber.H +++ b/Source/Diagnostics/ReducedDiags/ParticleNumber.H @@ -32,7 +32,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; }; diff --git a/Source/Diagnostics/ReducedDiags/ReducedDiags.H b/Source/Diagnostics/ReducedDiags/ReducedDiags.H index c4fc82fbddf..01847a1ff12 100644 --- a/Source/Diagnostics/ReducedDiags/ReducedDiags.H +++ b/Source/Diagnostics/ReducedDiags/ReducedDiags.H @@ -59,6 +59,13 @@ public: */ virtual ~ReducedDiags () = default; + // Default move and copy operations + ReducedDiags(const ReducedDiags&) = default; + ReducedDiags& operator=(const ReducedDiags&) = default; + ReducedDiags(ReducedDiags&&) = default; + ReducedDiags& operator=(ReducedDiags&&) = default; + + /** * function to initialize data after amr * levels are initialized. diff --git a/Source/Diagnostics/ReducedDiags/RhoMaximum.H b/Source/Diagnostics/ReducedDiags/RhoMaximum.H index e7386f73383..6f306a5e7ad 100644 --- a/Source/Diagnostics/ReducedDiags/RhoMaximum.H +++ b/Source/Diagnostics/ReducedDiags/RhoMaximum.H @@ -36,7 +36,7 @@ public: * * @param[in] step current time step */ - virtual void ComputeDiags(int step) override final; + void ComputeDiags(int step) final; private: /** Vector of (pointers to) functors to compute rho, per level, per species. We reuse here the diff --git a/Source/Diagnostics/WarpXOpenPMD.H b/Source/Diagnostics/WarpXOpenPMD.H index 62a1fa3755f..1acfc19b5a9 100644 --- a/Source/Diagnostics/WarpXOpenPMD.H +++ b/Source/Diagnostics/WarpXOpenPMD.H @@ -99,6 +99,11 @@ public: ~WarpXOpenPMDPlot (); + WarpXOpenPMDPlot ( WarpXOpenPMDPlot const &) = delete; + WarpXOpenPMDPlot& operator= ( WarpXOpenPMDPlot const & ) = delete; + WarpXOpenPMDPlot ( WarpXOpenPMDPlot&& ) = default; + WarpXOpenPMDPlot& operator= ( WarpXOpenPMDPlot&& ) = default; + /** Set Iteration Step for the series * * @note If an iteration has been written, then it will give a warning diff --git a/Source/Diagnostics/WarpXOpenPMD.cpp b/Source/Diagnostics/WarpXOpenPMD.cpp index 458760d3664..110806cbbaa 100644 --- a/Source/Diagnostics/WarpXOpenPMD.cpp +++ b/Source/Diagnostics/WarpXOpenPMD.cpp @@ -378,6 +378,8 @@ WarpXOpenPMDPlot::WarpXOpenPMDPlot ( std::map< std::string, std::string > engine_parameters, std::vector fieldPMLdirections) :m_Series(nullptr), + m_MPIRank{amrex::ParallelDescriptor::MyProc()}, + m_MPISize{amrex::ParallelDescriptor::NProcs()}, m_Encoding(ie), m_OpenPMDFileType(std::move(openPMDFileType)), m_fieldPMLdirections(std::move(fieldPMLdirections)) @@ -490,15 +492,11 @@ WarpXOpenPMDPlot::Init (openPMD::Access access, bool isBTD) amrex::ParallelDescriptor::Communicator(), m_OpenPMDoptions ); - m_MPISize = amrex::ParallelDescriptor::NProcs(); - m_MPIRank = amrex::ParallelDescriptor::MyProc(); #else WARPX_ABORT_WITH_MESSAGE("openPMD-api not built with MPI support!"); #endif } else { m_Series = std::make_unique(filepath, access, m_OpenPMDoptions); - m_MPISize = 1; - m_MPIRank = 1; } m_Series->setIterationEncoding( m_Encoding ); @@ -1481,11 +1479,10 @@ WarpXOpenPMDPlot::WriteOpenPMDFieldsAll ( //const std::string& filename, // // // -WarpXParticleCounter::WarpXParticleCounter (ParticleContainer* pc) +WarpXParticleCounter::WarpXParticleCounter (ParticleContainer* pc): + m_MPIRank{amrex::ParallelDescriptor::MyProc()}, + m_MPISize{amrex::ParallelDescriptor::NProcs()} { - m_MPISize = amrex::ParallelDescriptor::NProcs(); - m_MPIRank = amrex::ParallelDescriptor::MyProc(); - m_ParticleCounterByLevel.resize(pc->finestLevel()+1); m_ParticleOffsetAtRank.resize(pc->finestLevel()+1); m_ParticleSizeAtRank.resize(pc->finestLevel()+1); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/FiniteDifferenceSolver.cpp b/Source/FieldSolver/FiniteDifferenceSolver/FiniteDifferenceSolver.cpp index 1bb557d009d..8923ec1f6a5 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/FiniteDifferenceSolver.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/FiniteDifferenceSolver.cpp @@ -30,12 +30,11 @@ FiniteDifferenceSolver::FiniteDifferenceSolver ( int const fdtd_algo, std::array cell_size, - short grid_type) { - + short grid_type): // Register the type of finite-difference algorithm - m_fdtd_algo = fdtd_algo; - m_grid_type = grid_type; - + m_fdtd_algo{fdtd_algo}, + m_grid_type{grid_type} +{ // return if not FDTD if (fdtd_algo == ElectromagneticSolverAlgo::None || fdtd_algo == ElectromagneticSolverAlgo::PSATD) return; diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.H index be9dbf26b59..87dfe6f2d9a 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.H @@ -40,7 +40,7 @@ class PsatdAlgorithmComoving : public SpectralBaseAlgorithm /** * \brief Override the update equations in Fourier space */ - virtual void pushSpectralFields (SpectralFieldData& f) const override final; + void pushSpectralFields (SpectralFieldData& f) const final; /* \brief Initialize the coefficients needed in the update equations */ @@ -56,7 +56,7 @@ class PsatdAlgorithmComoving : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldData& field_data) override final; + void CurrentCorrection (SpectralFieldData& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space. @@ -66,7 +66,7 @@ class PsatdAlgorithmComoving : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldData& field_data) override final; + void VayDeposition (SpectralFieldData& field_data) final; private: diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmFirstOrder.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmFirstOrder.H index dea72bec396..8fde50d8172 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmFirstOrder.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmFirstOrder.H @@ -62,7 +62,7 @@ class PsatdAlgorithmFirstOrder : public SpectralBaseAlgorithm * * \param[in,out] f all the fields in spectral space */ - virtual void pushSpectralFields (SpectralFieldData& f) const override final; + void pushSpectralFields (SpectralFieldData& f) const final; /** * \brief Virtual function for current correction in Fourier space @@ -73,7 +73,7 @@ class PsatdAlgorithmFirstOrder : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldData& field_data) override final; + void CurrentCorrection (SpectralFieldData& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -84,7 +84,7 @@ class PsatdAlgorithmFirstOrder : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldData& field_data) override final; + void VayDeposition (SpectralFieldData& field_data) final; private: diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.H index 37d76373990..ad0aca1ecf5 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.H @@ -25,19 +25,19 @@ class PsatdAlgorithmGalileanRZ : public SpectralBaseAlgorithmRZ amrex::Real dt_step, bool update_with_rho); // Redefine functions from base class - virtual void pushSpectralFields (SpectralFieldDataRZ & f) override final; + void pushSpectralFields (SpectralFieldDataRZ & f) final; void InitializeSpectralCoefficients (SpectralFieldDataRZ const & f); /** * \brief Virtual function for current correction in Fourier space * This function overrides the virtual function \c CurrentCorrection in the - * base class \c SpectralBaseAlgorithmRZ (qualifier \c override) and cannot be + * base class \c SpectralBaseAlgorithmRZ and cannot be * overridden by further derived classes (qualifier \c final). * * \param[in,out] field_data all fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldDataRZ& field_data) override final; + void CurrentCorrection (SpectralFieldDataRZ& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -47,11 +47,11 @@ class PsatdAlgorithmGalileanRZ : public SpectralBaseAlgorithmRZ * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldDataRZ& field_data) override final; + void VayDeposition (SpectralFieldDataRZ& field_data) final; private: - bool coefficients_initialized; + bool coefficients_initialized = false; // Note that dt and v_galilean are saved to use in InitializeSpectralCoefficients amrex::Real m_dt; amrex::Vector m_v_galilean; diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp index 3188712d61a..dad10716655 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp @@ -23,14 +23,13 @@ PsatdAlgorithmGalileanRZ::PsatdAlgorithmGalileanRZ (SpectralKSpaceRZ const & spe short const grid_type, const amrex::Vector& v_galilean, amrex::Real const dt, - bool const update_with_rho) + bool const update_with_rho): // Initialize members of base class - : SpectralBaseAlgorithmRZ(spectral_kspace, dm, spectral_index, norder_z, grid_type), - m_dt(dt), - m_v_galilean(v_galilean), - m_update_with_rho(update_with_rho) + SpectralBaseAlgorithmRZ{spectral_kspace, dm, spectral_index, norder_z, grid_type}, + m_dt{dt}, + m_v_galilean{v_galilean}, + m_update_with_rho{update_with_rho} { - // Allocate the arrays of coefficients amrex::BoxArray const & ba = spectral_kspace.spectralspace_ba; C_coef = SpectralRealCoefficients(ba, dm, n_rz_azimuthal_modes, 0); @@ -41,8 +40,6 @@ PsatdAlgorithmGalileanRZ::PsatdAlgorithmGalileanRZ (SpectralKSpaceRZ const & spe X4_coef = SpectralComplexCoefficients(ba, dm, n_rz_azimuthal_modes, 0); Theta2_coef = SpectralComplexCoefficients(ba, dm, n_rz_azimuthal_modes, 0); T_rho_coef = SpectralComplexCoefficients(ba, dm, n_rz_azimuthal_modes, 0); - - coefficients_initialized = false; } /* Advance the E and B field in spectral space (stored in `f`) diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.H index ed293fd1efa..94f00d6d4eb 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.H @@ -65,7 +65,7 @@ class PsatdAlgorithmJConstantInTime : public SpectralBaseAlgorithm * * \param[in,out] f all the fields in spectral space */ - virtual void pushSpectralFields (SpectralFieldData& f) const override final; + void pushSpectralFields (SpectralFieldData& f) const final; /** * \brief Initializes the coefficients used in \c pushSpectralFields to update the E and B fields @@ -101,7 +101,7 @@ class PsatdAlgorithmJConstantInTime : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldData& field_data) override final; + void CurrentCorrection (SpectralFieldData& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -112,7 +112,7 @@ class PsatdAlgorithmJConstantInTime : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldData& field_data) override final; + void VayDeposition (SpectralFieldData& field_data) final; private: diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.cpp index 11d9bbd94d3..5513a65a2ed 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJConstantInTime.cpp @@ -60,12 +60,12 @@ PsatdAlgorithmJConstantInTime::PsatdAlgorithmJConstantInTime( m_update_with_rho(update_with_rho), m_time_averaging(time_averaging), m_dive_cleaning(dive_cleaning), - m_divb_cleaning(divb_cleaning) + m_divb_cleaning(divb_cleaning), + m_is_galilean{ + (v_galilean[0] != 0.) || (v_galilean[1] != 0.) || (v_galilean[2] != 0.)} { const amrex::BoxArray& ba = spectral_kspace.spectralspace_ba; - m_is_galilean = (v_galilean[0] != 0.) || (v_galilean[1] != 0.) || (v_galilean[2] != 0.); - // Always allocate these coefficients C_coef = SpectralRealCoefficients(ba, dm, 1, 0); S_ck_coef = SpectralRealCoefficients(ba, dm, 1, 0); diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJLinearInTime.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJLinearInTime.H index 94eda8a2f72..aa3bcc08e77 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJLinearInTime.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmJLinearInTime.H @@ -64,7 +64,7 @@ class PsatdAlgorithmJLinearInTime : public SpectralBaseAlgorithm * * \param[in,out] f all the fields in spectral space */ - virtual void pushSpectralFields (SpectralFieldData& f) const override final; + void pushSpectralFields (SpectralFieldData& f) const final; /** * \brief Initializes the coefficients used in \c pushSpectralFields to update the E and B fields @@ -100,7 +100,7 @@ class PsatdAlgorithmJLinearInTime : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldData& field_data) override final; + void CurrentCorrection (SpectralFieldData& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -111,7 +111,7 @@ class PsatdAlgorithmJLinearInTime : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldData& field_data) override final; + void VayDeposition (SpectralFieldData& field_data) final; private: diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H index cef6331888e..1c25ca91903 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.H @@ -74,7 +74,7 @@ class PsatdAlgorithmPml : public SpectralBaseAlgorithm * * \param[in,out] f All fields in spectral space */ - virtual void pushSpectralFields(SpectralFieldData& f) const override final; + void pushSpectralFields(SpectralFieldData& f) const final; /** * \brief Virtual function for current correction in Fourier space @@ -85,7 +85,7 @@ class PsatdAlgorithmPml : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldData& field_data) override final; + void CurrentCorrection (SpectralFieldData& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -96,7 +96,7 @@ class PsatdAlgorithmPml : public SpectralBaseAlgorithm * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldData& field_data) override final; + void VayDeposition (SpectralFieldData& field_data) final; private: diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp index be91d920cba..92b8b3e900d 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPml.cpp @@ -56,12 +56,11 @@ PsatdAlgorithmPml::PsatdAlgorithmPml( m_v_galilean(v_galilean), m_dt(dt), m_dive_cleaning(dive_cleaning), - m_divb_cleaning(divb_cleaning) + m_divb_cleaning(divb_cleaning), + m_is_galilean{(v_galilean[0] != 0.) || (v_galilean[1] != 0.) || (v_galilean[2] != 0.)} { const BoxArray& ba = spectral_kspace.spectralspace_ba; - m_is_galilean = (v_galilean[0] != 0.) || (v_galilean[1] != 0.) || (v_galilean[2] != 0.); - // Allocate arrays of coefficients C_coef = SpectralRealCoefficients(ba, dm, 1, 0); S_ck_coef = SpectralRealCoefficients(ba, dm, 1, 0); diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.H index 33dafd382ac..00264956900 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.H @@ -23,7 +23,7 @@ class PsatdAlgorithmPmlRZ : public SpectralBaseAlgorithmRZ short grid_type, amrex::Real dt_step); // Redefine functions from base class - virtual void pushSpectralFields (SpectralFieldDataRZ & f) override final; + void pushSpectralFields (SpectralFieldDataRZ & f) final; void InitializeSpectralCoefficients (SpectralFieldDataRZ const & f); @@ -36,7 +36,7 @@ class PsatdAlgorithmPmlRZ : public SpectralBaseAlgorithmRZ * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldDataRZ& field_data) override final; + void CurrentCorrection (SpectralFieldDataRZ& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -47,11 +47,11 @@ class PsatdAlgorithmPmlRZ : public SpectralBaseAlgorithmRZ * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldDataRZ& field_data) override final; + void VayDeposition (SpectralFieldDataRZ& field_data) final; private: - bool coefficients_initialized; + bool coefficients_initialized = false; // Note that dt is saved to use in InitializeSpectralCoefficients amrex::Real m_dt; SpectralRealCoefficients C_coef, S_ck_coef; diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp index 45a21f2ffec..3bca43f4e8f 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp @@ -20,17 +20,16 @@ PsatdAlgorithmPmlRZ::PsatdAlgorithmPmlRZ (SpectralKSpaceRZ const & spectral_kspa amrex::DistributionMapping const & dm, const SpectralFieldIndex& spectral_index, int const n_rz_azimuthal_modes, int const norder_z, - short const grid_type, amrex::Real const dt) - // Initialize members of base class - : SpectralBaseAlgorithmRZ(spectral_kspace, dm, spectral_index, norder_z, grid_type), - m_dt(dt) + short const grid_type, amrex::Real const dt): + // Initialize members of base class and member variables + SpectralBaseAlgorithmRZ{spectral_kspace, dm, spectral_index, norder_z, grid_type}, + coefficients_initialized{false}, + m_dt{dt} { // Allocate the arrays of coefficients amrex::BoxArray const & ba = spectral_kspace.spectralspace_ba; C_coef = SpectralRealCoefficients(ba, dm, n_rz_azimuthal_modes, 0); S_ck_coef = SpectralRealCoefficients(ba, dm, n_rz_azimuthal_modes, 0); - - coefficients_initialized = false; } /* Advance the E and B field in spectral space (stored in `f`) diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.H index 5f02498f7d0..10feafec38b 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.H @@ -28,7 +28,7 @@ class PsatdAlgorithmRZ : public SpectralBaseAlgorithmRZ bool dive_cleaning, bool divb_cleaning); // Redefine functions from base class - virtual void pushSpectralFields(SpectralFieldDataRZ & f) override final; + void pushSpectralFields(SpectralFieldDataRZ & f) final; void InitializeSpectralCoefficients(SpectralFieldDataRZ const & f); @@ -41,7 +41,7 @@ class PsatdAlgorithmRZ : public SpectralBaseAlgorithmRZ * * \param[in,out] field_data All fields in Fourier space */ - virtual void CurrentCorrection (SpectralFieldDataRZ& field_data) override final; + void CurrentCorrection (SpectralFieldDataRZ& field_data) final; /** * \brief Virtual function for Vay current deposition in Fourier space @@ -52,7 +52,7 @@ class PsatdAlgorithmRZ : public SpectralBaseAlgorithmRZ * * \param[in,out] field_data All fields in Fourier space */ - virtual void VayDeposition (SpectralFieldDataRZ& field_data) override final; + void VayDeposition (SpectralFieldDataRZ& field_data) final; private: diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.cpp index d5a95bc90bf..fcb89c39109 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmRZ.cpp @@ -26,15 +26,15 @@ PsatdAlgorithmRZ::PsatdAlgorithmRZ (SpectralKSpaceRZ const & spectral_kspace, const int J_in_time, const int rho_in_time, const bool dive_cleaning, - const bool divb_cleaning) - // Initialize members of base class - : SpectralBaseAlgorithmRZ(spectral_kspace, dm, spectral_index, norder_z, grid_type), - m_dt(dt), - m_update_with_rho(update_with_rho), - m_time_averaging(time_averaging), - m_J_in_time(J_in_time), - m_dive_cleaning(dive_cleaning), - m_divb_cleaning(divb_cleaning) + const bool divb_cleaning): + // Initialize members of base class and member variables + SpectralBaseAlgorithmRZ{spectral_kspace, dm, spectral_index, norder_z, grid_type}, + m_dt{dt}, + m_update_with_rho{update_with_rho}, + m_time_averaging{time_averaging}, + m_J_in_time{J_in_time}, + m_dive_cleaning{dive_cleaning}, + m_divb_cleaning{divb_cleaning} { amrex::ignore_unused(rho_in_time); @@ -46,8 +46,6 @@ PsatdAlgorithmRZ::PsatdAlgorithmRZ (SpectralKSpaceRZ const & spectral_kspace, X2_coef = SpectralRealCoefficients(ba, dm, n_rz_azimuthal_modes, 0); X3_coef = SpectralRealCoefficients(ba, dm, n_rz_azimuthal_modes, 0); - coefficients_initialized = false; - if (time_averaging && J_in_time == JInTime::Linear) { X5_coef = SpectralRealCoefficients(ba, dm, n_rz_azimuthal_modes, 0); diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.H index 22fc506cc5a..4af24cfa559 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.H @@ -41,7 +41,13 @@ class SpectralBaseAlgorithm // The destructor should also be a virtual function, so that // a pointer to subclass of `SpectraBaseAlgorithm` actually // calls the subclass's destructor. - virtual ~SpectralBaseAlgorithm() {} + virtual ~SpectralBaseAlgorithm() = default; + + // Default move and copy operations + SpectralBaseAlgorithm(const SpectralBaseAlgorithm&) = default; + SpectralBaseAlgorithm& operator=(const SpectralBaseAlgorithm&) = default; + SpectralBaseAlgorithm(SpectralBaseAlgorithm&&) = default; + SpectralBaseAlgorithm& operator=(SpectralBaseAlgorithm&&) = default; /** * \brief Virtual function for current correction in Fourier space diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H index 5758f81e304..1a935de8cdf 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H @@ -28,6 +28,27 @@ class SpectralBaseAlgorithmRZ // calls the subclass's destructor. virtual ~SpectralBaseAlgorithmRZ() {} + /** + * \brief Default Copy constructor + */ + SpectralBaseAlgorithmRZ ( SpectralBaseAlgorithmRZ const &) = default; + + /** + * \brief Default Copy operator + */ + SpectralBaseAlgorithmRZ& operator= ( SpectralBaseAlgorithmRZ const & ) = default; + + + /** + * \brief Default Move constructor + */ + SpectralBaseAlgorithmRZ ( SpectralBaseAlgorithmRZ&& ) = default; + + /** + * \brief Default Move operator + */ + SpectralBaseAlgorithmRZ& operator= ( SpectralBaseAlgorithmRZ&& ) = default; + /** * \brief Virtual function for current correction in Fourier space * ( Vay et al, 2013). diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldData.H b/Source/FieldSolver/SpectralSolver/SpectralFieldData.H index 8fd23fac747..a4ef6a39189 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldData.H +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldData.H @@ -75,6 +75,26 @@ class SpectralFieldIndex */ ~SpectralFieldIndex () = default; + /** + * \brief Default Copy constructor + */ + SpectralFieldIndex ( SpectralFieldIndex const &) = default; + + /** + * \brief Default Copy operator + */ + SpectralFieldIndex& operator= ( SpectralFieldIndex const & ) = default; + + /** + * \brief Default Move constructor + */ + SpectralFieldIndex ( SpectralFieldIndex&& ) = default; + + /** + * \brief Default Move operator + */ + SpectralFieldIndex& operator= ( SpectralFieldIndex&& ) = default; + // Total number of fields that are actually allocated int n_fields; @@ -129,9 +149,14 @@ class SpectralFieldData int n_field_required, bool periodic_single_box); SpectralFieldData() = default; // Default constructor - SpectralFieldData& operator=(SpectralFieldData&& field_data) = default; ~SpectralFieldData(); + // default move and copy operations + SpectralFieldData(const SpectralFieldData&) = delete; + SpectralFieldData& operator=(const SpectralFieldData&) = delete; + SpectralFieldData(SpectralFieldData&&) = default; + SpectralFieldData& operator=(SpectralFieldData&& field_data) = default; + void ForwardTransform (int lev, const amrex::MultiFab& mf, int field_index, int i_comp); diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp b/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp index 3547a5a55f5..c4d9a0dab35 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp @@ -122,13 +122,12 @@ SpectralFieldData::SpectralFieldData( const int lev, const SpectralKSpace& k_space, const amrex::DistributionMapping& dm, const int n_field_required, - const bool periodic_single_box) + const bool periodic_single_box): + m_periodic_single_box{periodic_single_box} { amrex::LayoutData* cost = WarpX::getCosts(lev); const bool do_costs = WarpXUtilLoadBalance::doCosts(cost, realspace_ba, dm); - m_periodic_single_box = periodic_single_box; - const BoxArray& spectralspace_ba = k_space.spectralspace_ba; // Allocate the arrays that contain the fields in spectral space diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H index b966c7c3c03..a8282fffbed 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H @@ -40,9 +40,18 @@ class SpectralFieldDataRZ int n_field_required, int n_modes); SpectralFieldDataRZ () = default; // Default constructor - SpectralFieldDataRZ& operator=(SpectralFieldDataRZ&& field_data) = default; ~SpectralFieldDataRZ (); + /** Delete Copy constructor */ + SpectralFieldDataRZ ( SpectralFieldDataRZ const &) = delete; + /** Delete Copy operator */ + SpectralFieldDataRZ& operator= ( SpectralFieldDataRZ const & ) = delete; + /** Default Move constructor */ + SpectralFieldDataRZ ( SpectralFieldDataRZ&& ) = default; + /** Default Move operator */ + SpectralFieldDataRZ& operator=(SpectralFieldDataRZ&& field_data) = default; + + void ForwardTransform (int lev, const amrex::MultiFab& mf, int field_index, int i_comp=0); void ForwardTransform (int lev, const amrex::MultiFab& mf_r, int field_index_r, diff --git a/Source/Filter/NCIGodfreyFilter.H b/Source/Filter/NCIGodfreyFilter.H index 18b93189d76..86446495869 100644 --- a/Source/Filter/NCIGodfreyFilter.H +++ b/Source/Filter/NCIGodfreyFilter.H @@ -39,6 +39,8 @@ public: private: +//NCIGodfreyFilter not implemented in 1D +#if (AMREX_SPACEDIM >= 2) // Set of coefficients (different fields require to read // different coefficients from the table) godfrey_coeff_set m_coeff_set; @@ -46,6 +48,8 @@ private: amrex::Real m_cdtodz; // Whether the gather is from nodal fields or staggered fields bool m_nodal_gather; +#endif + }; #endif // #ifndef WARPX_GODFREY_FILTER_H_ diff --git a/Source/Filter/NCIGodfreyFilter.cpp b/Source/Filter/NCIGodfreyFilter.cpp index ff866332521..1cfd42aed03 100644 --- a/Source/Filter/NCIGodfreyFilter.cpp +++ b/Source/Filter/NCIGodfreyFilter.cpp @@ -23,39 +23,36 @@ using namespace amrex; -NCIGodfreyFilter::NCIGodfreyFilter(godfrey_coeff_set coeff_set, amrex::Real cdtodz, bool nodal_gather){ - // Store parameters into class data members - m_coeff_set = coeff_set; - m_cdtodz = cdtodz; - m_nodal_gather = nodal_gather; +//NCIGodfreyFilter not implemented in 1D +#if (AMREX_SPACEDIM >= 2) +NCIGodfreyFilter::NCIGodfreyFilter(godfrey_coeff_set coeff_set, amrex::Real cdtodz, bool nodal_gather): + m_coeff_set{coeff_set}, // Store parameters into class data members + m_cdtodz{cdtodz}, + m_nodal_gather{nodal_gather} +{ // NCI Godfrey filter has fixed size, and is applied along z only. -#if defined(WARPX_DIM_3D) +# if defined(WARPX_DIM_3D) stencil_length_each_dir = {1,1,5}; slen = {1,1,5}; -#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) +# else stencil_length_each_dir = {1,5}; slen = {1,5,1}; -#else - amrex::ignore_unused(coeff_set, cdtodz, nodal_gather); - WARPX_ABORT_WITH_MESSAGE( - "NCIGodfreyFilter not implemented in 1D!"); -#endif +# endif } -void NCIGodfreyFilter::ComputeStencils(){ - -#if (AMREX_SPACEDIM >= 2) +void NCIGodfreyFilter::ComputeStencils() +{ using namespace warpx::nci_godfrey; // Sanity checks: filter length shoulz be 5 in z -#if defined(WARPX_DIM_3D) +# if defined(WARPX_DIM_3D) WARPX_ALWAYS_ASSERT_WITH_MESSAGE( slen.z==5,"ERROR: NCI filter requires 5 points in z"); -#else +# else WARPX_ALWAYS_ASSERT_WITH_MESSAGE( slen.y==5,"ERROR: NCI filter requires 5 points in z"); -#endif +# endif // Interpolate coefficients from the table, and store into prestencil. auto index = static_cast(tab_length*m_cdtodz); index = min(index, tab_length-2); @@ -111,33 +108,46 @@ void NCIGodfreyFilter::ComputeStencils(){ // so only 1 coeff, equal to 1) Vector h_stencil_x(1); h_stencil_x[0] = 1._rt; -#if defined(WARPX_DIM_3D) +# if defined(WARPX_DIM_3D) Vector h_stencil_y(1); h_stencil_y[0] = 1._rt; -#endif +# endif // Due to the way Filter::DoFilter() is written, // coefficient 0 has to be /2 h_stencil_x[0] /= 2._rt; -#if defined(WARPX_DIM_3D) +# if defined(WARPX_DIM_3D) h_stencil_y[0] /= 2._rt; -#endif +# endif h_stencil_z[0] /= 2._rt; stencil_x.resize(h_stencil_x.size()); -#if defined(WARPX_DIM_3D) +# if defined(WARPX_DIM_3D) stencil_y.resize(h_stencil_y.size()); -#endif +# endif stencil_z.resize(h_stencil_z.size()); Gpu::copyAsync(Gpu::hostToDevice,h_stencil_x.begin(),h_stencil_x.end(),stencil_x.begin()); -#if defined(WARPX_DIM_3D) +# if defined(WARPX_DIM_3D) Gpu::copyAsync(Gpu::hostToDevice,h_stencil_y.begin(),h_stencil_y.end(),stencil_y.begin()); -#endif +# endif Gpu::copyAsync(Gpu::hostToDevice,h_stencil_z.begin(),h_stencil_z.end(),stencil_z.begin()); Gpu::synchronize(); +} + #else - WARPX_ABORT_WITH_MESSAGE("NCIGodfreyFilter not implemented in 1D!"); -#endif + +NCIGodfreyFilter::NCIGodfreyFilter(godfrey_coeff_set, amrex::Real, bool) +{ + WARPX_ABORT_WITH_MESSAGE( + "NCIGodfreyFilter not implemented in 1D!"); } + +void NCIGodfreyFilter::ComputeStencils() +{ + WARPX_ABORT_WITH_MESSAGE( + "NCIGodfreyFilter not implemented in 1D!"); +} + +#endif diff --git a/Source/Fluids/MultiFluidContainer.H b/Source/Fluids/MultiFluidContainer.H index ef6cc6115cb..0576a211aa9 100644 --- a/Source/Fluids/MultiFluidContainer.H +++ b/Source/Fluids/MultiFluidContainer.H @@ -38,6 +38,11 @@ public: ~MultiFluidContainer() {} + MultiFluidContainer (MultiFluidContainer const &) = delete; + MultiFluidContainer& operator= (MultiFluidContainer const & ) = delete; + MultiFluidContainer(MultiFluidContainer&& ) = default; + MultiFluidContainer& operator=(MultiFluidContainer&& ) = default; + WarpXFluidContainer& GetFluidContainer (int ispecies) const {return *allcontainers[ispecies];} diff --git a/Source/Fluids/WarpXFluidContainer.H b/Source/Fluids/WarpXFluidContainer.H index 28f37ae393b..a4880f3c57a 100644 --- a/Source/Fluids/WarpXFluidContainer.H +++ b/Source/Fluids/WarpXFluidContainer.H @@ -33,6 +33,11 @@ public: WarpXFluidContainer (int nlevs_max, int ispecies, const std::string& name); ~WarpXFluidContainer() {} + WarpXFluidContainer (WarpXFluidContainer const &) = delete; + WarpXFluidContainer& operator= (WarpXFluidContainer const & ) = delete; + WarpXFluidContainer(WarpXFluidContainer&& ) = default; + WarpXFluidContainer& operator=(WarpXFluidContainer&& ) = default; + void AllocateLevelMFs (int lev, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm); void InitData (int lev, amrex::Box init_box, amrex::Real cur_time); diff --git a/Source/Fluids/WarpXFluidContainer.cpp b/Source/Fluids/WarpXFluidContainer.cpp index 48631035397..13063c2cfc0 100644 --- a/Source/Fluids/WarpXFluidContainer.cpp +++ b/Source/Fluids/WarpXFluidContainer.cpp @@ -19,11 +19,10 @@ using namespace ablastr::utils::communication; using namespace amrex; -WarpXFluidContainer::WarpXFluidContainer(int nlevs_max, int ispecies, const std::string &name) +WarpXFluidContainer::WarpXFluidContainer(int nlevs_max, int ispecies, const std::string &name): + species_id{ispecies}, + species_name{name} { - species_id = ispecies; - species_name = name; - ReadParameters(); // Initialize injection objects diff --git a/Source/Initialization/GetTemperature.cpp b/Source/Initialization/GetTemperature.cpp index 4aa1f7fbd74..2e8670bbe6c 100644 --- a/Source/Initialization/GetTemperature.cpp +++ b/Source/Initialization/GetTemperature.cpp @@ -8,8 +8,9 @@ #include "GetTemperature.H" -GetTemperature::GetTemperature (TemperatureProperties const& temp) noexcept { - m_type = temp.m_type; +GetTemperature::GetTemperature (TemperatureProperties const& temp) noexcept : + m_type{temp.m_type} +{ if (m_type == TempConstantValue) { m_temperature = temp.m_temperature; } diff --git a/Source/Initialization/GetVelocity.cpp b/Source/Initialization/GetVelocity.cpp index 2d2a342a7ed..4dd77a74383 100644 --- a/Source/Initialization/GetVelocity.cpp +++ b/Source/Initialization/GetVelocity.cpp @@ -7,10 +7,9 @@ #include "GetVelocity.H" -GetVelocity::GetVelocity (VelocityProperties const& vel) noexcept { - m_type = vel.m_type; - m_dir = vel.m_dir; - m_sign_dir = vel.m_sign_dir; +GetVelocity::GetVelocity (VelocityProperties const& vel) noexcept: + m_type{vel.m_type}, m_dir{vel.m_dir}, m_sign_dir{vel.m_sign_dir} +{ if (m_type == VelConstantValue) { m_velocity = vel.m_velocity; } diff --git a/Source/Initialization/InjectorDensity.H b/Source/Initialization/InjectorDensity.H index df85e0625f7..681bb334015 100644 --- a/Source/Initialization/InjectorDensity.H +++ b/Source/Initialization/InjectorDensity.H @@ -150,6 +150,9 @@ struct InjectorDensity void operator= (InjectorDensity const&) = delete; void operator= (InjectorDensity &&) = delete; + // Default destructor + ~InjectorDensity () = default; + void clear (); // call getDensity from the object stored in the union diff --git a/Source/Initialization/InjectorFlux.H b/Source/Initialization/InjectorFlux.H index adfe75b3ed2..74ae840a502 100644 --- a/Source/Initialization/InjectorFlux.H +++ b/Source/Initialization/InjectorFlux.H @@ -79,6 +79,9 @@ struct InjectorFlux void operator= (InjectorFlux const&) = delete; void operator= (InjectorFlux &&) = delete; + // Default destructor + ~InjectorFlux () = default; + void clear () { switch (type) diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index b597e361dfa..b1a3ac7b0c8 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -229,17 +229,14 @@ struct InjectorMomentumUniform amrex::Real a_uz_min, amrex::Real a_ux_max, amrex::Real a_uy_max, amrex::Real a_uz_max) noexcept : m_ux_min(a_ux_min), m_uy_min(a_uy_min), m_uz_min(a_uz_min), - m_ux_max(a_ux_max), m_uy_max(a_uy_max), m_uz_max(a_uz_max) - { - using namespace amrex; - - m_Dux = m_ux_max - m_ux_min; - m_Duy = m_uy_max - m_uy_min; - m_Duz = m_uz_max - m_uz_min; - m_ux_h = 0.5_rt * (m_ux_max + m_ux_min); - m_uy_h = 0.5_rt * (m_uy_max + m_uy_min); - m_uz_h = 0.5_rt * (m_uz_max + m_uz_min); - } + m_ux_max(a_ux_max), m_uy_max(a_uy_max), m_uz_max(a_uz_max), + m_Dux(m_ux_max - m_ux_min), + m_Duy(m_uy_max - m_uy_min), + m_Duz(m_uz_max - m_uz_min), + m_ux_h(amrex::Real(0.5) * (m_ux_max + m_ux_min)), + m_uy_h(amrex::Real(0.5) * (m_uy_max + m_uy_min)), + m_uz_h(amrex::Real(0.5) * (m_uz_max + m_uz_min)) + {} AMREX_GPU_HOST_DEVICE amrex::XDim3 @@ -261,8 +258,8 @@ struct InjectorMomentumUniform private: amrex::Real m_ux_min, m_uy_min, m_uz_min; amrex::Real m_ux_max, m_uy_max, m_uz_max; - amrex::Real m_ux_h, m_uy_h, m_uz_h; amrex::Real m_Dux, m_Duy, m_Duz; + amrex::Real m_ux_h, m_uy_h, m_uz_h; }; // struct whose getMomentum returns momentum for 1 particle with relativistic @@ -587,6 +584,9 @@ struct InjectorMomentum void operator= (InjectorMomentum const&) = delete; void operator= (InjectorMomentum &&) = delete; + // Default destructor + ~InjectorMomentum() = default; + void clear (); // call getMomentum from the object stored in the union diff --git a/Source/Initialization/InjectorPosition.H b/Source/Initialization/InjectorPosition.H index 17ce8325543..2d099c56319 100644 --- a/Source/Initialization/InjectorPosition.H +++ b/Source/Initialization/InjectorPosition.H @@ -154,12 +154,14 @@ struct InjectorPosition zmin(a_zmin), zmax(a_zmax) { } + ~InjectorPosition () = default; + // Explicitly prevent the compiler from generating copy constructors // and copy assignment operators. InjectorPosition (InjectorPosition const&) = delete; - InjectorPosition (InjectorPosition&&) = delete; - void operator= (InjectorPosition const&) = delete; - void operator= (InjectorPosition &&) = delete; + InjectorPosition (InjectorPosition&&) = delete; + void operator= (InjectorPosition const&) = delete; + void operator= (InjectorPosition &&) = delete; // call getPositionUnitBox from the object stored in the union // (the union is called Object, and the instance is called object). diff --git a/Source/Initialization/PlasmaInjector.H b/Source/Initialization/PlasmaInjector.H index ac64980b952..0b2fac02f51 100644 --- a/Source/Initialization/PlasmaInjector.H +++ b/Source/Initialization/PlasmaInjector.H @@ -42,10 +42,17 @@ class PlasmaInjector public: + /** Default constructor*/ PlasmaInjector () = default; PlasmaInjector (int ispecies, const std::string& name, const amrex::Geometry& geom); + // Default move and copy operations + PlasmaInjector(const PlasmaInjector&) = delete; + PlasmaInjector& operator=(const PlasmaInjector&) = delete; + PlasmaInjector(PlasmaInjector&&) = default; + PlasmaInjector& operator=(PlasmaInjector&&) = default; + ~PlasmaInjector (); // bool: whether the point (x, y, z) is inside the plasma region diff --git a/Source/Initialization/VelocityProperties.cpp b/Source/Initialization/VelocityProperties.cpp index 802eabc40c9..30cb64cb70a 100644 --- a/Source/Initialization/VelocityProperties.cpp +++ b/Source/Initialization/VelocityProperties.cpp @@ -11,11 +11,12 @@ #include "Utils/Parser/ParserUtils.H" #include "Utils/TextMsg.H" -VelocityProperties::VelocityProperties (const amrex::ParmParse& pp) { +VelocityProperties::VelocityProperties (const amrex::ParmParse& pp): + m_velocity{0} +{ // Set defaults std::string vel_dist_s = "constant"; std::string vel_dir_s = "x"; - m_velocity = 0; pp.query("bulk_vel_dir", vel_dir_s); if(vel_dir_s[0] == '-'){ diff --git a/Source/Laser/LaserProfiles.H b/Source/Laser/LaserProfiles.H index 487e2d9e527..73776fa3525 100644 --- a/Source/Laser/LaserProfiles.H +++ b/Source/Laser/LaserProfiles.H @@ -97,7 +97,13 @@ public: amrex::Real t, amrex::Real* AMREX_RESTRICT amplitude) const = 0; + ILaserProfile () = default; virtual ~ILaserProfile(){} + + ILaserProfile ( ILaserProfile const &) = default; + ILaserProfile& operator= ( ILaserProfile const & ) = default; + ILaserProfile ( ILaserProfile&& ) = default; + ILaserProfile& operator= ( ILaserProfile&& ) = default; }; /** @@ -110,11 +116,11 @@ public: void init ( const amrex::ParmParse& ppl, - CommonLaserParameters params) override final; + CommonLaserParameters params) final; //No update needed void - update (amrex::Real /*t */) override final {} + update (amrex::Real /*t */) final {} void fill_amplitude ( @@ -122,7 +128,7 @@ public: amrex::Real const * AMREX_RESTRICT Xp, amrex::Real const * AMREX_RESTRICT Yp, amrex::Real t, - amrex::Real * AMREX_RESTRICT amplitude) const override final; + amrex::Real * AMREX_RESTRICT amplitude) const final; private: struct { @@ -152,11 +158,11 @@ public: void init ( const amrex::ParmParse& ppl, - CommonLaserParameters params) override final; + CommonLaserParameters params) final; //No update needed void - update (amrex::Real /*t */) override final {} + update (amrex::Real /*t */) final {} void fill_amplitude ( @@ -164,7 +170,7 @@ public: amrex::Real const * AMREX_RESTRICT Xp, amrex::Real const * AMREX_RESTRICT Yp, amrex::Real t, - amrex::Real * AMREX_RESTRICT amplitude) const override final; + amrex::Real * AMREX_RESTRICT amplitude) const final; private: struct{ @@ -187,14 +193,14 @@ public: void init ( const amrex::ParmParse& ppl, - CommonLaserParameters params) override final; + CommonLaserParameters params) final; /** \brief Reads new field data chunk from file if needed * * @param[in] t simulation time (seconds) */ void - update (amrex::Real t) override final; + update (amrex::Real t) final; /** \brief compute field amplitude at particles' position for a laser beam * loaded from an E(x,y,t) file. @@ -215,7 +221,7 @@ public: amrex::Real const * AMREX_RESTRICT Xp, amrex::Real const * AMREX_RESTRICT Yp, amrex::Real t, - amrex::Real * AMREX_RESTRICT amplitude) const override final; + amrex::Real * AMREX_RESTRICT amplitude) const final; /** \brief Function to fill the amplitude in case of a uniform grid and for the lasy format in 3D Cartesian. * This function cannot be private due to restrictions related to diff --git a/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.H b/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.H index d47234d4571..be92e42d5fc 100644 --- a/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.H +++ b/Source/Particles/Collision/BackgroundMCC/BackgroundMCCCollision.H @@ -25,7 +25,12 @@ class BackgroundMCCCollision final public: BackgroundMCCCollision (std::string collision_name); - virtual ~BackgroundMCCCollision () = default; + ~BackgroundMCCCollision () override = default; + + BackgroundMCCCollision ( BackgroundMCCCollision const &) = delete; + BackgroundMCCCollision& operator= ( BackgroundMCCCollision const & ) = delete; + BackgroundMCCCollision ( BackgroundMCCCollision&& ) = delete; + BackgroundMCCCollision& operator= ( BackgroundMCCCollision&& ) = delete; amrex::ParticleReal get_nu_max (amrex::Vector const& mcc_processes); diff --git a/Source/Particles/Collision/BackgroundMCC/MCCProcess.H b/Source/Particles/Collision/BackgroundMCC/MCCProcess.H index 31f790153bd..58fc86b2c29 100644 --- a/Source/Particles/Collision/BackgroundMCC/MCCProcess.H +++ b/Source/Particles/Collision/BackgroundMCC/MCCProcess.H @@ -38,11 +38,12 @@ public: amrex::ParticleReal energy ); - MCCProcess (MCCProcess const&) = delete; - MCCProcess& operator= (MCCProcess const&) = delete; + ~MCCProcess() = default; - MCCProcess (MCCProcess &&) = default; - MCCProcess& operator= (MCCProcess &&) = default; + MCCProcess (MCCProcess const&) = delete; + MCCProcess& operator= (MCCProcess const&) = delete; + MCCProcess (MCCProcess &&) = default; + MCCProcess& operator= (MCCProcess &&) = default; /** Read the given cross-section data file to memory. * diff --git a/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.H b/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.H index ee4b3b01653..55fa4b9e1e3 100644 --- a/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.H +++ b/Source/Particles/Collision/BackgroundStopping/BackgroundStopping.H @@ -26,7 +26,12 @@ class BackgroundStopping final public: BackgroundStopping (std::string collision_name); - virtual ~BackgroundStopping () = default; + ~BackgroundStopping () override = default; + + BackgroundStopping ( BackgroundStopping const &) = delete; + BackgroundStopping& operator= ( BackgroundStopping const & ) = delete; + BackgroundStopping ( BackgroundStopping&& ) = delete; + BackgroundStopping& operator= ( BackgroundStopping&& ) = delete; /** Perform the stopping calculation * diff --git a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H index 2f9ddbbcb3f..58fe2c9911a 100644 --- a/Source/Particles/Collision/BinaryCollision/BinaryCollision.H +++ b/Source/Particles/Collision/BinaryCollision/BinaryCollision.H @@ -104,7 +104,12 @@ public: m_copy_transform_functor = CopyTransformFunctorType(collision_name, mypc); } - virtual ~BinaryCollision () = default; + ~BinaryCollision () override = default; + + BinaryCollision ( BinaryCollision const &) = default; + BinaryCollision& operator= ( BinaryCollision const & ) = default; + BinaryCollision ( BinaryCollision&& ) = default; + BinaryCollision& operator= ( BinaryCollision&& ) = default; /** Perform the collisions * diff --git a/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H b/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H index d8fdaee8bd0..feb7acf81d3 100644 --- a/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H +++ b/Source/Particles/Collision/BinaryCollision/Coulomb/PairWiseCoulombCollisionFunc.H @@ -45,15 +45,16 @@ public: */ PairWiseCoulombCollisionFunc (const std::string collision_name, [[maybe_unused]] MultiParticleContainer const * const mypc, - const bool isSameSpecies) + const bool isSameSpecies): + m_isSameSpecies{isSameSpecies} { using namespace amrex::literals; const amrex::ParmParse pp_collision_name(collision_name); // default Coulomb log, if < 0, will be computed automatically - m_CoulombLog = -1.0_prt; + amrex::ParticleReal CoulombLog = -1.0_prt; utils::parser::queryWithParser( - pp_collision_name, "CoulombLog", m_CoulombLog); - m_isSameSpecies = isSameSpecies; + pp_collision_name, "CoulombLog", CoulombLog); + m_CoulombLog = CoulombLog; } /** diff --git a/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H b/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H index 83952357ec5..397536b67bf 100644 --- a/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H +++ b/Source/Particles/Collision/BinaryCollision/NuclearFusion/NuclearFusionFunc.H @@ -54,9 +54,12 @@ public: * @param[in] isSameSpecies whether the two colliding species are the same */ NuclearFusionFunc (const std::string collision_name, MultiParticleContainer const * const mypc, - const bool isSameSpecies) : m_isSameSpecies(isSameSpecies) + const bool isSameSpecies): + m_fusion_multiplier{amrex::ParticleReal{1.0}}, // default fusion multiplier + m_probability_threshold{amrex::ParticleReal{0.02}}, // default fusion probability threshold + m_probability_target_value{amrex::ParticleReal{0.002}}, // default fusion probability target_value + m_isSameSpecies{isSameSpecies} { - using namespace amrex::literals; #ifdef AMREX_SINGLE_PRECISION_PARTICLES WARPX_ABORT_WITH_MESSAGE("Nuclear fusion module does not currently work with single precision"); @@ -65,16 +68,10 @@ public: m_fusion_type = BinaryCollisionUtils::get_nuclear_fusion_type(collision_name, mypc); const amrex::ParmParse pp_collision_name(collision_name); - // default fusion multiplier - m_fusion_multiplier = 1.0_prt; utils::parser::queryWithParser( pp_collision_name, "fusion_multiplier", m_fusion_multiplier); - // default fusion probability threshold - m_probability_threshold = 0.02_prt; utils::parser::queryWithParser( pp_collision_name, "fusion_probability_threshold", m_probability_threshold); - // default fusion probability target_value - m_probability_target_value = 0.002_prt; utils::parser::queryWithParser( pp_collision_name, "fusion_probability_target_value", m_probability_target_value); diff --git a/Source/Particles/Collision/CollisionBase.H b/Source/Particles/Collision/CollisionBase.H index 517f5e414c0..272cc056b1a 100644 --- a/Source/Particles/Collision/CollisionBase.H +++ b/Source/Particles/Collision/CollisionBase.H @@ -25,6 +25,7 @@ public: CollisionBase(CollisionBase const &) = delete; CollisionBase(CollisionBase &&) = delete; CollisionBase & operator=(CollisionBase const &) = delete; + CollisionBase & operator=(CollisionBase const &&) = delete; virtual ~CollisionBase() = default; diff --git a/Source/Particles/Collision/CollisionBase.cpp b/Source/Particles/Collision/CollisionBase.cpp index 701d5ff6bc6..8f799c83013 100644 --- a/Source/Particles/Collision/CollisionBase.cpp +++ b/Source/Particles/Collision/CollisionBase.cpp @@ -18,8 +18,8 @@ CollisionBase::CollisionBase (std::string collision_name) pp_collision_name.getarr("species", m_species_names); // number of time steps between collisions - m_ndt = 1; + int ndt = 1; utils::parser::queryWithParser( - pp_collision_name, "ndt", m_ndt); - + pp_collision_name, "ndt", ndt); + m_ndt = ndt; } diff --git a/Source/Particles/ElementaryProcess/Ionization.cpp b/Source/Particles/ElementaryProcess/Ionization.cpp index 8fcb953e9ff..98b49888d9d 100644 --- a/Source/Particles/ElementaryProcess/Ionization.cpp +++ b/Source/Particles/ElementaryProcess/Ionization.cpp @@ -30,18 +30,19 @@ IonizationFilterFunc::IonizationFilterFunc (const WarpXParIter& a_pti, int lev, const amrex::Real* const AMREX_RESTRICT a_adk_power, int a_comp, int a_atomic_number, - int a_offset) noexcept + int a_offset) noexcept: + m_ionization_energies{a_ionization_energies}, + m_adk_prefactor{a_adk_prefactor}, + m_adk_exp_prefactor{a_adk_exp_prefactor}, + m_adk_power{a_adk_power}, + comp{a_comp}, + m_atomic_number{a_atomic_number}, + m_galerkin_interpolation{WarpX::galerkin_interpolation}, + m_nox{WarpX::nox}, + m_n_rz_azimuthal_modes{WarpX::n_rz_azimuthal_modes} { - using namespace amrex::literals; - m_ionization_energies = a_ionization_energies; - m_adk_prefactor = a_adk_prefactor; - m_adk_exp_prefactor = a_adk_exp_prefactor; - m_adk_power = a_adk_power; - comp = a_comp; - m_atomic_number = a_atomic_number; - m_get_position = GetParticlePosition(a_pti, a_offset); m_get_externalEB = GetExternalEBField(a_pti, a_offset); @@ -69,9 +70,5 @@ IonizationFilterFunc::IonizationFilterFunc (const WarpXParIter& a_pti, int lev, const std::array& xyzmin = WarpX::LowerCorner(box, lev, 0._rt); m_xyzmin_arr = {xyzmin[0], xyzmin[1], xyzmin[2]}; - m_galerkin_interpolation = WarpX::galerkin_interpolation; - m_nox = WarpX::nox; - m_n_rz_azimuthal_modes = WarpX::n_rz_azimuthal_modes; - m_lo = amrex::lbound(box); } diff --git a/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp b/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp index 31aafbb1379..de027152be7 100644 --- a/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp +++ b/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp @@ -26,8 +26,11 @@ PairGenerationTransformFunc (BreitWheelerGeneratePairs const generate_functor, amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, - int a_offset) -: m_generate_functor(generate_functor) + int a_offset): + m_generate_functor{generate_functor}, + m_galerkin_interpolation{WarpX::galerkin_interpolation}, + m_nox{WarpX::nox}, + m_n_rz_azimuthal_modes{WarpX::n_rz_azimuthal_modes} { using namespace amrex::literals; @@ -59,9 +62,5 @@ PairGenerationTransformFunc (BreitWheelerGeneratePairs const generate_functor, const std::array& xyzmin = WarpX::LowerCorner(box, lev, 0._rt); m_xyzmin_arr = {xyzmin[0], xyzmin[1], xyzmin[2]}; - m_galerkin_interpolation = WarpX::galerkin_interpolation; - m_nox = WarpX::nox; - m_n_rz_azimuthal_modes = WarpX::n_rz_azimuthal_modes; - m_lo = amrex::lbound(box); } diff --git a/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp b/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp index a3b3c4103f7..4725f3748e2 100644 --- a/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp +++ b/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp @@ -27,10 +27,13 @@ PhotonEmissionTransformFunc (QuantumSynchrotronGetOpticalDepth opt_depth_functor amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, - int a_offset) -:m_opt_depth_functor{opt_depth_functor}, - m_opt_depth_runtime_comp{opt_depth_runtime_comp}, - m_emission_functor{emission_functor} + int a_offset): + m_opt_depth_functor{opt_depth_functor}, + m_opt_depth_runtime_comp{opt_depth_runtime_comp}, + m_emission_functor{emission_functor}, + m_galerkin_interpolation{WarpX::galerkin_interpolation}, + m_nox{WarpX::nox}, + m_n_rz_azimuthal_modes{WarpX::n_rz_azimuthal_modes} { using namespace amrex::literals; @@ -62,9 +65,7 @@ PhotonEmissionTransformFunc (QuantumSynchrotronGetOpticalDepth opt_depth_functor const std::array& xyzmin = WarpX::LowerCorner(box, lev, 0._rt); m_xyzmin_arr = {xyzmin[0], xyzmin[1], xyzmin[2]}; - m_galerkin_interpolation = WarpX::galerkin_interpolation; - m_nox = WarpX::nox; - m_n_rz_azimuthal_modes = WarpX::n_rz_azimuthal_modes; + m_lo = amrex::lbound(box); } diff --git a/Source/Particles/LaserParticleContainer.H b/Source/Particles/LaserParticleContainer.H index e125bab2ec5..464b411ac66 100644 --- a/Source/Particles/LaserParticleContainer.H +++ b/Source/Particles/LaserParticleContainer.H @@ -42,25 +42,31 @@ class LaserParticleContainer { public: LaserParticleContainer (amrex::AmrCore* amr_core, int ispecies, const std::string& name); - virtual ~LaserParticleContainer () {} + ~LaserParticleContainer () override = default; - virtual void InitData () final; + + LaserParticleContainer ( LaserParticleContainer const &) = delete; + LaserParticleContainer& operator= ( LaserParticleContainer const & ) = delete; + LaserParticleContainer ( LaserParticleContainer&& ) = default; + LaserParticleContainer& operator= ( LaserParticleContainer&& ) = default; + + void InitData () final; /** * \brief Method to initialize runtime attributes. Does nothing for LaserParticleContainer. */ - virtual void DefaultInitializeRuntimeAttributes ( + void DefaultInitializeRuntimeAttributes ( amrex::ParticleTile, NArrayReal, NArrayInt, amrex::PinnedArenaAllocator>& /*pinned_tile*/, const int /*n_external_attr_real*/, const int /*n_external_attr_int*/, - const amrex::RandomEngine& /*engine*/) override final {} + const amrex::RandomEngine& /*engine*/) final {} - virtual void ReadHeader (std::istream& is) final; + void ReadHeader (std::istream& is) final; - virtual void WriteHeader (std::ostream& os) const final; + void WriteHeader (std::ostream& os) const final; - virtual void Evolve (int lev, + void Evolve (int lev, const amrex::MultiFab&, const amrex::MultiFab&, const amrex::MultiFab&, const amrex::MultiFab&, const amrex::MultiFab&, const amrex::MultiFab&, amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz, @@ -71,7 +77,7 @@ public: amrex::Real t, amrex::Real dt, DtType a_dt_type=DtType::Full, bool skip_deposition=false) final; - virtual void PushP (int lev, amrex::Real dt, + void PushP (int lev, amrex::Real dt, const amrex::MultiFab& , const amrex::MultiFab& , const amrex::MultiFab& , @@ -79,7 +85,7 @@ public: const amrex::MultiFab& , const amrex::MultiFab& ) final; - virtual void PostRestart () final; + void PostRestart () final; void calculate_laser_plane_coordinates (const WarpXParIter& pti, int np, amrex::Real * AMREX_RESTRICT pplane_Xp, diff --git a/Source/Particles/MultiParticleContainer.H b/Source/Particles/MultiParticleContainer.H index 16817cbec07..64ce057e2ff 100644 --- a/Source/Particles/MultiParticleContainer.H +++ b/Source/Particles/MultiParticleContainer.H @@ -70,6 +70,11 @@ public: ~MultiParticleContainer() {} + MultiParticleContainer (MultiParticleContainer const &) = delete; + MultiParticleContainer& operator= (MultiParticleContainer const & ) = delete; + MultiParticleContainer(MultiParticleContainer&& ) = default; + MultiParticleContainer& operator=(MultiParticleContainer&& ) = default; + WarpXParticleContainer& GetParticleContainer (int index) const {return *allcontainers[index];} diff --git a/Source/Particles/NamedComponentParticleContainer.H b/Source/Particles/NamedComponentParticleContainer.H index e74dd8a6c7e..6ff9d157790 100644 --- a/Source/Particles/NamedComponentParticleContainer.H +++ b/Source/Particles/NamedComponentParticleContainer.H @@ -73,7 +73,7 @@ public: } /** Destructor for NamedComponentParticleContainer */ - virtual ~NamedComponentParticleContainer() = default; + ~NamedComponentParticleContainer() override = default; /** Construct a NamedComponentParticleContainer from a regular * amrex::ParticleContainer, and additional name-to-index maps diff --git a/Source/Particles/ParticleCreation/SmartCopy.H b/Source/Particles/ParticleCreation/SmartCopy.H index 4b29d90e065..94d10dfce6b 100644 --- a/Source/Particles/ParticleCreation/SmartCopy.H +++ b/Source/Particles/ParticleCreation/SmartCopy.H @@ -134,21 +134,17 @@ class SmartCopyFactory SmartCopyTag m_tag_int; PolicyVec m_policy_real; PolicyVec m_policy_int; - bool m_defined; + bool m_defined = false; public: template - SmartCopyFactory (const SrcPC& src, const DstPC& dst) noexcept - : m_defined(false) - { - m_tag_real = getSmartCopyTag(src.getParticleComps(), dst.getParticleComps()); - m_tag_int = getSmartCopyTag(src.getParticleiComps(), dst.getParticleiComps()); - - m_policy_real = getPolicies(dst.getParticleComps()); - m_policy_int = getPolicies(dst.getParticleiComps()); - - m_defined = true; - } + SmartCopyFactory (const SrcPC& src, const DstPC& dst) noexcept : + m_tag_real{getSmartCopyTag(src.getParticleComps(), dst.getParticleComps())}, + m_tag_int{getSmartCopyTag(src.getParticleiComps(), dst.getParticleiComps())}, + m_policy_real{getPolicies(dst.getParticleComps())}, + m_policy_int{getPolicies(dst.getParticleiComps())}, + m_defined{true} + {} SmartCopy getSmartCopy () const noexcept { diff --git a/Source/Particles/ParticleCreation/SmartCreate.H b/Source/Particles/ParticleCreation/SmartCreate.H index 875e69cab04..e91d39060b0 100644 --- a/Source/Particles/ParticleCreation/SmartCreate.H +++ b/Source/Particles/ParticleCreation/SmartCreate.H @@ -91,13 +91,11 @@ class SmartCreateFactory public: template - SmartCreateFactory (const PartTileData& part) noexcept - : m_defined(false) - { - m_policy_real = getPolicies(part.getParticleComps()); - m_policy_int = getPolicies(part.getParticleiComps()); - m_defined = true; - } + SmartCreateFactory (const PartTileData& part) noexcept: + m_policy_real{getPolicies(part.getParticleComps())}, + m_policy_int{getPolicies(part.getParticleiComps())}, + m_defined{true} + {} SmartCreate getSmartCreate () const noexcept { diff --git a/Source/Particles/PhotonParticleContainer.H b/Source/Particles/PhotonParticleContainer.H index 0f7b6e0e185..ada11857b34 100644 --- a/Source/Particles/PhotonParticleContainer.H +++ b/Source/Particles/PhotonParticleContainer.H @@ -36,11 +36,16 @@ public: PhotonParticleContainer (amrex::AmrCore* amr_core, int ispecies, const std::string& name); - virtual ~PhotonParticleContainer () {} + ~PhotonParticleContainer () override {} - virtual void InitData() override; + PhotonParticleContainer ( PhotonParticleContainer const &) = delete; + PhotonParticleContainer& operator= ( PhotonParticleContainer const & ) = delete; + PhotonParticleContainer ( PhotonParticleContainer&& ) = default; + PhotonParticleContainer& operator= ( PhotonParticleContainer&& ) = default; - virtual void Evolve (int lev, + void InitData() override; + + void Evolve (int lev, const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, @@ -66,7 +71,7 @@ public: DtType a_dt_type=DtType::Full, bool skip_deposition=false) override; - virtual void PushPX(WarpXParIter& pti, + void PushPX(WarpXParIter& pti, amrex::FArrayBox const * exfab, amrex::FArrayBox const * eyfab, amrex::FArrayBox const * ezfab, @@ -81,7 +86,7 @@ public: DtType a_dt_type) override; // Do nothing - virtual void PushP (int /*lev*/, + void PushP (int /*lev*/, amrex::Real /*dt*/, const amrex::MultiFab& /*Ex*/, const amrex::MultiFab& /*Ey*/, @@ -92,7 +97,7 @@ public: // DepositCharge should do nothing for photons - virtual void DepositCharge (WarpXParIter& /*pti*/, + void DepositCharge (WarpXParIter& /*pti*/, RealVector const & /*wp*/, const int * const /*ion_lev*/, amrex::MultiFab* /*rho*/, @@ -104,7 +109,7 @@ public: int /*depos_lev*/) override {} // DepositCurrent should do nothing for photons - virtual void DepositCurrent (WarpXParIter& /*pti*/, + void DepositCurrent (WarpXParIter& /*pti*/, RealVector const & /*wp*/, RealVector const & /*uxp*/, RealVector const & /*uyp*/, diff --git a/Source/Particles/PhysicalParticleContainer.H b/Source/Particles/PhysicalParticleContainer.H index 3e3ae069214..526e5e01b2d 100644 --- a/Source/Particles/PhysicalParticleContainer.H +++ b/Source/Particles/PhysicalParticleContainer.H @@ -56,20 +56,25 @@ public: * the run if one of them is specified. */ void BackwardCompatibility (); - virtual ~PhysicalParticleContainer () {} + ~PhysicalParticleContainer () override {} - virtual void InitData () override; + PhysicalParticleContainer (PhysicalParticleContainer const &) = delete; + PhysicalParticleContainer& operator= (PhysicalParticleContainer const & ) = delete; + PhysicalParticleContainer(PhysicalParticleContainer&& ) = default; + PhysicalParticleContainer& operator=(PhysicalParticleContainer&& ) = default; - virtual void ReadHeader (std::istream& is) override; + void InitData () override; - virtual void WriteHeader (std::ostream& os) const override; + void ReadHeader (std::istream& is) override; - virtual void InitIonizationModule () override; + void WriteHeader (std::ostream& os) const override; + + void InitIonizationModule () override; /* * \brief Returns a pointer to the plasma injector. */ - virtual PlasmaInjector* GetPlasmaInjector () override; + PlasmaInjector* GetPlasmaInjector () override; /** * \brief Evolve is the central function PhysicalParticleContainer that @@ -105,7 +110,7 @@ public: * field gather, particle push and current deposition for all particles * in the box. */ - virtual void Evolve (int lev, + void Evolve (int lev, const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, @@ -145,7 +150,7 @@ public: amrex::Real dt, ScaleFields scaleFields, DtType a_dt_type=DtType::Full); - virtual void PushP (int lev, amrex::Real dt, + void PushP (int lev, amrex::Real dt, const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, @@ -162,7 +167,7 @@ public: amrex::iMultiFab const* current_masks, amrex::iMultiFab const* gather_masks ); - virtual void PostRestart () final {} + void PostRestart () final {} void SplitParticles (int lev); @@ -240,13 +245,13 @@ public: * These are NOT initialized by this function. * @param[in] engine the random engine, used in initialization of QED optical depths */ - virtual void DefaultInitializeRuntimeAttributes ( + void DefaultInitializeRuntimeAttributes ( amrex::ParticleTile, NArrayReal, NArrayInt, amrex::PinnedArenaAllocator>& pinned_tile, int n_external_attr_real, int n_external_attr_int, - const amrex::RandomEngine& engine) override final; + const amrex::RandomEngine& engine) final; /** * \brief Apply NCI Godfrey filter to all components of E and B before gather @@ -307,7 +312,7 @@ public: * * @param[in] timestep the current timestep. */ - void resample (int timestep, bool verbose=true) override final; + void resample (int timestep, bool verbose) final; #ifdef WARPX_QED //Functions decleared in WarpXParticleContainer.H diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp index 632e3fca85b..7163f506ff8 100644 --- a/Source/Particles/PhysicalParticleContainer.cpp +++ b/Source/Particles/PhysicalParticleContainer.cpp @@ -144,12 +144,21 @@ namespace ParticleReal x, y, z; AMREX_GPU_HOST_DEVICE - PDim3(const PDim3&) = default; + PDim3(const amrex::XDim3& a): + x{a.x}, y{a.y}, z{a.z} + {} + AMREX_GPU_HOST_DEVICE - PDim3(const amrex::XDim3& a) : x(a.x), y(a.y), z(a.z) {} + ~PDim3() = default; AMREX_GPU_HOST_DEVICE - PDim3& operator=(const PDim3&) = default; + PDim3(PDim3 const &) = default; + AMREX_GPU_HOST_DEVICE + PDim3& operator=(PDim3 const &) = default; + AMREX_GPU_HOST_DEVICE + PDim3(PDim3&&) = default; + AMREX_GPU_HOST_DEVICE + PDim3& operator=(PDim3&&) = default; }; AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE diff --git a/Source/Particles/Resampling/LevelingThinning.H b/Source/Particles/Resampling/LevelingThinning.H index d7db1e7021c..fa05525e270 100644 --- a/Source/Particles/Resampling/LevelingThinning.H +++ b/Source/Particles/Resampling/LevelingThinning.H @@ -45,7 +45,7 @@ public: * @param[in] lev the index of the refinement level. * @param[in] pc a pointer to the particle container. */ - void operator() (WarpXParIter& pti, int lev, WarpXParticleContainer* pc) const override final; + void operator() (WarpXParIter& pti, int lev, WarpXParticleContainer* pc) const final; private: amrex::Real m_target_ratio = amrex::Real(1.5); diff --git a/Source/Particles/Resampling/Resampling.H b/Source/Particles/Resampling/Resampling.H index 820b14f5dc5..35439e905df 100644 --- a/Source/Particles/Resampling/Resampling.H +++ b/Source/Particles/Resampling/Resampling.H @@ -30,6 +30,15 @@ struct ResamplingAlgorithm * \brief Virtual destructor of the abstract ResamplingAlgorithm class */ virtual ~ResamplingAlgorithm () = default; + + /** Default constructor */ + ResamplingAlgorithm () = default; + + + ResamplingAlgorithm ( ResamplingAlgorithm const &) = default; + ResamplingAlgorithm& operator= ( ResamplingAlgorithm const & ) = default; + ResamplingAlgorithm ( ResamplingAlgorithm&& ) = default; + ResamplingAlgorithm& operator= ( ResamplingAlgorithm&& ) = default; }; /** diff --git a/Source/Particles/RigidInjectedParticleContainer.H b/Source/Particles/RigidInjectedParticleContainer.H index 0f6f20f5f71..29bf294f939 100644 --- a/Source/Particles/RigidInjectedParticleContainer.H +++ b/Source/Particles/RigidInjectedParticleContainer.H @@ -48,13 +48,20 @@ public: RigidInjectedParticleContainer (amrex::AmrCore* amr_core, int ispecies, const std::string& name); - virtual ~RigidInjectedParticleContainer () {} + ~RigidInjectedParticleContainer () override = default; - virtual void InitData() override; + + RigidInjectedParticleContainer ( RigidInjectedParticleContainer const &) = delete; + RigidInjectedParticleContainer& operator= ( RigidInjectedParticleContainer const & ) = delete; + RigidInjectedParticleContainer ( RigidInjectedParticleContainer&& ) = default; + RigidInjectedParticleContainer& operator= ( RigidInjectedParticleContainer&& ) = default; + + + void InitData() override; virtual void RemapParticles(); - virtual void Evolve (int lev, + void Evolve (int lev, const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, @@ -80,7 +87,7 @@ public: DtType a_dt_type=DtType::Full, bool skip_deposition=false ) override; - virtual void PushPX (WarpXParIter& pti, + void PushPX (WarpXParIter& pti, amrex::FArrayBox const * exfab, amrex::FArrayBox const * eyfab, amrex::FArrayBox const * ezfab, @@ -94,7 +101,7 @@ public: amrex::Real dt, ScaleFields scaleFields, DtType a_dt_type=DtType::Full) override; - virtual void PushP (int lev, amrex::Real dt, + void PushP (int lev, amrex::Real dt, const amrex::MultiFab& Ex, const amrex::MultiFab& Ey, const amrex::MultiFab& Ez, @@ -102,9 +109,9 @@ public: const amrex::MultiFab& By, const amrex::MultiFab& Bz) override; - virtual void ReadHeader (std::istream& is) override; + void ReadHeader (std::istream& is) override; - virtual void WriteHeader (std::ostream& os) const override; + void WriteHeader (std::ostream& os) const override; private: diff --git a/Source/Particles/Sorting/SortingUtils.H b/Source/Particles/Sorting/SortingUtils.H index dd53ad6f610..ac2c63e88f8 100644 --- a/Source/Particles/Sorting/SortingUtils.H +++ b/Source/Particles/Sorting/SortingUtils.H @@ -84,13 +84,13 @@ class fillBufferFlag public: fillBufferFlag( WarpXParIter const& pti, amrex::iMultiFab const* bmasks, amrex::Gpu::DeviceVector& inexflag, - amrex::Geometry const& geom ) { - + amrex::Geometry const& geom ): // Extract simple structure that can be used directly on the GPU - m_particles = pti.GetArrayOfStructs().data(); - m_buffer_mask = (*bmasks)[pti].array(); - m_inexflag_ptr = inexflag.dataPtr(); - m_domain = geom.Domain(); + m_domain{geom.Domain()}, + m_inexflag_ptr{inexflag.dataPtr()}, + m_particles{pti.GetArrayOfStructs().data()}, + m_buffer_mask{(*bmasks)[pti].array()} + { for (int idim=0; idim m_prob_lo; - amrex::GpuArray m_inv_cell_size; amrex::Box m_domain; int* m_inexflag_ptr; WarpXParticleContainer::ParticleType const* m_particles; amrex::Array4 m_buffer_mask; + amrex::GpuArray m_prob_lo; + amrex::GpuArray m_inv_cell_size; }; /** \brief Functor that fills the elements of the particle array `inexflag` @@ -143,14 +143,14 @@ class fillBufferFlagRemainingParticles amrex::Geometry const& geom, amrex::Gpu::DeviceVector const& particle_indices, long const start_index ) : - m_start_index(start_index) { - + m_domain{geom.Domain()}, // Extract simple structure that can be used directly on the GPU - m_particles = pti.GetArrayOfStructs().data(); - m_buffer_mask = (*bmasks)[pti].array(); - m_inexflag_ptr = inexflag.dataPtr(); - m_indices_ptr = particle_indices.dataPtr(); - m_domain = geom.Domain(); + m_inexflag_ptr{inexflag.dataPtr()}, + m_particles{pti.GetArrayOfStructs().data()}, + m_buffer_mask{(*bmasks)[pti].array()}, + m_start_index{start_index}, + m_indices_ptr{particle_indices.dataPtr()} + { for (int idim=0; idim const& src, amrex::Gpu::DeviceVector& dst, - amrex::Gpu::DeviceVector const& indices ) { - // Extract simple structure that can be used directly on the GPU - m_src_ptr = src.dataPtr(); - m_dst_ptr = dst.dataPtr(); - m_indices_ptr = indices.dataPtr(); - } + amrex::Gpu::DeviceVector const& indices ): + // Extract simple structure that can be used directly on the GPU + m_src_ptr{src.dataPtr()}, + m_dst_ptr{dst.dataPtr()}, + m_indices_ptr{indices.dataPtr()} + {} AMREX_GPU_DEVICE AMREX_FORCE_INLINE void operator()( const long ip ) const { diff --git a/Source/Particles/WarpXParticleContainer.H b/Source/Particles/WarpXParticleContainer.H index 2d3ba230bf1..3e3a43095e3 100644 --- a/Source/Particles/WarpXParticleContainer.H +++ b/Source/Particles/WarpXParticleContainer.H @@ -115,7 +115,13 @@ public: using DiagnosticParticles = amrex::Vector, DiagnosticParticleData> >; WarpXParticleContainer (amrex::AmrCore* amr_core, int ispecies); - virtual ~WarpXParticleContainer() {} + ~WarpXParticleContainer() override {} + + // Move and copy operations + WarpXParticleContainer(const WarpXParticleContainer&) = delete; + WarpXParticleContainer& operator=(const WarpXParticleContainer&) = delete; + WarpXParticleContainer(WarpXParticleContainer&&) = default; + WarpXParticleContainer& operator=(WarpXParticleContainer&&) = default; virtual void InitData () = 0; @@ -462,8 +468,7 @@ protected: TmpParticles tmp_particle_data; private: - virtual void particlePostLocate(ParticleType& p, const amrex::ParticleLocData& pld, - int lev) override; + void particlePostLocate(ParticleType& p, const amrex::ParticleLocData& pld, int lev) override; }; diff --git a/Source/Utils/Parser/IntervalsParser.cpp b/Source/Utils/Parser/IntervalsParser.cpp index 16595f6f49c..6af4ce6a8c3 100644 --- a/Source/Utils/Parser/IntervalsParser.cpp +++ b/Source/Utils/Parser/IntervalsParser.cpp @@ -16,9 +16,9 @@ #include -utils::parser::SliceParser::SliceParser (const std::string& instr, const bool isBTD) +utils::parser::SliceParser::SliceParser (const std::string& instr, const bool isBTD): + m_isBTD{isBTD} { - m_isBTD = isBTD; // split string and trim whitespaces auto insplit = ablastr::utils::text::split_string>( instr, m_separator, true); diff --git a/Source/WarpX.H b/Source/WarpX.H index e56d2e647c1..cd05d90555a 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -97,8 +97,7 @@ public: */ static void Finalize(); - /** Destructor */ - ~WarpX (); + ~WarpX () override; /** Copy constructor */ WarpX ( WarpX const &) = delete; @@ -1037,7 +1036,7 @@ public: // This needs to be public for CUDA. //! Tagging cells for refinement - virtual void ErrorEst (int lev, amrex::TagBoxArray& tags, amrex::Real time, int /*ngrow*/) final; + void ErrorEst (int lev, amrex::TagBoxArray& tags, amrex::Real time, int /*ngrow*/) final; // Return the accelerator lattice instance defined at the given refinement level const AcceleratorLattice& get_accelerator_lattice (int lev) {return *(m_accelerator_lattice[lev]);} @@ -1176,28 +1175,28 @@ protected: //! Use this function to override the Level 0 grids made by AMReX. //! This function is called in amrex::AmrCore::InitFromScratch. - virtual void PostProcessBaseGrids (amrex::BoxArray& ba0) const final; + void PostProcessBaseGrids (amrex::BoxArray& ba0) const final; //! Make a new level from scratch using provided BoxArray and //! DistributionMapping. Only used during initialization. Called //! by AmrCoreInitFromScratch. - virtual void MakeNewLevelFromScratch (int lev, amrex::Real time, const amrex::BoxArray& ba, + void MakeNewLevelFromScratch (int lev, amrex::Real time, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm) final; //! Make a new level using provided BoxArray and //! DistributionMapping and fill with interpolated coarse level //! data. Called by AmrCore::regrid. - virtual void MakeNewLevelFromCoarse (int /*lev*/, amrex::Real /*time*/, const amrex::BoxArray& /*ba*/, + void MakeNewLevelFromCoarse (int /*lev*/, amrex::Real /*time*/, const amrex::BoxArray& /*ba*/, const amrex::DistributionMapping& /*dm*/) final; //! Remake an existing level using provided BoxArray and //! DistributionMapping and fill with existing fine and coarse //! data. Called by AmrCore::regrid. - virtual void RemakeLevel (int lev, amrex::Real time, const amrex::BoxArray& ba, + void RemakeLevel (int lev, amrex::Real time, const amrex::BoxArray& ba, const amrex::DistributionMapping& dm) final; //! Delete level data. Called by AmrCore::regrid. - virtual void ClearLevel (int lev) final; + void ClearLevel (int lev) final; private: diff --git a/Source/ablastr/parallelization/KernelTimer.H b/Source/ablastr/parallelization/KernelTimer.H index 0d11220daba..0329948da75 100644 --- a/Source/ablastr/parallelization/KernelTimer.H +++ b/Source/ablastr/parallelization/KernelTimer.H @@ -71,6 +71,11 @@ public: #endif //AMREX_USE_GPU } + KernelTimer ( KernelTimer const &) = default; + KernelTimer& operator= ( KernelTimer const & ) = default; + KernelTimer ( KernelTimer&& ) = default; + KernelTimer& operator= ( KernelTimer&& ) = default; + #if (defined AMREX_USE_GPU) private: //! Stores whether kernel timer is active. diff --git a/Source/ablastr/profiler/ProfilerWrapper.H b/Source/ablastr/profiler/ProfilerWrapper.H index 1dbe80c59b1..f23f13773c1 100644 --- a/Source/ablastr/profiler/ProfilerWrapper.H +++ b/Source/ablastr/profiler/ProfilerWrapper.H @@ -38,6 +38,12 @@ namespace ablastr::profiler device_synchronize(m_do_device_synchronize); } + // default move and copy operations + SynchronizeOnDestruct(const SynchronizeOnDestruct&) = default; + SynchronizeOnDestruct& operator=(const SynchronizeOnDestruct&) = default; + SynchronizeOnDestruct(SynchronizeOnDestruct&&) = default; + SynchronizeOnDestruct& operator=(SynchronizeOnDestruct&& field_data) = default; + bool m_do_device_synchronize = false; }; diff --git a/Source/ablastr/warn_manager/WarnManager.H b/Source/ablastr/warn_manager/WarnManager.H index 75fd642a884..34a8102620e 100644 --- a/Source/ablastr/warn_manager/WarnManager.H +++ b/Source/ablastr/warn_manager/WarnManager.H @@ -52,12 +52,27 @@ namespace ablastr::warn_manager /** * A singleton class should not be cloneable. */ - WarnManager(WarnManager &other) = delete; + WarnManager( const WarnManager& ) = delete; + + /** + * A singleton class should not be cloneable. + */ + WarnManager( WarnManager&& ) = delete; /** * A singleton class should not be assignable. */ - void operator=(const WarnManager &) = delete; + WarnManager& operator=( const WarnManager & ) = delete; + + /** + * A singleton class should not be assignable. + */ + WarnManager& operator=( const WarnManager&& ) = delete; + + /** + * Default destructor + */ + ~WarnManager() = default; static WarnManager& GetInstance(); From 8af2c31353193df69f29b5bcf3205adda1ba29e8 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 26 Oct 2023 11:00:33 -0700 Subject: [PATCH 071/110] Fix External AMReX Builds (#4390) - print version - make AMReX external if pyAMReX is external - new AMReX 23.11+ installed CMake module paths for CMake scripts - `TINYP` is not really required for external packages and thus should not fail if configured not to be used --- .../ReducedDiags/FieldProbeParticleContainer.cpp | 1 - Source/Particles/LaserParticleContainer.cpp | 1 - Source/Particles/PhysicalParticleContainer.cpp | 1 - Source/Particles/WarpXParticleContainer.cpp | 1 - cmake/dependencies/AMReX.cmake | 14 +++++++++++++- cmake/dependencies/pyAMReX.cmake | 2 +- 6 files changed, 14 insertions(+), 6 deletions(-) diff --git a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp index 5d990b9eb07..9a943759542 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp +++ b/Source/Diagnostics/ReducedDiags/FieldProbeParticleContainer.cpp @@ -50,7 +50,6 @@ #include #include #include -#include #include diff --git a/Source/Particles/LaserParticleContainer.cpp b/Source/Particles/LaserParticleContainer.cpp index 782e630e805..7d918d5dfb5 100644 --- a/Source/Particles/LaserParticleContainer.cpp +++ b/Source/Particles/LaserParticleContainer.cpp @@ -46,7 +46,6 @@ #include #include #include -#include #include #include diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp index 7163f506ff8..47898a0344d 100644 --- a/Source/Particles/PhysicalParticleContainer.cpp +++ b/Source/Particles/PhysicalParticleContainer.cpp @@ -83,7 +83,6 @@ #include #include #include -#include #include #include #include diff --git a/Source/Particles/WarpXParticleContainer.cpp b/Source/Particles/WarpXParticleContainer.cpp index df3938ec909..6109cd830f9 100644 --- a/Source/Particles/WarpXParticleContainer.cpp +++ b/Source/Particles/WarpXParticleContainer.cpp @@ -58,7 +58,6 @@ #include #include #include -#include #include diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index fbdd39e4f43..99ef5c5e263 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -1,4 +1,10 @@ macro(find_amrex) + # if pyAMReX is external, AMReX must be as well + if(DEFINED WarpX_pyamrex_internal AND NOT WarpX_pyamrex_internal) + set(WarpX_amrex_internal OFF CACHE BOOL + "Download & build AMReX" FORCE) + endif() + if(WarpX_amrex_src) message(STATUS "Compiling local AMReX ...") message(STATUS "AMReX source path: ${WarpX_amrex_src}") @@ -10,6 +16,7 @@ macro(find_amrex) message(STATUS "AMReX repository: ${WarpX_amrex_repo} (${WarpX_amrex_branch})") include(FetchContent) endif() + if(WarpX_amrex_internal OR WarpX_amrex_src) set(CMAKE_POLICY_DEFAULT_CMP0077 NEW) @@ -243,7 +250,12 @@ macro(find_amrex) endif() set(COMPONENT_PRECISION ${WarpX_PRECISION} P${WarpX_PARTICLE_PRECISION}) - find_package(AMReX 23.10 CONFIG REQUIRED COMPONENTS ${COMPONENT_ASCENT} ${COMPONENT_DIMS} ${COMPONENT_EB} PARTICLES ${COMPONENT_PIC} ${COMPONENT_PRECISION} ${COMPONENT_SENSEI} TINYP LSOLVERS) + find_package(AMReX 23.10 CONFIG REQUIRED COMPONENTS ${COMPONENT_ASCENT} ${COMPONENT_DIMS} ${COMPONENT_EB} PARTICLES ${COMPONENT_PIC} ${COMPONENT_PRECISION} ${COMPONENT_SENSEI} LSOLVERS) + # note: TINYP skipped because user-configured and optional + + # AMReX CMake helper scripts + list(APPEND CMAKE_MODULE_PATH "${AMReX_DIR}/AMReXCMakeModules") + message(STATUS "AMReX: Found version '${AMReX_VERSION}'") endif() endmacro() diff --git a/cmake/dependencies/pyAMReX.cmake b/cmake/dependencies/pyAMReX.cmake index 2ff354785da..c94a41e838d 100644 --- a/cmake/dependencies/pyAMReX.cmake +++ b/cmake/dependencies/pyAMReX.cmake @@ -65,7 +65,7 @@ function(find_pyamrex) elseif(NOT WarpX_pyamrex_internal) # TODO: MPI control find_package(pyAMReX 23.10 CONFIG REQUIRED) - message(STATUS "pyAMReX: Found version '${pyamrex_VERSION}'") + message(STATUS "pyAMReX: Found version '${pyAMReX_VERSION}'") endif() endfunction() From ed13486ac0495b049958989022a9c76e8ef8f042 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 27 Oct 2023 09:02:06 -0700 Subject: [PATCH 072/110] CI: setup-python < 3.12 Requires #2821 --- .github/workflows/cuda.yml | 2 +- .github/workflows/windows.yml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index d3610179695..8eaccd7a2ef 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -27,7 +27,7 @@ jobs: - uses: actions/setup-python@v4 name: Install Python with: - python-version: '3.x' + python-version: '3.10' - name: install dependencies run: | .github/workflows/dependencies/nvcc11-3.sh diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml index e0629e3abca..83160f0a8fa 100644 --- a/.github/workflows/windows.yml +++ b/.github/workflows/windows.yml @@ -15,7 +15,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: - python-version: '3.x' + python-version: '3.10' - name: CCache Cache uses: actions/cache@v3 # - once stored under a key, they become immutable (even if local cache path content changes) @@ -64,7 +64,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: - python-version: '3.x' + python-version: '3.8' - uses: seanmiddleditch/gha-setup-ninja@master - name: CCache Cache uses: actions/cache@v3 From 8e4b7e811febdc9fd08670b39c21c58ca7ba076a Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 27 Oct 2023 09:02:40 -0700 Subject: [PATCH 073/110] Doc: Python Backtrace in CXX Segfaults (#4391) Document the workflow to find out which last Python line was called if the code crashes in the CXX part for PICMI runs. --- Docs/source/usage/workflows/debugging.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Docs/source/usage/workflows/debugging.rst b/Docs/source/usage/workflows/debugging.rst index 9a7d7efacd0..519649e4b46 100644 --- a/Docs/source/usage/workflows/debugging.rst +++ b/Docs/source/usage/workflows/debugging.rst @@ -23,6 +23,9 @@ Try the following steps to debug a simulation: Do you spot numerical artifacts or instabilities that could point to missing resolution or unexpected/incompatible numerical parameters? #. Did the job output files indicate a crash? Check the ``Backtrace.`` files for the location of the code that triggered the crash. Backtraces are read from bottom (high-level) to top (most specific line that crashed). + + #. Was this a segmentation fault in C++, but the run was controlled from Python (PICMI)? + To get the last called Python line for the backtrace, run again and add the Python ``faulthandler``, e.g., with ``python3 -X faulthandler PICMI_your_script_here.py``. #. Try to make the reproducible scenario as small as possible by modifying the inputs file. Reduce number of cells, particles and MPI processes to something as small and as quick to execute as possible. The next steps in debugging will increase runtime, so you will benefit from a fast reproducer. From dae7987c220aa29314d47b0d49c84fbd243bc9fb Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 30 Oct 2023 13:09:51 -0700 Subject: [PATCH 074/110] AMReX/pyAMReX/PICSAR: Weekly Update (#4396) * AMReX: Weekly Update * New Tool: `Release/updatepyAMReX.py` * pyAMReX: Weekly Update --- .github/workflows/cuda.yml | 2 +- Docs/source/maintenance/release.rst | 2 + Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- Tools/Release/updatepyAMReX.py | 124 ++++++++++++++++++++++++++++ cmake/dependencies/AMReX.cmake | 2 +- cmake/dependencies/pyAMReX.cmake | 2 +- run_test.sh | 2 +- 8 files changed, 132 insertions(+), 6 deletions(-) create mode 100755 Tools/Release/updatepyAMReX.py diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 8eaccd7a2ef..bdceac7a7c7 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -111,7 +111,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach da79aff8053058371a78d4bf85488384242368ee && cd - + cd ../amrex && git checkout --detach be6c6415467d09da6109d27cfa218868abc1f9db && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Docs/source/maintenance/release.rst b/Docs/source/maintenance/release.rst index e4047563f76..f25b8c313e4 100644 --- a/Docs/source/maintenance/release.rst +++ b/Docs/source/maintenance/release.rst @@ -13,6 +13,7 @@ The following scripts automate this workflow, in case one needs a newer commit o .. code-block:: sh ./Tools/Release/updateAMReX.py + ./Tools/Release/updatepyAMReX.py ./Tools/Release/updatePICSAR.py @@ -32,6 +33,7 @@ In order to create a GitHub release, you need to: .. code-block:: sh ./Tools/Release/updateAMReX.py + ./Tools/Release/updatepyAMReX.py ./Tools/Release/updatePICSAR.py ./Tools/Release/newVersion.sh diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index 17d98e4b76d..1e5f2a7a619 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = da79aff8053058371a78d4bf85488384242368ee +branch = be6c6415467d09da6109d27cfa218868abc1f9db [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 6f08f34bc2c..14896aff17e 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = da79aff8053058371a78d4bf85488384242368ee +branch = be6c6415467d09da6109d27cfa218868abc1f9db [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/Tools/Release/updatepyAMReX.py b/Tools/Release/updatepyAMReX.py new file mode 100755 index 00000000000..94fde0df1cc --- /dev/null +++ b/Tools/Release/updatepyAMReX.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +# +# Copyright 2021 Axel Huebl +# +# This file is part of WarpX. +# + +# This file is a maintainer tool to bump the pyAMReX version that we pull in +# when building WarpX. +# +import datetime +from pathlib import Path +import re +import sys + +import requests + +# Maintainer Inputs ########################################################### + +print("""Hi there, this is a WarpX maintainer tool to update the source +code of WarpX to a new commit/release of pyAMReX. +For it to work, you need write access on the source directory and +you should be working in a clean git branch without ongoing +rebase/merge/conflict resolves and without unstaged changes.""") + +# check source dir +REPO_DIR = Path(__file__).parent.parent.parent.absolute() +print(f"\nYour current source directory is: {REPO_DIR}") + +REPLY = input("Are you sure you want to continue? [y/N] ") +print() +if not REPLY in ["Y", "y"]: + print("You did not confirm with 'y', aborting.") + sys.exit(1) + + +# Current Versions ############################################################ + +# pyAMReX development HEAD +pyamrex_gh = requests.get('https://api.github.com/repos/AMReX-Codes/pyamrex/commits/development') +pyamrex_HEAD = pyamrex_gh.json()["sha"] + +# WarpX references to pyAMReX: cmake/dependencies/pyAMReX.cmake +pyamrex_cmake_path = str(REPO_DIR.joinpath("cmake/dependencies/pyAMReX.cmake")) +# branch/commit/tag (git fetcher) version +# set(WarpX_pyamrex_branch "development" ... +pyamrex_branch = f"unknown (format issue in {pyamrex_cmake_path})" +with open(pyamrex_cmake_path, encoding='utf-8') as f: + r_minimal = re.findall(r'.*set\(WarpX_pyamrex_branch\s+"(.+)"\s+.*', + f.read(), re.MULTILINE) + if len(r_minimal) >= 1: + pyamrex_branch = r_minimal[0] + +# minimal (external) version +# find_package(AMReX YY.MM CONFIG ... +pyamrex_minimal = f"unknown (format issue in {pyamrex_cmake_path})" +with open(pyamrex_cmake_path, encoding='utf-8') as f: + r_minimal = re.findall(r'.*find_package\(AMReX\s+(.+)\s+CONFIG\s+.*', + f.read(), re.MULTILINE) + if len(r_minimal) >= 1: + pyamrex_minimal = r_minimal[0] + + +# Ask for new ################################################################# + +print("""We will now run a few sed commands on your source directory. +Please answer the following questions about the version number +you want to require from pyAMReX:\n""") + +print(f"Currently, WarpX builds against this pyAMReX commit/branch/sha: {pyamrex_branch}") +print(f"pyAMReX HEAD commit (development branch): {pyamrex_HEAD}") +pyamrex_new_branch = input(f"Update pyAMReX commit/branch/sha: ").strip() +if not pyamrex_new_branch: + pyamrex_new_branch = pyamrex_branch + print(f"--> Nothing entered, will keep: {pyamrex_branch}") +print() + +print(f"Currently, a pre-installed pyAMReX is required at least at version: {pyamrex_minimal}") +today = datetime.date.today().strftime("%y.%m") +pyamrex_new_minimal = input(f"New minimal pyAMReX version (e.g. {today})? ").strip() +if not pyamrex_new_minimal: + pyamrex_new_minimal = pyamrex_minimal + print(f"--> Nothing entered, will keep: {pyamrex_minimal}") + +print() +print(f"New pyAMReX commit/branch/sha: {pyamrex_new_branch}") +print(f"New minimal pyAMReX version: {pyamrex_new_minimal}\n") + +REPLY = input("Is this information correct? Will now start updating! [y/N] ") +print() +if not REPLY in ["Y", "y"]: + print("You did not confirm with 'y', aborting.") + sys.exit(1) + + +# Updates ##################################################################### + +# WarpX references to pyAMReX: cmake/dependencies/pyAMReX.cmake +with open(pyamrex_cmake_path, encoding='utf-8') as f: + pyAMReX_cmake_content = f.read() + + # branch/commit/tag (git fetcher) version + # set(WarpX_pyamrex_branch "development" ... + pyAMReX_cmake_content = re.sub( + r'(.*set\(WarpX_pyamrex_branch\s+")(.+)("\s+.*)', + r'\g<1>{}\g<3>'.format(pyamrex_new_branch), + pyAMReX_cmake_content, flags = re.MULTILINE) + + # minimal (external) version + # find_package(AMReX YY.MM CONFIG ... + pyAMReX_cmake_content = re.sub( + r'(.*find_package\(pyAMReX\s+)(.+)(\s+CONFIG\s+.*)', + r'\g<1>{}\g<3>'.format(pyamrex_new_minimal), + pyAMReX_cmake_content, flags = re.MULTILINE) + +with open(pyamrex_cmake_path, "w", encoding='utf-8') as f: + f.write(pyAMReX_cmake_content) + + +# Epilogue #################################################################### + +print("""Done. Please check your source, e.g. via + git diff +now and commit the changes if no errors occurred.""") diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 99ef5c5e263..c03ea1ba828 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -269,7 +269,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "da79aff8053058371a78d4bf85488384242368ee" +set(WarpX_amrex_branch "be6c6415467d09da6109d27cfa218868abc1f9db" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/cmake/dependencies/pyAMReX.cmake b/cmake/dependencies/pyAMReX.cmake index c94a41e838d..13b84ee3522 100644 --- a/cmake/dependencies/pyAMReX.cmake +++ b/cmake/dependencies/pyAMReX.cmake @@ -79,7 +79,7 @@ option(WarpX_pyamrex_internal "Download & build pyAMReX" ON) set(WarpX_pyamrex_repo "https://github.com/AMReX-Codes/pyamrex.git" CACHE STRING "Repository URI to pull and build pyamrex from if(WarpX_pyamrex_internal)") -set(WarpX_pyamrex_branch "23.10" +set(WarpX_pyamrex_branch "056d332d68fa98b9e9509d05ec3b1a3d8ef37673" CACHE STRING "Repository branch for WarpX_pyamrex_repo if(WarpX_pyamrex_internal)") diff --git a/run_test.sh b/run_test.sh index 2e86618f2f2..78e9c4e3bda 100755 --- a/run_test.sh +++ b/run_test.sh @@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach da79aff8053058371a78d4bf85488384242368ee && cd - +cd amrex && git checkout --detach be6c6415467d09da6109d27cfa218868abc1f9db && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 996ea31ffe218c91949717b9e7cc6475dd4ae7ee Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 30 Oct 2023 16:31:14 -0700 Subject: [PATCH 075/110] Docs: Add Particles-in-PML Paper (#4398) Add paper: "Absorption of charged particles in perfectly matched layers by optimal damping of the deposited current" (2022). --- Docs/source/acknowledge_us.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Docs/source/acknowledge_us.rst b/Docs/source/acknowledge_us.rst index 2c0b1ce35f2..534a314298f 100644 --- a/Docs/source/acknowledge_us.rst +++ b/Docs/source/acknowledge_us.rst @@ -75,6 +75,11 @@ If your project uses a specific algorithm or component, please consider citing t *New Journal of Physics* **24** 025009, 2022. `DOI:10.1088/1367-2630/ac4ef1 `__ +- Lehe R, Blelly A, Giacomel L, Jambunathan R, Vay JL. + **Absorption of charged particles in perfectly matched layers by optimal damping of the deposited current**. + *Physical Review E* **106** 045306, 2022. + `DOI:10.1103/PhysRevE.106.045306 `__ + - Zoni E, Lehe R, Shapoval O, Belkin D, Zaim N, Fedeli L, Vincenti H, Vay JL. **A hybrid nodal-staggered pseudo-spectral electromagnetic particle-in-cell method with finite-order centering**. *Computer Physics Communications* **279**, 2022. `DOI:10.1016/j.cpc.2022.108457 `__ From f6cb58d795bc4024222a8a19f30abb27b2deb687 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 31 Oct 2023 03:14:52 -0700 Subject: [PATCH 076/110] Doc: Fix IPAC23 DOI (#4399) Changed since preprint. --- Docs/source/acknowledge_us.rst | 5 ++--- Docs/source/highlights.rst | 7 +++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/Docs/source/acknowledge_us.rst b/Docs/source/acknowledge_us.rst index 534a314298f..79bb9fa8884 100644 --- a/Docs/source/acknowledge_us.rst +++ b/Docs/source/acknowledge_us.rst @@ -55,9 +55,8 @@ If your project uses a specific algorithm or component, please consider citing t - Sandberg R T, Lehe R, Mitchell C E, Garten M, Qiang J, Vay J-L and Huebl A. **Hybrid Beamline Element ML-Training for Surrogates in the ImpactX Beam-Dynamics Code**. - 14th International Particle Accelerator Conference (IPAC'23), WEPA101, *in print*, 2023. - `preprint `__, - `DOI:10.18429/JACoW-IPAC-23-WEPA101 `__ + 14th International Particle Accelerator Conference (IPAC'23), WEPA101, 2023. + `DOI:10.18429/JACoW-IPAC2023-WEPA101 `__ - Huebl A, Lehe R, Zoni E, Shapoval O, Sandberg R T, Garten M, Formenti A, Jambunathan R, Kumar P, Gott K, Myers A, Zhang W, Almgren A, Mitchell C E, Qiang J, Sinn A, Diederichs S, Thevenet M, Grote D, Fedeli L, Clark T, Zaim N, Vincenti H, Vay JL. **From Compact Plasma Particle Sources to Advanced Accelerators with Modeling at Exascale**. diff --git a/Docs/source/highlights.rst b/Docs/source/highlights.rst index 1e99629b9c2..2fb5fd49263 100644 --- a/Docs/source/highlights.rst +++ b/Docs/source/highlights.rst @@ -24,11 +24,10 @@ Scientific works in laser-plasma and beam-plasma acceleration. Phys. Rev. Research **5**, 033112, 2023 `DOI:10.1103/PhysRevResearch.5.033112 `__ -#. Sandberg R T, Lehe R, Mitchell C E, Garten M, Qiang J, Vay J-L, Huebl A. +#. Sandberg R T, Lehe R, Mitchell C E, Garten M, Qiang J, Vay J-L and Huebl A. **Hybrid Beamline Element ML-Training for Surrogates in the ImpactX Beam-Dynamics Code**. - 14th International Particle Accelerator Conference (IPAC'23), WEPA101, *in print*, 2023. - `preprint `__, - `DOI:10.18429/JACoW-IPAC-23-WEPA101 `__ + 14th International Particle Accelerator Conference (IPAC'23), WEPA101, 2023. + `DOI:10.18429/JACoW-IPAC2023-WEPA101 `__ #. Wang J, Zeng M, Li D, Wang X, Gao J. **High quality beam produced by tightly focused laser driven wakefield accelerators**. From 5b4dfdcaad210ebb69af1f095aa077e449509cc8 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 2 Nov 2023 19:17:09 -0700 Subject: [PATCH 077/110] Python 3.12: Refactor out Distutils (#4392) * CI: Unconstrain Some Upper Py Versions * Replace distutils * Update Docs * no `clean` commmand https://github.com/pypa/setuptools/discussions/2838 * Remove `SETUPTOOLS_USE_DISTUTILS` Old workaround for old mpi4py. * Update Doc String * Update CI * Update Tools/machines/lassen-llnl/install_v100_ml.sh Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> --------- Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> --- .github/workflows/cuda.yml | 22 +++++++------------ .github/workflows/intel.yml | 7 +++--- .github/workflows/macos.yml | 6 +---- .github/workflows/ubuntu.yml | 7 ++---- .github/workflows/windows.yml | 10 ++++++--- CMakeLists.txt | 2 +- Docs/source/install/cmake.rst | 3 ++- Docs/source/install/hpc/lawrencium.rst | 2 ++ Docs/source/install/hpc/lxplus.rst | 3 ++- Docs/source/install/users.rst | 5 ++--- Docs/source/maintenance/performance_tests.rst | 1 + .../adastra-cines/install_dependencies.sh | 2 ++ .../crusher-olcf/install_dependencies.sh | 2 ++ .../frontier-olcf/install_dependencies.sh | 2 ++ .../hpc3-uci/install_gpu_dependencies.sh | 2 ++ .../karolina-it4i/install_cpu_dependencies.sh | 2 ++ .../karolina-it4i/install_gpu_dependencies.sh | 2 ++ .../lassen-llnl/install_v100_dependencies.sh | 2 ++ .../install_v100_dependencies_toss3.sh | 2 ++ Tools/machines/lassen-llnl/install_v100_ml.sh | 1 + .../install_gpu_dependencies.sh | 2 ++ .../machines/lumi-csc/install_dependencies.sh | 2 ++ .../install_cpu_dependencies.sh | 2 ++ .../install_gpu_dependencies.sh | 2 ++ .../quartz-llnl/install_dependencies.sh | 2 ++ .../summit-olcf/install_gpu_dependencies.sh | 2 ++ pyproject.toml | 3 ++- run_test.sh | 7 ++---- setup.py | 19 ++++++---------- 29 files changed, 72 insertions(+), 54 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index bdceac7a7c7..2efda054837 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -18,16 +18,12 @@ jobs: env: CXXFLAGS: "-Werror" CMAKE_GENERATOR: Ninja - # setuptools/mp4py work-around, see - # https://github.com/mpi4py/mpi4py/pull/159 - # https://github.com/mpi4py/mpi4py/issues/157#issuecomment-1001022274 - SETUPTOOLS_USE_DISTUTILS: stdlib steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 name: Install Python with: - python-version: '3.10' + python-version: '3.x' - name: install dependencies run: | .github/workflows/dependencies/nvcc11-3.sh @@ -75,7 +71,8 @@ jobs: -DAMReX_CUDA_ERROR_CAPTURE_THIS=ON cmake --build build_sp -j 2 - python3 -m pip install --upgrade pip setuptools wheel + python3 -m pip install --upgrade pip + python3 -m pip install --upgrade build packaging setuptools wheel export WARPX_MPI=ON export PYWARPX_LIB_DIR=$PWD/build_sp/lib/site-packages/pywarpx/ python3 -m pip wheel . @@ -118,13 +115,9 @@ jobs: name: NVHPC@21.11 NVCC/NVC++ Release [tests] runs-on: ubuntu-20.04 if: github.event.pull_request.draft == false - env: - # For NVHPC, Ninja is slower than the default: - #CMAKE_GENERATOR: Ninja - # setuptools/mp4py work-around, see - # https://github.com/mpi4py/mpi4py/pull/159 - # https://github.com/mpi4py/mpi4py/issues/157#issuecomment-1001022274 - SETUPTOOLS_USE_DISTUTILS: stdlib + #env: + # # For NVHPC, Ninja is slower than the default: + # CMAKE_GENERATOR: Ninja steps: - uses: actions/checkout@v3 - name: Dependencies @@ -175,7 +168,8 @@ jobs: # https://github.com/mpi4py/mpi4py/issues/114 #export CFLAGS="-noswitcherror" - #python3 -m pip install --upgrade pip setuptools wheel + #python3 -m pip install --upgrade pip + #python3 -m pip install --upgrade build packaging setuptools wheel #export WARPX_MPI=ON #export PYWARPX_LIB_DIR=$PWD/build/lib/site-packages/pywarpx/ #python3 -m pip wheel . diff --git a/.github/workflows/intel.yml b/.github/workflows/intel.yml index aba8200f1b2..b4a8869d498 100644 --- a/.github/workflows/intel.yml +++ b/.github/workflows/intel.yml @@ -41,7 +41,8 @@ jobs: export CXX=$(which icpc) export CC=$(which icc) - python3 -m pip install --upgrade pip setuptools wheel + python3 -m pip install --upgrade pip + python3 -m pip install --upgrade build packaging setuptools wheel cmake -S . -B build_dp \ -DCMAKE_VERBOSE_MAKEFILE=ON \ @@ -104,7 +105,7 @@ jobs: export CC=$(which icx) python3 -m pip install --upgrade pip - python3 -m pip install --upgrade setuptools wheel + python3 -m pip install --upgrade build packaging setuptools wheel cmake -S . -B build_sp \ -DCMAKE_CXX_FLAGS_RELEASE="-O1 -DNDEBUG" \ @@ -178,6 +179,6 @@ jobs: # Skip this as it will copy the binary artifacts and we are tight on disk space # python3 -m pip install --upgrade pip - # python3 -m pip install --upgrade setuptools wheel + # python3 -m pip install --upgrade build packaging setuptools wheel # PYWARPX_LIB_DIR=$PWD/build_sp/lib/site-packages/pywarpx/ python3 -m pip wheel . # python3 -m pip install *.whl diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml index c81ee598114..fda2cb77a51 100644 --- a/.github/workflows/macos.yml +++ b/.github/workflows/macos.yml @@ -16,10 +16,6 @@ jobs: HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: TRUE # For macOS, Ninja is slower than the default: #CMAKE_GENERATOR: Ninja - # setuptools/mp4py work-around, see - # https://github.com/mpi4py/mpi4py/pull/159 - # https://github.com/mpi4py/mpi4py/issues/157#issuecomment-1001022274 - SETUPTOOLS_USE_DISTUTILS: stdlib steps: - uses: actions/checkout@v3 - name: Brew Cache @@ -65,7 +61,7 @@ jobs: - name: build WarpX run: | python3 -m pip install --upgrade pip - python3 -m pip install --upgrade pip setuptools wheel + python3 -m pip install --upgrade build packaging setuptools wheel cmake -S . -B build_dp \ -DCMAKE_VERBOSE_MAKEFILE=ON \ diff --git a/.github/workflows/ubuntu.yml b/.github/workflows/ubuntu.yml index 0b43760b621..8fa83c99bef 100644 --- a/.github/workflows/ubuntu.yml +++ b/.github/workflows/ubuntu.yml @@ -167,10 +167,6 @@ jobs: CXX: clang++ # On CI for this test, Ninja is slower than the default: #CMAKE_GENERATOR: Ninja - # setuptools/mp4py work-around, see - # https://github.com/mpi4py/mpi4py/pull/159 - # https://github.com/mpi4py/mpi4py/issues/157#issuecomment-1001022274 - SETUPTOOLS_USE_DISTUTILS: stdlib steps: - uses: actions/checkout@v3 - name: install dependencies @@ -190,7 +186,8 @@ jobs: ccache-openmp-pyfull- - name: build WarpX run: | - python3 -m pip install --upgrade pip setuptools wheel + python3 -m pip install --upgrade pip + python3 -m pip install --upgrade build packaging setuptools wheel export CXXFLAGS="-Werror -Wno-error=pass-failed" diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml index 83160f0a8fa..8e2bb00f1db 100644 --- a/.github/workflows/windows.yml +++ b/.github/workflows/windows.yml @@ -15,7 +15,7 @@ jobs: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: - python-version: '3.10' + python-version: '3.x' - name: CCache Cache uses: actions/cache@v3 # - once stored under a key, they become immutable (even if local cache path content changes) @@ -41,7 +41,9 @@ jobs: cmake --build build --config Debug --parallel 2 if(!$?) { Exit $LASTEXITCODE } - python3 -m pip install --upgrade pip setuptools wheel + python3 -m pip install --upgrade pip + if(!$?) { Exit $LASTEXITCODE } + python3 -m pip install --upgrade build packaging setuptools wheel if(!$?) { Exit $LASTEXITCODE } cmake --build build --config Debug --target install if(!$?) { Exit $LASTEXITCODE } @@ -100,7 +102,9 @@ jobs: cmake --build build --config Release --target install if errorlevel 1 exit 1 - python3 -m pip install --upgrade pip setuptools wheel + python3 -m pip install --upgrade pip + if errorlevel 1 exit 1 + python3 -m pip install --upgrade build packaging setuptools wheel if errorlevel 1 exit 1 python3 -m pip install --upgrade -r requirements.txt if errorlevel 1 exit 1 diff --git a/CMakeLists.txt b/CMakeLists.txt index 4dc886be52d..6bc8fe8dd55 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -127,7 +127,7 @@ option(WarpX_PYTHON_IPO ) set(pyWarpX_VERSION_INFO "" CACHE STRING - "PEP-440 conformant version (set by distutils)") + "PEP-440 conformant version (set by setup.py)") # enforce consistency of dependent options if(WarpX_APP OR WarpX_PYTHON) diff --git a/Docs/source/install/cmake.rst b/Docs/source/install/cmake.rst index 457a352b3c2..a76e8c212a2 100644 --- a/Docs/source/install/cmake.rst +++ b/Docs/source/install/cmake.rst @@ -211,7 +211,8 @@ PICMI Python Bindings .. code-block:: bash - python3 -m pip install -U pip setuptools wheel + python3 -m pip install -U pip + python3 -m pip install -U build packaging setuptools wheel python3 -m pip install -U cmake python3 -m pip install -r requirements.txt diff --git a/Docs/source/install/hpc/lawrencium.rst b/Docs/source/install/hpc/lawrencium.rst index 36039e5eda4..23be9b75431 100644 --- a/Docs/source/install/hpc/lawrencium.rst +++ b/Docs/source/install/hpc/lawrencium.rst @@ -80,6 +80,8 @@ Optionally, download and install Python packages for :ref:`PICMI ` python3 -m venv $HOME/sw/v100/venvs/warpx source $HOME/sw/v100/venvs/warpx/bin/activate python3 -m pip install --upgrade pip + python3 -m pip install --upgrade build + python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Docs/source/install/hpc/lxplus.rst b/Docs/source/install/hpc/lxplus.rst index 413337f711d..5bd8a3c9795 100644 --- a/Docs/source/install/hpc/lxplus.rst +++ b/Docs/source/install/hpc/lxplus.rst @@ -147,7 +147,8 @@ Now, ensure Python tooling is up-to-date: .. code-block:: bash - python3 -m pip install -U pip setuptools wheel + python3 -m pip install -U pip + python3 -m pip install -U build packaging setuptools wheel python3 -m pip install -U cmake Then we compile WarpX as in the previous section (with or without CUDA) adding ``-DWarpX_PYTHON=ON`` and then we install it into our Python: diff --git a/Docs/source/install/users.rst b/Docs/source/install/users.rst index 5174a3b5343..b7893e248c0 100644 --- a/Docs/source/install/users.rst +++ b/Docs/source/install/users.rst @@ -109,12 +109,11 @@ Given that you have the :ref:`WarpX dependencies ` install .. code-block:: bash - # optional: --user - python3 -m pip install -U pip setuptools wheel + python3 -m pip install -U pip + python3 -m pip install -U build packaging setuptools wheel python3 -m pip install -U cmake python3 -m pip wheel -v git+https://github.com/ECP-WarpX/WarpX.git - # optional: --user python3 -m pip install *whl In the future, will publish pre-compiled binary packages on `PyPI `__ for faster installs. diff --git a/Docs/source/maintenance/performance_tests.rst b/Docs/source/maintenance/performance_tests.rst index 13b1f6e3acb..8b902297ccb 100644 --- a/Docs/source/maintenance/performance_tests.rst +++ b/Docs/source/maintenance/performance_tests.rst @@ -75,6 +75,7 @@ Then, in ``$AUTOMATED_PERF_TESTS``, create a file ``run_automated_performance_te # lines below this comment once, before submission. # The commented lines take too long for the job script. #python3 -m pip install --upgrade pip + #python3 -m pip install --upgrade build packaging setuptools wheel #python3 -m pip install --upgrade cython #python3 -m pip install --upgrade numpy #python3 -m pip install --upgrade markupsafe diff --git a/Tools/machines/adastra-cines/install_dependencies.sh b/Tools/machines/adastra-cines/install_dependencies.sh index b94d5224833..8a4cef4a2ec 100755 --- a/Tools/machines/adastra-cines/install_dependencies.sh +++ b/Tools/machines/adastra-cines/install_dependencies.sh @@ -101,6 +101,8 @@ rm -rf ${SW_DIR}/venvs/warpx-adastra python3 -m venv ${SW_DIR}/venvs/warpx-adastra source ${SW_DIR}/venvs/warpx-adastra/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/crusher-olcf/install_dependencies.sh b/Tools/machines/crusher-olcf/install_dependencies.sh index 2defd795786..ab5deb6b0a6 100755 --- a/Tools/machines/crusher-olcf/install_dependencies.sh +++ b/Tools/machines/crusher-olcf/install_dependencies.sh @@ -86,6 +86,8 @@ rm -rf ${SW_DIR}/venvs/warpx-crusher python3 -m venv ${SW_DIR}/venvs/warpx-crusher source ${SW_DIR}/venvs/warpx-crusher/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/frontier-olcf/install_dependencies.sh b/Tools/machines/frontier-olcf/install_dependencies.sh index 690941b2078..896cd9edbbc 100755 --- a/Tools/machines/frontier-olcf/install_dependencies.sh +++ b/Tools/machines/frontier-olcf/install_dependencies.sh @@ -86,6 +86,8 @@ rm -rf ${SW_DIR}/venvs/warpx-frontier python3 -m venv ${SW_DIR}/venvs/warpx-frontier source ${SW_DIR}/venvs/warpx-frontier/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/hpc3-uci/install_gpu_dependencies.sh b/Tools/machines/hpc3-uci/install_gpu_dependencies.sh index e7a5f9f086f..9334d0a2287 100755 --- a/Tools/machines/hpc3-uci/install_gpu_dependencies.sh +++ b/Tools/machines/hpc3-uci/install_gpu_dependencies.sh @@ -116,6 +116,8 @@ rm -rf ${SW_DIR}/venvs/warpx-gpu python3 -m venv ${SW_DIR}/venvs/warpx-gpu source ${SW_DIR}/venvs/warpx-gpu/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/karolina-it4i/install_cpu_dependencies.sh b/Tools/machines/karolina-it4i/install_cpu_dependencies.sh index f5067e03447..2695435738f 100755 --- a/Tools/machines/karolina-it4i/install_cpu_dependencies.sh +++ b/Tools/machines/karolina-it4i/install_cpu_dependencies.sh @@ -120,6 +120,8 @@ rm -rf ${SW_DIR}/venvs/warpx-cpu python3 -m venv ${SW_DIR}/venvs/warpx-cpu source ${SW_DIR}/venvs/warpx-cpu/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/karolina-it4i/install_gpu_dependencies.sh b/Tools/machines/karolina-it4i/install_gpu_dependencies.sh index 4cd1ce5851a..a03bb008816 100755 --- a/Tools/machines/karolina-it4i/install_gpu_dependencies.sh +++ b/Tools/machines/karolina-it4i/install_gpu_dependencies.sh @@ -120,6 +120,8 @@ rm -rf ${SW_DIR}/venvs/warpx-gpu python3 -m venv ${SW_DIR}/venvs/warpx-gpu source ${SW_DIR}/venvs/warpx-gpu/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies.sh b/Tools/machines/lassen-llnl/install_v100_dependencies.sh index dfbf139f27d..fe8285b7501 100755 --- a/Tools/machines/lassen-llnl/install_v100_dependencies.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies.sh @@ -117,6 +117,8 @@ python3 -m venv ${SW_DIR}/venvs/warpx-lassen source ${SW_DIR}/venvs/warpx-lassen/bin/activate python3 -m pip install --upgrade pip python3 -m pip cache purge +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools # Older version for h4py diff --git a/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh b/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh index b8273a69c87..916986ee119 100644 --- a/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh +++ b/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh @@ -117,6 +117,8 @@ python3 -m venv ${SW_DIR}/venvs/warpx-lassen-toss3 source ${SW_DIR}/venvs/warpx-lassen-toss3/bin/activate python3 -m pip install --upgrade pip python3 -m pip cache purge +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools # Older version for h4py diff --git a/Tools/machines/lassen-llnl/install_v100_ml.sh b/Tools/machines/lassen-llnl/install_v100_ml.sh index 6e00be035d6..72a4d04f2f1 100755 --- a/Tools/machines/lassen-llnl/install_v100_ml.sh +++ b/Tools/machines/lassen-llnl/install_v100_ml.sh @@ -64,6 +64,7 @@ rm -rf build cd - # optional: optimas dependencies (based on libEnsemble & ax->botorch->gpytorch->pytorch) +# TODO: scikit-learn needs a BLAS hint # commented because scikit-learn et al. compile > 2 hrs # please run manually on a login node if needed #python3 -m pip install -r ${SRC_DIR}/warpx/Tools/optimas/requirements.txt diff --git a/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh b/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh index 3a332f97420..bbaf0ab8464 100644 --- a/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh +++ b/Tools/machines/leonardo-cineca/install_gpu_dependencies.sh @@ -85,6 +85,8 @@ source ${SW_DIR}/venvs/warpx/bin/activate python3 -m ensurepip --upgrade python3 -m pip cache purge python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/lumi-csc/install_dependencies.sh b/Tools/machines/lumi-csc/install_dependencies.sh index 281ca10c449..0e466f1f57f 100755 --- a/Tools/machines/lumi-csc/install_dependencies.sh +++ b/Tools/machines/lumi-csc/install_dependencies.sh @@ -101,6 +101,8 @@ rm -rf ${SW_DIR}/venvs/warpx-lumi python3 -m venv ${SW_DIR}/venvs/warpx-lumi source ${SW_DIR}/venvs/warpx-lumi/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh b/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh index fa4e77cb309..dbe59438a16 100755 --- a/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh +++ b/Tools/machines/perlmutter-nersc/install_cpu_dependencies.sh @@ -116,6 +116,8 @@ rm -rf ${SW_DIR}/venvs/warpx python3 -m venv ${SW_DIR}/venvs/warpx source ${SW_DIR}/venvs/warpx/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh b/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh index 98e14f09ede..8dd64c56c58 100755 --- a/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh +++ b/Tools/machines/perlmutter-nersc/install_gpu_dependencies.sh @@ -116,6 +116,8 @@ rm -rf ${SW_DIR}/venvs/warpx python3 -m venv ${SW_DIR}/venvs/warpx source ${SW_DIR}/venvs/warpx/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/quartz-llnl/install_dependencies.sh b/Tools/machines/quartz-llnl/install_dependencies.sh index 114bef3d3c3..2920b00663b 100755 --- a/Tools/machines/quartz-llnl/install_dependencies.sh +++ b/Tools/machines/quartz-llnl/install_dependencies.sh @@ -99,6 +99,8 @@ python3 -m venv ${SW_DIR}/venvs/warpx-quartz source ${SW_DIR}/venvs/warpx-quartz/bin/activate python3 -m pip install --upgrade pip python3 -m pip cache purge +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/Tools/machines/summit-olcf/install_gpu_dependencies.sh b/Tools/machines/summit-olcf/install_gpu_dependencies.sh index 7679b999ea3..433b7ab0580 100755 --- a/Tools/machines/summit-olcf/install_gpu_dependencies.sh +++ b/Tools/machines/summit-olcf/install_gpu_dependencies.sh @@ -99,6 +99,8 @@ rm -rf ${SW_DIR}/venvs/warpx-summit python3 -m venv ${SW_DIR}/venvs/warpx-summit source ${SW_DIR}/venvs/warpx-summit/bin/activate python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build +python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel python3 -m pip install --upgrade setuptools python3 -m pip install --upgrade cython diff --git a/pyproject.toml b/pyproject.toml index d7096f82d01..f53522b5622 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,6 +2,7 @@ requires = [ "setuptools>=42", "wheel", - "cmake>=3.20.0,<4.0.0" + "cmake>=3.20.0,<4.0.0", + "packaging>=23", ] build-backend = "setuptools.build_meta" diff --git a/run_test.sh b/run_test.sh index 78e9c4e3bda..f683c5142f6 100755 --- a/run_test.sh +++ b/run_test.sh @@ -61,12 +61,9 @@ echo "cd $PWD" rm -rf py-venv python3 -m venv py-venv source py-venv/bin/activate -python3 -m pip install --upgrade pip setuptools wheel +python3 -m pip install --upgrade pip +python3 -m pip install --upgrade build packaging setuptools wheel python3 -m pip install --upgrade cmake -# setuptools/mp4py work-around, see -# https://github.com/mpi4py/mpi4py/pull/159 -# https://github.com/mpi4py/mpi4py/issues/157#issuecomment-1001022274 -export SETUPTOOLS_USE_DISTUTILS="stdlib" python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data diff --git a/setup.py b/setup.py index 4636d6995f6..e591bc9a506 100644 --- a/setup.py +++ b/setup.py @@ -1,6 +1,3 @@ -from distutils.command.build import build -from distutils.command.clean import clean -from distutils.version import LooseVersion import os import platform import re @@ -9,6 +6,7 @@ import sys from setuptools import Extension, setup +from setuptools.command.build import build from setuptools.command.build_ext import build_ext @@ -26,10 +24,8 @@ def run(self): # by default, this stays around. we want to make sure generated # files like libwarpx.(1d|2d|rz|3d).(so|pyd) are always only the # ones we want to package and not ones from an earlier wheel's stage - c = clean(self.distribution) - c.all = True - c.finalize_options() - c.run() + if os.path.exists(self.build_base): + shutil.rmtree(self.build_base) # call superclass build.run(self) @@ -59,6 +55,8 @@ def __init__(self, name, sourcedir=''): class CMakeBuild(build_ext): def run(self): + from packaging.version import parse + try: out = subprocess.check_output(['cmake', '--version']) except OSError: @@ -67,11 +65,8 @@ def run(self): "extensions: " + ", ".join(e.name for e in self.extensions)) - cmake_version = LooseVersion(re.search( - r'version\s*([\d.]+)', - out.decode() - ).group(1)) - if cmake_version < '3.20.0': + cmake_version = parse(re.search(r"version\s*([\d.]+)", out.decode()).group(1)) + if cmake_version < parse("3.20.0"): raise RuntimeError("CMake >= 3.20.0 is required") for ext in self.extensions: From c034b6637ccf21c209a0971a1c3049318052bb0c Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Fri, 3 Nov 2023 11:08:21 -0600 Subject: [PATCH 078/110] Properly flush metadata of incomplete BTD, at the end of simulation (#4372) --- Source/Diagnostics/BTDiagnostics.H | 2 +- Source/Diagnostics/BTDiagnostics.cpp | 4 ++-- Source/Diagnostics/BoundaryScrapingDiagnostics.H | 2 +- Source/Diagnostics/BoundaryScrapingDiagnostics.cpp | 2 +- Source/Diagnostics/Diagnostics.H | 4 +++- Source/Diagnostics/Diagnostics.cpp | 2 +- Source/Diagnostics/FullDiagnostics.H | 2 +- Source/Diagnostics/FullDiagnostics.cpp | 2 +- 8 files changed, 11 insertions(+), 9 deletions(-) diff --git a/Source/Diagnostics/BTDiagnostics.H b/Source/Diagnostics/BTDiagnostics.H index 9356fdbdaa2..4198846fa5b 100644 --- a/Source/Diagnostics/BTDiagnostics.H +++ b/Source/Diagnostics/BTDiagnostics.H @@ -47,7 +47,7 @@ private: * can be defined and implemented in Diagnostics.H, such that, * the function call to flush out buffer data for * FullDiagnostics and BTDiagnostics is the same */ - void Flush (int i_buffer) override; + void Flush (int i_buffer, bool force_flush) override; /** whether to write output files at this time step * The data is flushed when the buffer is full and/or * when the simulation ends or when forced. diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp index 6f976d9ee11..868dc94ba41 100644 --- a/Source/Diagnostics/BTDiagnostics.cpp +++ b/Source/Diagnostics/BTDiagnostics.cpp @@ -1000,7 +1000,7 @@ BTDiagnostics::GetKIndexInSnapshotBoxFlag (const int i_buffer, const int lev) } void -BTDiagnostics::Flush (int i_buffer) +BTDiagnostics::Flush (int i_buffer, bool force_flush) { auto & warpx = WarpX::GetInstance(); std::string file_name = m_file_prefix; @@ -1009,7 +1009,7 @@ BTDiagnostics::Flush (int i_buffer) file_name = file_name+"/buffer"; } SetSnapshotFullStatus(i_buffer); - const bool isLastBTDFlush = ( m_snapshot_full[i_buffer] == 1 ); + const bool isLastBTDFlush = ( m_snapshot_full[i_buffer] == 1 ) || force_flush; bool const use_pinned_pc = true; bool const isBTD = true; double const labtime = m_t_lab[i_buffer]; diff --git a/Source/Diagnostics/BoundaryScrapingDiagnostics.H b/Source/Diagnostics/BoundaryScrapingDiagnostics.H index 71ee7bfbed8..e9efcfe5abb 100644 --- a/Source/Diagnostics/BoundaryScrapingDiagnostics.H +++ b/Source/Diagnostics/BoundaryScrapingDiagnostics.H @@ -34,7 +34,7 @@ private: utils::parser::IntervalsParser m_intervals; /** \brief Flush data to file. */ - void Flush (int i_buffer) override; + void Flush (int i_buffer, bool /* force_flush */) override; /** \brief Return whether to dump data to file at this time step. * (i.e. whether to call Flush) * diff --git a/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp b/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp index a3ea097352c..9797b77ac1d 100644 --- a/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp +++ b/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp @@ -130,7 +130,7 @@ BoundaryScrapingDiagnostics::DoDump (int step, int /*i_buffer*/, bool force_flus } void -BoundaryScrapingDiagnostics::Flush (int i_buffer) +BoundaryScrapingDiagnostics::Flush (int i_buffer, bool /* force_flush */) { auto & warpx = WarpX::GetInstance(); ParticleBoundaryBuffer& particle_buffer = warpx.GetParticleBoundaryBuffer(); diff --git a/Source/Diagnostics/Diagnostics.H b/Source/Diagnostics/Diagnostics.H index 2b5c2362ec9..dc9656433c6 100644 --- a/Source/Diagnostics/Diagnostics.H +++ b/Source/Diagnostics/Diagnostics.H @@ -63,8 +63,10 @@ public: * multiple times yet. * When these are fixed, the implementation of Flush should be in Diagnostics.cpp * \param[in] i_buffer index of the buffer data to be flushed. + * \param[in] force_flush only used for BTD, whether to do a complete flush of the data + * (including metadata listing the total number of particles) even if the snapshot is incomplete */ - virtual void Flush (int i_buffer) = 0; + virtual void Flush (int i_buffer, bool force_flush) = 0; /** Initialize pointers to main fields and allocate output multifab m_mf_output. */ void InitData (); void InitDataBeforeRestart (); diff --git a/Source/Diagnostics/Diagnostics.cpp b/Source/Diagnostics/Diagnostics.cpp index 6e74fc2cfde..878504dda97 100644 --- a/Source/Diagnostics/Diagnostics.cpp +++ b/Source/Diagnostics/Diagnostics.cpp @@ -575,7 +575,7 @@ Diagnostics::FilterComputePackFlush (int step, bool force_flush) for (int i_buffer = 0; i_buffer < m_num_buffers; ++i_buffer) { if ( !DoDump (step, i_buffer, force_flush) ) continue; - Flush(i_buffer); + Flush(i_buffer, force_flush); } diff --git a/Source/Diagnostics/FullDiagnostics.H b/Source/Diagnostics/FullDiagnostics.H index 4bb3e27895a..4464d46ebac 100644 --- a/Source/Diagnostics/FullDiagnostics.H +++ b/Source/Diagnostics/FullDiagnostics.H @@ -27,7 +27,7 @@ private: */ bool m_solver_deposits_current = true; /** Flush m_mf_output and particles to file for the i^th buffer */ - void Flush (int i_buffer) override; + void Flush (int i_buffer, bool /* force_flush */) override; /** Flush raw data */ void FlushRaw (); /** whether to compute and pack cell-centered data in m_mf_output diff --git a/Source/Diagnostics/FullDiagnostics.cpp b/Source/Diagnostics/FullDiagnostics.cpp index c1c8f270bca..4ca45eeb59d 100644 --- a/Source/Diagnostics/FullDiagnostics.cpp +++ b/Source/Diagnostics/FullDiagnostics.cpp @@ -126,7 +126,7 @@ FullDiagnostics::BackwardCompatibility () } void -FullDiagnostics::Flush ( int i_buffer ) +FullDiagnostics::Flush ( int i_buffer, bool /* force_flush */ ) { // This function should be moved to Diagnostics when plotfiles/openpmd format // is supported for BackTransformed Diagnostics, in BTDiagnostics class. From 707e0bf02efcac3b1aea57772bea1da3b67d9691 Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Fri, 3 Nov 2023 21:35:55 +0100 Subject: [PATCH 079/110] Remove call to WarpX::GetInstance from InitializeMacroMultiFabUsingParser (#4138) * remove call to WarpX::GetInstance from InitializeMacroMultiFabUsingParser * fix bug * fix bugs --- .../MacroscopicProperties.H | 3 ++- .../MacroscopicProperties.cpp | 27 ++++++++++--------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H index d67978f5f3e..d0806c332e3 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H +++ b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H @@ -49,7 +49,8 @@ public: */ void InitializeMacroMultiFabUsingParser (amrex::MultiFab *macro_mf, amrex::ParserExecutor<3> const& macro_parser, - int lev); + const amrex::GpuArray& dx_lev, + const amrex::RealBox& prob_domain_lev); /** Gpu Vector with index type of the conductivity multifab */ amrex::GpuArray sigma_IndexType; diff --git a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.cpp b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.cpp index 858972f2eb4..32c07e0b9fe 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.cpp @@ -144,7 +144,8 @@ MacroscopicProperties::InitData () } else if (m_sigma_s == "parse_sigma_function") { - InitializeMacroMultiFabUsingParser(m_sigma_mf.get(), m_sigma_parser->compile<3>(), lev); + InitializeMacroMultiFabUsingParser(m_sigma_mf.get(), m_sigma_parser->compile<3>(), + warpx.Geom(lev).CellSizeArray(), warpx.Geom(lev).ProbDomain()); } // Initialize epsilon if (m_epsilon_s == "constant") { @@ -153,7 +154,8 @@ MacroscopicProperties::InitData () } else if (m_epsilon_s == "parse_epsilon_function") { - InitializeMacroMultiFabUsingParser(m_eps_mf.get(), m_epsilon_parser->compile<3>(), lev); + InitializeMacroMultiFabUsingParser(m_eps_mf.get(), m_epsilon_parser->compile<3>(), + warpx.Geom(lev).CellSizeArray(), warpx.Geom(lev).ProbDomain()); } // In the Maxwell solver, `epsilon` is used in the denominator. @@ -169,7 +171,8 @@ MacroscopicProperties::InitData () } else if (m_mu_s == "parse_mu_function") { - InitializeMacroMultiFabUsingParser(m_mu_mf.get(), m_mu_parser->compile<3>(), lev); + InitializeMacroMultiFabUsingParser(m_mu_mf.get(), m_mu_parser->compile<3>(), + warpx.Geom(lev).CellSizeArray(), warpx.Geom(lev).ProbDomain()); } @@ -204,11 +207,9 @@ void MacroscopicProperties::InitializeMacroMultiFabUsingParser ( amrex::MultiFab *macro_mf, amrex::ParserExecutor<3> const& macro_parser, - const int lev) + const amrex::GpuArray& dx_lev, + const amrex::RealBox& prob_domain_lev) { - WarpX& warpx = WarpX::GetInstance(); - const amrex::GpuArray dx_lev = warpx.Geom(lev).CellSizeArray(); - const amrex::RealBox& real_box = warpx.Geom(lev).ProbDomain(); const amrex::IntVect iv = macro_mf->ixType().toIntVect(); for ( amrex::MFIter mfi(*macro_mf, TilingIfNotGPU()); mfi.isValid(); ++mfi ) { // Initialize ghost cells in addition to valid cells @@ -223,20 +224,20 @@ MacroscopicProperties::InitializeMacroMultiFabUsingParser ( const amrex::Real x = 0._rt; const amrex::Real y = 0._rt; const amrex::Real fac_z = (1._rt - iv[0]) * dx_lev[0] * 0.5_rt; - const amrex::Real z = j * dx_lev[0] + real_box.lo(0) + fac_z; + const amrex::Real z = j * dx_lev[0] + prob_domain_lev.lo(0) + fac_z; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) const amrex::Real fac_x = (1._rt - iv[0]) * dx_lev[0] * 0.5_rt; - const amrex::Real x = i * dx_lev[0] + real_box.lo(0) + fac_x; + const amrex::Real x = i * dx_lev[0] + prob_domain_lev.lo(0) + fac_x; const amrex::Real y = 0._rt; const amrex::Real fac_z = (1._rt - iv[1]) * dx_lev[1] * 0.5_rt; - const amrex::Real z = j * dx_lev[1] + real_box.lo(1) + fac_z; + const amrex::Real z = j * dx_lev[1] + prob_domain_lev.lo(1) + fac_z; #else const amrex::Real fac_x = (1._rt - iv[0]) * dx_lev[0] * 0.5_rt; - const amrex::Real x = i * dx_lev[0] + real_box.lo(0) + fac_x; + const amrex::Real x = i * dx_lev[0] + prob_domain_lev.lo(0) + fac_x; const amrex::Real fac_y = (1._rt - iv[1]) * dx_lev[1] * 0.5_rt; - const amrex::Real y = j * dx_lev[1] + real_box.lo(1) + fac_y; + const amrex::Real y = j * dx_lev[1] + prob_domain_lev.lo(1) + fac_y; const amrex::Real fac_z = (1._rt - iv[2]) * dx_lev[2] * 0.5_rt; - const amrex::Real z = k * dx_lev[2] + real_box.lo(2) + fac_z; + const amrex::Real z = k * dx_lev[2] + prob_domain_lev.lo(2) + fac_z; #endif // initialize the macroparameter macro_fab(i,j,k) = macro_parser(x,y,z); From 19c32f5c766e1d44f8ae56fe25808728ab0711eb Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Fri, 3 Nov 2023 17:40:19 -0600 Subject: [PATCH 080/110] Remove Cori from documentation (#3939) * Remove Cori from documentation * rder clusters in alphabetical order * Update instructions to use mamba * Apply suggestions from code review * Apply suggestions from code review * Cleanup: Formatting & Robustness Improvements: - easier future updates: clean out existing envs - conda settings - doc links - remove outdated warning - `conda deactivate` will fall back into `nersc-python` conda env, instead we can `exit` the SSH session to be done Bugs: - `echo`: needs `-e` for line breaks - unpin `ipympl` - helper script was not yet used in Jupyter kernel - helper script lacked permissions - `!mamba install -c conda-forge`: needs `-y` or will hang Formatting: - small text updates - verbatim formatting - slightly clearer namings --------- Co-authored-by: Axel Huebl --- Docs/source/dataanalysis/formats.rst | 2 +- Docs/source/install/hpc.rst | 1 - Docs/source/install/hpc/cori.rst | 412 ------------------------- Docs/source/install/hpc/perlmutter.rst | 39 ++- 4 files changed, 33 insertions(+), 421 deletions(-) delete mode 100644 Docs/source/install/hpc/cori.rst diff --git a/Docs/source/dataanalysis/formats.rst b/Docs/source/dataanalysis/formats.rst index 85518abebb7..a7836d41fef 100644 --- a/Docs/source/dataanalysis/formats.rst +++ b/Docs/source/dataanalysis/formats.rst @@ -19,7 +19,7 @@ When using the AMReX `plotfile` format, users can set the ``amrex.async_out=1`` option to perform the IO in a non-blocking fashion, meaning that the simulation will continue to run while an IO thread controls writing the data to disk. This can significantly reduce the overall time spent in IO. This is primarily intended for -large runs on supercomputers such as Summit and Cori; depending on the MPI +large runs on supercomputers (e.g. at OLCF or NERSC); depending on the MPI implementation you are using, you may not see a benefit on your workstation. When writing plotfiles, each rank will write to a separate file, up to some maximum number diff --git a/Docs/source/install/hpc.rst b/Docs/source/install/hpc.rst index d6e901276d7..9617f2a7fd6 100644 --- a/Docs/source/install/hpc.rst +++ b/Docs/source/install/hpc.rst @@ -33,7 +33,6 @@ This section documents quick-start guides for a selection of supercomputers that :maxdepth: 1 hpc/adastra - hpc/cori hpc/crusher hpc/frontier hpc/fugaku diff --git a/Docs/source/install/hpc/cori.rst b/Docs/source/install/hpc/cori.rst deleted file mode 100644 index 35421982142..00000000000 --- a/Docs/source/install/hpc/cori.rst +++ /dev/null @@ -1,412 +0,0 @@ -.. _building-cori: - -Cori (NERSC) -============ - -The `Cori cluster `_ is located at NERSC. - - -Introduction ------------- - -If you are new to this system, **please see the following resources**: - -* `GPU nodes `__ - -* `Cori user guide `__ -* Batch system: `Slurm `__ -* `Jupyter service `__ -* `Production directories `__: - - * ``$SCRATCH``: per-user production directory, purged every 30 days (20TB) - * ``/global/cscratch1/sd/m3239``: shared production directory for users in the project ``m3239``, purged every 30 days (50TB) - * ``/global/cfs/cdirs/m3239/``: community file system for users in the project ``m3239`` (100TB) - -Installation ------------- - -Use the following commands to download the WarpX source code and switch to the correct branch: - -.. code-block:: bash - - git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx - -KNL -^^^ - -We use the following modules and environments on the system (``$HOME/knl_warpx.profile``). - -.. literalinclude:: ../../../../Tools/machines/cori-nersc/knl_warpx.profile.example - :language: bash - :caption: You can copy this file from ``Tools/machines/cori-nersc/knl_warpx.profile.example``. - -And install ADIOS2, BLAS++ and LAPACK++: - -.. code-block:: bash - - source $HOME/knl_warpx.profile - - # c-blosc (I/O compression) - git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc - rm -rf src/c-blosc-knl-build - cmake -S src/c-blosc -B src/c-blosc-knl-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/c-blosc-1.12.1-install - cmake --build src/c-blosc-knl-build --target install --parallel 16 - - # ADIOS2 - git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2 - rm -rf src/adios2-knl-build - cmake -S src/adios2 -B src/adios2-knl-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/adios2-2.7.1-install - cmake --build src/adios2-knl-build --target install --parallel 16 - - # BLAS++ (for PSATD+RZ) - git clone https://github.com/icl-utk-edu/blaspp.git src/blaspp - rm -rf src/blaspp-knl-build - cmake -S src/blaspp -B src/blaspp-knl-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/blaspp-master-install - cmake --build src/blaspp-knl-build --target install --parallel 16 - - # LAPACK++ (for PSATD+RZ) - git clone https://github.com/icl-utk-edu/lapackpp.git src/lapackpp - rm -rf src/lapackpp-knl-build - CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S src/lapackpp -B src/lapackpp-knl-build -Duse_cmake_find_lapack=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DLAPACK_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/knl/lapackpp-master-install - cmake --build src/lapackpp-knl-build --target install --parallel 16 - -For PICMI and Python workflows, also install a virtual environment: - -.. code-block:: bash - - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv - - python3 -m venv $HOME/sw/knl/venvs/knl_warpx - source $HOME/sw/knl/venvs/knl_warpx/bin/activate - - python3 -m pip install --upgrade pip - python3 -m pip install --upgrade wheel - python3 -m pip install --upgrade cython - python3 -m pip install --upgrade numpy - python3 -m pip install --upgrade pandas - python3 -m pip install --upgrade scipy - MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py - python3 -m pip install --upgrade openpmd-api - python3 -m pip install --upgrade matplotlib - python3 -m pip install --upgrade yt - # optional: for libEnsemble - #python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt - -Haswell -^^^^^^^ - -We use the following modules and environments on the system (``$HOME/haswell_warpx.profile``). - -.. literalinclude:: ../../../../Tools/machines/cori-nersc/haswell_warpx.profile.example - :language: bash - :caption: You can copy this file from ``Tools/machines/cori-nersc/haswell_warpx.profile.example``. - -And install ADIOS2, BLAS++ and LAPACK++: - -.. code-block:: bash - - source $HOME/haswell_warpx.profile - - # c-blosc (I/O compression) - git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc - rm -rf src/c-blosc-haswell-build - cmake -S src/c-blosc -B src/c-blosc-haswell-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/haswell/c-blosc-1.12.1-install - cmake --build src/c-blosc-haswell-build --target install --parallel 16 - - # ADIOS2 - git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2 - rm -rf src/adios2-haswell-build - cmake -S src/adios2 -B src/adios2-haswell-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/haswell/adios2-2.7.1-install - cmake --build src/adios2-haswell-build --target install --parallel 16 - - # BLAS++ (for PSATD+RZ) - git clone https://github.com/icl-utk-edu/blaspp.git src/blaspp - rm -rf src/blaspp-haswell-build - cmake -S src/blaspp -B src/blaspp-haswell-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/blaspp-master-haswell-install - cmake --build src/blaspp-haswell-build --target install --parallel 16 - - # LAPACK++ (for PSATD+RZ) - git clone https://github.com/icl-utk-edu/lapackpp.git src/lapackpp - rm -rf src/lapackpp-haswell-build - CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S src/lapackpp -B src/lapackpp-haswell-build -Duse_cmake_find_lapack=ON -DBLAS_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DLAPACK_LIBRARIES=${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_gnu.a -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=$HOME/sw/haswell/lapackpp-master-install - cmake --build src/lapackpp-haswell-build --target install --parallel 16 - -For PICMI and Python workflows, also install a virtual environment: - -.. code-block:: bash - - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv - - python3 -m venv $HOME/sw/haswell/venvs/haswell_warpx - source $HOME/sw/haswell/venvs/haswell_warpx/bin/activate - - python3 -m pip install --upgrade pip - python3 -m pip install --upgrade wheel - python3 -m pip install --upgrade cython - python3 -m pip install --upgrade numpy - python3 -m pip install --upgrade pandas - python3 -m pip install --upgrade scipy - MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py - python3 -m pip install --upgrade openpmd-api - python3 -m pip install --upgrade matplotlib - python3 -m pip install --upgrade yt - # optional: for libEnsemble - #python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt - -GPU (V100) -^^^^^^^^^^ - -Cori provides a partition with `18 nodes that include V100 (16 GB) GPUs `__. -We use the following modules and environments on the system (``$HOME/gpu_warpx.profile``). -You can copy this file from ``Tools/machines/cori-nersc/gpu_warpx.profile.example``: - -.. literalinclude:: ../../../../Tools/machines/cori-nersc/gpu_warpx.profile.example - :language: bash - :caption: You can copy this file from ``Tools/machines/cori-nersc/gpu_warpx.profile.example``. - -And install ADIOS2: - -.. code-block:: bash - - source $HOME/gpu_warpx.profile - - # c-blosc (I/O compression) - git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc - rm -rf src/c-blosc-gpu-build - cmake -S src/c-blosc -B src/c-blosc-gpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/cori_gpu/c-blosc-1.12.1-install - cmake --build src/c-blosc-gpu-build --target install --parallel 16 - - git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2 - rm -rf src/adios2-gpu-build - cmake -S src/adios2 -B src/adios2-gpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/cori_gpu/adios2-2.7.1-install - cmake --build src/adios2-gpu-build --target install --parallel 16 - -For PICMI and Python workflows, also install a virtual environment: - -.. code-block:: bash - - # establish Python dependencies - python3 -m pip install --user --upgrade pip - python3 -m pip install --user virtualenv - - python3 -m venv $HOME/sw/cori_gpu/venvs/gpu_warpx - source $HOME/sw/cori_gpu/venvs/gpu_warpx/bin/activate - - python3 -m pip install --upgrade pip - python3 -m pip install --upgrade wheel - python3 -m pip install --upgrade cython - python3 -m pip install --upgrade numpy - python3 -m pip install --upgrade pandas - python3 -m pip install --upgrade scipy - python3 -m pip install -U --no-cache-dir -v mpi4py - python3 -m pip install --upgrade openpmd-api - python3 -m pip install --upgrade matplotlib - python3 -m pip install --upgrade yt - # optional: for libEnsemble - #python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt - -Building WarpX --------------- - -We recommend to store the above lines in individual ``warpx.profile`` files, as suggested above. -If you want to run on either of the three partitions of Cori, open a new terminal, log into Cori and *source* the environment you want to work with: - -.. code-block:: bash - - # KNL: - source $HOME/knl_warpx.profile - - # Haswell: - #source $HOME/haswell_warpx.profile - - # GPU: - #source $HOME/gpu_warpx.profile - -.. warning:: - - Consider that all three Cori partitions are *incompatible*. - - Do not *source* multiple ``...warpx.profile`` files in the same terminal session. - Open a new terminal and log into Cori again, if you want to switch the targeted Cori partition. - - If you re-submit an already compiled simulation that you ran on another day or in another session, *make sure to source* the corresponding ``...warpx.profile`` again after login! - -Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following commands to compile: - -.. code-block:: bash - - cd $HOME/src/warpx - rm -rf build - - # append if you target GPUs: -DWarpX_COMPUTE=CUDA - cmake -S . -B build -DWarpX_DIMS=3 - cmake --build build -j 16 - -The general :ref:`cmake compile-time options ` apply as usual. - -**That's it!** -A 3D WarpX executable is now in ``build/bin/`` and :ref:`can be run ` with a :ref:`3D example inputs file `. -Most people execute the binary directly or copy it out to a location in ``$SCRATCH``. - -The general :ref:`cmake compile-time options and instructions for Python (PICMI) bindings ` apply as usual: - -.. code-block:: bash - - # PICMI build - cd $HOME/src/warpx - - # install or update dependencies - python3 -m pip install -r requirements.txt - - # compile parallel PICMI interfaces with openPMD support and 3D, 2D, 1D and RZ - WARPX_MPI=ON BUILD_PARALLEL=16 python3 -m pip install --force-reinstall --no-deps -v . - - -.. _building-cori-tests: - -Testing -------- - -To run all tests (here on KNL), do: - -* change in ``Regressions/WarpX-tests.ini`` from ``mpiexec`` to ``srun``: ``MPIcommand = srun -n @nprocs@ @command@`` - -.. code-block:: bash - - # set test directory to a shared directory available on all nodes - # note: the tests will create the directory automatically - export WARPX_CI_TMP="$HOME/warpx-regression-tests" - - # compile with more cores - export WARPX_CI_NUM_MAKE_JOBS=16 - - # run all integration tests - # note: we set MPICC as a build-setting for mpi4py on KNL/Haswell - MPICC="cc -shared" ./run_test.sh - - -.. _running-cpp-cori: - -Running -------- - -Navigate (i.e. ``cd``) into one of the production directories (e.g. ``$SCRATCH``) before executing the instructions below. - -KNL -^^^ - -The batch script below can be used to run a WarpX simulation on 2 KNL nodes on -the supercomputer Cori at NERSC. Replace descriptions between chevrons ``<>`` -by relevant values, for instance ```` could be ``laserWakefield``. - -Do not forget to first ``source $HOME/knl_warpx.profile`` if you have not done so already for this terminal session. - -For PICMI Python runs, the ```` has to read ``python3`` and the ```` is the path to your PICMI input script. - -.. literalinclude:: ../../../../Tools/machines/cori-nersc/cori_knl.sbatch - :language: bash - :caption: You can copy this file from ``Tools/machines/cori-nersc/cori_knl.sbatch``. - -To run a simulation, copy the lines above to a file ``cori_knl.sbatch`` and run - -.. code-block:: bash - - sbatch cori_knl.sbatch - -to submit the job. - -For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell -solver on Cori KNL for a well load-balanced problem (in our case laser -wakefield acceleration simulation in a boosted frame in the quasi-linear -regime), the following set of parameters provided good performance: - -* ``amr.max_grid_size=64`` and ``amr.blocking_factor=64`` so that the size of - each grid is fixed to ``64**3`` (we are not using load-balancing here). - -* **8 MPI ranks per KNL node**, with ``OMP_NUM_THREADS=8`` (that is 64 threads - per KNL node, i.e. 1 thread per physical core, and 4 cores left to the - system). - -* **2 grids per MPI**, *i.e.*, 16 grids per KNL node. - -Haswell -^^^^^^^ - -The batch script below can be used to run a WarpX simulation on 1 `Haswell node `_ on the supercomputer Cori at NERSC. - -Do not forget to first ``source $HOME/haswell_warpx.profile`` if you have not done so already for this terminal session. - -.. literalinclude:: ../../../../Tools/machines/cori-nersc/cori_haswell.sbatch - :language: bash - :caption: You can copy this file from ``Tools/machines/cori-nersc/cori_haswell.sbatch``. - -To run a simulation, copy the lines above to a file ``cori_haswell.sbatch`` and -run - -.. code-block:: bash - - sbatch cori_haswell.sbatch - -to submit the job. - -For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell -solver on Cori Haswell for a well load-balanced problem (in our case laser -wakefield acceleration simulation in a boosted frame in the quasi-linear -regime), the following set of parameters provided good performance: - -* **4 MPI ranks per Haswell node** (2 MPI ranks per `Intel Xeon E5-2698 v3 `_), with ``OMP_NUM_THREADS=16`` (which uses `2x hyperthreading `_) - -GPU (V100) -^^^^^^^^^^ - -Do not forget to first ``source $HOME/gpu_warpx.profile`` if you have not done so already for this terminal session. - -Due to the limited amount of GPU development nodes, just request a single node with the above defined ``getNode`` function. -For single-node runs, try to run one grid per GPU. - -A multi-node batch script template can be found below: - -.. literalinclude:: ../../../../Tools/machines/cori-nersc/cori_gpu.sbatch - :language: bash - :caption: You can copy this file from ``Tools/machines/cori-nersc/cori_gpu.sbatch``. - - -.. _post-processing-cori: - -Post-Processing ---------------- - -For post-processing, most users use Python via NERSC's `Jupyter service `__ (`Docs `__). - -As a one-time preparatory setup, `create your own Conda environment as described in NERSC docs `__. -In this manual, we often use this ``conda create`` line over the officially documented one: - -.. code-block:: bash - - conda create -n myenv -c conda-forge python mamba ipykernel ipympl==0.8.6 matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow - -We then follow the `Customizing Kernels with a Helper Shell Script `__ section to finalize the setup of using this conda-environment as a custom Jupyter kernel. - -``kernel_helper.sh`` should read: - -.. code-block:: bash - - #!/bin/bash - module load python - source activate myenv - exec "$@" - -When opening a Jupyter notebook, just select the name you picked for your custom kernel on the top right of the notebook. - -Additional software can be installed later on, e.g., in a Jupyter cell using ``!mamba install -c conda-forge ...``. -Software that is not available via conda can be installed via ``!python -m pip install ...``. - -.. warning:: - - Jan 6th, 2022 (NERSC-INC0179165 and `ipympl #416 `__): - Above, we fixated the ``ipympl`` version to *not* take the latest release of `Matplotlib Jupyter Widgets `__. - This is an intentional work-around; the ``ipympl`` version needs to exactly fit the version pre-installed on the Jupyter base system. diff --git a/Docs/source/install/hpc/perlmutter.rst b/Docs/source/install/hpc/perlmutter.rst index c5d6a9e1898..e9ebcd4e1de 100644 --- a/Docs/source/install/hpc/perlmutter.rst +++ b/Docs/source/install/hpc/perlmutter.rst @@ -13,7 +13,7 @@ If you are new to this system, **please see the following resources**: * `NERSC user guide `__ * Batch system: `Slurm `__ -* `Jupyter service `__ +* `Jupyter service `__ (`documentation `__) * `Filesystems `__: * ``$HOME``: per-user directory, use only for inputs, source and scripts; backed up (40GB) @@ -271,11 +271,36 @@ Running Post-Processing --------------- -For post-processing, most users use Python via NERSC's `Jupyter service `__ (`Docs `__). +For post-processing, most users use Python via NERSC's `Jupyter service `__ (`documentation `__). -Please follow the same process as for :ref:`NERSC Cori post-processing `. -**Important:** The *environment + Jupyter kernel* must separate from the one you create for Cori. +As a one-time preparatory setup, log into Perlmutter via SSH and do *not* source the WarpX profile script above. +Create your own Conda environment and `Jupyter kernel `__ for post-processing: -The Perlmutter ``$PSCRATCH`` filesystem is only available on *Perlmutter* Jupyter nodes. -Likewise, Cori's ``$SCRATCH`` filesystem is only available on *Cori* Jupyter nodes. -You can use the Community FileSystem (CFS) from everywhere. +.. code-block:: bash + + module load python + + conda config --set auto_activate_base false + + # create conda environment + rm -rf $HOME/.conda/envs/warpx-pm-postproc + conda create --yes -n warpx-pm-postproc -c conda-forge mamba conda-libmamba-solver + conda activate warpx-pm-postproc + conda config --set solver libmamba + mamba install --yes -c conda-forge python ipykernel ipympl matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow + + # create Jupyter kernel + rm -rf $HOME/.local/share/jupyter/kernels/warpx-pm-postproc/ + python -m ipykernel install --user --name warpx-pm-postproc --display-name WarpX-PM-PostProcessing + echo -e '#!/bin/bash\nmodule load python\nsource activate warpx-pm-postproc\nexec "$@"' > $HOME/.local/share/jupyter/kernels/warpx-pm-postproc/kernel-helper.sh + chmod a+rx $HOME/.local/share/jupyter/kernels/warpx-pm-postproc/kernel-helper.sh + KERNEL_STR=$(jq '.argv |= ["{resource_dir}/kernel-helper.sh"] + .' $HOME/.local/share/jupyter/kernels/warpx-pm-postproc/kernel.json | jq '.argv[1] = "python"') + echo ${KERNEL_STR} | jq > $HOME/.local/share/jupyter/kernels/warpx-pm-postproc/kernel.json + + exit + + +When opening a Jupyter notebook on `https://jupyter.nersc.gov `__, just select ``WarpX-PM-PostProcessing`` from the list of available kernels on the top right of the notebook. + +Additional software can be installed later on, e.g., in a Jupyter cell using ``!mamba install -y -c conda-forge ...``. +Software that is not available via conda can be installed via ``!python -m pip install ...``. From b3ba07df16de4e9736b01a8cc7024740714dcad6 Mon Sep 17 00:00:00 2001 From: David Grote Date: Sat, 4 Nov 2023 00:03:20 -0700 Subject: [PATCH 081/110] Allow negative mean velocity with NFluxPerCell (#4397) * Allow negative drift velocity with Gaussian flux injection * Simplify by using sign of u_m * Add negative u_m test to flux_injection * Update FluxInjection3D benchmark * Save abs(u_m) as a temporary const --- .../analysis_flux_injection_3d.py | 48 ++++++++++++++----- Examples/Tests/flux_injection/inputs_3d | 33 ++++++++++++- .../benchmarks_json/FluxInjection3D.json | 46 ++++++++++++------ Source/Initialization/InjectorMomentum.H | 36 +++++++------- 4 files changed, 115 insertions(+), 48 deletions(-) diff --git a/Examples/Tests/flux_injection/analysis_flux_injection_3d.py b/Examples/Tests/flux_injection/analysis_flux_injection_3d.py index 048ef70f9cc..0998de25a7b 100755 --- a/Examples/Tests/flux_injection/analysis_flux_injection_3d.py +++ b/Examples/Tests/flux_injection/analysis_flux_injection_3d.py @@ -57,11 +57,8 @@ def gaussian_dist(u, u_th): return 1./((2*np.pi)**.5*u_th) * np.exp(-u**2/(2*u_th**2) ) def gaussian_flux_dist(u, u_th, u_m): - au_m = np.abs(u_m) - normalization_factor = u_th**2 * np.exp(-au_m**2/(2*u_th**2)) + (np.pi/2)**.5*au_m*u_th * (1 + erf(au_m/(2**.5*u_th))) - result = 1./normalization_factor * np.where( u>0, u * np.exp(-(u-au_m)**2/(2*u_th**2)), 0 ) - if u_m < 0.: - result = result[::-1] + normalization_factor = u_th**2 * np.exp(-u_m**2/(2*u_th**2)) + (np.pi/2)**.5*u_m*u_th * (1 + erf(u_m/(2**.5*u_th))) + result = 1./normalization_factor * np.where( u>0, u * np.exp(-(u-u_m)**2/(2*u_th**2)), 0 ) return result def compare_gaussian(u, w, u_th, label=''): @@ -84,10 +81,10 @@ def compare_gaussian_flux(u, w, u_th, u_m, label=''): # Load data and perform check -plt.figure(figsize=(5,7)) +plt.figure(figsize=(8,7)) -plt.subplot(211) -plt.title('Electrons') +plt.subplot(221) +plt.title('Electrons u_m=0.07') ux = ad['electron','particle_momentum_x'].to_ndarray()/(m_e*c) uy = ad['electron','particle_momentum_y'].to_ndarray()/(m_e*c) @@ -97,21 +94,46 @@ def compare_gaussian_flux(u, w, u_th, u_m, label=''): compare_gaussian(ux, w, u_th=0.1, label='u_x') compare_gaussian_flux(uy, w, u_th=0.1, u_m=0.07, label='u_y') compare_gaussian(uz, w, u_th=0.1, label='u_z') -plt.legend(loc=0) -plt.subplot(212) -plt.title('Protons') +plt.subplot(223) +plt.title('Protons u_m=0.05') ux = ad['proton','particle_momentum_x'].to_ndarray()/(m_p*c) uy = ad['proton','particle_momentum_y'].to_ndarray()/(m_p*c) uz = ad['proton','particle_momentum_z'].to_ndarray()/(m_p*c) w = ad['proton', 'particle_weight'].to_ndarray() -compare_gaussian_flux(ux, w, u_th=0.1, u_m=-0.05, label='u_x') +compare_gaussian_flux(-ux, w, u_th=0.1, u_m=0.05, label='u_x') compare_gaussian(uy, w, u_th=0.1, label='u_y') compare_gaussian(uz, w, u_th=0.1, label='u_z') -plt.legend(loc=0) +plt.subplot(222) +plt.title('Electrons u_m=-0.07') + +ux = ad['electron_negative','particle_momentum_x'].to_ndarray()/(m_e*c) +uy = ad['electron_negative','particle_momentum_y'].to_ndarray()/(m_e*c) +uz = ad['electron_negative','particle_momentum_z'].to_ndarray()/(m_e*c) +w = ad['electron_negative', 'particle_weight'].to_ndarray() + +compare_gaussian(ux, w, u_th=0.1, label='u_x') +compare_gaussian(uy, w, u_th=0.1, label='u_y') +compare_gaussian_flux(uz, w, u_th=0.1, u_m=-0.07, label='u_z') +plt.legend(loc=(1.02, 0.5)) + +plt.subplot(224) +plt.title('Protons u_m=-0.05') + +ux = ad['proton_negative','particle_momentum_x'].to_ndarray()/(m_p*c) +uy = ad['proton_negative','particle_momentum_y'].to_ndarray()/(m_p*c) +uz = ad['proton_negative','particle_momentum_z'].to_ndarray()/(m_p*c) +w = ad['proton_negative', 'particle_weight'].to_ndarray() + +compare_gaussian(ux, w, u_th=0.1, label='u_x') +compare_gaussian(uy, w, u_th=0.1, label='u_y') +compare_gaussian_flux(-uz, w, u_th=0.1, u_m=-0.05, label='u_z') +#plt.legend(loc=0) + +plt.tight_layout() plt.savefig('Distribution.png') # Verify checksum diff --git a/Examples/Tests/flux_injection/inputs_3d b/Examples/Tests/flux_injection/inputs_3d index 7d277e7c292..80ee23feff9 100644 --- a/Examples/Tests/flux_injection/inputs_3d +++ b/Examples/Tests/flux_injection/inputs_3d @@ -28,7 +28,7 @@ boundary.field_lo = periodic periodic periodic boundary.field_hi = periodic periodic periodic # particles -particles.species_names = electron proton +particles.species_names = electron proton electron_negative proton_negative algo.particle_shape = 3 electron.charge = -q_e @@ -61,6 +61,37 @@ proton.ux_m = 0.05 proton.uy_th = 0.1 proton.uz_th = 0.1 +# "negative" is negative u_m +electron_negative.charge = -q_e +electron_negative.mass = m_e +electron_negative.injection_style = NFluxPerCell +electron_negative.num_particles_per_cell = 100 +electron_negative.surface_flux_pos = -4. +electron_negative.flux_normal_axis = z +electron_negative.flux_direction = +1 +electron_negative.flux_profile = parse_flux_function +electron_negative.flux_function(x,y,z,t) = "1." +electron_negative.momentum_distribution_type = gaussianflux +electron_negative.ux_th = 0.1 +electron_negative.uy_th = 0.1 +electron_negative.uz_th = 0.1 +electron_negative.uz_m = -0.07 + +proton_negative.charge = +q_e +proton_negative.mass = m_p +proton_negative.injection_style = NFluxPerCell +proton_negative.num_particles_per_cell = 100 +proton_negative.surface_flux_pos = 4. +proton_negative.flux_normal_axis = z +proton_negative.flux_direction = -1 +proton_negative.flux_profile = constant +proton_negative.flux = 1. +proton_negative.momentum_distribution_type = gaussianflux +proton_negative.ux_th = 0.1 +proton_negative.uy_th = 0.1 +proton_negative.uz_th = 0.1 +proton_negative.uz_m = -0.05 + # Diagnostics diagnostics.diags_names = diag1 diag1.intervals = 1000 diff --git a/Regression/Checksum/benchmarks_json/FluxInjection3D.json b/Regression/Checksum/benchmarks_json/FluxInjection3D.json index b2b3733737e..aa30f988f68 100644 --- a/Regression/Checksum/benchmarks_json/FluxInjection3D.json +++ b/Regression/Checksum/benchmarks_json/FluxInjection3D.json @@ -1,21 +1,39 @@ { - "electron": { - "particle_momentum_x": 1.1192116199394354e-18, - "particle_momentum_y": 2.238114590066897e-18, - "particle_momentum_z": 1.1156457989239732e-18, - "particle_position_x": 102495.14197173176, - "particle_position_y": 188132.22608016344, - "particle_position_z": 102423.13701045913, + "lev=0": {}, + "electron_negative": { + "particle_momentum_x": 1.1222699783863554e-18, + "particle_momentum_y": 1.1202176725070554e-18, + "particle_momentum_z": 1.3925955132362978e-18, + "particle_position_x": 102352.09026544492, + "particle_position_y": 102418.88243172191, + "particle_position_z": 194298.7949373403, "particle_weight": 8.959999999999998e-07 }, - "lev=0": {}, "proton": { - "particle_momentum_x": 3.835423016604918e-15, - "particle_momentum_y": 2.0468371931479925e-15, - "particle_momentum_z": 2.055186547721331e-15, - "particle_position_x": 189256.1546041931, - "particle_position_y": 102293.00576740496, - "particle_position_z": 102314.93877691089, + "particle_momentum_x": 3.8338884590187296e-15, + "particle_momentum_y": 2.0442156829943128e-15, + "particle_momentum_z": 2.045804260395492e-15, + "particle_position_x": 189238.69249885075, + "particle_position_y": 102242.91543133644, + "particle_position_z": 102297.92915049737, + "particle_weight": 8.959999999999998e-07 + }, + "electron": { + "particle_momentum_x": 1.1150196665556376e-18, + "particle_momentum_y": 2.2311586451156107e-18, + "particle_momentum_z": 1.1115069298757383e-18, + "particle_position_x": 102477.68142952351, + "particle_position_y": 188137.20906834095, + "particle_position_z": 102443.44417709563, + "particle_weight": 8.959999999999998e-07 + }, + "proton_negative": { + "particle_momentum_x": 2.051429985038282e-15, + "particle_momentum_y": 2.053711846305655e-15, + "particle_momentum_z": 2.7212135003240815e-15, + "particle_position_x": 102307.73649638034, + "particle_position_y": 102475.13878406698, + "particle_position_z": 193638.89296895845, "particle_weight": 8.959999999999998e-07 } } diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index b1a3ac7b0c8..d8409258fa0 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -104,37 +104,40 @@ namespace { // Momentum to be returned at the end of this function amrex::Real u = 0._rt; + const amrex::Real abs_u_m = std::abs(u_m); + if (u_th == 0._rt) { u = u_m; // Trivial case ; avoids division by 0 in the rest of the code below - } else if (u_m < 0.6*u_th) { - // Mean velocity is lower than thermal velocity - // Use the distribution u*exp(-u**2*(1-u_m/u_th)/(2*u_th**2)) as an approximation + } else if (abs_u_m < 0.6*u_th) { + // Mean velocity magnitude is less than thermal velocity + // Use the distribution u*exp(-u**2*(1-abs(u_m)/u_th)/(2*u_th**2)) as an approximation // and then use the rejection method to correct it - // ( stop rejecting with probability exp(-u_m/(2*u_th**3)*(u-u_th)**2) ) + // ( stop rejecting with probability exp(-abs(u_m)/(2*u_th**3)*(u-sign(u_m)*u_th)**2) ) // Note that this is the method that is used in the common case u_m=0 - const amrex::Real approx_u_th = u_th/std::sqrt( 1._rt - u_m/u_th ); - const amrex::Real reject_prefactor = (u_m/u_th)/(2._rt*u_th*u_th); // To save computation + const amrex::Real umsign = std::copysign(1._rt, u_m); + const amrex::Real approx_u_th = u_th/std::sqrt( 1._rt - abs_u_m/u_th ); + const amrex::Real reject_prefactor = (abs_u_m/u_th)/(2._rt*u_th*u_th); // To save computation bool reject = true; while (reject) { - // Generates u according to u*exp(-u**2/(2*approx_u_th**2), + // Generates u according to u*exp(-u**2/(2*approx_u_th**2)), // using the method of the inverse cumulative function amrex::Real xrand = 1._rt - amrex::Random(engine); // ensures urand > 0 u = approx_u_th * std::sqrt(2._rt*std::log(1._rt/xrand)); // Rejection method xrand = amrex::Random(engine); - if (xrand < std::exp(-reject_prefactor*(u-u_th)*(u-u_th))) reject = false; + if (xrand < std::exp(-reject_prefactor*(u - umsign*u_th)*(u - umsign*u_th))) reject = false; } } else { - // Mean velocity is greater than thermal velocity - // Use the distribution exp(-(u-u_m-u_th**2/u_m)**2/(2*u_th**2)) as an approximation + // Mean velocity magnitude is greater than thermal velocity + // Use the distribution exp(-(u-u_m-u_th**2/abs(u_m))**2/(2*u_th**2)) as an approximation // and then use the rejection method to correct it - // ( stop rejecting with probability (u/u_m)*exp(1-(u/u_m)) ; note + // ( stop rejecting with probability (u/abs(u_m))*exp(1-(u/abs(u_m))) ; note // that this number is always between 0 and 1 ) // Note that in the common case `u_m = 0`, this rejection method // is not used, and the above rejection method is used instead. bool reject = true; - const amrex::Real approx_u_m = u_m + u_th*u_th/u_m; - const amrex::Real inv_um = 1._rt/u_m; // To save computation + const amrex::Real approx_u_m = u_m + u_th*u_th/abs_u_m; + const amrex::Real inv_um = 1._rt/abs_u_m; // To save computation while (reject) { // Approximate distribution: normal distribution, where we only retain positive u u = -1._rt; @@ -166,13 +169,6 @@ struct InjectorMomentumGaussianFlux m_flux_normal_axis(a_flux_normal_axis), m_flux_direction(a_flux_direction) { - // For now, do not allow negative `u_m` along the flux axis - bool raise_error = false; - if ((m_flux_normal_axis == 0) && (m_ux_m < 0)) raise_error = true; - if ((m_flux_normal_axis == 1) && (m_uy_m < 0)) raise_error = true; - if ((m_flux_normal_axis == 2) && (m_uz_m < 0)) raise_error = true; - WARPX_ALWAYS_ASSERT_WITH_MESSAGE( raise_error==false, - "When using the `gaussianflux` distribution, the central momentum along the flux axis must be positive or zero." ); } AMREX_GPU_HOST_DEVICE From 797eee67a4a537ae3f1f2eb0499ecc7cbd3fd0b3 Mon Sep 17 00:00:00 2001 From: Minghao Ma <139459631+MaMH111@users.noreply.github.com> Date: Tue, 7 Nov 2023 08:08:37 +0800 Subject: [PATCH 082/110] Bug fix: the electric field calculation for ionization had a bug if external electric field and grid electric field co-exist in the quasi-cylindrical geometry. This commit fixes this bug. (#4094) --- Source/Particles/ElementaryProcess/Ionization.H | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Source/Particles/ElementaryProcess/Ionization.H b/Source/Particles/ElementaryProcess/Ionization.H index 742d8127da9..5a70069a3a3 100644 --- a/Source/Particles/ElementaryProcess/Ionization.H +++ b/Source/Particles/ElementaryProcess/Ionization.H @@ -96,13 +96,13 @@ struct IonizationFilterFunc amrex::ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; amrex::ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; - m_get_externalEB(i, ex, ey, ez, bx, by, bz); doGatherShapeN(xp, yp, zp, ex, ey, ez, bx, by, bz, m_ex_arr, m_ey_arr, m_ez_arr, m_bx_arr, m_by_arr, m_bz_arr, m_ex_type, m_ey_type, m_ez_type, m_bx_type, m_by_type, m_bz_type, m_dx_arr, m_xyzmin_arr, m_lo, m_n_rz_azimuthal_modes, m_nox, m_galerkin_interpolation); + m_get_externalEB(i, ex, ey, ez, bx, by, bz); // Compute electric field amplitude in the particle's frame of // reference (particularly important when in boosted frame). From f8fef83c3b7d6483b3488ffaaac7ecdc0bd9ca1d Mon Sep 17 00:00:00 2001 From: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Mon, 6 Nov 2023 20:22:15 -0800 Subject: [PATCH 083/110] add Nicks (2023) to science highlights (#4407) --- Docs/source/highlights.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Docs/source/highlights.rst b/Docs/source/highlights.rst index 2fb5fd49263..0f9bee1808a 100644 --- a/Docs/source/highlights.rst +++ b/Docs/source/highlights.rst @@ -140,3 +140,11 @@ High-Performance Computing and Numerics Scientific works in High-Performance Computing, applied mathematics and numerics. Please see :ref:`this section `. + +Nuclear Fusion - Magnetically Confined Plasmas +********************************************** + +#. Nicks, B. S., Putvinski, S. and Tajima, T. + **Stabilization of the Alfvén-ion cyclotron instability through short plasmas: Fully kinetic simulations in a high-beta regime**. + Physics of Plasmas **30**, 102108, 2023. + `DOI:10.1063/5.0163889 `__ From 0b1c13bb7448c9e21f07e0aec089de9209e3277a Mon Sep 17 00:00:00 2001 From: David Grote Date: Wed, 8 Nov 2023 07:24:11 -0800 Subject: [PATCH 084/110] In doGatherShapeN, for RZ use temporaries for Er and Etheta (#4409) --- Source/Particles/Gather/FieldGather.H | 73 ++++++++++++++++++++------- 1 file changed, 55 insertions(+), 18 deletions(-) diff --git a/Source/Particles/Gather/FieldGather.H b/Source/Particles/Gather/FieldGather.H index 0b2aae52ecc..1b5cede3c6f 100644 --- a/Source/Particles/Gather/FieldGather.H +++ b/Source/Particles/Gather/FieldGather.H @@ -252,7 +252,7 @@ void doGatherShapeN (const amrex::ParticleReal xp, by_arr(lo.x+l_by+iz, 0, 0, 0); } -#elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) +#elif defined(WARPX_DIM_XZ) // Gather field on particle Eyp from field on grid ey_arr for (int iz=0; iz<=depos_order; iz++){ for (int ix=0; ix<=depos_order; ix++){ @@ -288,7 +288,47 @@ void doGatherShapeN (const amrex::ParticleReal xp, } } -#ifdef WARPX_DIM_RZ +#elif defined(WARPX_DIM_RZ) + + amrex::ParticleReal Erp = 0.; + amrex::ParticleReal Ethetap = 0.; + amrex::ParticleReal Brp = 0.; + amrex::ParticleReal Bthetap = 0.; + + // Gather field on particle Ethetap from field on grid ey_arr + for (int iz=0; iz<=depos_order; iz++){ + for (int ix=0; ix<=depos_order; ix++){ + Ethetap += sx_ey[ix]*sz_ey[iz]* + ey_arr(lo.x+j_ey+ix, lo.y+l_ey+iz, 0, 0); + } + } + // Gather field on particle Erp from field on grid ex_arr + // Gather field on particle Bzp from field on grid bz_arr + for (int iz=0; iz<=depos_order; iz++){ + for (int ix=0; ix<=depos_order-galerkin_interpolation; ix++){ + Erp += sx_ex[ix]*sz_ex[iz]* + ex_arr(lo.x+j_ex+ix, lo.y+l_ex+iz, 0, 0); + Bzp += sx_bz[ix]*sz_bz[iz]* + bz_arr(lo.x+j_bz+ix, lo.y+l_bz+iz, 0, 0); + } + } + // Gather field on particle Ezp from field on grid ez_arr + // Gather field on particle Brp from field on grid bx_arr + for (int iz=0; iz<=depos_order-galerkin_interpolation; iz++){ + for (int ix=0; ix<=depos_order; ix++){ + Ezp += sx_ez[ix]*sz_ez[iz]* + ez_arr(lo.x+j_ez+ix, lo.y+l_ez+iz, 0, 0); + Brp += sx_bx[ix]*sz_bx[iz]* + bx_arr(lo.x+j_bx+ix, lo.y+l_bx+iz, 0, 0); + } + } + // Gather field on particle Bthetap from field on grid by_arr + for (int iz=0; iz<=depos_order-galerkin_interpolation; iz++){ + for (int ix=0; ix<=depos_order-galerkin_interpolation; ix++){ + Bthetap += sx_by[ix]*sz_by[iz]* + by_arr(lo.x+j_by+ix, lo.y+l_by+iz, 0, 0); + } + } amrex::Real costheta; amrex::Real sintheta; @@ -304,28 +344,28 @@ void doGatherShapeN (const amrex::ParticleReal xp, for (int imode=1 ; imode < n_rz_azimuthal_modes ; imode++) { - // Gather field on particle Eyp from field on grid ey_arr + // Gather field on particle Ethetap from field on grid ey_arr for (int iz=0; iz<=depos_order; iz++){ for (int ix=0; ix<=depos_order; ix++){ const amrex::Real dEy = (+ ey_arr(lo.x+j_ey+ix, lo.y+l_ey+iz, 0, 2*imode-1)*xy.real() - ey_arr(lo.x+j_ey+ix, lo.y+l_ey+iz, 0, 2*imode)*xy.imag()); - Eyp += sx_ey[ix]*sz_ey[iz]*dEy; + Ethetap += sx_ey[ix]*sz_ey[iz]*dEy; } } - // Gather field on particle Exp from field on grid ex_arr + // Gather field on particle Erp from field on grid ex_arr // Gather field on particle Bzp from field on grid bz_arr for (int iz=0; iz<=depos_order; iz++){ for (int ix=0; ix<=depos_order-galerkin_interpolation; ix++){ const amrex::Real dEx = (+ ex_arr(lo.x+j_ex+ix, lo.y+l_ex+iz, 0, 2*imode-1)*xy.real() - ex_arr(lo.x+j_ex+ix, lo.y+l_ex+iz, 0, 2*imode)*xy.imag()); - Exp += sx_ex[ix]*sz_ex[iz]*dEx; + Erp += sx_ex[ix]*sz_ex[iz]*dEx; const amrex::Real dBz = (+ bz_arr(lo.x+j_bz+ix, lo.y+l_bz+iz, 0, 2*imode-1)*xy.real() - bz_arr(lo.x+j_bz+ix, lo.y+l_bz+iz, 0, 2*imode)*xy.imag()); Bzp += sx_bz[ix]*sz_bz[iz]*dBz; } } // Gather field on particle Ezp from field on grid ez_arr - // Gather field on particle Bxp from field on grid bx_arr + // Gather field on particle Brp from field on grid bx_arr for (int iz=0; iz<=depos_order-galerkin_interpolation; iz++){ for (int ix=0; ix<=depos_order; ix++){ const amrex::Real dEz = (+ ez_arr(lo.x+j_ez+ix, lo.y+l_ez+iz, 0, 2*imode-1)*xy.real() @@ -333,28 +373,25 @@ void doGatherShapeN (const amrex::ParticleReal xp, Ezp += sx_ez[ix]*sz_ez[iz]*dEz; const amrex::Real dBx = (+ bx_arr(lo.x+j_bx+ix, lo.y+l_bx+iz, 0, 2*imode-1)*xy.real() - bx_arr(lo.x+j_bx+ix, lo.y+l_bx+iz, 0, 2*imode)*xy.imag()); - Bxp += sx_bx[ix]*sz_bx[iz]*dBx; + Brp += sx_bx[ix]*sz_bx[iz]*dBx; } } - // Gather field on particle Byp from field on grid by_arr + // Gather field on particle Bthetap from field on grid by_arr for (int iz=0; iz<=depos_order-galerkin_interpolation; iz++){ for (int ix=0; ix<=depos_order-galerkin_interpolation; ix++){ const amrex::Real dBy = (+ by_arr(lo.x+j_by+ix, lo.y+l_by+iz, 0, 2*imode-1)*xy.real() - by_arr(lo.x+j_by+ix, lo.y+l_by+iz, 0, 2*imode)*xy.imag()); - Byp += sx_by[ix]*sz_by[iz]*dBy; + Bthetap += sx_by[ix]*sz_by[iz]*dBy; } } xy = xy*xy0; } - // Convert Exp and Eyp (which are actually Er and Etheta) to Ex and Ey - const amrex::Real Exp_save = Exp; - Exp = costheta*Exp - sintheta*Eyp; - Eyp = costheta*Eyp + sintheta*Exp_save; - const amrex::Real Bxp_save = Bxp; - Bxp = costheta*Bxp - sintheta*Byp; - Byp = costheta*Byp + sintheta*Bxp_save; -#endif + // Convert Erp and Ethetap to Ex and Ey + Exp += costheta*Erp - sintheta*Ethetap; + Eyp += costheta*Ethetap + sintheta*Erp; + Bxp += costheta*Brp - sintheta*Bthetap; + Byp += costheta*Bthetap + sintheta*Brp; #else // defined(WARPX_DIM_3D) // Gather field on particle Exp from field on grid ex_arr From fb723b95b5883ccf269bfc5f1a57ba2d48957a6c Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Wed, 8 Nov 2023 10:02:30 -0800 Subject: [PATCH 085/110] Release 23.11 (#4410) * AMReX: 23.11 * pyAMReX: 23.11 * WarpX: 23.11 --- .github/workflows/cuda.yml | 2 +- CMakeLists.txt | 2 +- Docs/source/conf.py | 4 ++-- Python/setup.py | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- Tools/Release/updatepyAMReX.py | 2 +- cmake/dependencies/AMReX.cmake | 4 ++-- cmake/dependencies/pyAMReX.cmake | 4 ++-- run_test.sh | 2 +- setup.py | 2 +- 11 files changed, 14 insertions(+), 14 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 2efda054837..71f25b9b31a 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -108,7 +108,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach be6c6415467d09da6109d27cfa218868abc1f9db && cd - + cd ../amrex && git checkout --detach 23.11 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/CMakeLists.txt b/CMakeLists.txt index 6bc8fe8dd55..b2744d9c768 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,7 +1,7 @@ # Preamble #################################################################### # cmake_minimum_required(VERSION 3.20.0) -project(WarpX VERSION 23.10) +project(WarpX VERSION 23.11) include(${WarpX_SOURCE_DIR}/cmake/WarpXFunctions.cmake) diff --git a/Docs/source/conf.py b/Docs/source/conf.py index f9e06bcc8c3..f6fa22f40f8 100644 --- a/Docs/source/conf.py +++ b/Docs/source/conf.py @@ -79,9 +79,9 @@ # built documents. # # The short X.Y version. -version = u'23.10' +version = u'23.11' # The full version, including alpha/beta/rc tags. -release = u'23.10' +release = u'23.11' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/Python/setup.py b/Python/setup.py index e5165b17c41..62294fcaae2 100644 --- a/Python/setup.py +++ b/Python/setup.py @@ -54,7 +54,7 @@ package_data = {} setup(name = 'pywarpx', - version = '23.10', + version = '23.11', packages = ['pywarpx'], package_dir = {'pywarpx': 'pywarpx'}, description = """Wrapper of WarpX""", diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index 1e5f2a7a619..e07dfdc68da 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = be6c6415467d09da6109d27cfa218868abc1f9db +branch = 23.11 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 14896aff17e..48cf32d6bf7 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = be6c6415467d09da6109d27cfa218868abc1f9db +branch = 23.11 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/Tools/Release/updatepyAMReX.py b/Tools/Release/updatepyAMReX.py index 94fde0df1cc..8ba10c895e0 100755 --- a/Tools/Release/updatepyAMReX.py +++ b/Tools/Release/updatepyAMReX.py @@ -55,7 +55,7 @@ # find_package(AMReX YY.MM CONFIG ... pyamrex_minimal = f"unknown (format issue in {pyamrex_cmake_path})" with open(pyamrex_cmake_path, encoding='utf-8') as f: - r_minimal = re.findall(r'.*find_package\(AMReX\s+(.+)\s+CONFIG\s+.*', + r_minimal = re.findall(r'.*find_package\(pyAMReX\s+(.+)\s+CONFIG\s+.*', f.read(), re.MULTILINE) if len(r_minimal) >= 1: pyamrex_minimal = r_minimal[0] diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index c03ea1ba828..d0ff04f8526 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -250,7 +250,7 @@ macro(find_amrex) endif() set(COMPONENT_PRECISION ${WarpX_PRECISION} P${WarpX_PARTICLE_PRECISION}) - find_package(AMReX 23.10 CONFIG REQUIRED COMPONENTS ${COMPONENT_ASCENT} ${COMPONENT_DIMS} ${COMPONENT_EB} PARTICLES ${COMPONENT_PIC} ${COMPONENT_PRECISION} ${COMPONENT_SENSEI} LSOLVERS) + find_package(AMReX 23.11 CONFIG REQUIRED COMPONENTS ${COMPONENT_ASCENT} ${COMPONENT_DIMS} ${COMPONENT_EB} PARTICLES ${COMPONENT_PIC} ${COMPONENT_PRECISION} ${COMPONENT_SENSEI} LSOLVERS) # note: TINYP skipped because user-configured and optional # AMReX CMake helper scripts @@ -269,7 +269,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "be6c6415467d09da6109d27cfa218868abc1f9db" +set(WarpX_amrex_branch "23.11" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/cmake/dependencies/pyAMReX.cmake b/cmake/dependencies/pyAMReX.cmake index 13b84ee3522..e7202e5dd3b 100644 --- a/cmake/dependencies/pyAMReX.cmake +++ b/cmake/dependencies/pyAMReX.cmake @@ -64,7 +64,7 @@ function(find_pyamrex) endif() elseif(NOT WarpX_pyamrex_internal) # TODO: MPI control - find_package(pyAMReX 23.10 CONFIG REQUIRED) + find_package(pyAMReX 23.11 CONFIG REQUIRED) message(STATUS "pyAMReX: Found version '${pyAMReX_VERSION}'") endif() endfunction() @@ -79,7 +79,7 @@ option(WarpX_pyamrex_internal "Download & build pyAMReX" ON) set(WarpX_pyamrex_repo "https://github.com/AMReX-Codes/pyamrex.git" CACHE STRING "Repository URI to pull and build pyamrex from if(WarpX_pyamrex_internal)") -set(WarpX_pyamrex_branch "056d332d68fa98b9e9509d05ec3b1a3d8ef37673" +set(WarpX_pyamrex_branch "23.11" CACHE STRING "Repository branch for WarpX_pyamrex_repo if(WarpX_pyamrex_internal)") diff --git a/run_test.sh b/run_test.sh index f683c5142f6..47992ed006a 100755 --- a/run_test.sh +++ b/run_test.sh @@ -68,7 +68,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach be6c6415467d09da6109d27cfa218868abc1f9db && cd - +cd amrex && git checkout --detach 23.11 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets diff --git a/setup.py b/setup.py index e591bc9a506..bb92d69b417 100644 --- a/setup.py +++ b/setup.py @@ -278,7 +278,7 @@ def build_extension(self, ext): setup( name='pywarpx', # note PEP-440 syntax: x.y.zaN but x.y.z.devN - version = '23.10', + version = '23.11', packages = ['pywarpx'], package_dir = {'pywarpx': 'Python/pywarpx'}, author='Jean-Luc Vay, David P. Grote, Maxence Thévenet, Rémi Lehe, Andrew Myers, Weiqun Zhang, Axel Huebl, et al.', From ce82a694bcf0bdcb425ad0ad73c9658430ea9663 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 9 Nov 2023 15:09:27 -0800 Subject: [PATCH 086/110] AMReX/pyAMReX/PICSAR: Weekly Update (#4417) * AMReX: Weekly Update * pyAMReX: Weekly Update * PICSAR: Weekly Update --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- cmake/dependencies/PICSAR.cmake | 4 ++-- cmake/dependencies/pyAMReX.cmake | 2 +- run_test.sh | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 71f25b9b31a..6b810e1cbb0 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -108,7 +108,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 23.11 && cd - + cd ../amrex && git checkout --detach d36463103daed09a40cdea235041a6ab79ff280c && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index e07dfdc68da..c9ddc4e377d 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 23.11 +branch = d36463103daed09a40cdea235041a6ab79ff280c [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 48cf32d6bf7..1dd8ac3366d 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 23.11 +branch = d36463103daed09a40cdea235041a6ab79ff280c [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index d0ff04f8526..488310b7cc2 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -269,7 +269,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "23.11" +set(WarpX_amrex_branch "d36463103daed09a40cdea235041a6ab79ff280c" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/cmake/dependencies/PICSAR.cmake b/cmake/dependencies/PICSAR.cmake index 8400dc06a4b..a1e93a9e33c 100644 --- a/cmake/dependencies/PICSAR.cmake +++ b/cmake/dependencies/PICSAR.cmake @@ -85,7 +85,7 @@ function(find_picsar) #message(STATUS "PICSAR: Using version '${PICSAR_VERSION}'") else() # not supported by PICSAR (yet) - #find_package(PICSAR 23.09 CONFIG REQUIRED QED) + #find_package(PICSAR 23.11 CONFIG REQUIRED QED) #message(STATUS "PICSAR: Found version '${PICSAR_VERSION}'") message(FATAL_ERROR "PICSAR: Cannot be used as externally installed " "library yet. " @@ -106,7 +106,7 @@ if(WarpX_QED) set(WarpX_picsar_repo "https://github.com/ECP-WarpX/picsar.git" CACHE STRING "Repository URI to pull and build PICSAR from if(WarpX_picsar_internal)") - set(WarpX_picsar_branch "23.09" + set(WarpX_picsar_branch "aa54e985398c1d575abc7e6737cdbc660a13765f" CACHE STRING "Repository branch for WarpX_picsar_repo if(WarpX_picsar_internal)") diff --git a/cmake/dependencies/pyAMReX.cmake b/cmake/dependencies/pyAMReX.cmake index e7202e5dd3b..20232969f75 100644 --- a/cmake/dependencies/pyAMReX.cmake +++ b/cmake/dependencies/pyAMReX.cmake @@ -79,7 +79,7 @@ option(WarpX_pyamrex_internal "Download & build pyAMReX" ON) set(WarpX_pyamrex_repo "https://github.com/AMReX-Codes/pyamrex.git" CACHE STRING "Repository URI to pull and build pyamrex from if(WarpX_pyamrex_internal)") -set(WarpX_pyamrex_branch "23.11" +set(WarpX_pyamrex_branch "d90e390971de4144fb380ae62d5c74cdc27dd3fc" CACHE STRING "Repository branch for WarpX_pyamrex_repo if(WarpX_pyamrex_internal)") diff --git a/run_test.sh b/run_test.sh index 47992ed006a..e6ab5b166ca 100755 --- a/run_test.sh +++ b/run_test.sh @@ -68,7 +68,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 23.11 && cd - +cd amrex && git checkout --detach d36463103daed09a40cdea235041a6ab79ff280c && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 727429c360bda1c7dc67c8625389eec98abe3a27 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 16 Nov 2023 13:28:02 -0800 Subject: [PATCH 087/110] CI: Unbreak macOS (#4427) Python: Use a fresh Virtualenv Brew Python is too broken/inconsistent in packages like `six`, `matplotlib`, `pandas` et al. --- .github/workflows/macos.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml index fda2cb77a51..68ea4737107 100644 --- a/.github/workflows/macos.yml +++ b/.github/workflows/macos.yml @@ -48,6 +48,15 @@ jobs: set -e brew tap openpmd/openpmd brew install openpmd-api + + python3 -m pip install --upgrade pip + python3 -m pip install --upgrade virtualenv + + python3 -m venv py-venv + source py-venv/bin/activate + python3 -m pip install --upgrade pip + python3 -m pip install --upgrade build packaging setuptools wheel + python3 -m pip install --upgrade mpi4py - name: CCache Cache uses: actions/cache@v3 # - once stored under a key, they become immutable (even if local cache path content changes) @@ -60,8 +69,7 @@ jobs: ccache-macos-appleclang- - name: build WarpX run: | - python3 -m pip install --upgrade pip - python3 -m pip install --upgrade build packaging setuptools wheel + source py-venv/bin/activate cmake -S . -B build_dp \ -DCMAKE_VERBOSE_MAKEFILE=ON \ @@ -83,5 +91,7 @@ jobs: - name: run pywarpx run: | + source py-venv/bin/activate export OMP_NUM_THREADS=1 + mpirun -n 2 Examples/Physics_applications/laser_acceleration/PICMI_inputs_3d.py From 01c55bf537d26b7e6c48c685007dc9b4608a3bb5 Mon Sep 17 00:00:00 2001 From: David Grote Date: Fri, 17 Nov 2023 08:02:21 -0800 Subject: [PATCH 088/110] Update AMReX to latest commit (#4423) --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- run_test.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 6b810e1cbb0..2bad793571e 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -108,7 +108,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach d36463103daed09a40cdea235041a6ab79ff280c && cd - + cd ../amrex && git checkout --detach 15a0bb9a8c1b34136632b16c5511375e9d56b184 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index c9ddc4e377d..5184106415c 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = d36463103daed09a40cdea235041a6ab79ff280c +branch = 15a0bb9a8c1b34136632b16c5511375e9d56b184 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index 1dd8ac3366d..d913684ae7c 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = d36463103daed09a40cdea235041a6ab79ff280c +branch = 15a0bb9a8c1b34136632b16c5511375e9d56b184 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 488310b7cc2..0a108a1b36d 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -269,7 +269,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "d36463103daed09a40cdea235041a6ab79ff280c" +set(WarpX_amrex_branch "15a0bb9a8c1b34136632b16c5511375e9d56b184" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/run_test.sh b/run_test.sh index e6ab5b166ca..d861dfb9cd4 100755 --- a/run_test.sh +++ b/run_test.sh @@ -68,7 +68,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach d36463103daed09a40cdea235041a6ab79ff280c && cd - +cd amrex && git checkout --detach 15a0bb9a8c1b34136632b16c5511375e9d56b184 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From d0b88713719b30f7e6f6f5295d4ede7d3a7caa56 Mon Sep 17 00:00:00 2001 From: David Grote Date: Fri, 17 Nov 2023 08:05:17 -0800 Subject: [PATCH 089/110] Remove legacy references to EvolveEM which had been renamed Evolve (#4422) --- Docs/source/developers/fields.rst | 2 +- Docs/source/developers/repo_organization.rst | 2 +- Docs/source/glossary.rst | 2 +- Source/WarpX.H | 5 ----- Tools/PerformanceTests/functions_perftest.py | 2 +- Tools/PerformanceTests/performance_log.txt | 2 +- Tools/PerformanceTests/run_alltests.py | 2 +- Tools/PerformanceTests/run_alltests_1node.py | 4 ++-- 8 files changed, 8 insertions(+), 13 deletions(-) diff --git a/Docs/source/developers/fields.rst b/Docs/source/developers/fields.rst index af834354dcd..9ed790e4562 100644 --- a/Docs/source/developers/fields.rst +++ b/Docs/source/developers/fields.rst @@ -40,7 +40,7 @@ By default, the ``MultiFab`` are set to ``0`` at initialization. They can be ass Field solver ------------ -The field solver is performed in ``WarpX::EvolveE`` for the electric field and ``WarpX::EvolveB`` for the magnetic field, called from ``WarpX::OneStep_nosub`` in ``WarpX::EvolveEM``. This section describes the FDTD field push. It is implemented in ``Source/FieldSolver/FiniteDifferenceSolver/``. +The field solver is performed in ``WarpX::EvolveE`` for the electric field and ``WarpX::EvolveB`` for the magnetic field, called from ``WarpX::OneStep_nosub`` in ``WarpX::Evolve``. This section describes the FDTD field push. It is implemented in ``Source/FieldSolver/FiniteDifferenceSolver/``. As all cell-wise operation, the field push is done as follows (this is split in multiple functions in the actual implementation to avoid code duplication) : diff --git a/Docs/source/developers/repo_organization.rst b/Docs/source/developers/repo_organization.rst index b8a964f9839..5eaed5fcce5 100644 --- a/Docs/source/developers/repo_organization.rst +++ b/Docs/source/developers/repo_organization.rst @@ -8,7 +8,7 @@ Repo Organization All the WarpX source code is located in ``Source/``. All sub-directories have a pretty straightforward name. -The PIC loop is part of the WarpX class, in function ``WarpX::EvolveEM`` implemented in ``Source/WarpXEvolveEM.cpp``. +The PIC loop is part of the WarpX class, in function ``WarpX::Evolve`` implemented in ``Source/WarpXEvolve.cpp``. The core of the PIC loop (i.e., without diagnostics etc.) is in ``WarpX::OneStep_nosub`` (when subcycling is OFF) or ``WarpX::OneStep_sub1`` (when subcycling is ON, with method 1). Here is a `visual representation `__ of the repository structure. diff --git a/Docs/source/glossary.rst b/Docs/source/glossary.rst index 131b3a63ae7..8329159a6ba 100644 --- a/Docs/source/glossary.rst +++ b/Docs/source/glossary.rst @@ -69,7 +69,7 @@ Terms * **Ascent:** `many-core capable flyweight in situ visualization and analysis infrastructure `__, a visualization backend usable with WarpX data * **boosted frame:** a :ref:`Lorentz-boosted frame of reference ` for a simulation * **evolve:** this is a generic term to advance a quantity (same nomenclature in AMReX). - For instance, ``WarpX::EvolveE(dt)`` advances the electric field for duration ``dt``, ``PhysicalParticleContainer::Evolve(...)`` does field gather + particle push + current deposition for all particles in ``PhysicalParticleContainer``, and ``WarpX::EvolveEM`` is the central ``WarpX`` function that performs 1 PIC iteration. + For instance, ``WarpX::EvolveE(dt)`` advances the electric field for duration ``dt``, ``PhysicalParticleContainer::Evolve(...)`` does field gather + particle push + current deposition for all particles in ``PhysicalParticleContainer``, and ``WarpX::Evolve`` is the central ``WarpX`` function that performs 1 PIC iteration. * **Frontier:** an `Exascale supercomputer at OLCF `__ * **hybrid-PIC:** a plasma simulation scheme that combines fluid and kinetic approaches, with (usually) the electrons treated as a fluid and the ions as kinetic particles (see :ref:`theory-kinetic-fluid-hybrid-model`) * **laser:** most of the time, we mean a `laser pulse `__ diff --git a/Source/WarpX.H b/Source/WarpX.H index cd05d90555a..94b06d1665a 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -1220,11 +1220,6 @@ private: //! Complete the asynchronous broadcast of signal flags, and initiate a checkpoint if requested void HandleSignals (); - /// - /// Advance the simulation by numsteps steps, electromagnetic case. - /// - void EvolveEM(int numsteps); - void FillBoundaryB (int lev, PatchType patch_type, amrex::IntVect ng, std::optional nodal_sync = std::nullopt); void FillBoundaryE (int lev, PatchType patch_type, amrex::IntVect ng, std::optional nodal_sync = std::nullopt); void FillBoundaryF (int lev, PatchType patch_type, amrex::IntVect ng, std::optional nodal_sync = std::nullopt); diff --git a/Tools/PerformanceTests/functions_perftest.py b/Tools/PerformanceTests/functions_perftest.py index 938bcb974de..311bdb68f1f 100644 --- a/Tools/PerformanceTests/functions_perftest.py +++ b/Tools/PerformanceTests/functions_perftest.py @@ -154,7 +154,7 @@ def read_run_perf(filename, n_steps): '\nPPC::FieldGather.*',\ '\nPPC::ParticlePush.*',\ '\nPPC::Evolve::Copy.*',\ - '\nWarpX::EvolveEM().*',\ + '\nWarpX::Evolve().*',\ 'Checkpoint().*',\ 'WriteParticles().*',\ '\nVisMF::Write(FabArray).*',\ diff --git a/Tools/PerformanceTests/performance_log.txt b/Tools/PerformanceTests/performance_log.txt index 7f2d2453466..72fece34939 100644 --- a/Tools/PerformanceTests/performance_log.txt +++ b/Tools/PerformanceTests/performance_log.txt @@ -1,4 +1,4 @@ -## year month day run_name compiler architecture n_node n_mpi n_omp time_initialization time_one_iteration Redistribute FillBoundary ParallelCopy CurrentDeposition FieldGather ParthiclePush Copy EvolveEM Checkpoint WriteParticles Write_FabArray WriteMultiLevelPlotfile(unit: second) RedistributeMPI +## year month day run_name compiler architecture n_node n_mpi n_omp time_initialization time_one_iteration Redistribute FillBoundary ParallelCopy CurrentDeposition FieldGather ParthiclePush Copy Evolve Checkpoint WriteParticles Write_FabArray WriteMultiLevelPlotfile(unit: second) RedistributeMPI 2018 01 31 automated_test_1_uniform_rest_32ppc intel knl 1 16 8 3.14 0.3986 0.1713 0.01719 0.01615 0.06987 0.03636 0.01901 0.01999 0.003602 0 0 0 0 0.007262 2018 01 31 automated_test_1_uniform_rest_32ppc intel knl 1 16 8 3.39 0.4009 0.1712 0.01676 0.01583 0.07061 0.03684 0.01926 0.02011 0.003687 0 0 0 0 0.007841 2018 01 31 automated_test_1_uniform_rest_32ppc intel knl 1 16 8 2.91 0.4024 0.1716 0.01826 0.01918 0.0703 0.0363 0.01912 0.01989 0.003017 0 0 0 0 0.007256 diff --git a/Tools/PerformanceTests/run_alltests.py b/Tools/PerformanceTests/run_alltests.py index af99df194e2..e9ff899fd2a 100644 --- a/Tools/PerformanceTests/run_alltests.py +++ b/Tools/PerformanceTests/run_alltests.py @@ -259,7 +259,7 @@ def process_analysis(): log_line = '## year month day run_name compiler architecture n_node n_mpi ' +\ 'n_omp time_initialization time_one_iteration Redistribute '+\ 'FillBoundary ParallelCopy CurrentDeposition FieldGather '+\ - 'ParthiclePush Copy EvolveEM Checkpoint '+\ + 'ParthiclePush Copy Evolve Checkpoint '+\ 'WriteParticles Write_FabArray '+\ 'WriteMultiLevelPlotfile '+\ 'RedistributeMPI(unit: second)\n' diff --git a/Tools/PerformanceTests/run_alltests_1node.py b/Tools/PerformanceTests/run_alltests_1node.py index 54900e9e06e..f2ebb73124e 100644 --- a/Tools/PerformanceTests/run_alltests_1node.py +++ b/Tools/PerformanceTests/run_alltests_1node.py @@ -228,7 +228,7 @@ def process_analysis(): log_line = '## year month day input_file compiler architecture n_node n_mpi ' +\ 'n_omp time_initialization time_one_iteration Redistribute '+\ 'FillBoundary ParallelCopy CurrentDeposition FieldGather '+\ - 'ParthiclePush Copy EvolveEM Checkpoint '+\ + 'ParthiclePush Copy Evolve Checkpoint '+\ 'WriteParticles Write_FabArray '+\ 'WriteMultiLevelPlotfile(unit: second) '+\ 'RedistributeMPI\n' @@ -318,7 +318,7 @@ def process_analysis(): nrepeat = 4 legends = [ 'n_node', 'n_mpi', 'n_omp', 'time_initialization', 'time_one_iteration', \ 'Redistribute', 'FillBoundary', 'ParallelCopy', 'CurrentDeposition', \ - 'FieldGather', 'ParthiclePush', 'Copy', 'EvolveEM', 'Checkpoint', \ + 'FieldGather', 'ParthiclePush', 'Copy', 'Evolve', 'Checkpoint', \ 'WriteParticles', 'Write_FabArray', 'WriteMultiLevelPlotfile', \ 'RedistributeMPI'] date = np.loadtxt( filename, usecols = np.arange(0, 3 )) From 5f0cfb455a8a761d9088956d2d68d4d7b6d3230a Mon Sep 17 00:00:00 2001 From: Eya D <81635404+EyaDammak@users.noreply.github.com> Date: Fri, 17 Nov 2023 08:05:53 -0800 Subject: [PATCH 090/110] Parallelize scraping (#4418) * 2 modifs * 1 modif --- Source/EmbeddedBoundary/ParticleScraper.H | 3 +++ Source/Particles/ParticleBoundaryBuffer.cpp | 6 ++++++ 2 files changed, 9 insertions(+) diff --git a/Source/EmbeddedBoundary/ParticleScraper.H b/Source/EmbeddedBoundary/ParticleScraper.H index f1aaf544b32..d6196c35f44 100644 --- a/Source/EmbeddedBoundary/ParticleScraper.H +++ b/Source/EmbeddedBoundary/ParticleScraper.H @@ -161,6 +161,9 @@ scrapeParticles (PC& pc, const amrex::Vector& distance_t { const auto plo = pc.Geom(lev).ProbLoArray(); const auto dxi = pc.Geom(lev).InvCellSizeArray(); +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif for(WarpXParIter pti(pc, lev); pti.isValid(); ++pti) { const auto getPosition = GetParticlePosition(pti); diff --git a/Source/Particles/ParticleBoundaryBuffer.cpp b/Source/Particles/ParticleBoundaryBuffer.cpp index b2dfb2f03ef..b6eba51347c 100644 --- a/Source/Particles/ParticleBoundaryBuffer.cpp +++ b/Source/Particles/ParticleBoundaryBuffer.cpp @@ -244,6 +244,9 @@ void ParticleBoundaryBuffer::gatherParticles (MultiParticleContainer& mypc, for (int lev = 0; lev < pc.numLevels(); ++lev) { const auto& plevel = pc.GetParticles(lev); +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif for(PIter pti(pc, lev); pti.isValid(); ++pti) { auto index = std::make_pair(pti.index(), pti.LocalTileIndex()); @@ -307,6 +310,9 @@ void ParticleBoundaryBuffer::gatherParticles (MultiParticleContainer& mypc, { const auto& plevel = pc.GetParticles(lev); auto dxi = warpx_instance.Geom(lev).InvCellSizeArray(); +#ifdef AMREX_USE_OMP +#pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) +#endif for(PIter pti(pc, lev); pti.isValid(); ++pti) { auto phiarr = (*distance_to_eb[lev])[pti].array(); // signed distance function From 169887a508551115cb9ca1513a6fa0772f2b0298 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 17 Nov 2023 08:06:20 -0800 Subject: [PATCH 091/110] Doc: Laser Pulse Manip. for LPI (#4414) * Doc: Laser Pulse Manip. for LPI Add commonly used acronyms for laser pulse manipulation and instability mitigation in laser-plasma interaction physics. * Update glossary.rst --- Docs/source/glossary.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Docs/source/glossary.rst b/Docs/source/glossary.rst index 8329159a6ba..b78d2446de8 100644 --- a/Docs/source/glossary.rst +++ b/Docs/source/glossary.rst @@ -34,6 +34,7 @@ Abbreviations * **GPU:** originally graphics processing unit, now used for fast `general purpose computing (GPGPU) `__; also called (hardware) accelerator * **IO:** input/output, usually files and/or data * **IPO:** `interprocedural optimization `__, a collection of compiler optimization techniques that analyze the whole code to avoid duplicate calculations and optimize performance +* **ISI:** Induced Spectral Incoherence (a laser pulse manipulation technique) * **LDRD:** Laboratory Directed Research and Development, a :ref:`funding program in U.S. DOE laboratories ` that kick-started ABLASTR development * **LPA:** laser-plasma acceleration, historically used for laser-electron acceleration * **LPI:** laser-plasma interaction (often for laser-solid physics) *or* laser-plasma instability (often in fusion physics), depending on context @@ -56,9 +57,11 @@ Abbreviations * **PWFA:** plasma-wakefield acceleration * **QED:** `quantum electrodynamics `__ * **RPA:** radiation-pressure acceleration (of protons/ions), e.g. hole-boring (HB) or light-sail (LS) acceleration +* **RPP:** Random Phase Plate (a laser pulse manipulation technique) * **RZ:** for the coordinate system ``r-z`` in cylindrical geometry; we use "RZ" when we refer to quasi-cylindrical geometry, decomposed in azimuthal modes (see details `here `__) * **SENSEI:** `Scalable in situ analysis and visualization `__, light weight framework for in situ data analysis offering access to multiple visualization and analysis backends * **SEE:** secondary electron emission +* **SSD:** Smoothing by Spectral Dispersion (a laser pulse manipulation technique) * **TNSA:** target-normal sheet acceleration (of protons/ions) Terms From dff22613e0b47a9fb998c1823acf992b3c4adfbb Mon Sep 17 00:00:00 2001 From: David Grote Date: Fri, 17 Nov 2023 08:07:16 -0800 Subject: [PATCH 092/110] Fix unused variable with OpenPMD=OFF (#4421) --- Source/Initialization/PlasmaInjector.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Source/Initialization/PlasmaInjector.cpp b/Source/Initialization/PlasmaInjector.cpp index 0486f6e9537..20add644176 100644 --- a/Source/Initialization/PlasmaInjector.cpp +++ b/Source/Initialization/PlasmaInjector.cpp @@ -437,11 +437,11 @@ void PlasmaInjector::setupExternalFile (const amrex::ParmParse& pp_species_name) utils::parser::queryWithParser(pp_species_name, "q_tot", q_tot); utils::parser::queryWithParser(pp_species_name, "z_shift",z_shift); +#ifdef WARPX_USE_OPENPMD const bool charge_is_specified = pp_species_name.contains("charge"); const bool mass_is_specified = pp_species_name.contains("mass"); const bool species_is_specified = pp_species_name.contains("species_type"); -#ifdef WARPX_USE_OPENPMD if (amrex::ParallelDescriptor::IOProcessor()) { m_openpmd_input_series = std::make_unique( str_injection_file, openPMD::Access::READ_ONLY); From d0db41a94b7805ca87d36097a7fb1f17be6a7e4e Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 20 Nov 2023 00:07:54 -0800 Subject: [PATCH 093/110] AMReX: Weekly Update (#4431) --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- run_test.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 2bad793571e..051ff0ea758 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -108,7 +108,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 15a0bb9a8c1b34136632b16c5511375e9d56b184 && cd - + cd ../amrex && git checkout --detach 175b99d913dc2748e43c53192737170c770fe0e8 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index 5184106415c..264106eb8ec 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 15a0bb9a8c1b34136632b16c5511375e9d56b184 +branch = 175b99d913dc2748e43c53192737170c770fe0e8 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index d913684ae7c..d67a0a03569 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 15a0bb9a8c1b34136632b16c5511375e9d56b184 +branch = 175b99d913dc2748e43c53192737170c770fe0e8 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 0a108a1b36d..7aea38148ea 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -269,7 +269,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "15a0bb9a8c1b34136632b16c5511375e9d56b184" +set(WarpX_amrex_branch "175b99d913dc2748e43c53192737170c770fe0e8" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/run_test.sh b/run_test.sh index d861dfb9cd4..b548f11e93f 100755 --- a/run_test.sh +++ b/run_test.sh @@ -68,7 +68,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 15a0bb9a8c1b34136632b16c5511375e9d56b184 && cd - +cd amrex && git checkout --detach 175b99d913dc2748e43c53192737170c770fe0e8 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From 3089349470dc0bf6e0e57eb97810380c60ae8b33 Mon Sep 17 00:00:00 2001 From: Remi Lehe Date: Mon, 20 Nov 2023 00:11:39 -0800 Subject: [PATCH 094/110] Add paper using WarpX for laser-driven fusion (#4428) --- Docs/source/highlights.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Docs/source/highlights.rst b/Docs/source/highlights.rst index 0f9bee1808a..420bb376d67 100644 --- a/Docs/source/highlights.rst +++ b/Docs/source/highlights.rst @@ -70,6 +70,11 @@ Laser-Plasma Interaction Scientific works in laser-ion acceleration and laser-matter interaction. +#. Knight B, Gautam C, Stoner C, Egner B, Smith J, Orban C, Manfredi J, Frische K, Dexter M, Chowdhury E, Patnaik A (2023). + **Detailed Characterization of a kHz-rate Laser-Driven Fusion at a Thin Liquid Sheet with a Neutron Detection Suite**. + High Power Laser Science and Engineering, 1-13, 2023. + `DOI:10.1017/hpl.2023.84 `__ + #. Fedeli L, Huebl A, Boillod-Cerneux F, Clark T, Gott K, Hillairet C, Jaure S, Leblanc A, Lehe R, Myers A, Piechurski C, Sato M, Zaim N, Zhang W, Vay J-L, Vincenti H. **Pushing the Frontier in the Design of Laser-Based Electron Accelerators with Groundbreaking Mesh-Refined Particle-In-Cell Simulations on Exascale-Class Supercomputers**. *SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (SC)*. ISSN:2167-4337, pp. 25-36, Dallas, TX, US, 2022. From 86d3ae68b69e4a729c33f242ba223aae6d142e05 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 20 Nov 2023 16:22:38 -0800 Subject: [PATCH 095/110] [pre-commit.ci] pre-commit autoupdate (#4433) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/hadialqattan/pycln: v2.3.0 → v2.4.0](https://github.com/hadialqattan/pycln/compare/v2.3.0...v2.4.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b514758caba..eb298d39b68 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -68,7 +68,7 @@ repos: # Autoremoves unused Python imports - repo: https://github.com/hadialqattan/pycln - rev: v2.3.0 + rev: v2.4.0 hooks: - id: pycln name: pycln (python) From 0f755e195d4f560f33678cc43d30a4b8e5a82885 Mon Sep 17 00:00:00 2001 From: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Mon, 20 Nov 2023 16:23:12 -0800 Subject: [PATCH 096/110] Add python binding to `WarpXParticleContainer::sumParticleCharge` (#4406) --- Python/pywarpx/particle_containers.py | 5 +---- Source/Python/Particles/WarpXParticleContainer.cpp | 4 ++++ 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Python/pywarpx/particle_containers.py b/Python/pywarpx/particle_containers.py index c77781f6544..cece7968079 100644 --- a/Python/pywarpx/particle_containers.py +++ b/Python/pywarpx/particle_containers.py @@ -618,10 +618,7 @@ def get_species_charge_sum(self, local=False): local : bool If True return total charge per processor ''' - raise NotImplementedError() - return self.libwarpx_so.warpx_sumParticleCharge( - ctypes.c_char_p(species_name.encode('utf-8')), local - ) + return self.particle_container.sum_particle_charge(local) def getex(self): diff --git a/Source/Python/Particles/WarpXParticleContainer.cpp b/Source/Python/Particles/WarpXParticleContainer.cpp index 386225c8968..80bb68cfa5b 100644 --- a/Source/Python/Particles/WarpXParticleContainer.cpp +++ b/Source/Python/Particles/WarpXParticleContainer.cpp @@ -102,6 +102,10 @@ void init_WarpXParticleContainer (py::module& m) &WarpXParticleContainer::TotalNumberOfParticles, py::arg("valid_particles_only"), py::arg("local") ) + .def("sum_particle_charge", + &WarpXParticleContainer::sumParticleCharge, + py::arg("local") + ) .def("deposit_charge", [](WarpXParticleContainer& pc, amrex::MultiFab* rho, const int lev) From 6ace71850745ae224a05b435c8bf6e3558bfaffc Mon Sep 17 00:00:00 2001 From: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Date: Tue, 21 Nov 2023 19:29:36 -0800 Subject: [PATCH 097/110] Refinement patch parser (#4299) * parser to define ref patches * fix def for 1D and add RZ * fix 1D Z def * remove function def for fine_taglo/hi * fix warning and error strig * fix warning message --- Docs/source/usage/parameters.rst | 7 +++++++ Source/Utils/WarpXTagging.cpp | 20 ++++++++++++++++++-- Source/WarpX.H | 2 ++ Source/WarpX.cpp | 27 +++++++++++++++++++++++---- 4 files changed, 50 insertions(+), 6 deletions(-) diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index 06b3edd0f2d..2a48766a46d 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -284,6 +284,13 @@ Setting up the field mesh This patch is rectangular, and thus its extent is given here by the coordinates of the lower corner (``warpx.fine_tag_lo``) and upper corner (``warpx.fine_tag_hi``). +* ``warpx.ref_patch_function(x,y,z)`` (`string`) optional + A function of `x`, `y`, `z` that defines the extent of the refined patch when + using static mesh refinement with ``amr.max_level``>0. Note that the function can be used + to define distinct regions for refinement, however, the refined regions should be such that + the pml layer surrounding the patches should not overlap. For this reason, when defining + distinct patches, please ensure that they are sufficiently separated. + * ``warpx.refine_plasma`` (`integer`) optional (default `0`) Increase the number of macro-particles that are injected "ahead" of a mesh refinement patch in a moving window simulation. diff --git a/Source/Utils/WarpXTagging.cpp b/Source/Utils/WarpXTagging.cpp index da810244088..807c92b2f0b 100644 --- a/Source/Utils/WarpXTagging.cpp +++ b/Source/Utils/WarpXTagging.cpp @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -30,9 +31,10 @@ WarpX::ErrorEst (int lev, TagBoxArray& tags, Real /*time*/, int /*ngrow*/) const auto problo = Geom(lev).ProbLoArray(); const auto dx = Geom(lev).CellSizeArray(); + amrex::ParserExecutor<3> ref_parser; + if (ref_patch_parser) ref_parser = ref_patch_parser->compile<3>(); const auto ftlo = fine_tag_lo; const auto fthi = fine_tag_hi; - #ifdef AMREX_USE_OMP #pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) #endif @@ -45,7 +47,21 @@ WarpX::ErrorEst (int lev, TagBoxArray& tags, Real /*time*/, int /*ngrow*/) const RealVect pos {AMREX_D_DECL((i+0.5_rt)*dx[0]+problo[0], (j+0.5_rt)*dx[1]+problo[1], (k+0.5_rt)*dx[2]+problo[2])}; - if (pos > ftlo && pos < fthi) { + bool tag_val = 0; + if (ref_parser) { +#if defined (WARPX_DIM_3D) + tag_val = (ref_parser(pos[0], pos[1], pos[2]) == 1); +#elif defined (WARPX_DIM_XZ) || defined (WARPX_DIM_RZ) + amrex::Real unused = 0.0; + tag_val = (ref_parser(pos[0], unused, pos[1]) == 1); +#elif defined (WARPX_DIM_1D_Z) + amrex::Real unused = 0.0; + tag_val = (ref_parser(unused, unused, pos[0]) == 1); +#endif + } else { + tag_val = (pos > ftlo && pos < fthi); + } + if ( tag_val == 1) { fab(i,j,k) = TagBox::SET; } }); diff --git a/Source/WarpX.H b/Source/WarpX.H index 94b06d1665a..71bbcab8364 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -1615,6 +1615,8 @@ private: amrex::RealVect fine_tag_lo; amrex::RealVect fine_tag_hi; + //! User-defined parser to define refinement patches + std::unique_ptr ref_patch_parser; bool is_synchronized = true; diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 306c4ba049e..142e48d459e 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -1004,10 +1004,29 @@ WarpX::ReadParameters () if (maxLevel() > 0) { Vector lo, hi; - utils::parser::getArrWithParser(pp_warpx, "fine_tag_lo", lo); - utils::parser::getArrWithParser(pp_warpx, "fine_tag_hi", hi); - fine_tag_lo = RealVect{lo}; - fine_tag_hi = RealVect{hi}; + bool fine_tag_lo_specified = utils::parser::queryArrWithParser(pp_warpx, "fine_tag_lo", lo); + bool fine_tag_hi_specified = utils::parser::queryArrWithParser(pp_warpx, "fine_tag_hi", hi); + std::string ref_patch_function; + bool parser_specified = pp_warpx.query("ref_patch_function(x,y,z)",ref_patch_function); + WARPX_ALWAYS_ASSERT_WITH_MESSAGE( ((fine_tag_lo_specified && fine_tag_hi_specified) || + parser_specified ), + "For max_level > 0, you need to either set\ + warpx.fine_tag_lo and warpx.fine_tag_hi\ + or warpx.ref_patch_function(x,y,z)"); + + if ( (fine_tag_lo_specified && fine_tag_hi_specified) && parser_specified) { + ablastr::warn_manager::WMRecordWarning("Refined patch", "Both fine_tag_lo,fine_tag_hi\ + and ref_patch_function(x,y,z) are provided. Note that fine_tag_lo/fine_tag_hi will\ + override the ref_patch_function(x,y,z) for defining the refinement patches"); + } + if (fine_tag_lo_specified && fine_tag_hi_specified) { + fine_tag_lo = RealVect{lo}; + fine_tag_hi = RealVect{hi}; + } else { + utils::parser::Store_parserString(pp_warpx, "ref_patch_function(x,y,z)", ref_patch_function); + ref_patch_parser = std::make_unique( + utils::parser::makeParser(ref_patch_function,{"x","y","z"})); + } } pp_warpx.query("do_dynamic_scheduling", do_dynamic_scheduling); From 245ab63947875d9dcc181d993f0aa41e8431aace Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Wed, 22 Nov 2023 04:31:02 +0100 Subject: [PATCH 098/110] Make the static variable "authors" of the WarpX class a private member of the WarpX class (#4404) * make authors a private variable of the WarpX class instead of a public static variable * update docstring --- .../Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp | 3 ++- Source/Diagnostics/WarpXOpenPMD.H | 7 ++++++- Source/Diagnostics/WarpXOpenPMD.cpp | 10 ++++++---- Source/WarpX.H | 11 +++++++++-- Source/WarpX.cpp | 3 +-- 5 files changed, 24 insertions(+), 10 deletions(-) diff --git a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp index 3e43d18d7fa..b6e960720d5 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp @@ -106,7 +106,8 @@ FlushFormatOpenPMD::FlushFormatOpenPMD (const std::string& diag_name) encoding, openpmd_backend, operator_type, operator_parameters, engine_type, engine_parameters, - warpx.getPMLdirections() + warpx.getPMLdirections(), + warpx.GetAuthors() ); } diff --git a/Source/Diagnostics/WarpXOpenPMD.H b/Source/Diagnostics/WarpXOpenPMD.H index 1acfc19b5a9..66ffd1eaa97 100644 --- a/Source/Diagnostics/WarpXOpenPMD.H +++ b/Source/Diagnostics/WarpXOpenPMD.H @@ -88,6 +88,7 @@ public: * @param engine_type ADIOS engine for output * @param engine_parameters map of parameters for the engine * @param fieldPMLdirections PML field solver, @see WarpX::getPMLdirections() + * @param authors a string specifying the authors of the simulation (can be empty) */ WarpXOpenPMDPlot (openPMD::IterationEncoding ie, std::string filetype, @@ -95,7 +96,8 @@ public: std::map< std::string, std::string > operator_parameters, std::string engine_type, std::map< std::string, std::string > engine_parameters, - std::vector fieldPMLdirections); + std::vector fieldPMLdirections, + const std::string& authors); ~WarpXOpenPMDPlot (); @@ -339,6 +341,9 @@ private: // meta data std::vector< bool > m_fieldPMLdirections; //! @see WarpX::getPMLdirections() + + // The authors' string + std::string m_authors; }; #endif // WARPX_USE_OPENPMD diff --git a/Source/Diagnostics/WarpXOpenPMD.cpp b/Source/Diagnostics/WarpXOpenPMD.cpp index 110806cbbaa..df1cf15f89a 100644 --- a/Source/Diagnostics/WarpXOpenPMD.cpp +++ b/Source/Diagnostics/WarpXOpenPMD.cpp @@ -376,13 +376,15 @@ WarpXOpenPMDPlot::WarpXOpenPMDPlot ( std::map< std::string, std::string > operator_parameters, std::string engine_type, std::map< std::string, std::string > engine_parameters, - std::vector fieldPMLdirections) + std::vector fieldPMLdirections, + const std::string& authors) :m_Series(nullptr), m_MPIRank{amrex::ParallelDescriptor::MyProc()}, m_MPISize{amrex::ParallelDescriptor::NProcs()}, m_Encoding(ie), m_OpenPMDFileType(std::move(openPMDFileType)), - m_fieldPMLdirections(std::move(fieldPMLdirections)) + m_fieldPMLdirections(std::move(fieldPMLdirections)), + m_authors{authors} { m_OpenPMDoptions = detail::getSeriesOptions(operator_type, operator_parameters, engine_type, engine_parameters); @@ -502,8 +504,8 @@ WarpXOpenPMDPlot::Init (openPMD::Access access, bool isBTD) m_Series->setIterationEncoding( m_Encoding ); // input file / simulation setup author - if( !WarpX::authors.empty()) - m_Series->setAuthor( WarpX::authors ); + if( !m_authors.empty()) + m_Series->setAuthor( m_authors ); // more natural naming for PIC m_Series->setMeshesPath( "fields" ); // conform to ED-PIC extension of openPMD diff --git a/Source/WarpX.H b/Source/WarpX.H index 71bbcab8364..6a4d76f3c01 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -131,8 +131,12 @@ public: amrex::Real external_field=0.0, bool useparser = false, amrex::ParserExecutor<3> const& field_parser={}); - //! Author of an input file / simulation setup - static std::string authors; + /** + * \brief + * If an authors' string is specified in the inputfile, this method returns that string. + * Otherwise, it returns an empty string. + */ + std::string GetAuthors () const { return m_authors; } //! Initial electric field on the grid static amrex::Vector E_external_grid; @@ -1400,6 +1404,9 @@ private: # endif #endif + //! Author of an input file / simulation setup + std::string m_authors; + amrex::Vector istep; // which step? amrex::Vector nsubsteps; // how many substeps on each level? diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 142e48d459e..bb1f5b36b84 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -88,7 +88,6 @@ using namespace amrex; Vector WarpX::E_external_grid(3, 0.0); Vector WarpX::B_external_grid(3, 0.0); -std::string WarpX::authors; std::string WarpX::B_ext_grid_s = "default"; std::string WarpX::E_ext_grid_s = "default"; bool WarpX::add_external_E_field = false; @@ -518,7 +517,7 @@ WarpX::ReadParameters () const ParmParse pp;// Traditionally, max_step and stop_time do not have prefix. utils::parser::queryWithParser(pp, "max_step", max_step); utils::parser::queryWithParser(pp, "stop_time", stop_time); - pp.query("authors", authors); + pp.query("authors", m_authors); } { From b719ee03d7ca7d34e931160d49396373b32a5b26 Mon Sep 17 00:00:00 2001 From: David Grote Date: Wed, 22 Nov 2023 02:50:15 -0800 Subject: [PATCH 099/110] Add gaussian_parse_momentum_function (#4400) * Add gaussian_parse_momentum_function * Update initial_distribution benchmarks * Remove blank line at end of initial_distribution.json * Update doc * Minor clean up in the CI test * Further clean up of the doc * Add new option to PICMI * Add documentation for option in PICMI --- Docs/source/usage/parameters.rst | 14 ++++ .../analysis_distribution.py | 29 +++++++ Examples/Tests/initial_distribution/inputs | 45 ++++++++++- Python/pywarpx/picmi.py | 34 +++++++-- .../benchmarks_json/initial_distribution.json | 70 +++++++++-------- Source/Fluids/WarpXFluidContainer.H | 3 + Source/Fluids/WarpXFluidContainer.cpp | 2 +- Source/Initialization/InjectorMomentum.H | 75 ++++++++++++++++++- Source/Initialization/InjectorMomentum.cpp | 1 + Source/Initialization/PlasmaInjector.H | 3 + Source/Initialization/PlasmaInjector.cpp | 17 ++++- Source/Utils/SpeciesUtils.H | 3 + Source/Utils/SpeciesUtils.cpp | 42 +++++++++++ 13 files changed, 291 insertions(+), 47 deletions(-) diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index 2a48766a46d..dd69f6087f7 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -915,6 +915,20 @@ Particle initialization ``.momentum_function_uy(x,y,z)`` and ``.momentum_function_uz(x,y,z)``, which gives the distribution of each component of the momentum as a function of space. + * ``gaussian_parse_momentum_function``: Gaussian momentum distribution where the mean and the standard deviation are given by functions of position in the input file. + Both are assumed to be non-relativistic. + The mean is the normalized momentum, :math:`u_m = \gamma v_m/c`. + The standard deviation is normalized, :math:`u_th = v_th/c`. + For example, this might be `u_th = sqrt(T*q_e/mass)/clight` given the temperature (in eV) and mass. + It requires the following arguments: + + * ``.momentum_function_ux_m(x,y,z)``: mean :math:`u_{x}` + * ``.momentum_function_uy_m(x,y,z)``: mean :math:`u_{y}` + * ``.momentum_function_uz_m(x,y,z)``: mean :math:`u_{z}` + * ``.momentum_function_ux_th(x,y,z)``: standard deviation of :math:`u_{x}` + * ``.momentum_function_uy_th(x,y,z)``: standard deviation of :math:`u_{y}` + * ``.momentum_function_uz_th(x,y,z)``: standard deviation of :math:`u_{z}` + * ``.theta_distribution_type`` (`string`) optional (default ``constant``) Only read if ``.momentum_distribution_type`` is ``maxwell_boltzmann`` or ``maxwell_juttner``. See documentation for these distributions (above) for constraints on values of theta. Temperatures less than zero are not allowed. diff --git a/Examples/Tests/initial_distribution/analysis_distribution.py b/Examples/Tests/initial_distribution/analysis_distribution.py index a15b688faad..d5301c077d2 100755 --- a/Examples/Tests/initial_distribution/analysis_distribution.py +++ b/Examples/Tests/initial_distribution/analysis_distribution.py @@ -14,6 +14,8 @@ # 5 denotes maxwell-juttner distribution w/ spatially varying temperature # 6 denotes maxwell-boltzmann distribution w/ constant velocity # 7 denotes maxwell-boltzmann distribution w/ spatially-varying velocity +# 8 denotes uniform distribution +# 9 denotes gaussian_parser distribution w/ spatially-varying mean and thermal velocity # The distribution is obtained through reduced diagnostic ParticleHistogram. import os @@ -340,6 +342,33 @@ def check_validity_uniform(bins, histogram, u_min, u_max, Ntrials=1000): check_validity_uniform(bin_value_y, h8y[timestep] / N0, uy_min, uy_max) check_validity_uniform(bin_value_z, h8z[timestep] / N0, uz_min, uz_max) +#================================================= +# Gaussian with parser mean and standard deviation +#================================================= + +# load data +bin_value_ux, bin_data_ux = read_reduced_diags_histogram("h9x.txt")[2:] +bin_value_uy, bin_data_uy = read_reduced_diags_histogram("h9y.txt")[2:] +bin_value_uz, bin_data_uz = read_reduced_diags_histogram("h9z.txt")[2:] + +def Gaussian(mean, sigma, u): + V = 8.0 # volume in m^3 + n = 1.0e21 # number density in 1/m^3 + return (n*V/(sigma*np.sqrt(2.*np.pi)))*np.exp(-(u - mean)**2/(2.*sigma**2)) + +du = 2./50 +f_ux = Gaussian(0.1 , 0.2 , bin_value_ux)*du +f_uy = Gaussian(0.12, 0.21, bin_value_uy)*du +f_uz = Gaussian(0.14, 0.22, bin_value_uz)*du + +f9_error = np.sum(np.abs(f_ux - bin_data_ux)/f_ux.max() + +np.abs(f_uy - bin_data_uy)/f_ux.max() + +np.abs(f_uz - bin_data_uz)/f_uz.max()) / bin_value_ux.size + +print('gaussian_parse_momentum_function velocity difference:', f9_error) + +assert(f9_error < tolerance) + test_name = os.path.split(os.getcwd())[1] checksumAPI.evaluate_checksum(test_name, filename) diff --git a/Examples/Tests/initial_distribution/inputs b/Examples/Tests/initial_distribution/inputs index 14683adeb91..5fa463604e2 100644 --- a/Examples/Tests/initial_distribution/inputs +++ b/Examples/Tests/initial_distribution/inputs @@ -29,7 +29,7 @@ algo.particle_shape = 1 ################################# ############ PLASMA ############# ################################# -particles.species_names = gaussian maxwell_boltzmann maxwell_juttner beam maxwell_juttner_parser velocity_constant velocity_parser uniform +particles.species_names = gaussian maxwell_boltzmann maxwell_juttner beam maxwell_juttner_parser velocity_constant velocity_parser uniform gaussian_parser particles.rigid_injected_species = beam gaussian.charge = -q_e @@ -133,6 +133,20 @@ uniform.uy_max = 0.1 uniform.uz_min = 10 uniform.uz_max = 11.2 +gaussian_parser.charge = -q_e +gaussian_parser.mass = m_e +gaussian_parser.injection_style = "NRandomPerCell" +gaussian_parser.num_particles_per_cell = 1000 +gaussian_parser.profile = constant +gaussian_parser.density = 1.0e21 +gaussian_parser.momentum_distribution_type = "gaussian_parse_momentum_function" +gaussian_parser.momentum_function_ux_m(x,y,z) = 0.1*z +gaussian_parser.momentum_function_uy_m(x,y,z) = 0.12*z +gaussian_parser.momentum_function_uz_m(x,y,z) = 0.14*z +gaussian_parser.momentum_function_ux_th(x,y,z) = 0.2*z +gaussian_parser.momentum_function_uy_th(x,y,z) = 0.21*z +gaussian_parser.momentum_function_uz_th(x,y,z) = 0.22*z + ################################# ########## DIAGNOSTIC ########### ################################# @@ -144,7 +158,7 @@ uniform.uz_max = 11.2 # 6 for maxwell-boltzmann with constant velocity # 7 for maxwell-boltzmann with parser velocity # 8 for cuboid in momentum space -warpx.reduced_diags_names = h1x h1y h1z h2x h2y h2z h3 h3_filtered h4x h4y h4z bmmntr h5_neg h5_pos h6 h6uy h7 h7uy_pos h7uy_neg h8x h8y h8z +warpx.reduced_diags_names = h1x h1y h1z h2x h2y h2z h3 h3_filtered h4x h4y h4z bmmntr h5_neg h5_pos h6 h6uy h7 h7uy_pos h7uy_neg h8x h8y h8z h9x h9y h9z h1x.type = ParticleHistogram h1x.intervals = 1 @@ -350,6 +364,33 @@ h8z.bin_min = 0 h8z.bin_max = 15 h8z.histogram_function(t,x,y,z,ux,uy,uz) = "uz" +h9x.type = ParticleHistogram +h9x.intervals = 1 +h9x.path = "./" +h9x.species = gaussian_parser +h9x.bin_number = 50 +h9x.bin_min = -1 +h9x.bin_max = 1 +h9x.histogram_function(t,x,y,z,ux,uy,uz) = "ux/z" + +h9y.type = ParticleHistogram +h9y.intervals = 1 +h9y.path = "./" +h9y.species = gaussian_parser +h9y.bin_number = 50 +h9y.bin_min = -1 +h9y.bin_max = 1 +h9y.histogram_function(t,x,y,z,ux,uy,uz) = "uy/z" + +h9z.type = ParticleHistogram +h9z.intervals = 1 +h9z.path = "./" +h9z.species = gaussian_parser +h9z.bin_number = 50 +h9z.bin_min = -1 +h9z.bin_max = 1 +h9z.histogram_function(t,x,y,z,ux,uy,uz) = "uz/z" + # our little beam monitor bmmntr.type = BeamRelevant bmmntr.intervals = 1 diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index b17035d739d..4f7ff13699e 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -392,11 +392,17 @@ def set_species_attributes(self, species, layout): species.do_continuous_injection = 1 # --- Note that WarpX takes gamma*beta as input - if (hasattr(self, "momentum_expressions") + if (hasattr(self, "momentum_spread_expressions") + and np.any(np.not_equal(self.momentum_spread_expressions, None)) + ): + species.momentum_distribution_type = 'gaussian_parse_momentum_function' + self.setup_parse_momentum_functions(species, self.momentum_expressions, '_m', self.directed_velocity) + self.setup_parse_momentum_functions(species, self.momentum_spread_expressions, '_th', [0.,0.,0.]) + elif (hasattr(self, "momentum_expressions") and np.any(np.not_equal(self.momentum_expressions, None)) ): species.momentum_distribution_type = 'parse_momentum_function' - self.setup_parse_momentum_functions(species) + self.setup_parse_momentum_functions(species, self.momentum_expressions, '', self.directed_velocity) elif np.any(np.not_equal(self.rms_velocity, 0.)): species.momentum_distribution_type = "gaussian" species.ux_m = self.directed_velocity[0]/constants.c @@ -411,13 +417,13 @@ def set_species_attributes(self, species, layout): species.uy = self.directed_velocity[1]/constants.c species.uz = self.directed_velocity[2]/constants.c - def setup_parse_momentum_functions(self, species): + def setup_parse_momentum_functions(self, species, expressions, suffix, defaults): for sdir, idir in zip(['x', 'y', 'z'], [0, 1, 2]): - if self.momentum_expressions[idir] is not None: - expression = pywarpx.my_constants.mangle_expression(self.momentum_expressions[idir], self.mangle_dict) + if expressions[idir] is not None: + expression = pywarpx.my_constants.mangle_expression(expressions[idir], self.mangle_dict) else: - expression = f'{self.directed_velocity[idir]}' - species.__setattr__(f'momentum_function_u{sdir}(x,y,z)', f'({expression})/{constants.c}') + expression = f'{defaults[idir]}' + species.__setattr__(f'momentum_function_u{sdir}{suffix}(x,y,z)', f'({expression})/{constants.c}') class UniformFluxDistribution(picmistandard.PICMI_UniformFluxDistribution, DensityDistributionBase): @@ -458,6 +464,20 @@ def initialize_inputs(self, species_number, layout, species, density_scale): class AnalyticDistribution(picmistandard.PICMI_AnalyticDistribution, DensityDistributionBase): + """ + Parameters + ---------- + + warpx_momentum_spread_expressions: list of string + Analytic expressions describing the gamma*velocity spread for each axis [m/s]. + Expressions should be in terms of the position, written as 'x', 'y', and 'z'. + Parameters can be used in the expression with the values given as keyword arguments. + For any axis not supplied (set to None), zero will be used. + + """ + def init(self, kw): + self.momentum_spread_expressions = kw.pop('warpx_momentum_spread_expressions', [None, None, None]) + def initialize_inputs(self, species_number, layout, species, density_scale): self.set_mangle_dict() diff --git a/Regression/Checksum/benchmarks_json/initial_distribution.json b/Regression/Checksum/benchmarks_json/initial_distribution.json index d251f1dc93e..0366526e6c6 100644 --- a/Regression/Checksum/benchmarks_json/initial_distribution.json +++ b/Regression/Checksum/benchmarks_json/initial_distribution.json @@ -1,54 +1,54 @@ { "lev=0": { - "Bx": 1.4391873150708752e-11, - "By": 2.649275763336405e-12, - "Bz": 2.264858284005838e-12, - "Ex": 116126.69618118233, - "Ey": 2721568.897218672, - "Ez": 13311634.798029516, - "jx": 213561086649.1517, - "jy": 5005061111647.613, - "jz": 24480565503287.883 + "Bx": 1.1440874094834086e-11, + "By": 5.118726025680838e-12, + "Bz": 2.2926252330134977e-12, + "Ex": 485137.7295298721, + "Ey": 2693064.232411463, + "Ez": 13309483.565569244, + "jx": 892185381139.6952, + "jy": 4952640028546.276, + "jz": 24476609310982.766 }, "gaussian": { - "particle_momentum_x": 1.114869485442012e-18, - "particle_momentum_y": 1.113950484707249e-18, - "particle_momentum_z": 1.1145095462051466e-18, + "particle_momentum_x": 1.1148694854425342e-18, + "particle_momentum_y": 1.113950484707441e-18, + "particle_momentum_z": 1.1145095462059476e-18, "particle_position_x": 256080.8791703085, "particle_position_y": 255990.74838430836, "particle_position_z": 255985.3138786131, "particle_weight": 8e+21 }, "velocity_constant": { - "particle_momentum_x": 3.525652955891248e-21, - "particle_momentum_y": 2.854131883579884e-17, - "particle_momentum_z": 3.529207120167883e-21, + "particle_momentum_x": 3.525652955494033e-21, + "particle_momentum_y": 2.854131883579772e-17, + "particle_momentum_z": 3.529207119583616e-21, "particle_position_x": 256041.594025335, "particle_position_y": 255990.80243810173, "particle_position_z": 256058.6711995684, "particle_weight": 8e+21 }, "velocity_parser": { - "particle_momentum_x": 3.5275715705676264e-21, - "particle_momentum_y": 2.85413213756175e-17, - "particle_momentum_z": 3.524858764586134e-21, + "particle_momentum_x": 3.527571570555911e-21, + "particle_momentum_y": 2.8541321375838457e-17, + "particle_momentum_z": 3.5248587639885176e-21, "particle_position_x": 256005.32957331865, "particle_position_y": 256024.073334694, "particle_position_z": 255884.2303468903, "particle_weight": 8e+21 }, "maxwell_boltzmann": { - "particle_momentum_x": 1.115726254239367e-18, - "particle_momentum_y": 1.1158627328262547e-18, - "particle_momentum_z": 1.116611006192975e-18, + "particle_momentum_x": 1.1157262542392384e-18, + "particle_momentum_y": 1.115862732826051e-18, + "particle_momentum_z": 1.1166110061936783e-18, "particle_position_x": 255961.1556903974, "particle_position_y": 255863.7685374788, "particle_position_z": 255905.39310238374, "particle_weight": 8e+21 }, "maxwell_juttner_parser": { - "particle_momentum_x": 3.244003577912503e-16, - "particle_momentum_y": 3.25040141276013e-16, + "particle_momentum_x": 3.2440035779125025e-16, + "particle_momentum_y": 3.2504014127601276e-16, "particle_momentum_z": 3.2451492327611594e-16, "particle_position_x": 255982.1948847503, "particle_position_y": 255914.12054794806, @@ -56,9 +56,9 @@ "particle_weight": 8e+21 }, "uniform": { - "particle_momentum_x": 1.8165231916763963e-17, + "particle_momentum_x": 1.8165231916763772e-17, "particle_momentum_y": 6.992054835330534e-18, - "particle_momentum_z": 1.482077848751643e-15, + "particle_momentum_z": 1.4820778487516442e-15, "particle_position_x": 256022.44604413788, "particle_position_y": 256023.176965898, "particle_position_z": 255947.8223255734, @@ -66,21 +66,29 @@ }, "maxwell_juttner": { "particle_momentum_x": 2.2134518777013892e-16, - "particle_momentum_y": 2.2161036396263055e-16, - "particle_momentum_z": 2.2128980139002375e-16, + "particle_momentum_y": 2.2161036396263068e-16, + "particle_momentum_z": 2.2128980139002385e-16, "particle_position_x": 256016.2534041072, "particle_position_y": 255955.27567227924, "particle_position_z": 255979.3536929577, "particle_weight": 8e+21 }, "beam": { - "particle_momentum_x": 1.0656686508664945e-16, - "particle_momentum_y": 1.0652904561298736e-16, - "particle_momentum_z": 1.0646689144385907e-16, + "particle_momentum_x": 1.065668650866494e-16, + "particle_momentum_y": 1.065290456129872e-16, + "particle_momentum_z": 1.0646689144385903e-16, "particle_position_x": 97438.20096731099, "particle_position_y": 97358.839485784, "particle_position_z": 88428.30909066088, "particle_weight": 0.05958446885576042 + }, + "gaussian_parser": { + "particle_momentum_x": 1.250319517250506e-17, + "particle_momentum_y": 1.3593384741126328e-17, + "particle_momentum_z": 1.4688563133041566e-17, + "particle_position_x": 255966.4923850546, + "particle_position_y": 255982.45790441945, + "particle_position_z": 256101.68177582137, + "particle_weight": 8e+21 } } - diff --git a/Source/Fluids/WarpXFluidContainer.H b/Source/Fluids/WarpXFluidContainer.H index a4880f3c57a..32f9faf7562 100644 --- a/Source/Fluids/WarpXFluidContainer.H +++ b/Source/Fluids/WarpXFluidContainer.H @@ -174,6 +174,9 @@ protected: std::unique_ptr ux_parser; std::unique_ptr uy_parser; std::unique_ptr uz_parser; + std::unique_ptr ux_th_parser; + std::unique_ptr uy_th_parser; + std::unique_ptr uz_th_parser; // Keep a pointer to TemperatureProperties to ensure the lifetime of the // contained Parser diff --git a/Source/Fluids/WarpXFluidContainer.cpp b/Source/Fluids/WarpXFluidContainer.cpp index 13063c2cfc0..03cc9b66101 100644 --- a/Source/Fluids/WarpXFluidContainer.cpp +++ b/Source/Fluids/WarpXFluidContainer.cpp @@ -29,7 +29,7 @@ WarpXFluidContainer::WarpXFluidContainer(int nlevs_max, int ispecies, const std: const ParmParse pp_species_name(species_name); SpeciesUtils::parseDensity(species_name, h_inj_rho, density_parser); SpeciesUtils::parseMomentum(species_name, "none", h_inj_mom, - ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); + ux_parser, uy_parser, uz_parser, ux_th_parser, uy_th_parser, uz_th_parser, h_mom_temp, h_mom_vel); if (h_inj_rho) { #ifdef AMREX_USE_GPU d_inj_rho = static_cast diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index d8409258fa0..8e3dfec1417 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -500,6 +500,45 @@ struct InjectorMomentumParser amrex::ParserExecutor<3> m_ux_parser, m_uy_parser, m_uz_parser; }; +// struct whose getMomentumm returns local momentum and thermal spread computed from parser. +struct InjectorMomentumGaussianParser +{ + InjectorMomentumGaussianParser (amrex::ParserExecutor<3> const& a_ux_m_parser, + amrex::ParserExecutor<3> const& a_uy_m_parser, + amrex::ParserExecutor<3> const& a_uz_m_parser, + amrex::ParserExecutor<3> const& a_ux_th_parser, + amrex::ParserExecutor<3> const& a_uy_th_parser, + amrex::ParserExecutor<3> const& a_uz_th_parser) noexcept + : m_ux_m_parser(a_ux_m_parser), m_uy_m_parser(a_uy_m_parser), m_uz_m_parser(a_uz_m_parser), + m_ux_th_parser(a_ux_th_parser), m_uy_th_parser(a_uy_th_parser), m_uz_th_parser(a_uz_th_parser) {} + + AMREX_GPU_HOST_DEVICE + amrex::XDim3 + getMomentum (amrex::Real x, amrex::Real y, amrex::Real z, + amrex::RandomEngine const& engine) const noexcept + { + amrex::Real const ux_m = m_ux_m_parser(x,y,z); + amrex::Real const uy_m = m_uy_m_parser(x,y,z); + amrex::Real const uz_m = m_uz_m_parser(x,y,z); + amrex::Real const ux_th = m_ux_th_parser(x,y,z); + amrex::Real const uy_th = m_uy_th_parser(x,y,z); + amrex::Real const uz_th = m_uz_th_parser(x,y,z); + return amrex::XDim3{amrex::RandomNormal(ux_m, ux_th, engine), + amrex::RandomNormal(uy_m, uy_th, engine), + amrex::RandomNormal(uz_m, uz_th, engine)}; + } + + AMREX_GPU_HOST_DEVICE + amrex::XDim3 + getBulkMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept + { + return amrex::XDim3{m_ux_m_parser(x,y,z), m_uy_m_parser(x,y,z), m_uz_m_parser(x,y,z)}; + } + + amrex::ParserExecutor<3> m_ux_m_parser, m_uy_m_parser, m_uz_m_parser; + amrex::ParserExecutor<3> m_ux_th_parser, m_uy_th_parser, m_uz_th_parser; +}; + // Base struct for momentum injector. // InjectorMomentum contains a union (called Object) that holds any one // instance of: @@ -508,6 +547,7 @@ struct InjectorMomentumParser // - InjectorMomentumGaussianFlux : to generate v*gaussian distribution; // - InjectorMomentumRadialExpansion: to generate radial expansion; // - InjectorMomentumParser : to generate momentum from parser; +// - InjectorMomentumGaussianParser : to generate momentum and thermal spread from parser; // The choice is made at runtime, depending in the constructor called. // This mimics virtual functions. struct InjectorMomentum @@ -528,6 +568,19 @@ struct InjectorMomentum object(t, a_ux_parser, a_uy_parser, a_uz_parser) { } + // This constructor stores a InjectorMomentumGaussianParser in union object. + InjectorMomentum (InjectorMomentumGaussianParser* t, + amrex::ParserExecutor<3> const& a_ux_m_parser, + amrex::ParserExecutor<3> const& a_uy_m_parser, + amrex::ParserExecutor<3> const& a_uz_m_parser, + amrex::ParserExecutor<3> const& a_ux_th_parser, + amrex::ParserExecutor<3> const& a_uy_th_parser, + amrex::ParserExecutor<3> const& a_uz_th_parser) + : type(Type::gaussianparser), + object(t, a_ux_m_parser, a_uy_m_parser, a_uz_m_parser, + a_ux_th_parser, a_uy_th_parser, a_uz_th_parser) + { } + // This constructor stores a InjectorMomentumGaussian in union object. InjectorMomentum (InjectorMomentumGaussian* t, amrex::Real a_ux_m, amrex::Real a_uy_m, amrex::Real a_uz_m, @@ -602,6 +655,10 @@ struct InjectorMomentum { return object.gaussian.getMomentum(x,y,z,engine); } + case Type::gaussianparser: + { + return object.gaussianparser.getMomentum(x,y,z,engine); + } case Type::gaussianflux: { return object.gaussianflux.getMomentum(x,y,z,engine); @@ -650,6 +707,10 @@ struct InjectorMomentum { return object.gaussian.getBulkMomentum(x,y,z); } + case Type::gaussianparser: + { + return object.gaussianparser.getBulkMomentum(x,y,z); + } case Type::gaussianflux: { return object.gaussianflux.getBulkMomentum(x,y,z); @@ -682,7 +743,7 @@ struct InjectorMomentum } } - enum struct Type { constant, gaussian, gaussianflux, uniform, boltzmann, juttner, radial_expansion, parser }; + enum struct Type { constant, gaussian, gaussianflux, uniform, boltzmann, juttner, radial_expansion, parser, gaussianparser }; Type type; private: @@ -724,6 +785,15 @@ private: amrex::ParserExecutor<3> const& a_uy_parser, amrex::ParserExecutor<3> const& a_uz_parser) noexcept : parser(a_ux_parser, a_uy_parser, a_uz_parser) {} + Object (InjectorMomentumGaussianParser*, + amrex::ParserExecutor<3> const& a_ux_m_parser, + amrex::ParserExecutor<3> const& a_uy_m_parser, + amrex::ParserExecutor<3> const& a_uz_m_parser, + amrex::ParserExecutor<3> const& a_ux_th_parser, + amrex::ParserExecutor<3> const& a_uy_th_parser, + amrex::ParserExecutor<3> const& a_uz_th_parser) noexcept + : gaussianparser(a_ux_m_parser, a_uy_m_parser, a_uz_m_parser, + a_ux_th_parser, a_uy_th_parser, a_uz_th_parser) {} InjectorMomentumConstant constant; InjectorMomentumGaussian gaussian; InjectorMomentumGaussianFlux gaussianflux; @@ -731,7 +801,8 @@ private: InjectorMomentumBoltzmann boltzmann; InjectorMomentumJuttner juttner; InjectorMomentumRadialExpansion radial_expansion; - InjectorMomentumParser parser; + InjectorMomentumParser parser; + InjectorMomentumGaussianParser gaussianparser; }; Object object; }; diff --git a/Source/Initialization/InjectorMomentum.cpp b/Source/Initialization/InjectorMomentum.cpp index 13f39ef95a5..35513260827 100644 --- a/Source/Initialization/InjectorMomentum.cpp +++ b/Source/Initialization/InjectorMomentum.cpp @@ -15,6 +15,7 @@ void InjectorMomentum::clear () { case Type::parser: case Type::gaussian: + case Type::gaussianparser: case Type::gaussianflux: case Type::uniform: case Type::boltzmann: diff --git a/Source/Initialization/PlasmaInjector.H b/Source/Initialization/PlasmaInjector.H index 0b2fac02f51..22d5c7711e9 100644 --- a/Source/Initialization/PlasmaInjector.H +++ b/Source/Initialization/PlasmaInjector.H @@ -173,6 +173,9 @@ protected: std::unique_ptr ux_parser; std::unique_ptr uy_parser; std::unique_ptr uz_parser; + std::unique_ptr ux_th_parser; + std::unique_ptr uy_th_parser; + std::unique_ptr uz_th_parser; // Keep a pointer to TemperatureProperties to ensure the lifetime of the // contained Parser diff --git a/Source/Initialization/PlasmaInjector.cpp b/Source/Initialization/PlasmaInjector.cpp index 20add644176..e5b503af786 100644 --- a/Source/Initialization/PlasmaInjector.cpp +++ b/Source/Initialization/PlasmaInjector.cpp @@ -248,7 +248,9 @@ void PlasmaInjector::setupGaussianBeam (const amrex::ParmParse& pp_species_name) "Error: Symmetrization only supported to orders 4 or 8 "); gaussian_beam = true; SpeciesUtils::parseMomentum(species_name, "gaussian_beam", h_inj_mom, - ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); + ux_parser, uy_parser, uz_parser, + ux_th_parser, uy_th_parser, uz_th_parser, + h_mom_temp, h_mom_vel); #if defined(WARPX_DIM_XZ) WARPX_ALWAYS_ASSERT_WITH_MESSAGE( y_rms > 0._rt, "Error: Gaussian beam y_rms must be strictly greater than 0 in 2D " @@ -289,7 +291,9 @@ void PlasmaInjector::setupNRandomPerCell (const amrex::ParmParse& pp_species_nam #endif SpeciesUtils::parseDensity(species_name, h_inj_rho, density_parser); SpeciesUtils::parseMomentum(species_name, "nrandompercell", h_inj_mom, - ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); + ux_parser, uy_parser, uz_parser, + ux_th_parser, uy_th_parser, uz_th_parser, + h_mom_temp, h_mom_vel); } void PlasmaInjector::setupNFluxPerCell (const amrex::ParmParse& pp_species_name) @@ -367,7 +371,10 @@ void PlasmaInjector::setupNFluxPerCell (const amrex::ParmParse& pp_species_name) parseFlux(pp_species_name); SpeciesUtils::parseMomentum(species_name, "nfluxpercell", h_inj_mom, - ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel, flux_normal_axis, flux_direction); + ux_parser, uy_parser, uz_parser, + ux_th_parser, uy_th_parser, uz_th_parser, + h_mom_temp, h_mom_vel, + flux_normal_axis, flux_direction); } void PlasmaInjector::setupNuniformPerCell (const amrex::ParmParse& pp_species_name) @@ -420,7 +427,9 @@ void PlasmaInjector::setupNuniformPerCell (const amrex::ParmParse& pp_species_na num_particles_per_cell_each_dim[2]; SpeciesUtils::parseDensity(species_name, h_inj_rho, density_parser); SpeciesUtils::parseMomentum(species_name, "nuniformpercell", h_inj_mom, - ux_parser, uy_parser, uz_parser, h_mom_temp, h_mom_vel); + ux_parser, uy_parser, uz_parser, + ux_th_parser, uy_th_parser, uz_th_parser, + h_mom_temp, h_mom_vel); } void PlasmaInjector::setupExternalFile (const amrex::ParmParse& pp_species_name) diff --git a/Source/Utils/SpeciesUtils.H b/Source/Utils/SpeciesUtils.H index b1a2618a002..f95525ecb4c 100644 --- a/Source/Utils/SpeciesUtils.H +++ b/Source/Utils/SpeciesUtils.H @@ -30,6 +30,9 @@ namespace SpeciesUtils { std::unique_ptr& ux_parser, std::unique_ptr& uy_parser, std::unique_ptr& uz_parser, + std::unique_ptr& ux_th_parser, + std::unique_ptr& uy_th_parser, + std::unique_ptr& uz_th_parser, std::unique_ptr& h_mom_temp, std::unique_ptr& h_mom_vel, int flux_normal_axis=0, int flux_direction=0); diff --git a/Source/Utils/SpeciesUtils.cpp b/Source/Utils/SpeciesUtils.cpp index bdba97f10b2..50729188e33 100644 --- a/Source/Utils/SpeciesUtils.cpp +++ b/Source/Utils/SpeciesUtils.cpp @@ -118,6 +118,9 @@ namespace SpeciesUtils { std::unique_ptr& ux_parser, std::unique_ptr& uy_parser, std::unique_ptr& uz_parser, + std::unique_ptr& ux_th_parser, + std::unique_ptr& uy_th_parser, + std::unique_ptr& uz_th_parser, std::unique_ptr& h_mom_temp, std::unique_ptr& h_mom_vel, int flux_normal_axis, int flux_direction) @@ -240,6 +243,45 @@ namespace SpeciesUtils { ux_parser->compile<3>(), uy_parser->compile<3>(), uz_parser->compile<3>())); + } else if (mom_dist_s == "gaussian_parse_momentum_function") { + std::string str_momentum_function_ux_m; + std::string str_momentum_function_uy_m; + std::string str_momentum_function_uz_m; + std::string str_momentum_function_ux_th; + std::string str_momentum_function_uy_th; + std::string str_momentum_function_uz_th; + utils::parser::Store_parserString(pp_species_name, + "momentum_function_ux_m(x,y,z)", str_momentum_function_ux_m); + utils::parser::Store_parserString(pp_species_name, + "momentum_function_uy_m(x,y,z)", str_momentum_function_uy_m); + utils::parser::Store_parserString(pp_species_name, + "momentum_function_uz_m(x,y,z)", str_momentum_function_uz_m); + utils::parser::Store_parserString(pp_species_name, + "momentum_function_ux_th(x,y,z)", str_momentum_function_ux_th); + utils::parser::Store_parserString(pp_species_name, + "momentum_function_uy_th(x,y,z)", str_momentum_function_uy_th); + utils::parser::Store_parserString(pp_species_name, + "momentum_function_uz_th(x,y,z)", str_momentum_function_uz_th); + // Construct InjectorMomentum with InjectorMomentumParser. + ux_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_ux_m, {"x","y","z"})); + uy_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_uy_m, {"x","y","z"})); + uz_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_uz_m, {"x","y","z"})); + ux_th_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_ux_th, {"x","y","z"})); + uy_th_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_uy_th, {"x","y","z"})); + uz_th_parser = std::make_unique( + utils::parser::makeParser(str_momentum_function_uz_th, {"x","y","z"})); + h_inj_mom.reset(new InjectorMomentum((InjectorMomentumGaussianParser*)nullptr, + ux_parser->compile<3>(), + uy_parser->compile<3>(), + uz_parser->compile<3>(), + ux_th_parser->compile<3>(), + uy_th_parser->compile<3>(), + uz_th_parser->compile<3>())); } else { StringParseAbortMessage("Momentum distribution type", mom_dist_s); } From 7da4b2be5035a527f0f5b11e794931979ad20f4a Mon Sep 17 00:00:00 2001 From: Edward Basso Date: Thu, 23 Nov 2023 11:15:49 -0600 Subject: [PATCH 100/110] Make whitespace alignment more consistent (#4440) * Make sure code is aligned properly * Minor adjustments, mainly to method arguments * More corrections * More corrections * More corrections 2 --- .../analysis_2d_binary.py | 6 +- .../LatticeElements/HardEdged_K.H | 16 +- Source/BoundaryConditions/WarpX_PML_kernels.H | 184 ++++----- Source/Diagnostics/BTDiagnostics.cpp | 8 +- .../Diagnostics/BoundaryScrapingDiagnostics.H | 2 +- Source/Diagnostics/Diagnostics.cpp | 2 +- .../FlushFormats/FlushFormatOpenPMD.cpp | 134 +++---- Source/Diagnostics/FullDiagnostics.cpp | 2 +- Source/Diagnostics/SliceDiagnostic.H | 22 +- Source/Diagnostics/SliceDiagnostic.cpp | 172 ++++---- Source/Diagnostics/WarpXOpenPMD.cpp | 377 +++++++++--------- .../FiniteDifferenceSolver/ComputeDivE.cpp | 4 +- .../FiniteDifferenceSolver/EvolveB.cpp | 4 +- .../FiniteDifferenceSolver/EvolveBPML.cpp | 6 +- .../FiniteDifferenceSolver/EvolveEPML.cpp | 4 +- .../FiniteDifferenceSolver/EvolveF.cpp | 4 +- .../FiniteDifferenceSolver/EvolveFPML.cpp | 4 +- .../HybridPICSolveE.cpp | 8 +- .../MacroscopicEvolveE.cpp | 4 +- .../MacroscopicProperties.H | 186 ++++----- .../PsatdAlgorithmComoving.cpp | 2 +- .../PsatdAlgorithmGalileanRZ.cpp | 2 +- .../SpectralBaseAlgorithm.cpp | 2 +- .../SpectralBaseAlgorithmRZ.H | 2 +- .../SpectralSolver/SpectralKSpace.cpp | 2 +- Source/FieldSolver/WarpX_QED_K.H | 4 +- Source/Fluids/WarpXFluidContainer.H | 2 +- Source/Fluids/WarpXFluidContainer.cpp | 6 +- Source/Initialization/InjectorMomentum.H | 6 +- Source/Initialization/WarpXInitData.cpp | 154 +++---- .../LaserProfileFromFile.cpp | 2 +- Source/Make.WarpX | 66 +-- Source/Parallelization/GuardCellManager.cpp | 2 +- Source/Parallelization/WarpXComm.cpp | 16 +- .../BinaryCollision/ParticleCreationFunc.H | 2 +- .../BreitWheelerEngineWrapper.cpp | 2 +- .../QEDInternals/QuantumSyncEngineWrapper.cpp | 2 +- Source/Particles/MultiParticleContainer.cpp | 8 +- .../Particles/ParticleCreation/SmartCreate.H | 20 +- Source/Particles/WarpXParticleContainer.H | 2 +- Source/Utils/Parser/ParserUtils.cpp | 6 +- Source/Utils/WarpXAlgorithmSelection.H | 12 +- Source/Utils/WarpXMovingWindow.cpp | 28 +- .../Utils/check_interp_points_and_weights.py | 4 +- Source/WarpX.cpp | 38 +- Source/ablastr/coarsen/sample.cpp | 12 +- Source/ablastr/utils/SignalHandling.H | 2 +- Tools/PerformanceTests/functions_perftest.py | 20 +- 48 files changed, 787 insertions(+), 788 deletions(-) diff --git a/Examples/Tests/laser_injection_from_file/analysis_2d_binary.py b/Examples/Tests/laser_injection_from_file/analysis_2d_binary.py index fd59fcf33dd..c5bdd84d023 100755 --- a/Examples/Tests/laser_injection_from_file/analysis_2d_binary.py +++ b/Examples/Tests/laser_injection_from_file/analysis_2d_binary.py @@ -121,9 +121,9 @@ def write_file(fname, x, y, t, E): file.write(E.tobytes()) def create_gaussian_2d(): - T, X, Y = np.meshgrid(tcoords, xcoords, np.array([0.0]), indexing='ij') - E_t = gauss(T,X,Y,'2d') - write_file("gauss_2d", xcoords, np.array([0.0]), tcoords, E_t) + T, X, Y = np.meshgrid(tcoords, xcoords, np.array([0.0]), indexing='ij') + E_t = gauss(T,X,Y,'2d') + write_file("gauss_2d", xcoords, np.array([0.0]), tcoords, E_t) def do_analysis(fname, compname, steps): ds = yt.load(fname) diff --git a/Source/AcceleratorLattice/LatticeElements/HardEdged_K.H b/Source/AcceleratorLattice/LatticeElements/HardEdged_K.H index 35e2dde7a8f..6be88deb04d 100644 --- a/Source/AcceleratorLattice/LatticeElements/HardEdged_K.H +++ b/Source/AcceleratorLattice/LatticeElements/HardEdged_K.H @@ -34,14 +34,14 @@ amrex::ParticleReal hard_edged_fraction(const amrex::ParticleReal z, amrex::ParticleReal const zl = std::min(z, zpvdt); amrex::ParticleReal const zr = std::max(z, zpvdt); - // Calculate the residence correction - // frac will be 1 if the step is completely inside the lens, between 0 and 1 - // when entering or leaving the lens, and otherwise 0. - // This accounts for the case when particles step over the element without landing in it. - // This assumes that vzp != 0. - amrex::ParticleReal const zl_bounded = std::min(std::max(zl, zs), ze); - amrex::ParticleReal const zr_bounded = std::min(std::max(zr, zs), ze); - const amrex::ParticleReal frac = (zr_bounded - zl_bounded)/(zr - zl); + // Calculate the residence correction + // frac will be 1 if the step is completely inside the lens, between 0 and 1 + // when entering or leaving the lens, and otherwise 0. + // This accounts for the case when particles step over the element without landing in it. + // This assumes that vzp != 0. + amrex::ParticleReal const zl_bounded = std::min(std::max(zl, zs), ze); + amrex::ParticleReal const zr_bounded = std::min(std::max(zr, zs), ze); + const amrex::ParticleReal frac = (zr_bounded - zl_bounded)/(zr - zl); return frac; } diff --git a/Source/BoundaryConditions/WarpX_PML_kernels.H b/Source/BoundaryConditions/WarpX_PML_kernels.H index 1396613b4f5..7aaf5a57369 100644 --- a/Source/BoundaryConditions/WarpX_PML_kernels.H +++ b/Source/BoundaryConditions/WarpX_PML_kernels.H @@ -278,12 +278,12 @@ void warpx_damp_pml_bx (int i, int j, int k, amrex::Array4 const& B } } - // Bxz - if (sz == 0) { - Bx(i,j,k,PMLComp::xz) *= sigma_star_fac_z[j-zlo]; - } else { - Bx(i,j,k,PMLComp::xz) *= sigma_fac_z[j-zlo]; - } + // Bxz + if (sz == 0) { + Bx(i,j,k,PMLComp::xz) *= sigma_star_fac_z[j-zlo]; + } else { + Bx(i,j,k,PMLComp::xz) *= sigma_fac_z[j-zlo]; + } #elif defined(WARPX_DIM_3D) @@ -302,19 +302,19 @@ void warpx_damp_pml_bx (int i, int j, int k, amrex::Array4 const& B } } - // Bxy - if (sy == 0) { - Bx(i,j,k,PMLComp::xy) *= sigma_star_fac_y[j-ylo]; - } else { - Bx(i,j,k,PMLComp::xy) *= sigma_fac_y[j-ylo]; - } + // Bxy + if (sy == 0) { + Bx(i,j,k,PMLComp::xy) *= sigma_star_fac_y[j-ylo]; + } else { + Bx(i,j,k,PMLComp::xy) *= sigma_fac_y[j-ylo]; + } - // Bxz - if (sz == 0) { - Bx(i,j,k,PMLComp::xz) *= sigma_star_fac_z[k-zlo]; - } else { - Bx(i,j,k,PMLComp::xz) *= sigma_fac_z[k-zlo]; - } + // Bxz + if (sz == 0) { + Bx(i,j,k,PMLComp::xz) *= sigma_star_fac_z[k-zlo]; + } else { + Bx(i,j,k,PMLComp::xz) *= sigma_fac_z[k-zlo]; + } #endif } @@ -345,19 +345,19 @@ void warpx_damp_pml_by (int i, int j, int k, amrex::Array4 const& B const int sx = By_stag[0]; const int sz = By_stag[1]; - // Byx - if (sx == 0) { - By(i,j,k,PMLComp::yx) *= sigma_star_fac_x[i-xlo]; - } else { - By(i,j,k,PMLComp::yx) *= sigma_fac_x[i-xlo]; - } + // Byx + if (sx == 0) { + By(i,j,k,PMLComp::yx) *= sigma_star_fac_x[i-xlo]; + } else { + By(i,j,k,PMLComp::yx) *= sigma_fac_x[i-xlo]; + } - // Byz - if (sz == 0) { - By(i,j,k,PMLComp::yz) *= sigma_star_fac_z[j-zlo]; - } else { - By(i,j,k,PMLComp::yz) *= sigma_fac_z[j-zlo]; - } + // Byz + if (sz == 0) { + By(i,j,k,PMLComp::yz) *= sigma_star_fac_z[j-zlo]; + } else { + By(i,j,k,PMLComp::yz) *= sigma_fac_z[j-zlo]; + } #elif defined(WARPX_DIM_3D) @@ -366,12 +366,12 @@ void warpx_damp_pml_by (int i, int j, int k, amrex::Array4 const& B const int sy = By_stag[1]; const int sz = By_stag[2]; - // Byx - if (sx == 0) { - By(i,j,k,PMLComp::yx) *= sigma_star_fac_x[i-xlo]; - } else { - By(i,j,k,PMLComp::yx) *= sigma_fac_x[i-xlo]; - } + // Byx + if (sx == 0) { + By(i,j,k,PMLComp::yx) *= sigma_star_fac_x[i-xlo]; + } else { + By(i,j,k,PMLComp::yx) *= sigma_fac_x[i-xlo]; + } if (divb_cleaning) { @@ -383,12 +383,12 @@ void warpx_damp_pml_by (int i, int j, int k, amrex::Array4 const& B } } - // Byz - if (sz == 0) { - By(i,j,k,PMLComp::yz) *= sigma_star_fac_z[k-zlo]; - } else { - By(i,j,k,PMLComp::yz) *= sigma_fac_z[k-zlo]; - } + // Byz + if (sz == 0) { + By(i,j,k,PMLComp::yz) *= sigma_star_fac_z[k-zlo]; + } else { + By(i,j,k,PMLComp::yz) *= sigma_fac_z[k-zlo]; + } #endif } @@ -419,12 +419,12 @@ void warpx_damp_pml_bz (int i, int j, int k, amrex::Array4 const& B const int sx = Bz_stag[0]; const int sz = Bz_stag[1]; - // Bzx - if (sx == 0) { - Bz(i,j,k,PMLComp::zx) *= sigma_star_fac_x[i-xlo]; - } else { - Bz(i,j,k,PMLComp::zx) *= sigma_fac_x[i-xlo]; - } + // Bzx + if (sx == 0) { + Bz(i,j,k,PMLComp::zx) *= sigma_star_fac_x[i-xlo]; + } else { + Bz(i,j,k,PMLComp::zx) *= sigma_fac_x[i-xlo]; + } if (divb_cleaning) { @@ -443,19 +443,19 @@ void warpx_damp_pml_bz (int i, int j, int k, amrex::Array4 const& B const int sy = Bz_stag[1]; const int sz = Bz_stag[2]; - // Bzx - if (sx == 0) { - Bz(i,j,k,PMLComp::zx) *= sigma_star_fac_x[i-xlo]; - } else { - Bz(i,j,k,PMLComp::zx) *= sigma_fac_x[i-xlo]; - } + // Bzx + if (sx == 0) { + Bz(i,j,k,PMLComp::zx) *= sigma_star_fac_x[i-xlo]; + } else { + Bz(i,j,k,PMLComp::zx) *= sigma_fac_x[i-xlo]; + } - // Bzy - if (sy == 0) { - Bz(i,j,k,PMLComp::zy) *= sigma_star_fac_y[j-ylo]; - } else { - Bz(i,j,k,PMLComp::zy) *= sigma_fac_y[j-ylo]; - } + // Bzy + if (sy == 0) { + Bz(i,j,k,PMLComp::zy) *= sigma_star_fac_y[j-ylo]; + } else { + Bz(i,j,k,PMLComp::zy) *= sigma_fac_y[j-ylo]; + } if (divb_cleaning) { @@ -495,19 +495,19 @@ void warpx_damp_pml_scalar (int i, int j, int k, amrex::Array4 cons const int sx = arr_stag[0]; const int sz = arr_stag[1]; - // Component along x - if (sx == 0) { - arr(i,j,k,PMLComp::x) *= sigma_star_fac_x[i-xlo]; - } else { - arr(i,j,k,PMLComp::x) *= sigma_fac_x[i-xlo]; - } + // Component along x + if (sx == 0) { + arr(i,j,k,PMLComp::x) *= sigma_star_fac_x[i-xlo]; + } else { + arr(i,j,k,PMLComp::x) *= sigma_fac_x[i-xlo]; + } - // Component along z - if (sz == 0) { - arr(i,j,k,PMLComp::z) *= sigma_star_fac_z[j-zlo]; - } else { - arr(i,j,k,PMLComp::z) *= sigma_fac_z[j-zlo]; - } + // Component along z + if (sz == 0) { + arr(i,j,k,PMLComp::z) *= sigma_star_fac_z[j-zlo]; + } else { + arr(i,j,k,PMLComp::z) *= sigma_fac_z[j-zlo]; + } #elif defined(WARPX_DIM_3D) @@ -516,26 +516,26 @@ void warpx_damp_pml_scalar (int i, int j, int k, amrex::Array4 cons const int sy = arr_stag[1]; const int sz = arr_stag[2]; - // Component along x - if (sx == 0) { - arr(i,j,k,PMLComp::x) *= sigma_star_fac_x[i-xlo]; - } else { - arr(i,j,k,PMLComp::x) *= sigma_fac_x[i-xlo]; - } - - // Component along y - if (sy == 0) { - arr(i,j,k,PMLComp::y) *= sigma_star_fac_y[j-ylo]; - } else { - arr(i,j,k,PMLComp::y) *= sigma_fac_y[j-ylo]; - } - - // Component along z - if (sz == 0) { - arr(i,j,k,PMLComp::z) *= sigma_star_fac_z[k-zlo]; - } else { - arr(i,j,k,PMLComp::z) *= sigma_fac_z[k-zlo]; - } + // Component along x + if (sx == 0) { + arr(i,j,k,PMLComp::x) *= sigma_star_fac_x[i-xlo]; + } else { + arr(i,j,k,PMLComp::x) *= sigma_fac_x[i-xlo]; + } + + // Component along y + if (sy == 0) { + arr(i,j,k,PMLComp::y) *= sigma_star_fac_y[j-ylo]; + } else { + arr(i,j,k,PMLComp::y) *= sigma_fac_y[j-ylo]; + } + + // Component along z + if (sz == 0) { + arr(i,j,k,PMLComp::z) *= sigma_star_fac_z[k-zlo]; + } else { + arr(i,j,k,PMLComp::z) *= sigma_fac_z[k-zlo]; + } #endif } diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp index 868dc94ba41..fdda963ddae 100644 --- a/Source/Diagnostics/BTDiagnostics.cpp +++ b/Source/Diagnostics/BTDiagnostics.cpp @@ -884,10 +884,10 @@ BTDiagnostics::k_index_zlab (int i_buffer, int lev) void BTDiagnostics::SetSnapshotFullStatus (const int i_buffer) { - if (m_snapshot_full[i_buffer] == 1) return; - // if the last valid z-index of the snapshot, which is 0, is filled, then - // set the snapshot full integer to 1 - if (m_lastValidZSlice[i_buffer] == 1) m_snapshot_full[i_buffer] = 1; + if (m_snapshot_full[i_buffer] == 1) return; + // if the last valid z-index of the snapshot, which is 0, is filled, then + // set the snapshot full integer to 1 + if (m_lastValidZSlice[i_buffer] == 1) m_snapshot_full[i_buffer] = 1; } diff --git a/Source/Diagnostics/BoundaryScrapingDiagnostics.H b/Source/Diagnostics/BoundaryScrapingDiagnostics.H index e9efcfe5abb..097835297dd 100644 --- a/Source/Diagnostics/BoundaryScrapingDiagnostics.H +++ b/Source/Diagnostics/BoundaryScrapingDiagnostics.H @@ -52,7 +52,7 @@ private: * * This is not used for BoundaryScrapingDiagnostics: no field to output */ - void InitializeBufferData (int i_buffer, int lev, bool restart=false) override; + void InitializeBufferData (int i_buffer, int lev, bool restart=false) override; /** Initialize functors that point to the fields requested by the user. * * This is not used for BoundaryScrapingDiagnostics: no field to output diff --git a/Source/Diagnostics/Diagnostics.cpp b/Source/Diagnostics/Diagnostics.cpp index 878504dda97..068a082bce2 100644 --- a/Source/Diagnostics/Diagnostics.cpp +++ b/Source/Diagnostics/Diagnostics.cpp @@ -335,7 +335,7 @@ Diagnostics::InitDataAfterRestart () InitializeParticleBuffer(); InitializeParticleFunctors(); } - if (write_species == 0) { + if (write_species == 0) { WARPX_ALWAYS_ASSERT_WITH_MESSAGE( m_format != "checkpoint", "For checkpoint format, write_species flag must be 1." diff --git a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp index b6e960720d5..e15a7065f3a 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp @@ -38,77 +38,77 @@ FlushFormatOpenPMD::FlushFormatOpenPMD (const std::string& diag_name) openPMD::IterationEncoding encoding = openPMD::IterationEncoding::groupBased; if ( openpmd_encoding == "v" ) - encoding = openPMD::IterationEncoding::variableBased; + encoding = openPMD::IterationEncoding::variableBased; else if ( openpmd_encoding == "g" ) - encoding = openPMD::IterationEncoding::groupBased; + encoding = openPMD::IterationEncoding::groupBased; else if ( openpmd_encoding == "f" ) - encoding = openPMD::IterationEncoding::fileBased; + encoding = openPMD::IterationEncoding::fileBased; std::string diag_type_str; pp_diag_name.get("diag_type", diag_type_str); if (diag_type_str == "BackTransformed") { - if ( ( openPMD::IterationEncoding::fileBased != encoding ) && - ( openPMD::IterationEncoding::groupBased != encoding ) ) - { - const std::string warnMsg = diag_name+" Unable to support BTD with streaming. Using GroupBased "; - ablastr::warn_manager::WMRecordWarning("Diagnostics", warnMsg); - encoding = openPMD::IterationEncoding::groupBased; - } + if ( ( openPMD::IterationEncoding::fileBased != encoding ) && + ( openPMD::IterationEncoding::groupBased != encoding ) ) + { + const std::string warnMsg = diag_name+" Unable to support BTD with streaming. Using GroupBased "; + ablastr::warn_manager::WMRecordWarning("Diagnostics", warnMsg); + encoding = openPMD::IterationEncoding::groupBased; + } } - // - // if no encoding is defined, then check to see if tspf is defined. - // (backward compatibility) - // - if ( !encodingDefined ) + // + // if no encoding is defined, then check to see if tspf is defined. + // (backward compatibility) + // + if ( !encodingDefined ) { - bool openpmd_tspf = false; - const bool tspfDefined = pp_diag_name.query("openpmd_tspf", openpmd_tspf); - if ( tspfDefined && openpmd_tspf ) - encoding = openPMD::IterationEncoding::fileBased; + bool openpmd_tspf = false; + const bool tspfDefined = pp_diag_name.query("openpmd_tspf", openpmd_tspf); + if ( tspfDefined && openpmd_tspf ) + encoding = openPMD::IterationEncoding::fileBased; + } + + // ADIOS2 operator type & parameters + std::string operator_type; + pp_diag_name.query("adios2_operator.type", operator_type); + std::string const prefix = diag_name + ".adios2_operator.parameters"; + const ParmParse pp; + auto entr = amrex::ParmParse::getEntries(prefix); + + std::map< std::string, std::string > operator_parameters; + auto const prefix_len = prefix.size() + 1; + for (std::string k : entr) { + std::string v; + pp.get(k.c_str(), v); + k.erase(0, prefix_len); + operator_parameters.insert({k, v}); + } + + // ADIOS2 engine type & parameters + std::string engine_type; + pp_diag_name.query("adios2_engine.type", engine_type); + std::string const engine_prefix = diag_name + ".adios2_engine.parameters"; + const ParmParse ppe; + auto eng_entr = amrex::ParmParse::getEntries(engine_prefix); + + std::map< std::string, std::string > engine_parameters; + auto const prefixlen = engine_prefix.size() + 1; + for (std::string k : eng_entr) { + std::string v; + ppe.get(k.c_str(), v); + k.erase(0, prefixlen); + engine_parameters.insert({k, v}); } - // ADIOS2 operator type & parameters - std::string operator_type; - pp_diag_name.query("adios2_operator.type", operator_type); - std::string const prefix = diag_name + ".adios2_operator.parameters"; - const ParmParse pp; - auto entr = amrex::ParmParse::getEntries(prefix); - - std::map< std::string, std::string > operator_parameters; - auto const prefix_len = prefix.size() + 1; - for (std::string k : entr) { - std::string v; - pp.get(k.c_str(), v); - k.erase(0, prefix_len); - operator_parameters.insert({k, v}); - } - - // ADIOS2 engine type & parameters - std::string engine_type; - pp_diag_name.query("adios2_engine.type", engine_type); - std::string const engine_prefix = diag_name + ".adios2_engine.parameters"; - const ParmParse ppe; - auto eng_entr = amrex::ParmParse::getEntries(engine_prefix); - - std::map< std::string, std::string > engine_parameters; - auto const prefixlen = engine_prefix.size() + 1; - for (std::string k : eng_entr) { - std::string v; - ppe.get(k.c_str(), v); - k.erase(0, prefixlen); - engine_parameters.insert({k, v}); - } - - auto & warpx = WarpX::GetInstance(); - m_OpenPMDPlotWriter = std::make_unique( - encoding, openpmd_backend, - operator_type, operator_parameters, - engine_type, engine_parameters, - warpx.getPMLdirections(), - warpx.GetAuthors() - ); + auto & warpx = WarpX::GetInstance(); + m_OpenPMDPlotWriter = std::make_unique( + encoding, openpmd_backend, + operator_type, operator_parameters, + engine_type, engine_parameters, + warpx.getPMLdirections(), + warpx.GetAuthors() + ); } void @@ -129,15 +129,15 @@ FlushFormatOpenPMD::WriteToFile ( const std::string& filename = amrex::Concatenate(prefix, iteration[0], file_min_digits); if (!isBTD) { - amrex::Print() << Utils::TextMsg::Info("Writing openPMD file " + filename); + amrex::Print() << Utils::TextMsg::Info("Writing openPMD file " + filename); } else { - amrex::Print() << Utils::TextMsg::Info("Writing buffer " + std::to_string(bufferID+1) + " of " + std::to_string(numBuffers) - + " to snapshot " + std::to_string(snapshotID) + " to openPMD BTD " + prefix); - if (isLastBTDFlush) - { - amrex::Print() << Utils::TextMsg::Info("Finished writing snapshot " + std::to_string(snapshotID) + " in openPMD BTD " + prefix); - } + amrex::Print() << Utils::TextMsg::Info("Writing buffer " + std::to_string(bufferID+1) + " of " + std::to_string(numBuffers) + + " to snapshot " + std::to_string(snapshotID) + " to openPMD BTD " + prefix); + if (isLastBTDFlush) + { + amrex::Print() << Utils::TextMsg::Info("Finished writing snapshot " + std::to_string(snapshotID) + " in openPMD BTD " + prefix); + } } WARPX_ALWAYS_ASSERT_WITH_MESSAGE( @@ -160,7 +160,7 @@ FlushFormatOpenPMD::WriteToFile ( // particles: all (reside only on locally finest level) m_OpenPMDPlotWriter->WriteOpenPMDParticles( - particle_diags, static_cast(time), use_pinned_pc, isBTD, isLastBTDFlush, totalParticlesFlushedAlready); + particle_diags, static_cast(time), use_pinned_pc, isBTD, isLastBTDFlush, totalParticlesFlushedAlready); // signal that no further updates will be written to this iteration m_OpenPMDPlotWriter->CloseStep(isBTD, isLastBTDFlush); diff --git a/Source/Diagnostics/FullDiagnostics.cpp b/Source/Diagnostics/FullDiagnostics.cpp index 4ca45eeb59d..c471beb05b3 100644 --- a/Source/Diagnostics/FullDiagnostics.cpp +++ b/Source/Diagnostics/FullDiagnostics.cpp @@ -627,7 +627,7 @@ FullDiagnostics::InitializeFieldFunctors (int lev) m_all_field_functors[lev].resize(ntot); // Fill vector of functors for all components except individual cylindrical modes. for (int comp=0; comp(warpx.get_pointer_Efield_aux(lev, 2), lev, m_crse_ratio); } else if ( m_varnames[comp] == "Bz" ){ m_all_field_functors[lev][comp] = std::make_unique(warpx.get_pointer_Bfield_aux(lev, 2), lev, m_crse_ratio); diff --git a/Source/Diagnostics/SliceDiagnostic.H b/Source/Diagnostics/SliceDiagnostic.H index 977b147e4e8..000b01d470f 100644 --- a/Source/Diagnostics/SliceDiagnostic.H +++ b/Source/Diagnostics/SliceDiagnostic.H @@ -19,20 +19,20 @@ std::unique_ptr CreateSlice( const amrex::MultiFab& mf, amrex::IntVect &slice_cr_ratio ); void CheckSliceInput( amrex::RealBox real_box, - amrex::RealBox &slice_cc_nd_box, amrex::RealBox &slice_realbox, - amrex::IntVect &slice_cr_ratio, amrex::Vector dom_geom, - amrex::IntVect SliceType, amrex::IntVect &slice_lo, - amrex::IntVect &slice_hi, amrex::IntVect &interp_lo); + amrex::RealBox &slice_cc_nd_box, amrex::RealBox &slice_realbox, + amrex::IntVect &slice_cr_ratio, amrex::Vector dom_geom, + amrex::IntVect SliceType, amrex::IntVect &slice_lo, + amrex::IntVect &slice_hi, amrex::IntVect &interp_lo); void InterpolateSliceValues( amrex::MultiFab& smf, - amrex::IntVect interp_lo, amrex::RealBox slice_realbox, - amrex::Vector geom, int ncomp, int nghost, - amrex::IntVect slice_lo, amrex::IntVect slice_hi, - amrex::IntVect SliceType, amrex::RealBox real_box); + amrex::IntVect interp_lo, amrex::RealBox slice_realbox, + amrex::Vector geom, int ncomp, int nghost, + amrex::IntVect slice_lo, amrex::IntVect slice_hi, + amrex::IntVect SliceType, amrex::RealBox real_box); void InterpolateLo( const amrex::Box& bx, amrex::FArrayBox &fabox, - amrex::IntVect slice_lo, amrex::Vector geom, - int idir, amrex::IntVect IndType, amrex::RealBox slice_realbox, - int srccomp, int ncomp, int nghost, amrex::RealBox real_box); + amrex::IntVect slice_lo, amrex::Vector geom, + int idir, amrex::IntVect IndType, amrex::RealBox slice_realbox, + int srccomp, int ncomp, int nghost, amrex::RealBox real_box); #endif diff --git a/Source/Diagnostics/SliceDiagnostic.cpp b/Source/Diagnostics/SliceDiagnostic.cpp index aede8b303b8..9d3b3108b75 100644 --- a/Source/Diagnostics/SliceDiagnostic.cpp +++ b/Source/Diagnostics/SliceDiagnostic.cpp @@ -101,31 +101,31 @@ CreateSlice( const MultiFab& mf, const Vector &dom_geom, // Determine if interpolation is required and number of cells in slice // for (int idim = 0; idim < AMREX_SPACEDIM; ++idim) { - // Flag for interpolation if required // - if ( interp_lo[idim] == 1) { - interpolate = 1; - } - - // For the case when a dimension is reduced // - if ( ( slice_hi[idim] - slice_lo[idim]) == 1) { - slice_ncells[idim] = 1; - } - else { - slice_ncells[idim] = ( slice_hi[idim] - slice_lo[idim] + 1 ) - / slice_cr_ratio[idim]; - - const int refined_ncells = slice_hi[idim] - slice_lo[idim] + 1 ; - if ( slice_cr_ratio[idim] > 1) { - coarsen = true; - - // modify slice_grid_size if >= refines_cells // - if ( slice_grid_size >= refined_ncells ) { - slice_grid_size = refined_ncells - 1; - } - - } - configuration_dim += 1; - } + // Flag for interpolation if required // + if ( interp_lo[idim] == 1) { + interpolate = 1; + } + + // For the case when a dimension is reduced // + if ( ( slice_hi[idim] - slice_lo[idim]) == 1) { + slice_ncells[idim] = 1; + } + else { + slice_ncells[idim] = ( slice_hi[idim] - slice_lo[idim] + 1 ) + / slice_cr_ratio[idim]; + + const int refined_ncells = slice_hi[idim] - slice_lo[idim] + 1 ; + if ( slice_cr_ratio[idim] > 1) { + coarsen = true; + + // modify slice_grid_size if >= refines_cells // + if ( slice_grid_size >= refined_ncells ) { + slice_grid_size = refined_ncells - 1; + } + + } + configuration_dim += 1; + } } if (configuration_dim==1) { ablastr::warn_manager::WMRecordWarning("Diagnostics", @@ -162,70 +162,70 @@ CreateSlice( const MultiFab& mf, const Vector &dom_geom, return smf; } else { - Vector crse_ba(1); - crse_ba[0] = sba[0]; - crse_ba[0].coarsen(slice_cr_ratio); + Vector crse_ba(1); + crse_ba[0] = sba[0]; + crse_ba[0].coarsen(slice_cr_ratio); - AMREX_ALWAYS_ASSERT(crse_ba[0].size() == sba[0].size()); + AMREX_ALWAYS_ASSERT(crse_ba[0].size() == sba[0].size()); - cs_mf = std::make_unique(amrex::convert(crse_ba[0],SliceType), - sdmap[0], ncomp,nghost); + cs_mf = std::make_unique(amrex::convert(crse_ba[0],SliceType), + sdmap[0], ncomp,nghost); - const MultiFab& mfSrc = *smf; - MultiFab& mfDst = *cs_mf; + const MultiFab& mfSrc = *smf; + MultiFab& mfDst = *cs_mf; - MFIter mfi_dst(mfDst); - for (MFIter mfi(mfSrc); mfi.isValid(); ++mfi) { + MFIter mfi_dst(mfDst); + for (MFIter mfi(mfSrc); mfi.isValid(); ++mfi) { - Array4 const& Src_fabox = mfSrc.const_array(mfi); + Array4 const& Src_fabox = mfSrc.const_array(mfi); - const Box& Dst_bx = mfi_dst.validbox(); - Array4 const& Dst_fabox = mfDst.array(mfi_dst); + const Box& Dst_bx = mfi_dst.validbox(); + Array4 const& Dst_fabox = mfDst.array(mfi_dst); - const int scomp = 0; - const int dcomp = 0; + const int scomp = 0; + const int dcomp = 0; - const IntVect cctype(AMREX_D_DECL(0,0,0)); - if( SliceType==cctype ) { - amrex::amrex_avgdown(Dst_bx, Dst_fabox, Src_fabox, dcomp, scomp, - ncomp, slice_cr_ratio); - } - const IntVect ndtype(AMREX_D_DECL(1,1,1)); - if( SliceType == ndtype ) { - amrex::amrex_avgdown_nodes(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio); - } - if( SliceType == WarpX::GetInstance().getEfield(0,0).ixType().toIntVect() ) { - amrex::amrex_avgdown_edges(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio, 0); - } - if( SliceType == WarpX::GetInstance().getEfield(0,1).ixType().toIntVect() ) { - amrex::amrex_avgdown_edges(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio, 1); - } - if( SliceType == WarpX::GetInstance().getEfield(0,2).ixType().toIntVect() ) { - amrex::amrex_avgdown_edges(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio, 2); - } - if( SliceType == WarpX::GetInstance().getBfield(0,0).ixType().toIntVect() ) { - amrex::amrex_avgdown_faces(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio, 0); - } - if( SliceType == WarpX::GetInstance().getBfield(0,1).ixType().toIntVect() ) { - amrex::amrex_avgdown_faces(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio, 1); - } - if( SliceType == WarpX::GetInstance().getBfield(0,2).ixType().toIntVect() ) { - amrex::amrex_avgdown_faces(Dst_bx, Dst_fabox, Src_fabox, dcomp, - scomp, ncomp, slice_cr_ratio, 2); - } + const IntVect cctype(AMREX_D_DECL(0,0,0)); + if( SliceType==cctype ) { + amrex::amrex_avgdown(Dst_bx, Dst_fabox, Src_fabox, dcomp, scomp, + ncomp, slice_cr_ratio); + } + const IntVect ndtype(AMREX_D_DECL(1,1,1)); + if( SliceType == ndtype ) { + amrex::amrex_avgdown_nodes(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio); + } + if( SliceType == WarpX::GetInstance().getEfield(0,0).ixType().toIntVect() ) { + amrex::amrex_avgdown_edges(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio, 0); + } + if( SliceType == WarpX::GetInstance().getEfield(0,1).ixType().toIntVect() ) { + amrex::amrex_avgdown_edges(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio, 1); + } + if( SliceType == WarpX::GetInstance().getEfield(0,2).ixType().toIntVect() ) { + amrex::amrex_avgdown_edges(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio, 2); + } + if( SliceType == WarpX::GetInstance().getBfield(0,0).ixType().toIntVect() ) { + amrex::amrex_avgdown_faces(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio, 0); + } + if( SliceType == WarpX::GetInstance().getBfield(0,1).ixType().toIntVect() ) { + amrex::amrex_avgdown_faces(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio, 1); + } + if( SliceType == WarpX::GetInstance().getBfield(0,2).ixType().toIntVect() ) { + amrex::amrex_avgdown_faces(Dst_bx, Dst_fabox, Src_fabox, dcomp, + scomp, ncomp, slice_cr_ratio, 2); + } - if ( mfi_dst.isValid() ) { - ++mfi_dst; - } + if ( mfi_dst.isValid() ) { + ++mfi_dst; + } - } - return cs_mf; + } + return cs_mf; } } @@ -299,7 +299,7 @@ CheckSliceInput( const RealBox real_box, RealBox &slice_cc_nd_box, warnMsg.str(), ablastr::warn_manager::WarnPriority::low); } - const auto very_small_number = 1E-10; + const auto very_small_number = 1E-10; // Factor to ensure index values computation depending on index type // const double fac = ( 1.0 - SliceType[idim] )*dom_geom[0].CellSize(idim)*0.5; @@ -425,15 +425,15 @@ InterpolateSliceValues(MultiFab& smf, IntVect interp_lo, RealBox slice_realbox, { for (MFIter mfi(smf); mfi.isValid(); ++mfi) { - const Box& bx = mfi.tilebox(); - FArrayBox& fabox = smf[mfi]; + const Box& bx = mfi.tilebox(); + FArrayBox& fabox = smf[mfi]; - for ( int idim = 0; idim < AMREX_SPACEDIM; ++idim) { - if ( interp_lo[idim] == 1 ) { + for ( int idim = 0; idim < AMREX_SPACEDIM; ++idim) { + if ( interp_lo[idim] == 1 ) { InterpolateLo( bx, fabox, slice_lo, geom, idim, SliceType, slice_realbox, 0, ncomp, nghost, real_box); - } - } + } + } } } diff --git a/Source/Diagnostics/WarpXOpenPMD.cpp b/Source/Diagnostics/WarpXOpenPMD.cpp index df1cf15f89a..2e9746fe5c8 100644 --- a/Source/Diagnostics/WarpXOpenPMD.cpp +++ b/Source/Diagnostics/WarpXOpenPMD.cpp @@ -378,13 +378,13 @@ WarpXOpenPMDPlot::WarpXOpenPMDPlot ( std::map< std::string, std::string > engine_parameters, std::vector fieldPMLdirections, const std::string& authors) - :m_Series(nullptr), - m_MPIRank{amrex::ParallelDescriptor::MyProc()}, - m_MPISize{amrex::ParallelDescriptor::NProcs()}, - m_Encoding(ie), - m_OpenPMDFileType(std::move(openPMDFileType)), - m_fieldPMLdirections(std::move(fieldPMLdirections)), - m_authors{authors} + : m_Series(nullptr), + m_MPIRank{amrex::ParallelDescriptor::MyProc()}, + m_MPISize{amrex::ParallelDescriptor::NProcs()}, + m_Encoding(ie), + m_OpenPMDFileType(std::move(openPMDFileType)), + m_fieldPMLdirections(std::move(fieldPMLdirections)), + m_authors{authors} { m_OpenPMDoptions = detail::getSeriesOptions(operator_type, operator_parameters, engine_type, engine_parameters); @@ -402,23 +402,23 @@ WarpXOpenPMDPlot::~WarpXOpenPMDPlot () std::string WarpXOpenPMDPlot::GetFileName (std::string& filepath) { - filepath.append("/"); - // transform paths for Windows -#ifdef _WIN32 - filepath = openPMD::auxiliary::replace_all(filepath, "/", "\\"); -#endif + filepath.append("/"); + // transform paths for Windows + #ifdef _WIN32 + filepath = openPMD::auxiliary::replace_all(filepath, "/", "\\"); + #endif - std::string filename = "openpmd"; - // - // OpenPMD supports timestepped names - // - if (m_Encoding == openPMD::IterationEncoding::fileBased) { - const std::string fileSuffix = std::string("_%0") + std::to_string(m_file_min_digits) + std::string("T"); - filename = filename.append(fileSuffix); - } - filename.append(".").append(m_OpenPMDFileType); - filepath.append(filename); - return filename; + std::string filename = "openpmd"; + // + // OpenPMD supports timestepped names + // + if (m_Encoding == openPMD::IterationEncoding::fileBased) { + const std::string fileSuffix = std::string("_%0") + std::to_string(m_file_min_digits) + std::string("T"); + filename = filename.append(fileSuffix); + } + filename.append(".").append(m_OpenPMDFileType); + filepath.append(filename); + return filename; } void WarpXOpenPMDPlot::SetStep (int ts, const std::string& dirPrefix, int file_min_digits, @@ -521,9 +521,9 @@ WarpXOpenPMDPlot::WriteOpenPMDParticles (const amrex::Vector& part const bool isBTD, const bool isLastBTDFlush, const amrex::Vector& totalParticlesFlushedAlready) { - WARPX_PROFILE("WarpXOpenPMDPlot::WriteOpenPMDParticles()"); +WARPX_PROFILE("WarpXOpenPMDPlot::WriteOpenPMDParticles()"); - for (unsigned i = 0, n = particle_diags.size(); i < n; ++i) { +for (unsigned i = 0, n = particle_diags.size(); i < n; ++i) { WarpXParticleContainer* pc = particle_diags[i].getParticleContainer(); PinnedMemoryParticleContainer* pinned_pc = particle_diags[i].getPinnedParticleContainer(); @@ -572,65 +572,64 @@ WarpXOpenPMDPlot::WriteOpenPMDParticles (const amrex::Vector& part int_flags.resize(tmp.NumIntComps(), 1); const auto mass = pc->AmIA() ? PhysConst::m_e : pc->getMass(); - RandomFilter const random_filter(particle_diags[i].m_do_random_filter, - particle_diags[i].m_random_fraction); - UniformFilter const uniform_filter(particle_diags[i].m_do_uniform_filter, - particle_diags[i].m_uniform_stride); - ParserFilter parser_filter(particle_diags[i].m_do_parser_filter, - utils::parser::compileParser + RandomFilter const random_filter(particle_diags[i].m_do_random_filter, + particle_diags[i].m_random_fraction); + UniformFilter const uniform_filter(particle_diags[i].m_do_uniform_filter, + particle_diags[i].m_uniform_stride); + ParserFilter parser_filter(particle_diags[i].m_do_parser_filter, + utils::parser::compileParser (particle_diags[i].m_particle_filter_parser.get()), pc->getMass(), time); - parser_filter.m_units = InputUnits::SI; - GeometryFilter const geometry_filter(particle_diags[i].m_do_geom_filter, + parser_filter.m_units = InputUnits::SI; + GeometryFilter const geometry_filter(particle_diags[i].m_do_geom_filter, particle_diags[i].m_diag_domain); - if (isBTD || use_pinned_pc) { - tmp.copyParticles(*pinned_pc, true); - particlesConvertUnits(ConvertDirection::WarpX_to_SI, &tmp, mass); - } else { - particlesConvertUnits(ConvertDirection::WarpX_to_SI, pc, mass); - using SrcData = WarpXParticleContainer::ParticleTileType::ConstParticleTileDataType; - tmp.copyParticles(*pc, - [random_filter,uniform_filter,parser_filter,geometry_filter] - AMREX_GPU_HOST_DEVICE - (const SrcData& src, int ip, const amrex::RandomEngine& engine) - { - const SuperParticleType& p = src.getSuperParticle(ip); - return random_filter(p, engine) * uniform_filter(p, engine) - * parser_filter(p, engine) * geometry_filter(p, engine); - }, true); - particlesConvertUnits(ConvertDirection::SI_to_WarpX, pc, mass); - } + if (isBTD || use_pinned_pc) { + tmp.copyParticles(*pinned_pc, true); + particlesConvertUnits(ConvertDirection::WarpX_to_SI, &tmp, mass); + } else { + particlesConvertUnits(ConvertDirection::WarpX_to_SI, pc, mass); + using SrcData = WarpXParticleContainer::ParticleTileType::ConstParticleTileDataType; + tmp.copyParticles(*pc, + [random_filter,uniform_filter,parser_filter,geometry_filter] + AMREX_GPU_HOST_DEVICE + (const SrcData& src, int ip, const amrex::RandomEngine& engine) + { + const SuperParticleType& p = src.getSuperParticle(ip); + return random_filter(p, engine) * uniform_filter(p, engine) + * parser_filter(p, engine) * geometry_filter(p, engine); + }, true); + particlesConvertUnits(ConvertDirection::SI_to_WarpX, pc, mass); + } // real_names contains a list of all real particle attributes. // real_flags is 1 or 0, whether quantity is dumped or not. - { - if (isBTD) { - DumpToFile(&tmp, - particle_diags[i].getSpeciesName(), - m_CurrentStep, - real_flags, - int_flags, - real_names, int_names, - pc->getCharge(), pc->getMass(), - isBTD, isLastBTDFlush, - totalParticlesFlushedAlready[i] - ); - } else { - DumpToFile(&tmp, - particle_diags[i].getSpeciesName(), - m_CurrentStep, - real_flags, - int_flags, - real_names, int_names, - pc->getCharge(), pc->getMass(), - isBTD, isLastBTDFlush, - 0 - ); - } + if (isBTD) { + DumpToFile(&tmp, + particle_diags[i].getSpeciesName(), + m_CurrentStep, + real_flags, + int_flags, + real_names, int_names, + pc->getCharge(), pc->getMass(), + isBTD, isLastBTDFlush, + totalParticlesFlushedAlready[i] + ); + } else { + DumpToFile(&tmp, + particle_diags[i].getSpeciesName(), + m_CurrentStep, + real_flags, + int_flags, + real_names, int_names, + pc->getCharge(), pc->getMass(), + isBTD, isLastBTDFlush, + 0 + ); + } } - } +} } void @@ -948,23 +947,23 @@ WarpXOpenPMDPlot::SaveRealProperty (ParticleIter& pti, { // note: WarpX does not yet use extra AoS Real attributes for( auto idx=0; idx d( - new amrex::ParticleReal[numParticleOnTile], - [](amrex::ParticleReal const *p){ delete[] p; } - ); - - for( auto kk=0; kk d( + new amrex::ParticleReal[numParticleOnTile], + [](amrex::ParticleReal const *p){ delete[] p; } + ); + + for( auto kk=0; kk(), {np}, options); - auto idType = openPMD::Dataset(openPMD::determineDatatype< uint64_t >(), {np}, options); - - auto const positionComponents = detail::getParticlePositionComponentLabels(); - for( auto const& comp : positionComponents ) { - currSpecies["position"][comp].resetDataset( realType ); - } + std::string options = "{}"; + if (isBTD) options = "{ \"resizable\": true }"; + auto realType = openPMD::Dataset(openPMD::determineDatatype(), {np}, options); + auto idType = openPMD::Dataset(openPMD::determineDatatype< uint64_t >(), {np}, options); + + auto const positionComponents = detail::getParticlePositionComponentLabels(); + for( auto const& comp : positionComponents ) { + currSpecies["position"][comp].resetDataset( realType ); + } - auto const scalar = openPMD::RecordComponent::SCALAR; - currSpecies["id"][scalar].resetDataset( idType ); + auto const scalar = openPMD::RecordComponent::SCALAR; + currSpecies["id"][scalar].resetDataset( idType ); } void @@ -1139,69 +1138,69 @@ void WarpXOpenPMDPlot::SetupFields ( openPMD::Container< openPMD::Mesh >& meshes, amrex::Geometry& full_geom ) const { - // meta data for ED-PIC extension - auto const period = full_geom.periodicity(); // TODO double-check: is this the proper global bound or of some level? - std::vector fieldBoundary(6, "reflecting"); - std::vector particleBoundary(6, "absorbing"); - fieldBoundary.resize(AMREX_SPACEDIM * 2); - particleBoundary.resize(AMREX_SPACEDIM * 2); - - const auto HalfFieldBoundarySize = static_cast(fieldBoundary.size() / 2u); - - for (auto i = 0; i < HalfFieldBoundarySize; ++i) - if (m_fieldPMLdirections.at(i)) - fieldBoundary.at(i) = "open"; - - for (int i = 0; i < HalfFieldBoundarySize; ++i) - if (period.isPeriodic(i)) { - fieldBoundary.at(2u * i) = "periodic"; - fieldBoundary.at(2u * i + 1u) = "periodic"; - particleBoundary.at(2u * i) = "periodic"; - particleBoundary.at(2u * i + 1u) = "periodic"; - } - - meshes.setAttribute("fieldSolver", []() { - switch (WarpX::electromagnetic_solver_id) { - case ElectromagneticSolverAlgo::Yee : - return "Yee"; - case ElectromagneticSolverAlgo::CKC : - return "CK"; - case ElectromagneticSolverAlgo::PSATD : - return "PSATD"; - default: - return "other"; - } - }()); - meshes.setAttribute("fieldBoundary", fieldBoundary); - meshes.setAttribute("particleBoundary", particleBoundary); - meshes.setAttribute("currentSmoothing", []() { - if (WarpX::use_filter) return "Binomial"; - else return "none"; - }()); - if (WarpX::use_filter) - meshes.setAttribute("currentSmoothingParameters", []() { - std::stringstream ss; - ss << "period=1;compensator=false"; + // meta data for ED-PIC extension + auto const period = full_geom.periodicity(); // TODO double-check: is this the proper global bound or of some level? + std::vector fieldBoundary(6, "reflecting"); + std::vector particleBoundary(6, "absorbing"); + fieldBoundary.resize(AMREX_SPACEDIM * 2); + particleBoundary.resize(AMREX_SPACEDIM * 2); + + const auto HalfFieldBoundarySize = static_cast(fieldBoundary.size() / 2u); + + for (auto i = 0; i < HalfFieldBoundarySize; ++i) + if (m_fieldPMLdirections.at(i)) + fieldBoundary.at(i) = "open"; + + for (int i = 0; i < HalfFieldBoundarySize; ++i) + if (period.isPeriodic(i)) { + fieldBoundary.at(2u * i) = "periodic"; + fieldBoundary.at(2u * i + 1u) = "periodic"; + particleBoundary.at(2u * i) = "periodic"; + particleBoundary.at(2u * i + 1u) = "periodic"; + } + + meshes.setAttribute("fieldSolver", []() { + switch (WarpX::electromagnetic_solver_id) { + case ElectromagneticSolverAlgo::Yee : + return "Yee"; + case ElectromagneticSolverAlgo::CKC : + return "CK"; + case ElectromagneticSolverAlgo::PSATD : + return "PSATD"; + default: + return "other"; + } + }()); + meshes.setAttribute("fieldBoundary", fieldBoundary); + meshes.setAttribute("particleBoundary", particleBoundary); + meshes.setAttribute("currentSmoothing", []() { + if (WarpX::use_filter) return "Binomial"; + else return "none"; + }()); + if (WarpX::use_filter) + meshes.setAttribute("currentSmoothingParameters", []() { + std::stringstream ss; + ss << "period=1;compensator=false"; #if (AMREX_SPACEDIM >= 2) - ss << ";numPasses_x=" << WarpX::filter_npass_each_dir[0]; + ss << ";numPasses_x=" << WarpX::filter_npass_each_dir[0]; #endif #if defined(WARPX_DIM_3D) - ss << ";numPasses_y=" << WarpX::filter_npass_each_dir[1]; - ss << ";numPasses_z=" << WarpX::filter_npass_each_dir[2]; + ss << ";numPasses_y=" << WarpX::filter_npass_each_dir[1]; + ss << ";numPasses_z=" << WarpX::filter_npass_each_dir[2]; #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - ss << ";numPasses_z=" << WarpX::filter_npass_each_dir[1]; + ss << ";numPasses_z=" << WarpX::filter_npass_each_dir[1]; #elif defined(WARPX_DIM_1D_Z) - ss << ";numPasses_z=" << WarpX::filter_npass_each_dir[0]; + ss << ";numPasses_z=" << WarpX::filter_npass_each_dir[0]; #endif - std::string currentSmoothingParameters = ss.str(); - return currentSmoothingParameters; - }()); - meshes.setAttribute("chargeCorrection", []() { - if (WarpX::do_dive_cleaning) return "hyperbolic"; // TODO or "spectral" or something? double-check - else return "none"; - }()); - if (WarpX::do_dive_cleaning) - meshes.setAttribute("chargeCorrectionParameters", "period=1"); + std::string currentSmoothingParameters = ss.str(); + return currentSmoothingParameters; + }()); + meshes.setAttribute("chargeCorrection", []() { + if (WarpX::do_dive_cleaning) return "hyperbolic"; // TODO or "spectral" or something? double-check + else return "none"; + }()); + if (WarpX::do_dive_cleaning) + meshes.setAttribute("chargeCorrectionParameters", "period=1"); } @@ -1485,33 +1484,33 @@ WarpXParticleCounter::WarpXParticleCounter (ParticleContainer* pc): m_MPIRank{amrex::ParallelDescriptor::MyProc()}, m_MPISize{amrex::ParallelDescriptor::NProcs()} { - m_ParticleCounterByLevel.resize(pc->finestLevel()+1); - m_ParticleOffsetAtRank.resize(pc->finestLevel()+1); - m_ParticleSizeAtRank.resize(pc->finestLevel()+1); + m_ParticleCounterByLevel.resize(pc->finestLevel()+1); + m_ParticleOffsetAtRank.resize(pc->finestLevel()+1); + m_ParticleSizeAtRank.resize(pc->finestLevel()+1); - for (auto currentLevel = 0; currentLevel <= pc->finestLevel(); currentLevel++) + for (auto currentLevel = 0; currentLevel <= pc->finestLevel(); currentLevel++) { - long numParticles = 0; // numParticles in this processor + long numParticles = 0; // numParticles in this processor - for (ParticleIter pti(*pc, currentLevel); pti.isValid(); ++pti) { - auto numParticleOnTile = pti.numParticles(); - numParticles += numParticleOnTile; - } + for (ParticleIter pti(*pc, currentLevel); pti.isValid(); ++pti) { + auto numParticleOnTile = pti.numParticles(); + numParticles += numParticleOnTile; + } - unsigned long long offset=0; // offset of this level - unsigned long long sum=0; // numParticles in this level (sum from all processors) + unsigned long long offset=0; // offset of this level + unsigned long long sum=0; // numParticles in this level (sum from all processors) - GetParticleOffsetOfProcessor(numParticles, offset, sum); + GetParticleOffsetOfProcessor(numParticles, offset, sum); - m_ParticleCounterByLevel[currentLevel] = sum; - m_ParticleOffsetAtRank[currentLevel] = offset; - m_ParticleSizeAtRank[currentLevel] = numParticles; + m_ParticleCounterByLevel[currentLevel] = sum; + m_ParticleOffsetAtRank[currentLevel] = offset; + m_ParticleSizeAtRank[currentLevel] = numParticles; - // adjust offset, it should be numbered after particles from previous levels - for (auto lv=0; lv,3>& Efield, amrex::MultiFab& divEfield ) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ if (m_fdtd_algo == ElectromagneticSolverAlgo::Yee || m_fdtd_algo == ElectromagneticSolverAlgo::HybridPIC){ diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp index 0ccff8bac9a..4425e655917 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp @@ -63,8 +63,8 @@ void FiniteDifferenceSolver::EvolveB ( amrex::ignore_unused(area_mod, ECTRhofield, Venl, flag_info_cell, borrowing); #endif - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ if ((m_fdtd_algo == ElectromagneticSolverAlgo::Yee)|| (m_fdtd_algo == ElectromagneticSolverAlgo::HybridPIC)){ diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp index 35cbb6ede93..1fb2dd85e60 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveBPML.cpp @@ -46,11 +46,11 @@ void FiniteDifferenceSolver::EvolveBPML ( amrex::Real const dt, const bool dive_cleaning) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ amrex::ignore_unused(Bfield, Efield, dt, dive_cleaning); - WARPX_ABORT_WITH_MESSAGE( + WARPX_ABORT_WITH_MESSAGE( "PML are not implemented in cylindrical geometry."); #else if (m_grid_type == GridType::Collocated) { diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp index 87145daa79b..93352ce9896 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveEPML.cpp @@ -52,8 +52,8 @@ void FiniteDifferenceSolver::EvolveEPML ( MultiSigmaBox const& sigba, amrex::Real const dt, bool pml_has_particles ) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ amrex::ignore_unused(Efield, Bfield, Jfield, Ffield, sigba, dt, pml_has_particles, edge_lengths); WARPX_ABORT_WITH_MESSAGE( diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp index 752adb8e628..83014a08026 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveF.cpp @@ -50,8 +50,8 @@ void FiniteDifferenceSolver::EvolveF ( int const rhocomp, amrex::Real const dt ) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ if (m_fdtd_algo == ElectromagneticSolverAlgo::Yee){ diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp index ee9b7ec368e..4ef056c937a 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveFPML.cpp @@ -44,8 +44,8 @@ void FiniteDifferenceSolver::EvolveFPML ( std::array< amrex::MultiFab*, 3 > const Efield, amrex::Real const dt ) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ amrex::ignore_unused(Ffield, Efield, dt); WARPX_ABORT_WITH_MESSAGE( diff --git a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp index d58c2889c62..ff220887099 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICSolveE.cpp @@ -28,8 +28,8 @@ void FiniteDifferenceSolver::CalculateCurrentAmpere ( std::array< std::unique_ptr, 3 > const& edge_lengths, int lev ) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) if (m_fdtd_algo == ElectromagneticSolverAlgo::HybridPIC) { #ifdef WARPX_DIM_RZ CalculateCurrentAmpereCylindrical ( @@ -376,8 +376,8 @@ void FiniteDifferenceSolver::HybridPICSolveE ( int lev, HybridPICModel const* hybrid_model, const bool include_resistivity_term ) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) if (m_fdtd_algo == ElectromagneticSolverAlgo::HybridPIC) { #ifdef WARPX_DIM_RZ diff --git a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp index 3c9dc8db1c1..3aee7697073 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp +++ b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicEvolveE.cpp @@ -43,8 +43,8 @@ void FiniteDifferenceSolver::MacroscopicEvolveE ( std::unique_ptr const& macroscopic_properties) { - // Select algorithm (The choice of algorithm is a runtime option, - // but we compile code for each algorithm, using templates) + // Select algorithm (The choice of algorithm is a runtime option, + // but we compile code for each algorithm, using templates) #ifdef WARPX_DIM_RZ amrex::ignore_unused(Efield, Bfield, Jfield, edge_lengths, dt, macroscopic_properties); diff --git a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H index d0806c332e3..b26a918c9ae 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H +++ b/Source/FieldSolver/FiniteDifferenceSolver/MacroscopicProperties/MacroscopicProperties.H @@ -31,72 +31,72 @@ class MacroscopicProperties { public: - MacroscopicProperties (); // constructor - /** Read user-defined macroscopic properties. Called in constructor. */ - void ReadParameters (); - /** Initialize multifabs storing macroscopic multifabs */ - void InitData (); - - /** return MultiFab, sigma (conductivity) of the medium. */ - amrex::MultiFab& getsigma_mf () {return (*m_sigma_mf);} - /** return MultiFab, epsilon (permittivity) of the medium. */ - amrex::MultiFab& getepsilon_mf () {return (*m_eps_mf);} - /** return MultiFab, mu (permeability) of the medium. */ - amrex::MultiFab& getmu_mf () {return (*m_mu_mf);} - - /** Initializes the Multifabs storing macroscopic properties - * with user-defined functions(x,y,z). - */ - void InitializeMacroMultiFabUsingParser (amrex::MultiFab *macro_mf, + MacroscopicProperties (); // constructor + /** Read user-defined macroscopic properties. Called in constructor. */ + void ReadParameters (); + /** Initialize multifabs storing macroscopic multifabs */ + void InitData (); + + /** return MultiFab, sigma (conductivity) of the medium. */ + amrex::MultiFab& getsigma_mf () {return (*m_sigma_mf);} + /** return MultiFab, epsilon (permittivity) of the medium. */ + amrex::MultiFab& getepsilon_mf () {return (*m_eps_mf);} + /** return MultiFab, mu (permeability) of the medium. */ + amrex::MultiFab& getmu_mf () {return (*m_mu_mf);} + + /** Initializes the Multifabs storing macroscopic properties + * with user-defined functions(x,y,z). + */ + void InitializeMacroMultiFabUsingParser (amrex::MultiFab *macro_mf, amrex::ParserExecutor<3> const& macro_parser, const amrex::GpuArray& dx_lev, const amrex::RealBox& prob_domain_lev); - /** Gpu Vector with index type of the conductivity multifab */ - amrex::GpuArray sigma_IndexType; - /** Gpu Vector with index type of the permittivity multifab */ - amrex::GpuArray epsilon_IndexType; - /** Gpu Vector with index type of the permeability multifab */ - amrex::GpuArray mu_IndexType; - /** Gpu Vector with index type of the Ex multifab */ - amrex::GpuArray Ex_IndexType; - /** Gpu Vector with index type of the Ey multifab */ - amrex::GpuArray Ey_IndexType; - /** Gpu Vector with index type of the Ez multifab */ - amrex::GpuArray Ez_IndexType; - /** Gpu Vector with index type of coarsening ratio with default value (1,1,1) */ - amrex::GpuArray macro_cr_ratio; + /** Gpu Vector with index type of the conductivity multifab */ + amrex::GpuArray sigma_IndexType; + /** Gpu Vector with index type of the permittivity multifab */ + amrex::GpuArray epsilon_IndexType; + /** Gpu Vector with index type of the permeability multifab */ + amrex::GpuArray mu_IndexType; + /** Gpu Vector with index type of the Ex multifab */ + amrex::GpuArray Ex_IndexType; + /** Gpu Vector with index type of the Ey multifab */ + amrex::GpuArray Ey_IndexType; + /** Gpu Vector with index type of the Ez multifab */ + amrex::GpuArray Ez_IndexType; + /** Gpu Vector with index type of coarsening ratio with default value (1,1,1) */ + amrex::GpuArray macro_cr_ratio; private: - /** Conductivity, sigma, of the medium */ - amrex::Real m_sigma = 0.0; - /** Permittivity, epsilon, of the medium */ - amrex::Real m_epsilon = PhysConst::ep0; - /** Permeability, mu, of the medium */ - amrex::Real m_mu = PhysConst::mu0; - /** Multifab for m_sigma */ - std::unique_ptr m_sigma_mf; - /** Multifab for m_epsilon */ - std::unique_ptr m_eps_mf; - /** Multifab for m_mu */ - std::unique_ptr m_mu_mf; - - /** Stores initialization type for conductivity : constant or parser */ - std::string m_sigma_s = "constant"; - /** Stores initialization type for permittivity : constant or parser */ - std::string m_epsilon_s = "constant"; - /** Stores initialization type for permeability : constant or parser */ - std::string m_mu_s = "constant"; - - /** string for storing parser function */ - std::string m_str_sigma_function; - std::string m_str_epsilon_function; - std::string m_str_mu_function; - /** Parser Wrappers */ - std::unique_ptr m_sigma_parser; - std::unique_ptr m_epsilon_parser; - std::unique_ptr m_mu_parser; + /** Conductivity, sigma, of the medium */ + amrex::Real m_sigma = 0.0; + /** Permittivity, epsilon, of the medium */ + amrex::Real m_epsilon = PhysConst::ep0; + /** Permeability, mu, of the medium */ + amrex::Real m_mu = PhysConst::mu0; + /** Multifab for m_sigma */ + std::unique_ptr m_sigma_mf; + /** Multifab for m_epsilon */ + std::unique_ptr m_eps_mf; + /** Multifab for m_mu */ + std::unique_ptr m_mu_mf; + + /** Stores initialization type for conductivity : constant or parser */ + std::string m_sigma_s = "constant"; + /** Stores initialization type for permittivity : constant or parser */ + std::string m_epsilon_s = "constant"; + /** Stores initialization type for permeability : constant or parser */ + std::string m_mu_s = "constant"; + + /** string for storing parser function */ + std::string m_str_sigma_function; + std::string m_str_epsilon_function; + std::string m_str_mu_function; + /** Parser Wrappers */ + std::unique_ptr m_sigma_parser; + std::unique_ptr m_epsilon_parser; + std::unique_ptr m_mu_parser; }; @@ -110,25 +110,25 @@ private: */ struct LaxWendroffAlgo { - AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - static amrex::Real alpha (amrex::Real const sigma, - amrex::Real const epsilon, - amrex::Real dt) { - using namespace amrex; - const amrex::Real fac1 = 0.5_rt * sigma * dt / epsilon; - const amrex::Real alpha = (1._rt - fac1)/(1._rt + fac1); - return alpha; - } - - AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - static amrex::Real beta (amrex::Real const sigma, + AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE + static amrex::Real alpha (amrex::Real const sigma, amrex::Real const epsilon, amrex::Real dt) { - using namespace amrex; - const amrex::Real fac1 = 0.5_rt * sigma * dt / epsilon; - const amrex::Real beta = dt / ( epsilon * (1._rt + fac1) ); - return beta; - } + using namespace amrex; + const amrex::Real fac1 = 0.5_rt * sigma * dt / epsilon; + const amrex::Real alpha = (1._rt - fac1)/(1._rt + fac1); + return alpha; + } + + AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE + static amrex::Real beta (amrex::Real const sigma, + amrex::Real const epsilon, + amrex::Real dt) { + using namespace amrex; + const amrex::Real fac1 = 0.5_rt * sigma * dt / epsilon; + const amrex::Real beta = dt / ( epsilon * (1._rt + fac1) ); + return beta; + } }; @@ -142,25 +142,25 @@ struct LaxWendroffAlgo { */ struct BackwardEulerAlgo { - AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - static amrex::Real alpha (amrex::Real const sigma, - amrex::Real const epsilon, - amrex::Real dt) { - using namespace amrex; - const amrex::Real fac1 = sigma * dt / epsilon; - const amrex::Real alpha = (1._rt)/(1._rt + fac1); - return alpha; - } - - AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE - static amrex::Real beta (amrex::Real const sigma, + AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE + static amrex::Real alpha (amrex::Real const sigma, amrex::Real const epsilon, amrex::Real dt) { - using namespace amrex; - const amrex::Real fac1 = sigma * dt / epsilon; - const amrex::Real beta = dt / ( epsilon * (1._rt + fac1) ); - return beta; - } + using namespace amrex; + const amrex::Real fac1 = sigma * dt / epsilon; + const amrex::Real alpha = (1._rt)/(1._rt + fac1); + return alpha; + } + + AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE + static amrex::Real beta (amrex::Real const sigma, + amrex::Real const epsilon, + amrex::Real dt) { + using namespace amrex; + const amrex::Real fac1 = sigma * dt / epsilon; + const amrex::Real beta = dt / ( epsilon * (1._rt + fac1) ); + return beta; + } }; diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.cpp index 075645233ec..640cf1b82af 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmComoving.cpp @@ -31,7 +31,7 @@ PsatdAlgorithmComoving::PsatdAlgorithmComoving (const SpectralKSpace& spectral_k const amrex::Vector& v_comoving, const amrex::Real dt, const bool update_with_rho) - // Members initialization + // Members initialization : SpectralBaseAlgorithm{spectral_kspace, dm, spectral_index, norder_x, norder_y, norder_z, grid_type}, // Initialize the infinite-order k vectors (the argument n_order = -1 selects // the infinite order option, the argument grid_type=GridType::Staggered is then irrelevant) diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp index dad10716655..71cee95524e 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmGalileanRZ.cpp @@ -24,7 +24,7 @@ PsatdAlgorithmGalileanRZ::PsatdAlgorithmGalileanRZ (SpectralKSpaceRZ const & spe const amrex::Vector& v_galilean, amrex::Real const dt, bool const update_with_rho): - // Initialize members of base class + // Initialize members of base class SpectralBaseAlgorithmRZ{spectral_kspace, dm, spectral_index, norder_z, grid_type}, m_dt{dt}, m_v_galilean{v_galilean}, diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.cpp index 172715b988a..d635a5debe3 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithm.cpp @@ -70,7 +70,7 @@ SpectralBaseAlgorithm::ComputeSpectralDivE ( const Box& bx = field_data.fields[mfi].box(); // Extract arrays for the fields to be updated - const Array4 fields = field_data.fields[mfi].array(); + const Array4 fields = field_data.fields[mfi].array(); // Extract pointers for the k vectors const Real* modified_kx_arr = modified_kx_vec[mfi].dataPtr(); #if defined(WARPX_DIM_3D) diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H index 1a935de8cdf..ce3c26b6297 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H @@ -88,7 +88,7 @@ class SpectralBaseAlgorithmRZ // Compute and assign the modified k vectors : m_spectral_index(spectral_index), modified_kz_vec(spectral_kspace.getModifiedKComponent(dm, 1, norder_z, grid_type)) - {} + {} SpectralFieldIndex m_spectral_index; diff --git a/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp b/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp index d882d32d758..6d35234f0ed 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp @@ -147,7 +147,7 @@ SpectralKSpace::getSpectralShiftFactor( const DistributionMapping& dm, { // Initialize an empty DeviceVector in each box SpectralShiftFactor shift_factor( spectralspace_ba, dm ); - // Loop over boxes and allocate the corresponding DeviceVector + // Loop over boxes and allocate the corresponding DeviceVector // for each box owned by the local MPI proc for ( MFIter mfi(spectralspace_ba, dm); mfi.isValid(); ++mfi ){ const Gpu::DeviceVector& k = k_vec[i_dim][mfi]; diff --git a/Source/FieldSolver/WarpX_QED_K.H b/Source/FieldSolver/WarpX_QED_K.H index a20dde8b9a7..94903f3cb35 100644 --- a/Source/FieldSolver/WarpX_QED_K.H +++ b/Source/FieldSolver/WarpX_QED_K.H @@ -220,7 +220,7 @@ const amrex::Real dyi = 1._rt/dy; invAz[1]*Omega[1] + invAz[2]*Omega[2]); - // Adding the QED corrections to the origional fields + // Adding the QED corrections to the origional fields Ex(j,k,l) = Ex(j,k,l) + 0.5_rt*dt*dEx; @@ -346,7 +346,7 @@ const amrex::Real dyi = 1._rt/dy; invAz[1]*Omega[1] + invAz[2]*Omega[2]); - // Adding the QED corrections to the origional fields + // Adding the QED corrections to the origional fields Ex(j,k,0) = Ex(j,k,0) + 0.5_rt*dt*dEx; diff --git a/Source/Fluids/WarpXFluidContainer.H b/Source/Fluids/WarpXFluidContainer.H index 32f9faf7562..ac6824479e6 100644 --- a/Source/Fluids/WarpXFluidContainer.H +++ b/Source/Fluids/WarpXFluidContainer.H @@ -185,7 +185,7 @@ protected: public: - // MultiFabs that contain the density (N) and momentum density (NU) of this fluid species, for each refinement level + // MultiFabs that contain the density (N) and momentum density (NU) of this fluid species, for each refinement level amrex::Vector< std::unique_ptr > N; amrex::Vector, 3 > > NU; diff --git a/Source/Fluids/WarpXFluidContainer.cpp b/Source/Fluids/WarpXFluidContainer.cpp index 03cc9b66101..68cb65b6174 100644 --- a/Source/Fluids/WarpXFluidContainer.cpp +++ b/Source/Fluids/WarpXFluidContainer.cpp @@ -263,8 +263,8 @@ void WarpXFluidContainer::Evolve( WARPX_PROFILE("WarpXFluidContainer::Evolve"); if (rho && ! skip_deposition && ! do_not_deposit) { - // Deposit charge before particle push, in component 0 of MultiFab rho. - DepositCharge(lev, *rho, 0); + // Deposit charge before particle push, in component 0 of MultiFab rho. + DepositCharge(lev, *rho, 0); } // Step the Lorentz Term @@ -963,7 +963,7 @@ void WarpXFluidContainer::GatherAndPush ( bool external_b_fields; // Needs intializing - // Prepare interpolation of current components to cell center + // Prepare interpolation of current components to cell center amrex::GpuArray Nodal_type = amrex::GpuArray{0, 0, 0}; amrex::GpuArray Ex_type = amrex::GpuArray{0, 0, 0}; amrex::GpuArray Ey_type = amrex::GpuArray{0, 0, 0}; diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index 8e3dfec1417..c1dea61f2b7 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -612,9 +612,9 @@ struct InjectorMomentum object(t, temperature, velocity) { } - // This constructor stores a InjectorMomentumJuttner in union object. - InjectorMomentum (InjectorMomentumJuttner* t, - GetTemperature const& temperature, GetVelocity const& velocity) + // This constructor stores a InjectorMomentumJuttner in union object. + InjectorMomentum (InjectorMomentumJuttner* t, + GetTemperature const& temperature, GetVelocity const& velocity) : type(Type::juttner), object(t, temperature, velocity) { } diff --git a/Source/Initialization/WarpXInitData.cpp b/Source/Initialization/WarpXInitData.cpp index fbf173992c2..683955e764a 100644 --- a/Source/Initialization/WarpXInitData.cpp +++ b/Source/Initialization/WarpXInitData.cpp @@ -875,40 +875,40 @@ WarpX::InitLevelData (int lev, Real /*time*/) if (E_ext_grid_s == "parse_e_ext_grid_function") { #ifdef WARPX_DIM_RZ - WARPX_ABORT_WITH_MESSAGE( - "E and B parser for external fields does not work with RZ -- TO DO"); + WARPX_ABORT_WITH_MESSAGE( + "E and B parser for external fields does not work with RZ -- TO DO"); #endif - //! Strings storing parser function to initialize the components of the electric field on the grid - std::string str_Ex_ext_grid_function; - std::string str_Ey_ext_grid_function; - std::string str_Ez_ext_grid_function; - - utils::parser::Store_parserString(pp_warpx, "Ex_external_grid_function(x,y,z)", - str_Ex_ext_grid_function); - utils::parser::Store_parserString(pp_warpx, "Ey_external_grid_function(x,y,z)", - str_Ey_ext_grid_function); - utils::parser::Store_parserString(pp_warpx, "Ez_external_grid_function(x,y,z)", - str_Ez_ext_grid_function); - - Exfield_parser = std::make_unique( - utils::parser::makeParser(str_Ex_ext_grid_function,{"x","y","z"})); - Eyfield_parser = std::make_unique( - utils::parser::makeParser(str_Ey_ext_grid_function,{"x","y","z"})); - Ezfield_parser = std::make_unique( - utils::parser::makeParser(str_Ez_ext_grid_function,{"x","y","z"})); - - // Initialize Efield_fp with external function - InitializeExternalFieldsOnGridUsingParser(Efield_fp[lev][0].get(), - Efield_fp[lev][1].get(), - Efield_fp[lev][2].get(), - Exfield_parser->compile<3>(), - Eyfield_parser->compile<3>(), - Ezfield_parser->compile<3>(), - m_edge_lengths[lev], - m_face_areas[lev], - 'E', - lev, PatchType::fine); + //! Strings storing parser function to initialize the components of the electric field on the grid + std::string str_Ex_ext_grid_function; + std::string str_Ey_ext_grid_function; + std::string str_Ez_ext_grid_function; + + utils::parser::Store_parserString(pp_warpx, "Ex_external_grid_function(x,y,z)", + str_Ex_ext_grid_function); + utils::parser::Store_parserString(pp_warpx, "Ey_external_grid_function(x,y,z)", + str_Ey_ext_grid_function); + utils::parser::Store_parserString(pp_warpx, "Ez_external_grid_function(x,y,z)", + str_Ez_ext_grid_function); + + Exfield_parser = std::make_unique( + utils::parser::makeParser(str_Ex_ext_grid_function,{"x","y","z"})); + Eyfield_parser = std::make_unique( + utils::parser::makeParser(str_Ey_ext_grid_function,{"x","y","z"})); + Ezfield_parser = std::make_unique( + utils::parser::makeParser(str_Ez_ext_grid_function,{"x","y","z"})); + + // Initialize Efield_fp with external function + InitializeExternalFieldsOnGridUsingParser(Efield_fp[lev][0].get(), + Efield_fp[lev][1].get(), + Efield_fp[lev][2].get(), + Exfield_parser->compile<3>(), + Eyfield_parser->compile<3>(), + Ezfield_parser->compile<3>(), + m_edge_lengths[lev], + m_face_areas[lev], + 'E', + lev, PatchType::fine); #ifdef AMREX_USE_EB // We initialize ECTRhofield consistently with the Efield @@ -919,35 +919,35 @@ WarpX::InitLevelData (int lev, Real /*time*/) } #endif - if (lev > 0) { - InitializeExternalFieldsOnGridUsingParser(Efield_aux[lev][0].get(), - Efield_aux[lev][1].get(), - Efield_aux[lev][2].get(), - Exfield_parser->compile<3>(), - Eyfield_parser->compile<3>(), - Ezfield_parser->compile<3>(), - m_edge_lengths[lev], - m_face_areas[lev], - 'E', - lev, PatchType::fine); - - InitializeExternalFieldsOnGridUsingParser(Efield_cp[lev][0].get(), - Efield_cp[lev][1].get(), - Efield_cp[lev][2].get(), - Exfield_parser->compile<3>(), - Eyfield_parser->compile<3>(), - Ezfield_parser->compile<3>(), - m_edge_lengths[lev], - m_face_areas[lev], - 'E', - lev, PatchType::coarse); + if (lev > 0) { + InitializeExternalFieldsOnGridUsingParser(Efield_aux[lev][0].get(), + Efield_aux[lev][1].get(), + Efield_aux[lev][2].get(), + Exfield_parser->compile<3>(), + Eyfield_parser->compile<3>(), + Ezfield_parser->compile<3>(), + m_edge_lengths[lev], + m_face_areas[lev], + 'E', + lev, PatchType::fine); + + InitializeExternalFieldsOnGridUsingParser(Efield_cp[lev][0].get(), + Efield_cp[lev][1].get(), + Efield_cp[lev][2].get(), + Exfield_parser->compile<3>(), + Eyfield_parser->compile<3>(), + Ezfield_parser->compile<3>(), + m_edge_lengths[lev], + m_face_areas[lev], + 'E', + lev, PatchType::coarse); #ifdef AMREX_USE_EB - if (WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::ECT) { - // We initialize ECTRhofield consistently with the Efield - m_fdtd_solver_cp[lev]->EvolveECTRho(Efield_cp[lev], m_edge_lengths[lev], - m_face_areas[lev], ECTRhofield[lev], lev); + if (WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::ECT) { + // We initialize ECTRhofield consistently with the Efield + m_fdtd_solver_cp[lev]->EvolveECTRho(Efield_cp[lev], m_edge_lengths[lev], + m_face_areas[lev], ECTRhofield[lev], lev); - } + } #endif } } @@ -1018,27 +1018,27 @@ WarpX::InitializeExternalFieldsOnGridUsingParser ( for ( MFIter mfi(*mfx, TilingIfNotGPU()); mfi.isValid(); ++mfi) { - const amrex::Box& tbx = mfi.tilebox( x_nodal_flag, mfx->nGrowVect() ); - const amrex::Box& tby = mfi.tilebox( y_nodal_flag, mfy->nGrowVect() ); - const amrex::Box& tbz = mfi.tilebox( z_nodal_flag, mfz->nGrowVect() ); + const amrex::Box& tbx = mfi.tilebox( x_nodal_flag, mfx->nGrowVect() ); + const amrex::Box& tby = mfi.tilebox( y_nodal_flag, mfy->nGrowVect() ); + const amrex::Box& tbz = mfi.tilebox( z_nodal_flag, mfz->nGrowVect() ); - auto const& mfxfab = mfx->array(mfi); - auto const& mfyfab = mfy->array(mfi); - auto const& mfzfab = mfz->array(mfi); + auto const& mfxfab = mfx->array(mfi); + auto const& mfyfab = mfy->array(mfi); + auto const& mfzfab = mfz->array(mfi); #ifdef AMREX_USE_EB - amrex::Array4 const& lx = edge_lengths[0]->array(mfi); - amrex::Array4 const& ly = edge_lengths[1]->array(mfi); - amrex::Array4 const& lz = edge_lengths[2]->array(mfi); - amrex::Array4 const& Sx = face_areas[0]->array(mfi); - amrex::Array4 const& Sy = face_areas[1]->array(mfi); - amrex::Array4 const& Sz = face_areas[2]->array(mfi); + amrex::Array4 const& lx = edge_lengths[0]->array(mfi); + amrex::Array4 const& ly = edge_lengths[1]->array(mfi); + amrex::Array4 const& lz = edge_lengths[2]->array(mfi); + amrex::Array4 const& Sx = face_areas[0]->array(mfi); + amrex::Array4 const& Sy = face_areas[1]->array(mfi); + amrex::Array4 const& Sz = face_areas[2]->array(mfi); #if defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - const amrex::Dim3 lx_lo = amrex::lbound(lx); - const amrex::Dim3 lx_hi = amrex::ubound(lx); - const amrex::Dim3 lz_lo = amrex::lbound(lz); - const amrex::Dim3 lz_hi = amrex::ubound(lz); + const amrex::Dim3 lx_lo = amrex::lbound(lx); + const amrex::Dim3 lx_hi = amrex::ubound(lx); + const amrex::Dim3 lz_lo = amrex::lbound(lz); + const amrex::Dim3 lz_hi = amrex::ubound(lz); #endif #if defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) @@ -1048,7 +1048,7 @@ WarpX::InitializeExternalFieldsOnGridUsingParser ( #endif #else - amrex::ignore_unused(edge_lengths, face_areas, field); + amrex::ignore_unused(edge_lengths, face_areas, field); #endif amrex::ParallelFor (tbx, tby, tbz, @@ -1412,7 +1412,7 @@ WarpX::ReadExternalFieldFromFile ( "XZ expects axisLabels {x, z}"); #elif defined(WARPX_DIM_1D_Z) WARPX_ABORT_WITH_MESSAGE( - "Reading from openPMD for external fields is not known to work with 1D3V (see #3830)"); + "Reading from openPMD for external fields is not known to work with 1D3V (see #3830)"); WARPX_ALWAYS_ASSERT_WITH_MESSAGE(fileGeom == "cartesian", "1D3V can only read from files with cartesian geometry"); WARPX_ALWAYS_ASSERT_WITH_MESSAGE(axisLabels[0] == "z"); #elif defined(WARPX_DIM_RZ) diff --git a/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp b/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp index 714e4bd52e1..6cb9c50dc21 100644 --- a/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp +++ b/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp @@ -216,7 +216,7 @@ WarpXLaserProfiles::FromFileLaserProfile::parse_lasy_file(std::string lasy_file_ } //Broadcast parameters - ParallelDescriptor::Bcast(&m_params.file_in_cartesian_geom, 1, ParallelDescriptor::IOProcessorNumber()); + ParallelDescriptor::Bcast(&m_params.file_in_cartesian_geom, 1, ParallelDescriptor::IOProcessorNumber()); ParallelDescriptor::Bcast(&m_params.nt, 1, ParallelDescriptor::IOProcessorNumber()); ParallelDescriptor::Bcast(&m_params.nx, 1, ParallelDescriptor::IOProcessorNumber()); ParallelDescriptor::Bcast(&m_params.ny, 1, ParallelDescriptor::IOProcessorNumber()); diff --git a/Source/Make.WarpX b/Source/Make.WarpX index d1c77482f75..06fff2052e2 100644 --- a/Source/Make.WarpX +++ b/Source/Make.WarpX @@ -121,7 +121,7 @@ ifeq ($(QED),TRUE) CFLAGS += -DWARPX_QED_TABLE_GEN FFLAGS += -DWARPX_QED_TABLE_GEN F90FLAGS += -DWARPX_QED_TABLE_GEN - USERSuffix := $(USERSuffix).GENTABLES + USERSuffix := $(USERSuffix).GENTABLES endif endif @@ -149,25 +149,25 @@ ifeq ($(USE_OPENBC_POISSON),TRUE) endif ifeq ($(USE_OPENPMD), TRUE) - # try pkg-config query - ifeq (0, $(shell pkg-config "openPMD >= 0.15.1"; echo $$?)) - CXXFLAGS += $(shell pkg-config --cflags openPMD) - LIBRARY_LOCATIONS += $(shell pkg-config --variable=libdir openPMD) - libraries += $(shell pkg-config --libs-only-l openPMD) - # fallback to manual settings - else - OPENPMD_LIB_PATH ?= NOT_SET - ifneq ($(OPENPMD_LIB_PATH),NOT_SET) - LIBRARY_LOCATIONS += $(OPENPMD_LIB_PATH) - endif - OPENPMD_INCLUDE_PATH ?= NOT_SET - ifneq ($(OPENPMD_INCLUDE_PATH),NOT_SET) - INCLUDE_LOCATIONS += $(OPENPMD_INCLUDE_PATH) - endif - libraries += -lopenPMD - endif - DEFINES += -DWARPX_USE_OPENPMD - USERSuffix := $(USERSuffix).OPMD + # try pkg-config query + ifeq (0, $(shell pkg-config "openPMD >= 0.15.1"; echo $$?)) + CXXFLAGS += $(shell pkg-config --cflags openPMD) + LIBRARY_LOCATIONS += $(shell pkg-config --variable=libdir openPMD) + libraries += $(shell pkg-config --libs-only-l openPMD) + # fallback to manual settings + else + OPENPMD_LIB_PATH ?= NOT_SET + ifneq ($(OPENPMD_LIB_PATH),NOT_SET) + LIBRARY_LOCATIONS += $(OPENPMD_LIB_PATH) + endif + OPENPMD_INCLUDE_PATH ?= NOT_SET + ifneq ($(OPENPMD_INCLUDE_PATH),NOT_SET) + INCLUDE_LOCATIONS += $(OPENPMD_INCLUDE_PATH) + endif + libraries += -lopenPMD + endif + DEFINES += -DWARPX_USE_OPENPMD + USERSuffix := $(USERSuffix).OPMD endif @@ -183,18 +183,18 @@ ifeq ($(USE_PSATD),TRUE) LIBRARY_LOCATIONS += $(ROC_PATH)/rocfft/lib libraries += -lrocfft else # Running on CPU - # Use FFTW - ifeq ($(PRECISION),FLOAT) - libraries += -lfftw3f_mpi -lfftw3f -lfftw3f_threads - else - libraries += -lfftw3_mpi -lfftw3 -lfftw3_threads - endif - FFTW_HOME ?= NOT_SET - ifneq ($(FFTW_HOME),NOT_SET) - VPATH_LOCATIONS += $(FFTW_HOME)/include - INCLUDE_LOCATIONS += $(FFTW_HOME)/include - LIBRARY_LOCATIONS += $(FFTW_HOME)/lib - endif + # Use FFTW + ifeq ($(PRECISION),FLOAT) + libraries += -lfftw3f_mpi -lfftw3f -lfftw3f_threads + else + libraries += -lfftw3_mpi -lfftw3 -lfftw3_threads + endif + FFTW_HOME ?= NOT_SET + ifneq ($(FFTW_HOME),NOT_SET) + VPATH_LOCATIONS += $(FFTW_HOME)/include + INCLUDE_LOCATIONS += $(FFTW_HOME)/include + LIBRARY_LOCATIONS += $(FFTW_HOME)/lib + endif endif ifeq ($(USE_RZ),TRUE) # Use blas and lapack @@ -211,7 +211,7 @@ ifeq ($(USE_RZ),TRUE) endif ifeq ($(USE_HDF5),TRUE) - DEFINES += -DWARPX_USE_HDF5 + DEFINES += -DWARPX_USE_HDF5 endif ifeq ($(USE_GPUCLOCK),TRUE) diff --git a/Source/Parallelization/GuardCellManager.cpp b/Source/Parallelization/GuardCellManager.cpp index 4d31cad9a1c..d71a65446d7 100644 --- a/Source/Parallelization/GuardCellManager.cpp +++ b/Source/Parallelization/GuardCellManager.cpp @@ -222,7 +222,7 @@ guardCellManager::Init ( } } #else - amrex::ignore_unused(do_pml, do_pml_in_domain, pml_ncell); + amrex::ignore_unused(do_pml, do_pml_in_domain, pml_ncell); #endif // All boxes should have the same number of guard cells, to avoid temporary parallel copies: diff --git a/Source/Parallelization/WarpXComm.cpp b/Source/Parallelization/WarpXComm.cpp index e5ede95272c..179afe86ae2 100644 --- a/Source/Parallelization/WarpXComm.cpp +++ b/Source/Parallelization/WarpXComm.cpp @@ -674,9 +674,9 @@ WarpX::FillBoundaryE_avg (int lev, PatchType patch_type, IntVect ng) if (patch_type == PatchType::fine) { if (do_pml && pml[lev]->ok()) - { + { WARPX_ABORT_WITH_MESSAGE("Averaged Galilean PSATD with PML is not yet implemented"); - } + } const amrex::Periodicity& period = Geom(lev).periodicity(); if ( safe_guard_cells ){ @@ -694,9 +694,9 @@ WarpX::FillBoundaryE_avg (int lev, PatchType patch_type, IntVect ng) else if (patch_type == PatchType::coarse) { if (do_pml && pml[lev]->ok()) - { + { WARPX_ABORT_WITH_MESSAGE("Averaged Galilean PSATD with PML is not yet implemented"); - } + } const amrex::Periodicity& cperiod = Geom(lev-1).periodicity(); if ( safe_guard_cells ) { @@ -728,9 +728,9 @@ WarpX::FillBoundaryB_avg (int lev, PatchType patch_type, IntVect ng) if (patch_type == PatchType::fine) { if (do_pml && pml[lev]->ok()) - { + { WARPX_ABORT_WITH_MESSAGE("Averaged Galilean PSATD with PML is not yet implemented"); - } + } const amrex::Periodicity& period = Geom(lev).periodicity(); if ( safe_guard_cells ) { const Vector mf{Bfield_avg_fp[lev][0].get(),Bfield_avg_fp[lev][1].get(),Bfield_avg_fp[lev][2].get()}; @@ -747,9 +747,9 @@ WarpX::FillBoundaryB_avg (int lev, PatchType patch_type, IntVect ng) else if (patch_type == PatchType::coarse) { if (do_pml && pml[lev]->ok()) - { + { WARPX_ABORT_WITH_MESSAGE("Averaged Galilean PSATD with PML is not yet implemented"); - } + } const amrex::Periodicity& cperiod = Geom(lev-1).periodicity(); if ( safe_guard_cells ){ diff --git a/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H b/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H index 8d85dab11d7..ca994ed1f56 100644 --- a/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H +++ b/Source/Particles/Collision/BinaryCollision/ParticleCreationFunc.H @@ -153,7 +153,7 @@ public: amrex::Gpu::copyAsync(amrex::Gpu::hostToDevice, products_np.begin(), products_np.end(), device_products_np.begin()); - amrex::Gpu::copyAsync(amrex::Gpu::hostToDevice, products_mass.begin(), + amrex::Gpu::copyAsync(amrex::Gpu::hostToDevice, products_mass.begin(), products_mass.end(), device_products_mass.begin()); amrex::Gpu::streamSynchronize(); diff --git a/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp b/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp index 911a5c64ea0..a43453da925 100644 --- a/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp +++ b/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.cpp @@ -104,7 +104,7 @@ void BreitWheelerEngine::init_builtin_tables( vector BreitWheelerEngine::export_lookup_tables_data () const { - if(!m_lookup_tables_initialized) + if(!m_lookup_tables_initialized) return vector{}; const auto data_dndt = m_dndt_table.serialize(); diff --git a/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp b/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp index 180bd5babce..2a52829272d 100644 --- a/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp +++ b/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.cpp @@ -103,7 +103,7 @@ void QuantumSynchrotronEngine::init_builtin_tables( vector QuantumSynchrotronEngine::export_lookup_tables_data () const { - if(!m_lookup_tables_initialized) + if(!m_lookup_tables_initialized) return vector{}; const auto data_dndt = m_dndt_table.serialize(); diff --git a/Source/Particles/MultiParticleContainer.cpp b/Source/Particles/MultiParticleContainer.cpp index 67dfda1dbe0..5cfd4e2b68d 100644 --- a/Source/Particles/MultiParticleContainer.cpp +++ b/Source/Particles/MultiParticleContainer.cpp @@ -1349,8 +1349,8 @@ MultiParticleContainer::doQEDSchwinger () * geom.CellSize(2); #endif - // Get the temporal step - const auto dt = warpx.getdt(level_0); + // Get the temporal step + const auto dt = warpx.getdt(level_0); auto& pc_product_ele = allcontainers[m_qed_schwinger_ele_product]; @@ -1370,8 +1370,8 @@ MultiParticleContainer::doQEDSchwinger () #ifdef AMREX_USE_OMP #pragma omp parallel if (amrex::Gpu::notInLaunchRegion()) #endif - for (MFIter mfi(Ex, TilingIfNotGPU()); mfi.isValid(); ++mfi ) - { + for (MFIter mfi(Ex, TilingIfNotGPU()); mfi.isValid(); ++mfi ) + { // Make the box cell centered to avoid creating particles twice on the tile edges amrex::Box box = enclosedCells(mfi.nodaltilebox()); diff --git a/Source/Particles/ParticleCreation/SmartCreate.H b/Source/Particles/ParticleCreation/SmartCreate.H index e91d39060b0..82e3cbce710 100644 --- a/Source/Particles/ParticleCreation/SmartCreate.H +++ b/Source/Particles/ParticleCreation/SmartCreate.H @@ -62,17 +62,17 @@ struct SmartCreate prt.m_aos[i_prt].cpu() = cpu; prt.m_aos[i_prt].id() = id; - // initialize the real components - for (int j = 0; j < PartData::NAR; ++j) - prt.m_rdata[j][i_prt] = initializeRealValue(m_policy_real[j], engine); - for (int j = 0; j < prt.m_num_runtime_real; ++j) - prt.m_runtime_rdata[j][i_prt] = initializeRealValue(m_policy_real[j+PartData::NAR], engine); + // initialize the real components + for (int j = 0; j < PartData::NAR; ++j) + prt.m_rdata[j][i_prt] = initializeRealValue(m_policy_real[j], engine); + for (int j = 0; j < prt.m_num_runtime_real; ++j) + prt.m_runtime_rdata[j][i_prt] = initializeRealValue(m_policy_real[j+PartData::NAR], engine); - // initialize the int components - for (int j = 0; j < PartData::NAI; ++j) - prt.m_idata[j][i_prt] = initializeIntValue(m_policy_int[j]); - for (int j = 0; j < prt.m_num_runtime_int; ++j) - prt.m_runtime_idata[j][i_prt] = initializeIntValue(m_policy_int[j+PartData::NAI]); + // initialize the int components + for (int j = 0; j < PartData::NAI; ++j) + prt.m_idata[j][i_prt] = initializeIntValue(m_policy_int[j]); + for (int j = 0; j < prt.m_num_runtime_int; ++j) + prt.m_runtime_idata[j][i_prt] = initializeIntValue(m_policy_int[j+PartData::NAI]); } }; diff --git a/Source/Particles/WarpXParticleContainer.H b/Source/Particles/WarpXParticleContainer.H index 3e3a43095e3..8587f74544d 100644 --- a/Source/Particles/WarpXParticleContainer.H +++ b/Source/Particles/WarpXParticleContainer.H @@ -393,7 +393,7 @@ public: * they do not exist (or if they were defined by default, i.e., * without runtime component). */ - void defineAllParticleTiles () noexcept; + void defineAllParticleTiles () noexcept; protected: int species_id; diff --git a/Source/Utils/Parser/ParserUtils.cpp b/Source/Utils/Parser/ParserUtils.cpp index 0add95df1ae..650d46c9e84 100644 --- a/Source/Utils/Parser/ParserUtils.cpp +++ b/Source/Utils/Parser/ParserUtils.cpp @@ -133,9 +133,9 @@ amrex::Parser utils::parser::makeParser ( const auto constant = warpx_constants.find(*it); if (constant != warpx_constants.end()) { - parser.setConstant(*it, constant->second); - it = symbols.erase(it); - continue; + parser.setConstant(*it, constant->second); + it = symbols.erase(it); + continue; } ++it; diff --git a/Source/Utils/WarpXAlgorithmSelection.H b/Source/Utils/WarpXAlgorithmSelection.H index 24d6edccdbf..7ad33409900 100644 --- a/Source/Utils/WarpXAlgorithmSelection.H +++ b/Source/Utils/WarpXAlgorithmSelection.H @@ -74,23 +74,23 @@ struct ParticlePusherAlgo { struct CurrentDepositionAlgo { enum { - Esirkepov = 0, - Direct = 1, - Vay = 2 + Esirkepov = 0, + Direct = 1, + Vay = 2 }; }; struct ChargeDepositionAlgo { // Only the Standard algorithm is implemented enum { - Standard = 0 + Standard = 0 }; }; struct GatheringAlgo { enum { - EnergyConserving = 0, - MomentumConserving + EnergyConserving = 0, + MomentumConserving }; }; diff --git a/Source/Utils/WarpXMovingWindow.cpp b/Source/Utils/WarpXMovingWindow.cpp index 3fd993ef9da..60b31af2f14 100644 --- a/Source/Utils/WarpXMovingWindow.cpp +++ b/Source/Utils/WarpXMovingWindow.cpp @@ -180,20 +180,20 @@ WarpX::MoveWindow (const int step, bool move_j) // slice box is modified only if slice diagnostics is initialized in input // if ( slice_plot_int > 0 ) { - amrex::Real new_slice_lo[AMREX_SPACEDIM]; - amrex::Real new_slice_hi[AMREX_SPACEDIM]; - const amrex::Real* current_slice_lo = slice_realbox.lo(); - const amrex::Real* current_slice_hi = slice_realbox.hi(); - for ( int i = 0; i < AMREX_SPACEDIM; i++) { - new_slice_lo[i] = current_slice_lo[i]; - new_slice_hi[i] = current_slice_hi[i]; - } - const int num_shift_base_slice = static_cast ((moving_window_x - - current_slice_lo[dir]) / cdx[dir]); - new_slice_lo[dir] = current_slice_lo[dir] + num_shift_base_slice*cdx[dir]; - new_slice_hi[dir] = current_slice_hi[dir] + num_shift_base_slice*cdx[dir]; - slice_realbox.setLo(new_slice_lo); - slice_realbox.setHi(new_slice_hi); + amrex::Real new_slice_lo[AMREX_SPACEDIM]; + amrex::Real new_slice_hi[AMREX_SPACEDIM]; + const amrex::Real* current_slice_lo = slice_realbox.lo(); + const amrex::Real* current_slice_hi = slice_realbox.hi(); + for ( int i = 0; i < AMREX_SPACEDIM; i++) { + new_slice_lo[i] = current_slice_lo[i]; + new_slice_hi[i] = current_slice_hi[i]; + } + const int num_shift_base_slice = static_cast ((moving_window_x - + current_slice_lo[dir]) / cdx[dir]); + new_slice_lo[dir] = current_slice_lo[dir] + num_shift_base_slice*cdx[dir]; + new_slice_hi[dir] = current_slice_hi[dir] + num_shift_base_slice*cdx[dir]; + slice_realbox.setLo(new_slice_lo); + slice_realbox.setHi(new_slice_hi); } int num_shift = num_shift_base; diff --git a/Source/Utils/check_interp_points_and_weights.py b/Source/Utils/check_interp_points_and_weights.py index a1d17c8dd3d..8bf2cf08490 100644 --- a/Source/Utils/check_interp_points_and_weights.py +++ b/Source/Utils/check_interp_points_and_weights.py @@ -100,8 +100,8 @@ def refinement_points_and_weights( ii, sc, sf, cr ): # Input coarsening ratio cr = int( input( "\n Select coarsening ratio (cr=1,2,4): cr=" ) ) if ( cr!=1 and cr!=2 and cr!=4 ): - print() - sys.exit( 'coarsening ratio cr={} is not valid'.format( cr ) ) + print() + sys.exit( 'coarsening ratio cr={} is not valid'.format( cr ) ) # Loop over possible staggering of coarse and fine grid (cell-centered or nodal) for sc in [0,1]: diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index bb1f5b36b84..240c7fbb4e0 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -3070,29 +3070,29 @@ amrex::Vector WarpX::getFornbergStencilCoefficients(const int n_ord // Coefficients for collocated (nodal) finite-difference approximation if (a_grid_type == GridType::Collocated) { - // First coefficient - coeffs.at(0) = m * 2._rt / (m+1); - // Other coefficients by recurrence - for (int n = 1; n < m; n++) - { - coeffs.at(n) = - (m-n) * 1._rt / (m+n+1) * coeffs.at(n-1); - } + // First coefficient + coeffs.at(0) = m * 2._rt / (m+1); + // Other coefficients by recurrence + for (int n = 1; n < m; n++) + { + coeffs.at(n) = - (m-n) * 1._rt / (m+n+1) * coeffs.at(n-1); + } } // Coefficients for staggered finite-difference approximation else { - Real prod = 1.; - for (int k = 1; k < m+1; k++) - { - prod *= (m + k) / (4._rt * k); - } - // First coefficient - coeffs.at(0) = 4_rt * m * prod * prod; - // Other coefficients by recurrence - for (int n = 1; n < m; n++) - { - coeffs.at(n) = - ((2_rt*n-1) * (m-n)) * 1._rt / ((2_rt*n+1) * (m+n)) * coeffs.at(n-1); - } + Real prod = 1.; + for (int k = 1; k < m+1; k++) + { + prod *= (m + k) / (4._rt * k); + } + // First coefficient + coeffs.at(0) = 4_rt * m * prod * prod; + // Other coefficients by recurrence + for (int n = 1; n < m; n++) + { + coeffs.at(n) = - ((2_rt*n-1) * (m-n)) * 1._rt / ((2_rt*n+1) * (m+n)) * coeffs.at(n-1); + } } return coeffs; diff --git a/Source/ablastr/coarsen/sample.cpp b/Source/ablastr/coarsen/sample.cpp index 65ada612905..ab5b135309b 100644 --- a/Source/ablastr/coarsen/sample.cpp +++ b/Source/ablastr/coarsen/sample.cpp @@ -29,12 +29,12 @@ namespace ablastr::coarsen::sample void Loop ( amrex::MultiFab& mf_dst, - const amrex::MultiFab& mf_src, - const int dcomp, - const int scomp, - const int ncomp, - const amrex::IntVect ngrowvect, - const amrex::IntVect crse_ratio + const amrex::MultiFab& mf_src, + const int dcomp, + const int scomp, + const int ncomp, + const amrex::IntVect ngrowvect, + const amrex::IntVect crse_ratio ) { // Staggering of source fine MultiFab and destination coarse MultiFab diff --git a/Source/ablastr/utils/SignalHandling.H b/Source/ablastr/utils/SignalHandling.H index 559fb93a080..29ea2db9f99 100644 --- a/Source/ablastr/utils/SignalHandling.H +++ b/Source/ablastr/utils/SignalHandling.H @@ -82,7 +82,7 @@ private: //! HandleSignals() to indicate actions requested by signals static bool signal_actions_requested[SIGNAL_REQUESTS_SIZE]; - // Don't allow clients to incorrectly try to construct and use an instance of this type + // Don't allow clients to incorrectly try to construct and use an instance of this type SignalHandling () = delete; }; diff --git a/Tools/PerformanceTests/functions_perftest.py b/Tools/PerformanceTests/functions_perftest.py index 311bdb68f1f..8d7f4e29246 100644 --- a/Tools/PerformanceTests/functions_perftest.py +++ b/Tools/PerformanceTests/functions_perftest.py @@ -41,13 +41,13 @@ def scale_n_cell(self, n_node=0): self.n_cell = n_cell_scaled def scale_n_cell(ncell, n_node): - ncell_scaled = ncell[:] - index_dim = 0 - while n_node > 1: - ncell_scaled[index_dim] *= 2 - n_node /= 2 - index_dim = (index_dim+1) % 3 - return ncell_scaled + ncell_scaled = ncell[:] + index_dim = 0 + while n_node > 1: + ncell_scaled[index_dim] *= 2 + n_node /= 2 + index_dim = (index_dim+1) % 3 + return ncell_scaled def store_git_hash(repo_path=None, filename=None, name=None): repo = git.Repo(path=repo_path) @@ -108,7 +108,7 @@ def run_batch(run_name, res_dir, bin_name, config_command, architecture='knl',\ def run_batch_nnode(test_list, res_dir, cwd, bin_name, config_command, batch_string, submit_job_command): # Clean res_dir if os.path.exists(res_dir): - shutil.rmtree(res_dir, ignore_errors=True) + shutil.rmtree(res_dir, ignore_errors=True) os.makedirs(res_dir) # Copy files to res_dir bin_dir = cwd + 'Bin/' @@ -198,9 +198,9 @@ def extract_dataframe(filename, n_steps): # New, might break something line_match_WritePlotFile = re.search('\nDiagnostics::FilterComputePackFlush().*', search_area) if line_match_WritePlotFile is not None: - time_WritePlotFile = float(line_match_WritePlotFile.group(0).split()[3]) + time_WritePlotFile = float(line_match_WritePlotFile.group(0).split()[3]) else: - time_WritePlotFile = 0. + time_WritePlotFile = 0. # Get timers for all routines # Where to start and stop in the output_file partition_limit_start = 'NCalls Excl. Min Excl. Avg Excl. Max Max %' From 67d6000174fc331b1c1c78733fbc950060686d86 Mon Sep 17 00:00:00 2001 From: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Sun, 26 Nov 2023 03:55:15 -0800 Subject: [PATCH 101/110] fix minor bug in picmi `ParticleDiagnostic` (#4442) --- Python/pywarpx/picmi.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index 4f7ff13699e..fa7776b1def 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -2319,7 +2319,7 @@ def initialize_inputs(self): elif np.iterable(self.species): species_names = [specie.name for specie in self.species] else: - species_names = [species.name] + species_names = [self.species.name] if self.mangle_dict is None: # Only do this once so that the same variables are used in this distribution From f2c0e2422e7927cebeecf1c548095ad56dfc826f Mon Sep 17 00:00:00 2001 From: David Grote Date: Mon, 27 Nov 2023 09:26:08 -0800 Subject: [PATCH 102/110] Set the particle fields directly to the constant external fields (#4339) * Set the particle fields directly to the constant external fields * Change "get" to "query" for the external fields * Fix PairGenerationTransformFunc * More fixes * Yet more fixes for QED * Make temporary a const * Broke vector into separate reals * Fix ColliderRelevant.cpp * Fix ParticleExtrema * Updated comment regarding static * Clean up for clang-tidy --- .../ReducedDiags/ColliderRelevant.cpp | 16 ++++++++-- .../ReducedDiags/ParticleExtrema.cpp | 16 ++++++++-- .../Particles/ElementaryProcess/Ionization.H | 17 ++++++++-- .../ElementaryProcess/Ionization.cpp | 8 +++++ .../ElementaryProcess/QEDPairGeneration.H | 17 ++++++++-- .../ElementaryProcess/QEDPairGeneration.cpp | 8 +++++ .../ElementaryProcess/QEDPhotonEmission.H | 17 ++++++++-- .../ElementaryProcess/QEDPhotonEmission.cpp | 8 +++++ Source/Particles/Gather/GetExternalFields.H | 21 ++----------- Source/Particles/Gather/GetExternalFields.cpp | 18 ++--------- Source/Particles/MultiParticleContainer.H | 3 -- Source/Particles/MultiParticleContainer.cpp | 29 ++++------------- Source/Particles/PhotonParticleContainer.cpp | 15 +++++++-- .../Particles/PhysicalParticleContainer.cpp | 31 ++++++++++++++++--- .../RigidInjectedParticleContainer.cpp | 15 +++++++-- Source/Particles/WarpXParticleContainer.H | 4 +++ Source/Particles/WarpXParticleContainer.cpp | 12 +++++++ 17 files changed, 178 insertions(+), 77 deletions(-) diff --git a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp index c4eaf78745d..a1a7b6cd61b 100644 --- a/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp +++ b/Source/Diagnostics/ReducedDiags/ColliderRelevant.cpp @@ -465,6 +465,13 @@ void ColliderRelevant::ComputeDiags (int step) // declare external fields const int offset = 0; const auto getExternalEB = GetExternalEBField(pti, offset); + const amrex::ParticleReal Ex_external_particle = myspc.m_E_external_particle[0]; + const amrex::ParticleReal Ey_external_particle = myspc.m_E_external_particle[1]; + const amrex::ParticleReal Ez_external_particle = myspc.m_E_external_particle[2]; + const amrex::ParticleReal Bx_external_particle = myspc.m_B_external_particle[0]; + const amrex::ParticleReal By_external_particle = myspc.m_B_external_particle[1]; + const amrex::ParticleReal Bz_external_particle = myspc.m_B_external_particle[2]; + // define variables in preparation for field gathering amrex::Box box = pti.tilebox(); box.grow(ngEB); @@ -491,8 +498,13 @@ void ColliderRelevant::ComputeDiags (int step) // get external fields amrex::ParticleReal xp, yp, zp; GetPosition(i, xp, yp, zp); - amrex::ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; - amrex::ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; + amrex::ParticleReal ex = Ex_external_particle; + amrex::ParticleReal ey = Ey_external_particle; + amrex::ParticleReal ez = Ez_external_particle; + amrex::ParticleReal bx = Bx_external_particle; + amrex::ParticleReal by = By_external_particle; + amrex::ParticleReal bz = Bz_external_particle; + getExternalEB(i, ex, ey, ez, bx, by, bz); // gather E and B diff --git a/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp b/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp index 78f8be8914e..ba8e7e373de 100644 --- a/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp +++ b/Source/Diagnostics/ReducedDiags/ParticleExtrema.cpp @@ -396,6 +396,13 @@ void ParticleExtrema::ComputeDiags (int step) // declare external fields const int offset = 0; const auto getExternalEB = GetExternalEBField(pti, offset); + const amrex::ParticleReal Ex_external_particle = myspc.m_E_external_particle[0]; + const amrex::ParticleReal Ey_external_particle = myspc.m_E_external_particle[1]; + const amrex::ParticleReal Ez_external_particle = myspc.m_E_external_particle[2]; + const amrex::ParticleReal Bx_external_particle = myspc.m_B_external_particle[0]; + const amrex::ParticleReal By_external_particle = myspc.m_B_external_particle[1]; + const amrex::ParticleReal Bz_external_particle = myspc.m_B_external_particle[2]; + // define variables in preparation for field gathering amrex::Box box = pti.tilebox(); box.grow(ngEB); @@ -422,8 +429,13 @@ void ParticleExtrema::ComputeDiags (int step) // get external fields ParticleReal xp, yp, zp; GetPosition(i, xp, yp, zp); - ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; - ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; + amrex::ParticleReal ex = Ex_external_particle; + amrex::ParticleReal ey = Ey_external_particle; + amrex::ParticleReal ez = Ez_external_particle; + amrex::ParticleReal bx = Bx_external_particle; + amrex::ParticleReal by = By_external_particle; + amrex::ParticleReal bz = Bz_external_particle; + getExternalEB(i, ex, ey, ez, bx, by, bz); // gather E and B diff --git a/Source/Particles/ElementaryProcess/Ionization.H b/Source/Particles/ElementaryProcess/Ionization.H index 5a70069a3a3..2c96c1a15a2 100644 --- a/Source/Particles/ElementaryProcess/Ionization.H +++ b/Source/Particles/ElementaryProcess/Ionization.H @@ -39,6 +39,12 @@ struct IonizationFilterFunc GetParticlePosition m_get_position; GetExternalEBField m_get_externalEB; + amrex::ParticleReal m_Ex_external_particle; + amrex::ParticleReal m_Ey_external_particle; + amrex::ParticleReal m_Ez_external_particle; + amrex::ParticleReal m_Bx_external_particle; + amrex::ParticleReal m_By_external_particle; + amrex::ParticleReal m_Bz_external_particle; amrex::Array4 m_ex_arr; amrex::Array4 m_ey_arr; @@ -70,6 +76,8 @@ struct IonizationFilterFunc amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, + amrex::Vector& E_external_particle, + amrex::Vector& B_external_particle, const amrex::Real* AMREX_RESTRICT a_ionization_energies, const amrex::Real* AMREX_RESTRICT a_adk_prefactor, const amrex::Real* AMREX_RESTRICT a_adk_exp_prefactor, @@ -94,8 +102,13 @@ struct IonizationFilterFunc amrex::ParticleReal xp, yp, zp; m_get_position(i, xp, yp, zp); - amrex::ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; - amrex::ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; + amrex::ParticleReal ex = m_Ex_external_particle; + amrex::ParticleReal ey = m_Ey_external_particle; + amrex::ParticleReal ez = m_Ez_external_particle; + amrex::ParticleReal bx = m_Bx_external_particle; + amrex::ParticleReal by = m_By_external_particle; + amrex::ParticleReal bz = m_Bz_external_particle; + m_get_externalEB(i, ex, ey, ez, bx, by, bz); doGatherShapeN(xp, yp, zp, ex, ey, ez, bx, by, bz, m_ex_arr, m_ey_arr, m_ez_arr, m_bx_arr, m_by_arr, m_bz_arr, diff --git a/Source/Particles/ElementaryProcess/Ionization.cpp b/Source/Particles/ElementaryProcess/Ionization.cpp index 98b49888d9d..c3681a30cad 100644 --- a/Source/Particles/ElementaryProcess/Ionization.cpp +++ b/Source/Particles/ElementaryProcess/Ionization.cpp @@ -24,6 +24,8 @@ IonizationFilterFunc::IonizationFilterFunc (const WarpXParIter& a_pti, int lev, amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, + amrex::Vector& E_external_particle, + amrex::Vector& B_external_particle, const amrex::Real* const AMREX_RESTRICT a_ionization_energies, const amrex::Real* const AMREX_RESTRICT a_adk_prefactor, const amrex::Real* const AMREX_RESTRICT a_adk_exp_prefactor, @@ -37,6 +39,12 @@ IonizationFilterFunc::IonizationFilterFunc (const WarpXParIter& a_pti, int lev, m_adk_power{a_adk_power}, comp{a_comp}, m_atomic_number{a_atomic_number}, + m_Ex_external_particle{E_external_particle[0]}, + m_Ey_external_particle{E_external_particle[1]}, + m_Ez_external_particle{E_external_particle[2]}, + m_Bx_external_particle{B_external_particle[0]}, + m_By_external_particle{B_external_particle[1]}, + m_Bz_external_particle{B_external_particle[2]}, m_galerkin_interpolation{WarpX::galerkin_interpolation}, m_nox{WarpX::nox}, m_n_rz_azimuthal_modes{WarpX::n_rz_azimuthal_modes} diff --git a/Source/Particles/ElementaryProcess/QEDPairGeneration.H b/Source/Particles/ElementaryProcess/QEDPairGeneration.H index 5a3efe483d7..5abc9282d4f 100644 --- a/Source/Particles/ElementaryProcess/QEDPairGeneration.H +++ b/Source/Particles/ElementaryProcess/QEDPairGeneration.H @@ -104,6 +104,8 @@ public: amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, + amrex::Vector& E_external_particle, + amrex::Vector& B_external_particle, int a_offset = 0); /** @@ -130,8 +132,13 @@ public: amrex::ParticleReal xp, yp, zp; m_get_position(i_src, xp, yp, zp); - amrex::ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; - amrex::ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; + amrex::ParticleReal ex = m_Ex_external_particle; + amrex::ParticleReal ey = m_Ey_external_particle; + amrex::ParticleReal ez = m_Ez_external_particle; + amrex::ParticleReal bx = m_Bx_external_particle; + amrex::ParticleReal by = m_By_external_particle; + amrex::ParticleReal bz = m_Bz_external_particle; + m_get_externalEB(i_src, ex, ey, ez, bx, by, bz); doGatherShapeN(xp, yp, zp, ex, ey, ez, bx, by, bz, @@ -170,6 +177,12 @@ private: GetParticlePosition m_get_position; GetExternalEBField m_get_externalEB; + amrex::ParticleReal m_Ex_external_particle; + amrex::ParticleReal m_Ey_external_particle; + amrex::ParticleReal m_Ez_external_particle; + amrex::ParticleReal m_Bx_external_particle; + amrex::ParticleReal m_By_external_particle; + amrex::ParticleReal m_Bz_external_particle; amrex::Array4 m_ex_arr; amrex::Array4 m_ey_arr; diff --git a/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp b/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp index de027152be7..2b380d454f4 100644 --- a/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp +++ b/Source/Particles/ElementaryProcess/QEDPairGeneration.cpp @@ -26,8 +26,16 @@ PairGenerationTransformFunc (BreitWheelerGeneratePairs const generate_functor, amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, + amrex::Vector& E_external_particle, + amrex::Vector& B_external_particle, int a_offset): m_generate_functor{generate_functor}, + m_Ex_external_particle{E_external_particle[0]}, + m_Ey_external_particle{E_external_particle[1]}, + m_Ez_external_particle{E_external_particle[2]}, + m_Bx_external_particle{B_external_particle[0]}, + m_By_external_particle{B_external_particle[1]}, + m_Bz_external_particle{B_external_particle[2]}, m_galerkin_interpolation{WarpX::galerkin_interpolation}, m_nox{WarpX::nox}, m_n_rz_azimuthal_modes{WarpX::n_rz_azimuthal_modes} diff --git a/Source/Particles/ElementaryProcess/QEDPhotonEmission.H b/Source/Particles/ElementaryProcess/QEDPhotonEmission.H index 0a889077036..8ba5c63ad57 100644 --- a/Source/Particles/ElementaryProcess/QEDPhotonEmission.H +++ b/Source/Particles/ElementaryProcess/QEDPhotonEmission.H @@ -118,6 +118,8 @@ public: amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, + amrex::Vector& E_external_particle, + amrex::Vector& B_external_particle, int a_offset = 0); /** @@ -141,8 +143,13 @@ public: amrex::ParticleReal xp, yp, zp; m_get_position(i_src, xp, yp, zp); - amrex::ParticleReal ex = 0._rt, ey = 0._rt, ez = 0._rt; - amrex::ParticleReal bx = 0._rt, by = 0._rt, bz = 0._rt; + amrex::ParticleReal ex = m_Ex_external_particle; + amrex::ParticleReal ey = m_Ey_external_particle; + amrex::ParticleReal ez = m_Ez_external_particle; + amrex::ParticleReal bx = m_Bx_external_particle; + amrex::ParticleReal by = m_By_external_particle; + amrex::ParticleReal bz = m_Bz_external_particle; + m_get_externalEB(i_src, ex, ey, ez, bx, by, bz); doGatherShapeN(xp, yp, zp, ex, ey, ez, bx, by, bz, @@ -180,6 +187,12 @@ private: GetParticlePosition m_get_position; GetExternalEBField m_get_externalEB; + amrex::ParticleReal m_Ex_external_particle; + amrex::ParticleReal m_Ey_external_particle; + amrex::ParticleReal m_Ez_external_particle; + amrex::ParticleReal m_Bx_external_particle; + amrex::ParticleReal m_By_external_particle; + amrex::ParticleReal m_Bz_external_particle; amrex::Array4 m_ex_arr; amrex::Array4 m_ey_arr; diff --git a/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp b/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp index 4725f3748e2..077a4659ce5 100644 --- a/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp +++ b/Source/Particles/ElementaryProcess/QEDPhotonEmission.cpp @@ -27,10 +27,18 @@ PhotonEmissionTransformFunc (QuantumSynchrotronGetOpticalDepth opt_depth_functor amrex::FArrayBox const& bxfab, amrex::FArrayBox const& byfab, amrex::FArrayBox const& bzfab, + amrex::Vector& E_external_particle, + amrex::Vector& B_external_particle, int a_offset): m_opt_depth_functor{opt_depth_functor}, m_opt_depth_runtime_comp{opt_depth_runtime_comp}, m_emission_functor{emission_functor}, + m_Ex_external_particle{E_external_particle[0]}, + m_Ey_external_particle{E_external_particle[1]}, + m_Ez_external_particle{E_external_particle[2]}, + m_Bx_external_particle{B_external_particle[0]}, + m_By_external_particle{B_external_particle[1]}, + m_Bz_external_particle{B_external_particle[2]}, m_galerkin_interpolation{WarpX::galerkin_interpolation}, m_nox{WarpX::nox}, m_n_rz_azimuthal_modes{WarpX::n_rz_azimuthal_modes} diff --git a/Source/Particles/Gather/GetExternalFields.H b/Source/Particles/Gather/GetExternalFields.H index 8db7565a485..5e74a409824 100644 --- a/Source/Particles/Gather/GetExternalFields.H +++ b/Source/Particles/Gather/GetExternalFields.H @@ -23,7 +23,7 @@ */ struct GetExternalEBField { - enum ExternalFieldInitType { None, Constant, Parser, RepeatedPlasmaLens, Unknown }; + enum ExternalFieldInitType { None, Parser, RepeatedPlasmaLens, Unknown }; GetExternalEBField () = default; @@ -35,9 +35,6 @@ struct GetExternalEBField amrex::ParticleReal m_gamma_boost; amrex::ParticleReal m_uz_boost; - amrex::GpuArray m_Efield_value; - amrex::GpuArray m_Bfield_value; - amrex::ParserExecutor<4> m_Exfield_partparser; amrex::ParserExecutor<4> m_Eyfield_partparser; amrex::ParserExecutor<4> m_Ezfield_partparser; @@ -92,13 +89,7 @@ struct GetExternalEBField constexpr amrex::ParticleReal inv_c2 = 1._prt/(PhysConst::c*PhysConst::c); - if (m_Etype == Constant) - { - Ex = m_Efield_value[0]; - Ey = m_Efield_value[1]; - Ez = m_Efield_value[2]; - } - else if (m_Etype == ExternalFieldInitType::Parser) + if (m_Etype == ExternalFieldInitType::Parser) { amrex::ParticleReal x, y, z; m_get_position(i, x, y, z); @@ -112,13 +103,7 @@ struct GetExternalEBField Ez = m_Ezfield_partparser((amrex::ParticleReal) x, (amrex::ParticleReal) y, (amrex::ParticleReal) z, lab_time); } - if (m_Btype == Constant) - { - Bx = m_Bfield_value[0]; - By = m_Bfield_value[1]; - Bz = m_Bfield_value[2]; - } - else if (m_Btype == ExternalFieldInitType::Parser) + if (m_Btype == ExternalFieldInitType::Parser) { amrex::ParticleReal x, y, z; m_get_position(i, x, y, z); diff --git a/Source/Particles/Gather/GetExternalFields.cpp b/Source/Particles/Gather/GetExternalFields.cpp index af00e092080..4aeb6ce05b9 100644 --- a/Source/Particles/Gather/GetExternalFields.cpp +++ b/Source/Particles/Gather/GetExternalFields.cpp @@ -34,21 +34,9 @@ GetExternalEBField::GetExternalEBField (const WarpXParIter& a_pti, long a_offset if (mypc.m_E_ext_particle_s == "none") m_Etype = None; if (mypc.m_B_ext_particle_s == "none") m_Btype = None; - if (mypc.m_E_ext_particle_s == "constant") - { - m_Etype = Constant; - m_Efield_value[0] = mypc.m_E_external_particle[0]; - m_Efield_value[1] = mypc.m_E_external_particle[1]; - m_Efield_value[2] = mypc.m_E_external_particle[2]; - } - - if (mypc.m_B_ext_particle_s == "constant") - { - m_Btype = Constant; - m_Bfield_value[0] = mypc.m_B_external_particle[0]; - m_Bfield_value[1] = mypc.m_B_external_particle[1]; - m_Bfield_value[2] = mypc.m_B_external_particle[2]; - } + // These lines will be removed once the user interface is redefined and the CI tests updated + if (mypc.m_E_ext_particle_s == "constant") m_Etype = None; + if (mypc.m_B_ext_particle_s == "constant") m_Btype = None; if (mypc.m_E_ext_particle_s == "parse_e_ext_particle_function" || mypc.m_B_ext_particle_s == "parse_b_ext_particle_function" || diff --git a/Source/Particles/MultiParticleContainer.H b/Source/Particles/MultiParticleContainer.H index 64ce057e2ff..64c8dbb085a 100644 --- a/Source/Particles/MultiParticleContainer.H +++ b/Source/Particles/MultiParticleContainer.H @@ -294,9 +294,6 @@ public: std::string m_B_ext_particle_s = "none"; std::string m_E_ext_particle_s = "none"; - // External fields added to particle fields. - amrex::Vector m_B_external_particle; - amrex::Vector m_E_external_particle; // Parser for B_external on the particle std::unique_ptr m_Bx_particle_parser; std::unique_ptr m_By_particle_parser; diff --git a/Source/Particles/MultiParticleContainer.cpp b/Source/Particles/MultiParticleContainer.cpp index 5cfd4e2b68d..58b490af10a 100644 --- a/Source/Particles/MultiParticleContainer.cpp +++ b/Source/Particles/MultiParticleContainer.cpp @@ -133,14 +133,6 @@ MultiParticleContainer::ReadParameters () { const ParmParse pp_particles("particles"); - // allocating and initializing default values of external fields for particles - m_E_external_particle.resize(3); - m_B_external_particle.resize(3); - // initialize E and B fields to 0.0 - for (int idim = 0; idim < 3; ++idim) { - m_E_external_particle[idim] = 0.0; - m_B_external_particle[idim] = 0.0; - } // default values of E_external_particle and B_external_particle // are used to set the E and B field when "constant" or "parser" // is not explicitly used in the input @@ -154,19 +146,6 @@ MultiParticleContainer::ReadParameters () m_E_ext_particle_s.end(), m_E_ext_particle_s.begin(), ::tolower); - // if the input string for B_external on particles is "constant" - // then the values for the external B on particles must - // be provided in the input file. - if (m_B_ext_particle_s == "constant") - utils::parser::getArrWithParser( - pp_particles, "B_external_particle", m_B_external_particle); - - // if the input string for E_external on particles is "constant" - // then the values for the external E on particles must - // be provided in the input file. - if (m_E_ext_particle_s == "constant") - utils::parser::getArrWithParser( - pp_particles, "E_external_particle", m_E_external_particle); // if the input string for B_ext_particle_s is // "parse_b_ext_particle_function" then the mathematical expression @@ -1547,7 +1526,9 @@ void MultiParticleContainer::doQedBreitWheeler (int lev, auto Transform = PairGenerationTransformFunc(pair_gen_functor, pti, lev, Ex.nGrowVect(), Ex[pti], Ey[pti], Ez[pti], - Bx[pti], By[pti], Bz[pti]); + Bx[pti], By[pti], Bz[pti], + phys_pc_ptr->m_E_external_particle, + phys_pc_ptr->m_B_external_particle); auto& src_tile = pc_source->ParticlesAt(lev, pti); auto& dst_ele_tile = pc_product_ele->ParticlesAt(lev, pti); @@ -1625,7 +1606,9 @@ void MultiParticleContainer::doQedQuantumSync (int lev, m_shr_p_qs_engine->build_phot_em_functor(), pti, lev, Ex.nGrowVect(), Ex[pti], Ey[pti], Ez[pti], - Bx[pti], By[pti], Bz[pti]); + Bx[pti], By[pti], Bz[pti], + phys_pc_ptr->m_E_external_particle, + phys_pc_ptr->m_B_external_particle); auto& src_tile = pc_source->ParticlesAt(lev, pti); auto& dst_tile = pc_product_phot->ParticlesAt(lev, pti); diff --git a/Source/Particles/PhotonParticleContainer.cpp b/Source/Particles/PhotonParticleContainer.cpp index 7609bd67b13..96091b9067f 100644 --- a/Source/Particles/PhotonParticleContainer.cpp +++ b/Source/Particles/PhotonParticleContainer.cpp @@ -134,6 +134,13 @@ PhotonParticleContainer::PushPX (WarpXParIter& pti, const auto getExternalEB = GetExternalEBField(pti, offset); + const amrex::ParticleReal Ex_external_particle = m_E_external_particle[0]; + const amrex::ParticleReal Ey_external_particle = m_E_external_particle[1]; + const amrex::ParticleReal Ez_external_particle = m_E_external_particle[2]; + const amrex::ParticleReal Bx_external_particle = m_B_external_particle[0]; + const amrex::ParticleReal By_external_particle = m_B_external_particle[1]; + const amrex::ParticleReal Bz_external_particle = m_B_external_particle[2]; + // Lower corner of tile box physical domain (take into account Galilean shift) const std::array& xyzmin = WarpX::LowerCorner(box, gather_lev, 0._rt); @@ -182,8 +189,12 @@ PhotonParticleContainer::PushPX (WarpXParIter& pti, ParticleReal x, y, z; GetPosition(i, x, y, z); - amrex::ParticleReal Exp=0, Eyp=0, Ezp=0; - amrex::ParticleReal Bxp=0, Byp=0, Bzp=0; + amrex::ParticleReal Exp = Ex_external_particle; + amrex::ParticleReal Eyp = Ey_external_particle; + amrex::ParticleReal Ezp = Ez_external_particle; + amrex::ParticleReal Bxp = Bx_external_particle; + amrex::ParticleReal Byp = By_external_particle; + amrex::ParticleReal Bzp = Bz_external_particle; if(!t_do_not_gather){ // first gather E and B to the particle positions diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp index 47898a0344d..70614702ca5 100644 --- a/Source/Particles/PhysicalParticleContainer.cpp +++ b/Source/Particles/PhysicalParticleContainer.cpp @@ -2506,6 +2506,13 @@ PhysicalParticleContainer::PushP (int lev, Real dt, const auto getExternalEB = GetExternalEBField(pti); + const amrex::ParticleReal Ex_external_particle = m_E_external_particle[0]; + const amrex::ParticleReal Ey_external_particle = m_E_external_particle[1]; + const amrex::ParticleReal Ez_external_particle = m_E_external_particle[2]; + const amrex::ParticleReal Bx_external_particle = m_B_external_particle[0]; + const amrex::ParticleReal By_external_particle = m_B_external_particle[1]; + const amrex::ParticleReal Bz_external_particle = m_B_external_particle[2]; + const std::array& xyzmin = WarpX::LowerCorner(box, lev, 0._rt); const Dim3 lo = lbound(box); @@ -2561,8 +2568,12 @@ PhysicalParticleContainer::PushP (int lev, Real dt, amrex::ParticleReal xp, yp, zp; getPosition(ip, xp, yp, zp); - amrex::ParticleReal Exp = 0._rt, Eyp = 0._rt, Ezp = 0._rt; - amrex::ParticleReal Bxp = 0._rt, Byp = 0._rt, Bzp = 0._rt; + amrex::ParticleReal Exp = Ex_external_particle; + amrex::ParticleReal Eyp = Ey_external_particle; + amrex::ParticleReal Ezp = Ez_external_particle; + amrex::ParticleReal Bxp = Bx_external_particle; + amrex::ParticleReal Byp = By_external_particle; + amrex::ParticleReal Bzp = Bz_external_particle; if (!t_do_not_gather){ // first gather E and B to the particle positions @@ -2683,6 +2694,13 @@ PhysicalParticleContainer::PushPX (WarpXParIter& pti, const auto getExternalEB = GetExternalEBField(pti, offset); + const amrex::ParticleReal Ex_external_particle = m_E_external_particle[0]; + const amrex::ParticleReal Ey_external_particle = m_E_external_particle[1]; + const amrex::ParticleReal Ez_external_particle = m_E_external_particle[2]; + const amrex::ParticleReal Bx_external_particle = m_B_external_particle[0]; + const amrex::ParticleReal By_external_particle = m_B_external_particle[1]; + const amrex::ParticleReal Bz_external_particle = m_B_external_particle[2]; + // Lower corner of tile box physical domain (take into account Galilean shift) const std::array& xyzmin = WarpX::LowerCorner(box, gather_lev, 0._rt); @@ -2797,8 +2815,12 @@ PhysicalParticleContainer::PushPX (WarpXParIter& pti, z_old[ip] = zp; } - amrex::ParticleReal Exp = 0._rt, Eyp = 0._rt, Ezp = 0._rt; - amrex::ParticleReal Bxp = 0._rt, Byp = 0._rt, Bzp = 0._rt; + amrex::ParticleReal Exp = Ex_external_particle; + amrex::ParticleReal Eyp = Ey_external_particle; + amrex::ParticleReal Ezp = Ez_external_particle; + amrex::ParticleReal Bxp = Bx_external_particle; + amrex::ParticleReal Byp = By_external_particle; + amrex::ParticleReal Bzp = Bz_external_particle; if(!t_do_not_gather){ // first gather E and B to the particle positions @@ -2943,6 +2965,7 @@ PhysicalParticleContainer::getIonizationFunc (const WarpXParIter& pti, WARPX_PROFILE("PhysicalParticleContainer::getIonizationFunc()"); return {pti, lev, ngEB, Ex, Ey, Ez, Bx, By, Bz, + m_E_external_particle, m_B_external_particle, ionization_energies.dataPtr(), adk_prefactor.dataPtr(), adk_exp_prefactor.dataPtr(), diff --git a/Source/Particles/RigidInjectedParticleContainer.cpp b/Source/Particles/RigidInjectedParticleContainer.cpp index e013d41268a..4e4d644123b 100644 --- a/Source/Particles/RigidInjectedParticleContainer.cpp +++ b/Source/Particles/RigidInjectedParticleContainer.cpp @@ -361,6 +361,13 @@ RigidInjectedParticleContainer::PushP (int lev, Real dt, const auto getExternalEB = GetExternalEBField(pti); + const amrex::ParticleReal Ex_external_particle = m_E_external_particle[0]; + const amrex::ParticleReal Ey_external_particle = m_E_external_particle[1]; + const amrex::ParticleReal Ez_external_particle = m_E_external_particle[2]; + const amrex::ParticleReal Bx_external_particle = m_B_external_particle[0]; + const amrex::ParticleReal By_external_particle = m_B_external_particle[1]; + const amrex::ParticleReal Bz_external_particle = m_B_external_particle[2]; + const std::array& xyzmin = WarpX::LowerCorner(box, lev, 0._rt); const Dim3 lo = lbound(box); @@ -426,8 +433,12 @@ RigidInjectedParticleContainer::PushP (int lev, Real dt, amrex::ParticleReal xp, yp, zp; getPosition(ip, xp, yp, zp); - amrex::ParticleReal Exp = 0._prt, Eyp = 0._prt, Ezp = 0._prt; - amrex::ParticleReal Bxp = 0._prt, Byp = 0._prt, Bzp = 0._prt; + amrex::ParticleReal Exp = Ex_external_particle; + amrex::ParticleReal Eyp = Ey_external_particle; + amrex::ParticleReal Ezp = Ez_external_particle; + amrex::ParticleReal Bxp = Bx_external_particle; + amrex::ParticleReal Byp = By_external_particle; + amrex::ParticleReal Bzp = Bz_external_particle; // first gather E and B to the particle positions doGatherShapeN(xp, yp, zp, Exp, Eyp, Ezp, Bxp, Byp, Bzp, diff --git a/Source/Particles/WarpXParticleContainer.H b/Source/Particles/WarpXParticleContainer.H index 8587f74544d..f42863d5357 100644 --- a/Source/Particles/WarpXParticleContainer.H +++ b/Source/Particles/WarpXParticleContainer.H @@ -335,6 +335,10 @@ public: int self_fields_max_iters = 200; int self_fields_verbosity = 2; + // External fields added to particle fields. + amrex::Vector m_B_external_particle; + amrex::Vector m_E_external_particle; + //! Current injection position amrex::Real m_current_injection_position; diff --git a/Source/Particles/WarpXParticleContainer.cpp b/Source/Particles/WarpXParticleContainer.cpp index 6109cd830f9..98868d1cf14 100644 --- a/Source/Particles/WarpXParticleContainer.cpp +++ b/Source/Particles/WarpXParticleContainer.cpp @@ -20,6 +20,7 @@ #include "Utils/WarpXAlgorithmSelection.H" #include "Utils/WarpXConst.H" #include "Utils/WarpXProfilerWrapper.H" +#include "Utils/Parser/ParserUtils.H" #include "WarpX.H" #include @@ -89,6 +90,17 @@ WarpXParticleContainer::WarpXParticleContainer (AmrCore* amr_core, int ispecies) SetParticleSize(); ReadParameters(); + // Reading the external fields needs to be here since ReadParameters + // is static but the m_E_external_particle and B are not + const ParmParse pp_particles("particles"); + + // allocating and initializing default values of external fields for particles + m_E_external_particle.resize(3, 0.); + m_B_external_particle.resize(3, 0.); + + utils::parser::queryArrWithParser(pp_particles, "E_external_particle", m_E_external_particle); + utils::parser::queryArrWithParser(pp_particles, "B_external_particle", m_B_external_particle); + // Initialize temporary local arrays for charge/current deposition #ifdef AMREX_USE_OMP int num_threads = 1; From 6628b414683a7877327409e1865640af4a95e97c Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Mon, 27 Nov 2023 18:51:32 +0100 Subject: [PATCH 103/110] Clang tidy CI test: add almost all modernize-* checks (#4319) * Clang-tidy CI test: add almost all modernize-* checks * address issues found with clang-tidy * address issues found with clang-tidy * fix bug * fixed bug * fix bugs * address issue found with clang-tidy * fix residual issues found with clang-tidy * fix issue related to modernize-use-default-member-init check * address issues found with clang-tidy * fix issues found with clang-tidy --- .clang-tidy | 19 +-- .../AcceleratorLattice/AcceleratorLattice.H | 3 +- .../LatticeElements/HardEdgedPlasmaLens.H | 2 +- .../LatticeElements/HardEdgedQuadrupole.H | 4 +- Source/BoundaryConditions/PML.H | 40 ++++-- Source/BoundaryConditions/PML.cpp | 12 +- Source/BoundaryConditions/PML_RZ.cpp | 2 +- Source/Diagnostics/BTD_Plotfile_Header_Impl.H | 28 ++-- Source/Diagnostics/BTDiagnostics.cpp | 6 +- .../ComputeDiagFunctors/ComputeDiagFunctor.H | 2 +- .../ComputeParticleDiagFunctor.H | 2 +- Source/Diagnostics/Diagnostics.H | 2 +- Source/Diagnostics/FlushFormats/FlushFormat.H | 2 +- .../FlushFormats/FlushFormatPlotfile.H | 2 +- .../FlushFormats/FlushFormatSensei.cpp | 6 - .../Diagnostics/ParticleDiag/ParticleDiag.H | 6 +- .../Diagnostics/ReducedDiags/FieldProbe.cpp | 2 +- Source/Diagnostics/SliceDiagnostic.cpp | 2 +- Source/Diagnostics/WarpXOpenPMD.H | 2 +- Source/Diagnostics/WarpXOpenPMD.cpp | 6 +- Source/FieldSolver/ElectrostaticSolver.H | 4 +- .../HybridPICModel/HybridPICModel.H | 13 +- .../PsatdAlgorithmPmlRZ.cpp | 1 - .../SpectralBaseAlgorithmRZ.H | 2 +- .../SpectralSolver/SpectralBinomialFilter.H | 2 +- .../SpectralHankelTransform/BesselRoots.cpp | 2 +- .../SpectralHankelTransformer.H | 4 +- Source/FieldSolver/WarpXPushFieldsEM_K.H | 4 +- Source/Fluids/MultiFluidContainer.H | 6 +- Source/Fluids/WarpXFluidContainer.H | 6 +- Source/Fluids/WarpXFluidContainer.cpp | 22 ++-- Source/Initialization/GetVelocity.H | 3 +- Source/Initialization/InjectorDensity.H | 6 +- Source/Initialization/InjectorDensity.cpp | 1 - Source/Initialization/InjectorFlux.H | 3 + Source/Initialization/InjectorMomentum.H | 25 +++- Source/Initialization/InjectorPosition.H | 7 + Source/Initialization/PlasmaInjector.H | 19 +-- Source/Initialization/VelocityProperties.H | 2 +- Source/Initialization/VelocityProperties.cpp | 3 +- Source/Laser/LaserProfiles.H | 4 +- .../LaserProfileFromFile.cpp | 2 +- Source/Parallelization/GuardCellManager.cpp | 8 +- .../BackgroundMCC/ImpactIonization.H | 2 +- .../Collision/BackgroundMCC/MCCProcess.H | 14 +- .../Particles/ElementaryProcess/Ionization.H | 2 +- .../QEDInternals/BreitWheelerEngineWrapper.H | 14 +- .../QEDInternals/QuantumSyncEngineWrapper.H | 14 +- Source/Particles/Filter/FilterFunctors.H | 8 +- Source/Particles/LaserParticleContainer.H | 1 - Source/Particles/MultiParticleContainer.H | 44 ++++--- Source/Particles/MultiParticleContainer.cpp | 8 +- .../NamedComponentParticleContainer.H | 8 +- Source/Particles/ParticleBoundaryBuffer.H | 2 +- Source/Particles/ParticleCreation/SmartCopy.H | 4 +- .../Particles/ParticleCreation/SmartCreate.H | 6 +- .../Particles/ParticleCreation/SmartUtils.H | 2 +- Source/Particles/PhotonParticleContainer.H | 2 +- Source/Particles/PhysicalParticleContainer.H | 4 +- .../Particles/PhysicalParticleContainer.cpp | 3 +- .../RigidInjectedParticleContainer.H | 1 - Source/Particles/WarpXParticleContainer.H | 17 ++- Source/Utils/Parser/IntervalsParser.H | 34 ++--- Source/Utils/Parser/ParserUtils.cpp | 2 +- Source/Utils/WarpXTagging.cpp | 2 +- Source/WarpX.H | 122 +++++++++--------- Source/WarpX.cpp | 8 +- Source/ablastr/parallelization/KernelTimer.H | 23 ++-- Source/ablastr/utils/SignalHandling.H | 6 +- Source/ablastr/utils/msg_logger/MsgLogger.H | 21 +-- Source/ablastr/utils/timer/Timer.H | 2 +- Source/ablastr/utils/timer/Timer.cpp | 2 - Source/ablastr/warn_manager/WarnManager.H | 12 +- 73 files changed, 376 insertions(+), 313 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index c7fb58ae117..a659b8dc5be 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -24,23 +24,10 @@ Checks: ' misc-*, -misc-no-recursion, -misc-non-private-member-variables-in-classes, - modernize-avoid-bind, - modernize-concat-nested-namespaces, - modernize-deprecated-headers, - modernize-deprecated-ios-base-aliases, - modernize-loop-convert, - modernize-make-shared, - modernize-make-unique, - modernize-pass-by-value, - modernize-raw-string-literal, - modernize-redundant-void-arg, - modernize-replace-auto-ptr, - modernize-replace-disallow-copy-and-assign-macro, - modernize-replace-random-shuffle, - modernize-shrink-to-fit, - modernize-unary-static-assert, - modernize-use-nullptr, + modernize-*, + -modernize-avoid-c-arrays, -modernize-return-braced-init-list, + -modernize-use-trailing-return-type, mpi-*, performance-faster-string-find, performance-for-range-copy, diff --git a/Source/AcceleratorLattice/AcceleratorLattice.H b/Source/AcceleratorLattice/AcceleratorLattice.H index 67611743586..4b3eff46094 100644 --- a/Source/AcceleratorLattice/AcceleratorLattice.H +++ b/Source/AcceleratorLattice/AcceleratorLattice.H @@ -66,7 +66,8 @@ public: * @param[in] a_pti the grid where the finder is needed * @param[in] a_offset the particle offset since the finded needs information about the particles as well */ - LatticeElementFinderDevice GetFinderDeviceInstance (WarpXParIter const& a_pti, int a_offset) const; + [[nodiscard]] LatticeElementFinderDevice + GetFinderDeviceInstance (WarpXParIter const& a_pti, int a_offset) const; /* All of the available lattice element types */ Drift h_drift; diff --git a/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H b/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H index c30a91867e6..2d15dfa3bd4 100644 --- a/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H +++ b/Source/AcceleratorLattice/LatticeElements/HardEdgedPlasmaLens.H @@ -53,7 +53,7 @@ struct HardEdgedPlasmaLens /** * \brief Returns the device level instance with the lattice information */ - HardEdgedPlasmaLensDevice GetDeviceInstance () const; + [[nodiscard]] HardEdgedPlasmaLensDevice GetDeviceInstance () const; }; diff --git a/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H b/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H index 0cdefea769e..5ac4f7fe7c1 100644 --- a/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H +++ b/Source/AcceleratorLattice/LatticeElements/HardEdgedQuadrupole.H @@ -53,8 +53,8 @@ struct HardEdgedQuadrupole /** * \brief Returns the device level instance with the lattice information */ - HardEdgedQuadrupoleDevice GetDeviceInstance () const; - + [[nodiscard]] HardEdgedQuadrupoleDevice + GetDeviceInstance () const; }; diff --git a/Source/BoundaryConditions/PML.H b/Source/BoundaryConditions/PML.H index 649d4a6b40b..5f41ecbd706 100644 --- a/Source/BoundaryConditions/PML.H +++ b/Source/BoundaryConditions/PML.H @@ -35,8 +35,8 @@ struct Sigma : amrex::Gpu::DeviceVector { - int lo() const { return m_lo; } - int hi() const { return m_hi; } + [[nodiscard]] int lo() const { return m_lo; } + [[nodiscard]] int hi() const { return m_hi; } int m_lo, m_hi; }; @@ -90,15 +90,23 @@ public: SigmaBoxFactory& operator= (const SigmaBoxFactory&) = delete; SigmaBoxFactory& operator= (SigmaBoxFactory&&) = delete; - SigmaBox* create (const amrex::Box& box, int /*ncomps*/, - const amrex::FabInfo& /*info*/, int /*box_index*/) const final - { return new SigmaBox(box, m_grids, m_dx, m_ncell, m_delta, m_regdomain, m_v_sigma_sb); } - void destroy (SigmaBox* fab) const final { + [[nodiscard]] SigmaBox* create (const amrex::Box& box, int /*ncomps*/, + const amrex::FabInfo& /*info*/, int /*box_index*/) const final + { + return new SigmaBox(box, m_grids, m_dx, m_ncell, m_delta, m_regdomain, m_v_sigma_sb); + } + + void destroy (SigmaBox* fab) const final + { delete fab; } - SigmaBoxFactory* clone () const final { + + [[nodiscard]] SigmaBoxFactory* + clone () const final + { return new SigmaBoxFactory(*this); } + private: const amrex::BoxArray& m_grids; const amrex::Real* m_dx; @@ -161,11 +169,15 @@ public: amrex::MultiFab* GetG_fp (); amrex::MultiFab* GetG_cp (); - const MultiSigmaBox& GetMultiSigmaBox_fp () const - { return *sigba_fp; } + [[nodiscard]] const MultiSigmaBox& GetMultiSigmaBox_fp () const + { + return *sigba_fp; + } - const MultiSigmaBox& GetMultiSigmaBox_cp () const - { return *sigba_cp; } + [[nodiscard]] const MultiSigmaBox& GetMultiSigmaBox_cp () const + { + return *sigba_cp; + } #ifdef WARPX_USE_PSATD void PushPSATD (int lev); @@ -193,7 +205,7 @@ public: void FillBoundaryF (PatchType patch_type, std::optional nodal_sync=std::nullopt); void FillBoundaryG (PatchType patch_type, std::optional nodal_sync=std::nullopt); - bool ok () const { return m_ok; } + [[nodiscard]] bool ok () const { return m_ok; } void CheckPoint (const std::string& dir) const; void Restart (const std::string& dir); @@ -241,9 +253,11 @@ private: // Factory for field data std::unique_ptr > pml_field_factory; - amrex::FabFactory const& fieldFactory () const noexcept { + [[nodiscard]] amrex::FabFactory const& fieldFactory () const noexcept + { return *pml_field_factory; } + #ifdef AMREX_USE_EB amrex::EBFArrayBoxFactory const& fieldEBFactory () const noexcept { return static_cast(*pml_field_factory); diff --git a/Source/BoundaryConditions/PML.cpp b/Source/BoundaryConditions/PML.cpp index 98f2b9fa546..b2016e65d2a 100644 --- a/Source/BoundaryConditions/PML.cpp +++ b/Source/BoundaryConditions/PML.cpp @@ -594,11 +594,11 @@ PML::PML (const int lev, const BoxArray& grid_ba, const DistributionMapping& gri } // Define the number of guard cells in each direction, for E, B, and F - IntVect nge = IntVect(AMREX_D_DECL(2, 2, 2)); - IntVect ngb = IntVect(AMREX_D_DECL(2, 2, 2)); + auto nge = IntVect(AMREX_D_DECL(2, 2, 2)); + auto ngb = IntVect(AMREX_D_DECL(2, 2, 2)); int ngf_int = 0; if (WarpX::electromagnetic_solver_id == ElectromagneticSolverAlgo::CKC) ngf_int = std::max( ngf_int, 1 ); - IntVect ngf = IntVect(AMREX_D_DECL(ngf_int, ngf_int, ngf_int)); + auto ngf = IntVect(AMREX_D_DECL(ngf_int, ngf_int, ngf_int)); if (do_moving_window) { WARPX_ALWAYS_ASSERT_WITH_MESSAGE(lev <= 1, @@ -622,11 +622,11 @@ PML::PML (const int lev, const BoxArray& grid_ba, const DistributionMapping& gri utils::parser::queryWithParser(pp_psatd, "nz_guard", ngFFt_z); #if defined(WARPX_DIM_3D) - IntVect ngFFT = IntVect(ngFFt_x, ngFFt_y, ngFFt_z); + auto ngFFT = IntVect(ngFFt_x, ngFFt_y, ngFFt_z); #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - IntVect ngFFT = IntVect(ngFFt_x, ngFFt_z); + auto ngFFT = IntVect(ngFFt_x, ngFFt_z); #elif defined(WARPX_DIM_1D_Z) - IntVect ngFFT = IntVect(ngFFt_z); + auto ngFFT = IntVect(ngFFt_z); #endif // Set the number of guard cells to the maximum of each field diff --git a/Source/BoundaryConditions/PML_RZ.cpp b/Source/BoundaryConditions/PML_RZ.cpp index d04c6e53bf8..7eb011a6bb2 100644 --- a/Source/BoundaryConditions/PML_RZ.cpp +++ b/Source/BoundaryConditions/PML_RZ.cpp @@ -95,7 +95,7 @@ PML_RZ::ApplyDamping (amrex::MultiFab* Et_fp, amrex::MultiFab* Ez_fp, amrex::ParallelFor( tilebox, Et_fp->nComp(), [=] AMREX_GPU_DEVICE (int i, int j, int k, int icomp) { - const amrex::Real rr = static_cast(i - nr_damp_min); + const auto rr = static_cast(i - nr_damp_min); const amrex::Real wr = rr/nr_damp; const amrex::Real damp_factor = std::exp( -4._rt * cdt_over_dr * wr*wr ); diff --git a/Source/Diagnostics/BTD_Plotfile_Header_Impl.H b/Source/Diagnostics/BTD_Plotfile_Header_Impl.H index 2becaba85b3..209113dedeb 100644 --- a/Source/Diagnostics/BTD_Plotfile_Header_Impl.H +++ b/Source/Diagnostics/BTD_Plotfile_Header_Impl.H @@ -33,44 +33,44 @@ public: BTDPlotfileHeaderImpl (std::string const& Headerfile_path); /** Returns the Header file version for plotfile */ - std::string fileVersion () const noexcept {return m_file_version; } + [[nodiscard]] std::string fileVersion () const noexcept {return m_file_version; } /** Returns the number of components written in the Headerfile */ - int ncomp () const noexcept {return m_nComp; } + [[nodiscard]] int ncomp () const noexcept {return m_nComp; } /** Returns the names of components in the Headerfile */ - const amrex::Vector& varnames () const noexcept {return m_varnames; } + [[nodiscard]] const amrex::Vector& varnames () const noexcept {return m_varnames; } /** Returns the number of dimensions in the Headerfile */ - int spaceDim () const noexcept {return m_spacedim; } + [[nodiscard]] int spaceDim () const noexcept {return m_spacedim; } /** Returns the physical time in the simulation in the boosted-frame */ - amrex::Real time () const noexcept {return m_time; } + [[nodiscard]] amrex::Real time () const noexcept {return m_time; } /** Returns finest level output in the Headerfile */ - int finestLevel () const noexcept { return m_finest_level; } + [[nodiscard]] int finestLevel () const noexcept { return m_finest_level; } /** Returns the physical co-ordinates of the lower corner in dimension, idim, * that corresponds the to the respective plotfile data */ - amrex::Real problo (int dim) const noexcept {return m_prob_lo[dim]; } + [[nodiscard]] amrex::Real problo (int dim) const noexcept {return m_prob_lo[dim]; } /** Returns the physical co-ordinates of the upper corner in dimension, idim, * that corresponds the to the respective plotfile data. */ - amrex::Real probhi (int dim) const noexcept {return m_prob_hi[dim]; } + [[nodiscard]] amrex::Real probhi (int dim) const noexcept {return m_prob_hi[dim]; } /** Returns the bounding box of the domain spanned in the plotfile */ - amrex::Box probDomain () const noexcept {return m_prob_domain; } + [[nodiscard]] amrex::Box probDomain () const noexcept {return m_prob_domain; } /** Returns timestep at which the plotfile was written */ - int timestep () const noexcept {return m_timestep; } - int numFabs () const noexcept {return m_numFabs; } + [[nodiscard]] int timestep () const noexcept {return m_timestep; } + [[nodiscard]] int numFabs () const noexcept {return m_numFabs; } /** Returns physical co-ordinates of the lower-corner for the i-th Fab. * \param[in] iFab id of the ith Fab in the list of Multifabs * \returns Array of lower-corner physical co-ordinates corresponding to the ith Fab */ - amrex::Array FabLo (int iFab) const noexcept {return m_glo[iFab]; } + [[nodiscard]] amrex::Array FabLo (int iFab) const noexcept {return m_glo[iFab]; } /** Returns physical co-ordinates of the lower-corner for the i-th Fab. * \param[in] iFab id of the ith Fab in the list of Multifabs * \returns Array of lower-corner physical co-ordinates corresponding to the ith Fab */ - amrex::Array FabHi (int iFab) const noexcept {return m_ghi[iFab]; } + [[nodiscard]] amrex::Array FabHi (int iFab) const noexcept {return m_ghi[iFab]; } /** Returns path to location of multifabs */ - std::string CellPath () const noexcept {return m_CellPath; } + [[nodiscard]] std::string CellPath () const noexcept {return m_CellPath; } /** Reads the Header file data for BTD */ void ReadHeaderData (); diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp index fdda963ddae..52d3656a425 100644 --- a/Source/Diagnostics/BTDiagnostics.cpp +++ b/Source/Diagnostics/BTDiagnostics.cpp @@ -423,7 +423,7 @@ BTDiagnostics::InitializeBufferData ( int i_buffer , int lev, bool restart) // For the z-dimension, number of cells in the lab-frame is // computed using the coarsened cell-size in the lab-frame obtained using // the ref_ratio at level, lev-1. - amrex::IntVect ref_ratio = amrex::IntVect(1); + auto ref_ratio = amrex::IntVect(1); if (lev > 0 ) ref_ratio = WarpX::RefRatio(lev-1); // Number of lab-frame cells in z-direction at level, lev const int num_zcells_lab = static_cast( std::floor ( @@ -871,7 +871,7 @@ BTDiagnostics::k_index_zlab (int i_buffer, int lev) { auto & warpx = WarpX::GetInstance(); const amrex::Real prob_domain_zmin_lab = m_snapshot_domain_lab[i_buffer].lo( m_moving_window_dir ); - amrex::IntVect ref_ratio = amrex::IntVect(1); + auto ref_ratio = amrex::IntVect(1); if (lev > 0 ) ref_ratio = WarpX::RefRatio(lev-1); const int k_lab = static_cast(floor ( ( m_current_z_lab[i_buffer] @@ -910,7 +910,7 @@ BTDiagnostics::DefineFieldBufferMultiFab (const int i_buffer, const int lev) static_cast(m_varnames.size()), ngrow ); m_mf_output[i_buffer][lev].setVal(0.); - amrex::IntVect ref_ratio = amrex::IntVect(1); + auto ref_ratio = amrex::IntVect(1); if (lev > 0 ) ref_ratio = WarpX::RefRatio(lev-1); for (int idim = 0; idim < AMREX_SPACEDIM; ++idim) { const amrex::Real cellsize = (idim < WARPX_ZINDEX)? diff --git a/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H index 9e21160242f..68f9c38bc8e 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/ComputeDiagFunctor.H @@ -38,7 +38,7 @@ public: virtual void operator() (amrex::MultiFab& mf_dst, int dcomp, int i_buffer = 0) const = 0; /** Number of component from the input multifab to write to the output * multifab */ - int nComp () const { return m_ncomp; } + [[nodiscard]] int nComp () const { return m_ncomp; } /** \brief Prepare data required to process fields in the operator() * Note that this function has parameters that are specific to diff --git a/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H b/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H index 4a5e8976943..3309dbed584 100644 --- a/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H +++ b/Source/Diagnostics/ComputeDiagFunctors/ComputeParticleDiagFunctor.H @@ -20,7 +20,7 @@ ComputeParticleDiagFunctor { public: - ComputeParticleDiagFunctor( ) {} + ComputeParticleDiagFunctor( ) = default; /** Virtual Destructor to handle clean destruction of derived classes */ virtual ~ComputeParticleDiagFunctor() = default; diff --git a/Source/Diagnostics/Diagnostics.H b/Source/Diagnostics/Diagnostics.H index dc9656433c6..5d5a300dea9 100644 --- a/Source/Diagnostics/Diagnostics.H +++ b/Source/Diagnostics/Diagnostics.H @@ -112,7 +112,7 @@ public: */ void FilterComputePackFlush (int step, bool force_flush=false); /** Whether the last timestep is always dumped */ - bool DoDumpLastTimestep () const {return m_dump_last_timestep;} + [[nodiscard]] bool DoDumpLastTimestep () const {return m_dump_last_timestep;} /** Returns the number of snapshots used in BTD. For Full-Diagnostics, the value is 1*/ int getnumbuffers() {return m_num_buffers;} /** Time in lab-frame associated with the ith snapshot diff --git a/Source/Diagnostics/FlushFormats/FlushFormat.H b/Source/Diagnostics/FlushFormats/FlushFormat.H index f5a693a5fa4..403e9df7857 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormat.H +++ b/Source/Diagnostics/FlushFormats/FlushFormat.H @@ -28,7 +28,7 @@ public: const amrex::Vector& totalParticlesFlushedAlready = amrex::Vector() ) const = 0; FlushFormat () = default; - virtual ~FlushFormat() {} + virtual ~FlushFormat() = default; FlushFormat ( FlushFormat const &) = default; FlushFormat& operator= ( FlushFormat const & ) = default; diff --git a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H index 62c7311804e..5e7d9b7b8e5 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H +++ b/Source/Diagnostics/FlushFormats/FlushFormatPlotfile.H @@ -57,7 +57,7 @@ public: bool isBTD = false) const; FlushFormatPlotfile () = default; - ~FlushFormatPlotfile() override {} + ~FlushFormatPlotfile() override = default; FlushFormatPlotfile ( FlushFormatPlotfile const &) = default; FlushFormatPlotfile& operator= ( FlushFormatPlotfile const & ) = default; diff --git a/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp b/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp index 7d047913988..e162b8b3121 100644 --- a/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp +++ b/Source/Diagnostics/FlushFormats/FlushFormatSensei.cpp @@ -7,14 +7,8 @@ # include #endif -FlushFormatSensei::FlushFormatSensei () : - m_insitu_pin_mesh(0), m_insitu_bridge(nullptr), - m_amr_mesh(nullptr) -{} - FlushFormatSensei::FlushFormatSensei (amrex::AmrMesh *amr_mesh, std::string diag_name) : - m_insitu_pin_mesh(0), m_insitu_bridge(nullptr), m_amr_mesh(amr_mesh) { #ifndef AMREX_USE_SENSEI_INSITU diff --git a/Source/Diagnostics/ParticleDiag/ParticleDiag.H b/Source/Diagnostics/ParticleDiag/ParticleDiag.H index 03537ebfe43..7d8b5819f5f 100644 --- a/Source/Diagnostics/ParticleDiag/ParticleDiag.H +++ b/Source/Diagnostics/ParticleDiag/ParticleDiag.H @@ -18,9 +18,9 @@ class ParticleDiag { public: ParticleDiag(std::string diag_name, std::string name, WarpXParticleContainer* pc, PinnedMemoryParticleContainer *pinned_pc = nullptr); - WarpXParticleContainer* getParticleContainer() const { return m_pc; } - PinnedMemoryParticleContainer* getPinnedParticleContainer() const { return m_pinned_pc; } - std::string getSpeciesName() const { return m_name; } + [[nodiscard]] WarpXParticleContainer* getParticleContainer() const { return m_pc; } + [[nodiscard]] PinnedMemoryParticleContainer* getPinnedParticleContainer() const { return m_pinned_pc; } + [[nodiscard]] std::string getSpeciesName() const { return m_name; } amrex::Vector m_plot_flags; bool m_do_random_filter = false; diff --git a/Source/Diagnostics/ReducedDiags/FieldProbe.cpp b/Source/Diagnostics/ReducedDiags/FieldProbe.cpp index 98cba72cfae..9d35c1896b3 100644 --- a/Source/Diagnostics/ReducedDiags/FieldProbe.cpp +++ b/Source/Diagnostics/ReducedDiags/FieldProbe.cpp @@ -631,7 +631,7 @@ void FieldProbe::WriteToFile (int step) const if (!(ProbeInDomain() && amrex::ParallelDescriptor::IOProcessor())) return; // loop over num valid particles to find the lowest particle ID for later sorting - long int first_id = static_cast(m_data_out[0]); + auto first_id = static_cast(m_data_out[0]); for (long int i = 0; i < m_valid_particles; i++) { if (m_data_out[i*noutputs] < first_id) diff --git a/Source/Diagnostics/SliceDiagnostic.cpp b/Source/Diagnostics/SliceDiagnostic.cpp index 9d3b3108b75..ddadc70d343 100644 --- a/Source/Diagnostics/SliceDiagnostic.cpp +++ b/Source/Diagnostics/SliceDiagnostic.cpp @@ -103,7 +103,7 @@ CreateSlice( const MultiFab& mf, const Vector &dom_geom, // Flag for interpolation if required // if ( interp_lo[idim] == 1) { - interpolate = 1; + interpolate = true; } // For the case when a dimension is reduced // diff --git a/Source/Diagnostics/WarpXOpenPMD.H b/Source/Diagnostics/WarpXOpenPMD.H index 66ffd1eaa97..4675e219fdd 100644 --- a/Source/Diagnostics/WarpXOpenPMD.H +++ b/Source/Diagnostics/WarpXOpenPMD.H @@ -167,7 +167,7 @@ private: * @param[in] isBTD is this a backtransformed diagnostics write? * @return the iteration object */ - inline openPMD::Iteration GetIteration (int const iteration, bool const isBTD) const + [[nodiscard]] inline openPMD::Iteration GetIteration (int const iteration, bool const isBTD) const { if (isBTD) { diff --git a/Source/Diagnostics/WarpXOpenPMD.cpp b/Source/Diagnostics/WarpXOpenPMD.cpp index 2e9746fe5c8..a91511dcced 100644 --- a/Source/Diagnostics/WarpXOpenPMD.cpp +++ b/Source/Diagnostics/WarpXOpenPMD.cpp @@ -703,12 +703,12 @@ WarpXOpenPMDPlot::DumpToFile (ParticleContainer* pc, // dump individual particles bool contributed_particles = false; // did the local MPI rank contribute particles? for (auto currentLevel = 0; currentLevel <= pc->finestLevel(); currentLevel++) { - uint64_t offset = static_cast( counter.m_ParticleOffsetAtRank[currentLevel] ); + auto offset = static_cast( counter.m_ParticleOffsetAtRank[currentLevel] ); // For BTD, the offset include the number of particles already flushed if (isBTD) offset += ParticleFlushOffset; for (ParticleIter pti(*pc, currentLevel); pti.isValid(); ++pti) { auto const numParticleOnTile = pti.numParticles(); - uint64_t const numParticleOnTile64 = static_cast( numParticleOnTile ); + auto const numParticleOnTile64 = static_cast( numParticleOnTile ); // Do not call storeChunk() with zero-sized particle tiles: // https://github.com/openPMD/openPMD-api/issues/1147 @@ -940,7 +940,7 @@ WarpXOpenPMDPlot::SaveRealProperty (ParticleIter& pti, { auto const numParticleOnTile = pti.numParticles(); - uint64_t const numParticleOnTile64 = static_cast( numParticleOnTile ); + auto const numParticleOnTile64 = static_cast( numParticleOnTile ); auto const& aos = pti.GetArrayOfStructs(); // size = numParticlesOnTile auto const& soa = pti.GetStructOfArrays(); // first we concatinate the AoS into contiguous arrays diff --git a/Source/FieldSolver/ElectrostaticSolver.H b/Source/FieldSolver/ElectrostaticSolver.H index 1e0f180b0fe..7bd982bb4e0 100644 --- a/Source/FieldSolver/ElectrostaticSolver.H +++ b/Source/FieldSolver/ElectrostaticSolver.H @@ -61,7 +61,9 @@ namespace ElectrostaticSolver { buildParsersEB(); } - PhiCalculatorEB getPhiEB(amrex::Real t) const noexcept { + [[nodiscard]] PhiCalculatorEB + getPhiEB(amrex::Real t) const noexcept + { return PhiCalculatorEB{t, potential_eb}; } diff --git a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H index 452dbc3057a..399177c82bb 100644 --- a/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H +++ b/Source/FieldSolver/FiniteDifferenceSolver/HybridPICModel/HybridPICModel.H @@ -140,8 +140,17 @@ public: amrex::Vector< std::unique_ptr > electron_pressure_fp; // Helper functions to retrieve hybrid-PIC multifabs - amrex::MultiFab * get_pointer_current_fp_ampere (int lev, int direction) const { return current_fp_ampere[lev][direction].get(); } - amrex::MultiFab * get_pointer_electron_pressure_fp (int lev) const { return electron_pressure_fp[lev].get(); } + [[nodiscard]] amrex::MultiFab* + get_pointer_current_fp_ampere (int lev, int direction) const + { + return current_fp_ampere[lev][direction].get(); + } + + [[nodiscard]] amrex::MultiFab* + get_pointer_electron_pressure_fp (int lev) const + { + return electron_pressure_fp[lev].get(); + } /** Gpu Vector with index type of the Jx multifab */ amrex::GpuArray Jx_IndexType; diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp index 3bca43f4e8f..66aa463503f 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/PsatdAlgorithmPmlRZ.cpp @@ -23,7 +23,6 @@ PsatdAlgorithmPmlRZ::PsatdAlgorithmPmlRZ (SpectralKSpaceRZ const & spectral_kspa short const grid_type, amrex::Real const dt): // Initialize members of base class and member variables SpectralBaseAlgorithmRZ{spectral_kspace, dm, spectral_index, norder_z, grid_type}, - coefficients_initialized{false}, m_dt{dt} { // Allocate the arrays of coefficients diff --git a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H index ce3c26b6297..95c68f6d61b 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralAlgorithms/SpectralBaseAlgorithmRZ.H @@ -26,7 +26,7 @@ class SpectralBaseAlgorithmRZ // The destructor should also be a virtual function, so that // a pointer to subclass of `SpectraBaseAlgorithm` actually // calls the subclass's destructor. - virtual ~SpectralBaseAlgorithmRZ() {} + virtual ~SpectralBaseAlgorithmRZ() = default; /** * \brief Default Copy constructor diff --git a/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.H b/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.H index ca30fb907ae..9d3af0f40a7 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.H +++ b/Source/FieldSolver/SpectralSolver/SpectralBinomialFilter.H @@ -23,7 +23,7 @@ class SpectralBinomialFilter // implements the filter. using KFilterArray = amrex::Gpu::DeviceVector; - SpectralBinomialFilter () {} + SpectralBinomialFilter () = default; void InitFilterArray (RealKVector const & kvec, amrex::Real dels, int npasses, diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp index 7a74acfadf0..c2309512596 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/BesselRoots.cpp @@ -139,7 +139,7 @@ void GetBesselRoots(int n, int nk, amrex::Vector& roots, amrex::Vec zeroj = b0 - (t1/b1) - (t3/b3) - (t5/b5) - (t7/b7); - const amrex::Real errj = static_cast(std::abs(jn(n, zeroj))); + const auto errj = static_cast(std::abs(jn(n, zeroj))); // improve solution using procedure SecantRootFinder if (errj > tol) ::SecantRootFinder(n, nitmx, tol, &zeroj, &ierror); diff --git a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/SpectralHankelTransformer.H b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/SpectralHankelTransformer.H index d14bb16681b..881a219c279 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/SpectralHankelTransformer.H +++ b/Source/FieldSolver/SpectralSolver/SpectralHankelTransform/SpectralHankelTransformer.H @@ -23,7 +23,7 @@ class SpectralHankelTransformer { public: - SpectralHankelTransformer () {} + SpectralHankelTransformer () = default; SpectralHankelTransformer (int nr, int n_rz_azimuthal_modes, @@ -33,7 +33,7 @@ class SpectralHankelTransformer ExtractKrArray (); // Returns an array that holds the kr for all of the modes - HankelTransform::RealVector const & getKrArray () const {return m_kr;} + [[nodiscard]] HankelTransform::RealVector const & getKrArray () const {return m_kr;} // Converts a scalar field from the physical to the spectral space void diff --git a/Source/FieldSolver/WarpXPushFieldsEM_K.H b/Source/FieldSolver/WarpXPushFieldsEM_K.H index e8e675cd276..d6e7e353a54 100644 --- a/Source/FieldSolver/WarpXPushFieldsEM_K.H +++ b/Source/FieldSolver/WarpXPushFieldsEM_K.H @@ -94,7 +94,7 @@ void damp_field_in_guards( // Apply damping factor in guards cells below the lower end of the domain const int n_guard = -tb_smallEnd; - const amrex::Real cell = static_cast(idx + n_guard); + const auto cell = static_cast(idx + n_guard); const amrex::Real phase = MathConst::pi * cell / n_guard; const amrex::Real sin_phase = std::sin(phase); @@ -107,7 +107,7 @@ void damp_field_in_guards( // Apply damping factor in guards cells above the upper end of the domain const int n_guard = tb_bigEnd - n_domain; - const amrex::Real cell = static_cast(tb_bigEnd - idx); + const auto cell = static_cast(tb_bigEnd - idx); const amrex::Real phase = MathConst::pi * cell / n_guard; const amrex::Real sin_phase = std::sin(phase); diff --git a/Source/Fluids/MultiFluidContainer.H b/Source/Fluids/MultiFluidContainer.H index 0576a211aa9..23f0c46590b 100644 --- a/Source/Fluids/MultiFluidContainer.H +++ b/Source/Fluids/MultiFluidContainer.H @@ -36,14 +36,14 @@ public: MultiFluidContainer (int nlevs_max); - ~MultiFluidContainer() {} + ~MultiFluidContainer() = default; MultiFluidContainer (MultiFluidContainer const &) = delete; MultiFluidContainer& operator= (MultiFluidContainer const & ) = delete; MultiFluidContainer(MultiFluidContainer&& ) = default; MultiFluidContainer& operator=(MultiFluidContainer&& ) = default; - WarpXFluidContainer& + [[nodiscard]] WarpXFluidContainer& GetFluidContainer (int ispecies) const {return *allcontainers[ispecies];} #ifdef WARPX_USE_OPENPMD @@ -66,7 +66,7 @@ public: amrex::MultiFab* rho, amrex::MultiFab& jx, amrex::MultiFab& jy, amrex::MultiFab& jz, amrex::Real cur_time, bool skip_deposition=false); - int nSpecies() const {return static_cast(species_names.size());} + [[nodiscard]] int nSpecies() const {return static_cast(species_names.size());} void DepositCharge (int lev, amrex::MultiFab &rho); void DepositCurrent (int lev, diff --git a/Source/Fluids/WarpXFluidContainer.H b/Source/Fluids/WarpXFluidContainer.H index ac6824479e6..172ef4c4209 100644 --- a/Source/Fluids/WarpXFluidContainer.H +++ b/Source/Fluids/WarpXFluidContainer.H @@ -31,7 +31,7 @@ public: friend MultiFluidContainer; WarpXFluidContainer (int nlevs_max, int ispecies, const std::string& name); - ~WarpXFluidContainer() {} + ~WarpXFluidContainer() = default; WarpXFluidContainer (WarpXFluidContainer const &) = delete; WarpXFluidContainer& operator= (WarpXFluidContainer const & ) = delete; @@ -131,8 +131,8 @@ public: */ void DepositCharge (int lev, amrex::MultiFab &rho, int icomp = 0); - amrex::Real getCharge () const {return charge;} - amrex::Real getMass () const {return mass;} + [[nodiscard]] amrex::Real getCharge () const {return charge;} + [[nodiscard]] amrex::Real getMass () const {return mass;} protected: int species_id; diff --git a/Source/Fluids/WarpXFluidContainer.cpp b/Source/Fluids/WarpXFluidContainer.cpp index 68cb65b6174..5b6eacf2fc7 100644 --- a/Source/Fluids/WarpXFluidContainer.cpp +++ b/Source/Fluids/WarpXFluidContainer.cpp @@ -964,13 +964,13 @@ void WarpXFluidContainer::GatherAndPush ( // Prepare interpolation of current components to cell center - amrex::GpuArray Nodal_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray Ex_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray Ey_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray Ez_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray Bx_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray By_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray Bz_type = amrex::GpuArray{0, 0, 0}; + auto Nodal_type = amrex::GpuArray{0, 0, 0}; + auto Ex_type = amrex::GpuArray{0, 0, 0}; + auto Ey_type = amrex::GpuArray{0, 0, 0}; + auto Ez_type = amrex::GpuArray{0, 0, 0}; + auto Bx_type = amrex::GpuArray{0, 0, 0}; + auto By_type = amrex::GpuArray{0, 0, 0}; + auto Bz_type = amrex::GpuArray{0, 0, 0}; for (int i = 0; i < AMREX_SPACEDIM; ++i) { Nodal_type[i] = N[lev]->ixType()[i]; @@ -1244,10 +1244,10 @@ void WarpXFluidContainer::DepositCurrent( const amrex::Real q = getCharge(); // Prepare interpolation of current components to cell center - amrex::GpuArray j_nodal_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray jx_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray jy_type = amrex::GpuArray{0, 0, 0}; - amrex::GpuArray jz_type = amrex::GpuArray{0, 0, 0}; + auto j_nodal_type = amrex::GpuArray{0, 0, 0}; + auto jx_type = amrex::GpuArray{0, 0, 0}; + auto jy_type = amrex::GpuArray{0, 0, 0}; + auto jz_type = amrex::GpuArray{0, 0, 0}; for (int i = 0; i < AMREX_SPACEDIM; ++i) { j_nodal_type[i] = tmp_jx_fluid.ixType()[i]; diff --git a/Source/Initialization/GetVelocity.H b/Source/Initialization/GetVelocity.H index 7b1d5f21350..3123708bf06 100644 --- a/Source/Initialization/GetVelocity.H +++ b/Source/Initialization/GetVelocity.H @@ -27,7 +27,7 @@ struct GetVelocity int m_sign_dir; //! Sign of the velocity direction positive=1, negative=-1 /** Constant velocity value, if m_type == VelConstantValue */ - amrex::Real m_velocity; + amrex::Real m_velocity{0}; /** Velocity parser function, if m_type == VelParserFunction */ amrex::ParserExecutor<3> m_velocity_parser; @@ -79,6 +79,7 @@ struct GetVelocity * 1: y * 2: z */ + [[nodiscard]] AMREX_GPU_HOST_DEVICE int direction () const noexcept { diff --git a/Source/Initialization/InjectorDensity.H b/Source/Initialization/InjectorDensity.H index 681bb334015..3848a42f4ee 100644 --- a/Source/Initialization/InjectorDensity.H +++ b/Source/Initialization/InjectorDensity.H @@ -26,6 +26,7 @@ struct InjectorDensityConstant { InjectorDensityConstant (amrex::Real a_rho) noexcept : m_rho(a_rho) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getDensity (amrex::Real, amrex::Real, amrex::Real) const noexcept @@ -43,6 +44,7 @@ struct InjectorDensityParser InjectorDensityParser (amrex::ParserExecutor<3> const& a_parser) noexcept : m_parser(a_parser) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getDensity (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -60,6 +62,7 @@ struct InjectorDensityPredefined void clear (); + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getDensity (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -111,7 +114,7 @@ struct InjectorDensityPredefined private: enum struct Profile { null, parabolic_channel }; - Profile profile; + Profile profile{Profile::null}; amrex::GpuArray p; }; @@ -157,6 +160,7 @@ struct InjectorDensity // call getDensity from the object stored in the union // (the union is called Object, and the instance is called object). + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getDensity (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept diff --git a/Source/Initialization/InjectorDensity.cpp b/Source/Initialization/InjectorDensity.cpp index 959f412c8b3..92a7b5193dd 100644 --- a/Source/Initialization/InjectorDensity.cpp +++ b/Source/Initialization/InjectorDensity.cpp @@ -39,7 +39,6 @@ void InjectorDensity::clear () InjectorDensityPredefined::InjectorDensityPredefined ( std::string const& a_species_name) noexcept - : profile(Profile::null) { const ParmParse pp_species_name(a_species_name); diff --git a/Source/Initialization/InjectorFlux.H b/Source/Initialization/InjectorFlux.H index 74ae840a502..e889c8b8966 100644 --- a/Source/Initialization/InjectorFlux.H +++ b/Source/Initialization/InjectorFlux.H @@ -24,6 +24,7 @@ struct InjectorFluxConstant { InjectorFluxConstant (amrex::Real a_flux) noexcept : m_flux(a_flux) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getFlux (amrex::Real, amrex::Real, amrex::Real, amrex::Real) const noexcept @@ -41,6 +42,7 @@ struct InjectorFluxParser InjectorFluxParser (amrex::ParserExecutor<4> const& a_parser) noexcept : m_parser(a_parser) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getFlux (amrex::Real x, amrex::Real y, amrex::Real z, amrex::Real t) const noexcept @@ -96,6 +98,7 @@ struct InjectorFlux // call getFlux from the object stored in the union // (the union is called Object, and the instance is called object). + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::Real getFlux (amrex::Real x, amrex::Real y, amrex::Real z, amrex::Real t) const noexcept diff --git a/Source/Initialization/InjectorMomentum.H b/Source/Initialization/InjectorMomentum.H index c1dea61f2b7..b0e71b52fa7 100644 --- a/Source/Initialization/InjectorMomentum.H +++ b/Source/Initialization/InjectorMomentum.H @@ -34,6 +34,7 @@ struct InjectorMomentumConstant InjectorMomentumConstant (amrex::Real a_ux, amrex::Real a_uy, amrex::Real a_uz) noexcept : m_ux(a_ux), m_uy(a_uy), m_uz(a_uz) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real, amrex::Real, amrex::Real, @@ -42,6 +43,7 @@ struct InjectorMomentumConstant return amrex::XDim3{m_ux,m_uy,m_uz}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real, amrex::Real, amrex::Real) const noexcept @@ -64,6 +66,7 @@ struct InjectorMomentumGaussian m_ux_th(a_ux_th), m_uy_th(a_uy_th), m_uz_th(a_uz_th) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real /*x*/, amrex::Real /*y*/, amrex::Real /*z*/, @@ -74,6 +77,7 @@ struct InjectorMomentumGaussian amrex::RandomNormal(m_uz_m, m_uz_th, engine)}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real /*x*/, amrex::Real /*y*/, amrex::Real /*z*/) const noexcept @@ -94,6 +98,7 @@ namespace { * @param u_th Momentum spread * @param engine Object used to generate random numbers */ + [[nodiscard]] AMREX_FORCE_INLINE AMREX_GPU_HOST_DEVICE amrex::Real @@ -171,6 +176,7 @@ struct InjectorMomentumGaussianFlux { } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real /*x*/, amrex::Real /*y*/, amrex::Real /*z*/, @@ -202,6 +208,7 @@ struct InjectorMomentumGaussianFlux return amrex::XDim3{ux, uy, uz}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real /*x*/, amrex::Real /*y*/, amrex::Real /*z*/) const noexcept @@ -234,6 +241,7 @@ struct InjectorMomentumUniform m_uz_h(amrex::Real(0.5) * (m_uz_max + m_uz_min)) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real /*x*/, amrex::Real /*y*/, amrex::Real /*z*/, @@ -244,6 +252,7 @@ struct InjectorMomentumUniform m_uz_min + amrex::Random(engine) * m_Duz}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real /*x*/, amrex::Real /*y*/, amrex::Real /*z*/) const noexcept @@ -269,6 +278,7 @@ struct InjectorMomentumBoltzmann : velocity(b), temperature(t) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real const x, amrex::Real const y, amrex::Real const z, @@ -321,6 +331,7 @@ struct InjectorMomentumBoltzmann return amrex::XDim3 {u[0],u[1],u[2]}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real const x, amrex::Real const y, amrex::Real const z) const noexcept @@ -330,7 +341,7 @@ struct InjectorMomentumBoltzmann for (auto& el : u) el = 0.0_rt; const amrex::Real beta = velocity(x,y,z); int const dir = velocity.direction(); - const amrex::Real gamma = static_cast(1._rt/sqrt(1._rt-beta*beta)); + const auto gamma = static_cast(1._rt/sqrt(1._rt-beta*beta)); u[dir] = gamma*beta; return amrex::XDim3 {u[0],u[1],u[2]}; } @@ -352,6 +363,7 @@ struct InjectorMomentumJuttner : velocity(b), temperature(t) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real const x, amrex::Real const y, amrex::Real const z, @@ -422,6 +434,7 @@ struct InjectorMomentumJuttner return amrex::XDim3 {u[0],u[1],u[2]}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real const x, amrex::Real const y, amrex::Real const z) const noexcept @@ -431,7 +444,7 @@ struct InjectorMomentumJuttner for (auto& el : u) el = 0.0_rt; amrex::Real const beta = velocity(x,y,z); int const dir = velocity.direction(); - amrex::Real const gamma = static_cast(1._rt/sqrt(1._rt-beta*beta)); + auto const gamma = static_cast(1._rt/sqrt(1._rt-beta*beta)); u[dir] = gamma*beta; return amrex::XDim3 {u[0],u[1],u[2]}; } @@ -454,6 +467,7 @@ struct InjectorMomentumRadialExpansion : u_over_r(a_u_over_r) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real x, amrex::Real y, amrex::Real z, @@ -462,6 +476,7 @@ struct InjectorMomentumRadialExpansion return {x*u_over_r, y*u_over_r, z*u_over_r}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -482,6 +497,7 @@ struct InjectorMomentumParser : m_ux_parser(a_ux_parser), m_uy_parser(a_uy_parser), m_uz_parser(a_uz_parser) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real x, amrex::Real y, amrex::Real z, @@ -490,6 +506,7 @@ struct InjectorMomentumParser return amrex::XDim3{m_ux_parser(x,y,z),m_uy_parser(x,y,z),m_uz_parser(x,y,z)}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -512,6 +529,7 @@ struct InjectorMomentumGaussianParser : m_ux_m_parser(a_ux_m_parser), m_uy_m_parser(a_uy_m_parser), m_uz_m_parser(a_uz_m_parser), m_ux_th_parser(a_ux_th_parser), m_uy_th_parser(a_uy_th_parser), m_uz_th_parser(a_uz_th_parser) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real x, amrex::Real y, amrex::Real z, @@ -528,6 +546,7 @@ struct InjectorMomentumGaussianParser amrex::RandomNormal(uz_m, uz_th, engine)}; } + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -640,6 +659,7 @@ struct InjectorMomentum // call getMomentum from the object stored in the union // (the union is called Object, and the instance is called object). + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getMomentum (amrex::Real x, amrex::Real y, amrex::Real z, @@ -693,6 +713,7 @@ struct InjectorMomentum // call getBulkMomentum from the object stored in the union // (the union is called Object, and the instance is called object). + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getBulkMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept diff --git a/Source/Initialization/InjectorPosition.H b/Source/Initialization/InjectorPosition.H index 2d099c56319..99aba7bc88b 100644 --- a/Source/Initialization/InjectorPosition.H +++ b/Source/Initialization/InjectorPosition.H @@ -18,6 +18,7 @@ // random distribution inside a unit cell. struct InjectorPositionRandom { + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getPositionUnitBox (int /*i_part*/, amrex::IntVect const /*ref_fac*/, @@ -33,6 +34,7 @@ struct InjectorPositionRandomPlane { InjectorPositionRandomPlane (int const& a_dir) noexcept : dir(a_dir) {} + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getPositionUnitBox (int /*i_part*/, amrex::IntVect const /*ref_fac*/, @@ -70,6 +72,7 @@ struct InjectorPositionRegular // particles within the cell. // ref_fac: the number of particles evenly-spaced within a cell // is a_ppc*(ref_fac**AMREX_SPACEDIM). + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getPositionUnitBox (int const i_part, amrex::IntVect const ref_fac, @@ -165,6 +168,7 @@ struct InjectorPosition // call getPositionUnitBox from the object stored in the union // (the union is called Object, and the instance is called object). + [[nodiscard]] AMREX_GPU_HOST_DEVICE amrex::XDim3 getPositionUnitBox (int const i_part, amrex::IntVect const ref_fac, @@ -192,6 +196,7 @@ struct InjectorPosition * \param x, y, z the point to check * \returns bool flag */ + [[nodiscard]] AMREX_GPU_HOST_DEVICE bool insideBounds (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -206,6 +211,7 @@ struct InjectorPosition * \param x, y, z the point to check * \returns bool flag */ + [[nodiscard]] AMREX_GPU_HOST_DEVICE bool insideBoundsInclusive (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept @@ -216,6 +222,7 @@ struct InjectorPosition } // bool: whether the region defined by lo and hi overaps with the plasma region + [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE bool overlapsWith (const amrex::XDim3& lo, const amrex::XDim3& hi) const noexcept diff --git a/Source/Initialization/PlasmaInjector.H b/Source/Initialization/PlasmaInjector.H index 22d5c7711e9..cec84440159 100644 --- a/Source/Initialization/PlasmaInjector.H +++ b/Source/Initialization/PlasmaInjector.H @@ -56,10 +56,12 @@ public: ~PlasmaInjector (); // bool: whether the point (x, y, z) is inside the plasma region - bool insideBounds (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept; + [[nodiscard]] bool insideBounds ( + amrex::Real x, amrex::Real y, amrex::Real z) const noexcept; // bool: whether the region defined by lo and hi overlaps with the plasma region - bool overlapsWith (const amrex::XDim3& lo, const amrex::XDim3& hi) const noexcept; + [[nodiscard]] bool overlapsWith ( + const amrex::XDim3& lo, const amrex::XDim3& hi) const noexcept; int num_particles_per_cell; amrex::Real num_particles_per_cell_real; @@ -67,18 +69,19 @@ public: amrex::Vector num_particles_per_cell_each_dim; // gamma * beta - amrex::XDim3 getMomentum (amrex::Real x, amrex::Real y, amrex::Real z) const noexcept; + [[nodiscard]] amrex::XDim3 getMomentum ( + amrex::Real x, amrex::Real y, amrex::Real z) const noexcept; - amrex::Real getCharge () {return charge;} - amrex::Real getMass () {return mass;} - PhysicalSpecies getPhysicalSpecies() const {return physical_species;} + [[nodiscard]] amrex::Real getCharge () {return charge;} + [[nodiscard]] amrex::Real getMass () {return mass;} + [[nodiscard]] PhysicalSpecies getPhysicalSpecies() const {return physical_species;} // bool: whether the initial injection of particles should be done // This routine is called during initialization of the plasma. - bool doInjection () const noexcept { return h_inj_pos != nullptr;} + [[nodiscard]] bool doInjection () const noexcept { return h_inj_pos != nullptr;} // bool: whether the flux injection of particles should be done. - bool doFluxInjection () const noexcept { return h_flux_pos != nullptr;} + [[nodiscard]] bool doFluxInjection () const noexcept { return h_flux_pos != nullptr;} bool add_single_particle = false; amrex::Vector single_particle_pos; diff --git a/Source/Initialization/VelocityProperties.H b/Source/Initialization/VelocityProperties.H index 6b981849dab..a66d0fca653 100644 --- a/Source/Initialization/VelocityProperties.H +++ b/Source/Initialization/VelocityProperties.H @@ -44,7 +44,7 @@ struct VelocityProperties int m_sign_dir; // Sign of the velocity direction positive=1, negative=-1 /* Constant velocity value, if m_type == VelConstantValue */ - amrex::Real m_velocity; + amrex::Real m_velocity{0}; /* Storage of the parser function, if m_type == VelParserFunction */ std::unique_ptr m_ptr_velocity_parser; }; diff --git a/Source/Initialization/VelocityProperties.cpp b/Source/Initialization/VelocityProperties.cpp index 30cb64cb70a..799b29e5be8 100644 --- a/Source/Initialization/VelocityProperties.cpp +++ b/Source/Initialization/VelocityProperties.cpp @@ -11,8 +11,7 @@ #include "Utils/Parser/ParserUtils.H" #include "Utils/TextMsg.H" -VelocityProperties::VelocityProperties (const amrex::ParmParse& pp): - m_velocity{0} +VelocityProperties::VelocityProperties (const amrex::ParmParse& pp) { // Set defaults std::string vel_dist_s = "constant"; diff --git a/Source/Laser/LaserProfiles.H b/Source/Laser/LaserProfiles.H index 73776fa3525..3aea88ac956 100644 --- a/Source/Laser/LaserProfiles.H +++ b/Source/Laser/LaserProfiles.H @@ -98,7 +98,7 @@ public: amrex::Real* AMREX_RESTRICT amplitude) const = 0; ILaserProfile () = default; - virtual ~ILaserProfile(){} + virtual ~ILaserProfile() = default; ILaserProfile ( ILaserProfile const &) = default; ILaserProfile& operator= ( ILaserProfile const & ) = default; @@ -308,7 +308,7 @@ private: * * \param t: simulation time */ - std::pair find_left_right_time_indices(amrex::Real t) const; + [[nodiscard]] std::pair find_left_right_time_indices(amrex::Real t) const; /** \brief Load field data within the temporal range [t_begin, t_end] * diff --git a/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp b/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp index 6cb9c50dc21..e29439c4bc0 100644 --- a/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp +++ b/Source/Laser/LaserProfilesImpl/LaserProfileFromFile.cpp @@ -172,7 +172,7 @@ WarpXLaserProfiles::FromFileLaserProfile::parse_lasy_file(std::string lasy_file_ WARPX_ALWAYS_ASSERT_WITH_MESSAGE(E.getAttribute("dataOrder").get() == "C", "Reading from files with non-C dataOrder is not implemented"); - std::string fileGeom = E.getAttribute("geometry").get(); + auto fileGeom = E.getAttribute("geometry").get(); auto E_laser = E[io::RecordComponent::SCALAR]; auto extent = E_laser.getExtent(); // Extract grid offset and grid spacing diff --git a/Source/Parallelization/GuardCellManager.cpp b/Source/Parallelization/GuardCellManager.cpp index d71a65446d7..c1ecc5db34c 100644 --- a/Source/Parallelization/GuardCellManager.cpp +++ b/Source/Parallelization/GuardCellManager.cpp @@ -208,11 +208,11 @@ guardCellManager::Init ( utils::parser::queryWithParser(pp_psatd, "nz_guard", ngFFt_z); #if defined(WARPX_DIM_3D) - IntVect ngFFT = IntVect(ngFFt_x, ngFFt_y, ngFFt_z); + auto ngFFT = IntVect(ngFFt_x, ngFFt_y, ngFFt_z); #elif defined(WARPX_DIM_XZ) || defined(WARPX_DIM_RZ) - IntVect ngFFT = IntVect(ngFFt_x, ngFFt_z); + auto ngFFT = IntVect(ngFFt_x, ngFFt_z); #elif defined(WARPX_DIM_1D_Z) - IntVect ngFFT = IntVect(ngFFt_z); + auto ngFFT = IntVect(ngFFt_z); #endif #ifdef WARPX_DIM_RZ @@ -309,7 +309,7 @@ guardCellManager::Init ( // factor grows symmetrically by half a cell on each side. So every // +2 orders, one touches one more cell point. int const FGcell = (nox + 1) / 2; // integer division - IntVect ng_FieldGather_noNCI = IntVect(AMREX_D_DECL(FGcell,FGcell,FGcell)); + auto ng_FieldGather_noNCI = IntVect(AMREX_D_DECL(FGcell,FGcell,FGcell)); ng_FieldGather_noNCI = ng_FieldGather_noNCI.min(ng_alloc_EB); // If NCI filter, add guard cells in the z direction diff --git a/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H b/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H index 88ec7693b4a..cd9b74c72b8 100644 --- a/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H +++ b/Source/Particles/Collision/BackgroundMCC/ImpactIonization.H @@ -202,7 +202,7 @@ public: ParticleUtils::getEnergy(u_coll2, m_mass1, E_coll); // each electron gets half the energy (could change this later) - const amrex::ParticleReal E_out = static_cast((E_coll - m_energy_cost) / 2.0_prt * PhysConst::q_e); + const auto E_out = static_cast((E_coll - m_energy_cost) / 2.0_prt * PhysConst::q_e); // precalculate often used value constexpr auto c2 = PhysConst::c * PhysConst::c; diff --git a/Source/Particles/Collision/BackgroundMCC/MCCProcess.H b/Source/Particles/Collision/BackgroundMCC/MCCProcess.H index 58fc86b2c29..9b844959e52 100644 --- a/Source/Particles/Collision/BackgroundMCC/MCCProcess.H +++ b/Source/Particles/Collision/BackgroundMCC/MCCProcess.H @@ -74,6 +74,7 @@ public: * @param E_coll collision energy in eV * */ + [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE amrex::ParticleReal getCrossSection (amrex::ParticleReal E_coll) const { @@ -101,6 +102,7 @@ public: MCCProcessType m_type; }; + [[nodiscard]] Executor const& executor () const { #ifdef AMREX_USE_GPU return m_exe_d; @@ -109,17 +111,17 @@ public: #endif } - amrex::ParticleReal getCrossSection (amrex::ParticleReal E_coll) const + [[nodiscard]] amrex::ParticleReal getCrossSection (amrex::ParticleReal E_coll) const { return m_exe_h.getCrossSection(E_coll); } - amrex::ParticleReal getEnergyPenalty () const { return m_exe_h.m_energy_penalty; } - amrex::ParticleReal getMinEnergyInput () const { return m_exe_h.m_energy_lo; } - amrex::ParticleReal getMaxEnergyInput () const { return m_exe_h.m_energy_hi; } - amrex::ParticleReal getEnergyInputStep () const { return m_exe_h.m_dE; } + [[nodiscard]] amrex::ParticleReal getEnergyPenalty () const { return m_exe_h.m_energy_penalty; } + [[nodiscard]] amrex::ParticleReal getMinEnergyInput () const { return m_exe_h.m_energy_lo; } + [[nodiscard]] amrex::ParticleReal getMaxEnergyInput () const { return m_exe_h.m_energy_hi; } + [[nodiscard]] amrex::ParticleReal getEnergyInputStep () const { return m_exe_h.m_dE; } - MCCProcessType type () const { return m_exe_h.m_type; } + [[nodiscard]] MCCProcessType type () const { return m_exe_h.m_type; } private: diff --git a/Source/Particles/ElementaryProcess/Ionization.H b/Source/Particles/ElementaryProcess/Ionization.H index 2c96c1a15a2..573491645ea 100644 --- a/Source/Particles/ElementaryProcess/Ionization.H +++ b/Source/Particles/ElementaryProcess/Ionization.H @@ -123,7 +123,7 @@ struct IonizationFilterFunc const amrex::ParticleReal uy = ptd.m_rdata[PIdx::uy][i]; const amrex::ParticleReal uz = ptd.m_rdata[PIdx::uz][i]; - const amrex::Real ga = static_cast( + const auto ga = static_cast( std::sqrt(1. + (ux*ux + uy*uy + uz*uz) * c2_inv)); const amrex::Real E = std::sqrt( - ( ux*ex + uy*ey + uz*ez ) * ( ux*ex + uy*ey + uz*ez ) * c2_inv diff --git a/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H b/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H index 2abd31fb4e4..f80705fd043 100644 --- a/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H +++ b/Source/Particles/ElementaryProcess/QEDInternals/BreitWheelerEngineWrapper.H @@ -300,22 +300,22 @@ public: /** * Builds the functor to initialize the optical depth */ - BreitWheelerGetOpticalDepth build_optical_depth_functor () const; + [[nodiscard]] BreitWheelerGetOpticalDepth build_optical_depth_functor () const; /** * Builds the functor to evolve the optical depth */ - BreitWheelerEvolveOpticalDepth build_evolve_functor () const; + [[nodiscard]] BreitWheelerEvolveOpticalDepth build_evolve_functor () const; /** * Builds the functor to generate the pairs */ - BreitWheelerGeneratePairs build_pair_functor () const; + [[nodiscard]] BreitWheelerGeneratePairs build_pair_functor () const; /** * Checks if the optical tables are properly initialized */ - bool are_lookup_tables_initialized () const; + [[nodiscard]] bool are_lookup_tables_initialized () const; /** * Export lookup tables data into a raw binary Vector @@ -323,7 +323,7 @@ public: * @return the data in binary format. The Vector is empty if tables were * not previously initialized. */ - std::vector export_lookup_tables_data () const; + [[nodiscard]] std::vector export_lookup_tables_data () const; /** * Init lookup tables from raw binary data. @@ -357,9 +357,9 @@ public: * * @return default control params to generate the tables */ - PicsarBreitWheelerCtrl get_default_ctrl() const; + [[nodiscard]] PicsarBreitWheelerCtrl get_default_ctrl() const; - amrex::ParticleReal get_minimum_chi_phot() const; + [[nodiscard]] amrex::ParticleReal get_minimum_chi_phot() const; private: bool m_lookup_tables_initialized = false; diff --git a/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.H b/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.H index 33660cde233..ceacb2016e8 100644 --- a/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.H +++ b/Source/Particles/ElementaryProcess/QEDInternals/QuantumSyncEngineWrapper.H @@ -279,22 +279,22 @@ public: /** * Builds the functor to initialize the optical depth */ - QuantumSynchrotronGetOpticalDepth build_optical_depth_functor (); + [[nodiscard]] QuantumSynchrotronGetOpticalDepth build_optical_depth_functor (); /** * Builds the functor to evolve the optical depth */ - QuantumSynchrotronEvolveOpticalDepth build_evolve_functor (); + [[nodiscard]] QuantumSynchrotronEvolveOpticalDepth build_evolve_functor (); /** * Builds the functor to generate photons */ - QuantumSynchrotronPhotonEmission build_phot_em_functor (); + [[nodiscard]] QuantumSynchrotronPhotonEmission build_phot_em_functor (); /** * Checks if the optical tables are properly initialized */ - bool are_lookup_tables_initialized () const; + [[nodiscard]] bool are_lookup_tables_initialized () const; /** * Export lookup tables data into a raw binary Vector @@ -302,7 +302,7 @@ public: * @return the data in binary format. The Vector is empty if tables were * not previously initialized. */ - std::vector export_lookup_tables_data () const; + [[nodiscard]] std::vector export_lookup_tables_data () const; /** * Init lookup tables from raw binary data. @@ -335,9 +335,9 @@ public: * * @return default control params to generate the tables */ - PicsarQuantumSyncCtrl get_default_ctrl() const; + [[nodiscard]] PicsarQuantumSyncCtrl get_default_ctrl() const; - amrex::ParticleReal get_minimum_chi_part() const; + [[nodiscard]] amrex::ParticleReal get_minimum_chi_part() const; private: bool m_lookup_tables_initialized = false; diff --git a/Source/Particles/Filter/FilterFunctors.H b/Source/Particles/Filter/FilterFunctors.H index fb83df1184a..3b2ddbc9dc8 100644 --- a/Source/Particles/Filter/FilterFunctors.H +++ b/Source/Particles/Filter/FilterFunctors.H @@ -97,8 +97,8 @@ struct ParserFilter m_is_active{a_is_active}, m_function_partparser{a_filter_parser}, m_mass{a_mass}, - m_t{time}, - m_units{InputUnits::WarpX}{} + m_t{time} + {} /** * \brief return 1 if the particle is selected by the parser @@ -143,7 +143,7 @@ public: /** Store physical time on the coarsest level. */ amrex::Real m_t; /** keep track of momentum units particles will come in with **/ - InputUnits m_units; + InputUnits m_units{InputUnits::WarpX}; }; @@ -164,7 +164,7 @@ struct GeometryFilter AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE bool operator () (const SuperParticleType& p, const amrex::RandomEngine&) const noexcept { - if ( !m_is_active ) return 1; + if ( !m_is_active ) return true; return ! (AMREX_D_TERM( (p.pos(0) < m_domain.lo(0)) || (p.pos(0) > m_domain.hi(0) ), || (p.pos(1) < m_domain.lo(1)) || (p.pos(1) > m_domain.hi(1) ), || (p.pos(2) < m_domain.lo(2)) || (p.pos(2) > m_domain.hi(2) ))); diff --git a/Source/Particles/LaserParticleContainer.H b/Source/Particles/LaserParticleContainer.H index 464b411ac66..b722daa40fc 100644 --- a/Source/Particles/LaserParticleContainer.H +++ b/Source/Particles/LaserParticleContainer.H @@ -44,7 +44,6 @@ public: LaserParticleContainer (amrex::AmrCore* amr_core, int ispecies, const std::string& name); ~LaserParticleContainer () override = default; - LaserParticleContainer ( LaserParticleContainer const &) = delete; LaserParticleContainer& operator= ( LaserParticleContainer const & ) = delete; LaserParticleContainer ( LaserParticleContainer&& ) = default; diff --git a/Source/Particles/MultiParticleContainer.H b/Source/Particles/MultiParticleContainer.H index 64c8dbb085a..a45d9fd0dc0 100644 --- a/Source/Particles/MultiParticleContainer.H +++ b/Source/Particles/MultiParticleContainer.H @@ -68,20 +68,20 @@ public: MultiParticleContainer (amrex::AmrCore* amr_core); - ~MultiParticleContainer() {} + ~MultiParticleContainer() = default; MultiParticleContainer (MultiParticleContainer const &) = delete; MultiParticleContainer& operator= (MultiParticleContainer const & ) = delete; MultiParticleContainer(MultiParticleContainer&& ) = default; MultiParticleContainer& operator=(MultiParticleContainer&& ) = default; - WarpXParticleContainer& + [[nodiscard]] WarpXParticleContainer& GetParticleContainer (int index) const {return *allcontainers[index];} - WarpXParticleContainer* + [[nodiscard]] WarpXParticleContainer* GetParticleContainerPtr (int index) const {return allcontainers[index].get();} - WarpXParticleContainer& + [[nodiscard]] WarpXParticleContainer& GetParticleContainerFromName (const std::string& name) const; std::array meanParticleVelocity(int index) { @@ -195,7 +195,7 @@ public: * defined by m_qed_schwinger_xmin/xmax/ymin/ymax/zmin/zmax and the warpx level 0 geometry * object (to make the link between Real and int quatities). */ - amrex::Box ComputeSchwingerGlobalBox () const; + [[nodiscard]] amrex::Box ComputeSchwingerGlobalBox () const; #endif void Restart (const std::string& dir); @@ -225,18 +225,18 @@ public: * * @param[in] lev the index of the refinement level. */ - amrex::Vector GetZeroParticlesInGrid(int lev) const; + [[nodiscard]] amrex::Vector GetZeroParticlesInGrid(int lev) const; - amrex::Vector NumberOfParticlesInGrid(int lev) const; + [[nodiscard]] amrex::Vector NumberOfParticlesInGrid(int lev) const; void Increment (amrex::MultiFab& mf, int lev); void SetParticleBoxArray (int lev, amrex::BoxArray& new_ba); void SetParticleDistributionMap (int lev, amrex::DistributionMapping& new_dm); - int nSpecies () const {return static_cast(species_names.size());} - int nLasers () const {return static_cast(lasers_names.size());} - int nContainers () const {return static_cast(allcontainers.size());} + [[nodiscard]] int nSpecies () const {return static_cast(species_names.size());} + [[nodiscard]] int nLasers () const {return static_cast(lasers_names.size());} + [[nodiscard]] int nContainers () const {return static_cast(allcontainers.size());} /** Whether back-transformed diagnostics need to be performed for any plasma species. * @@ -250,13 +250,15 @@ public: */ void SetDoBackTransformedParticles (std::string species_name, bool do_back_transformed_particles); - int nSpeciesDepositOnMainGrid () const { + [[nodiscard]] int nSpeciesDepositOnMainGrid () const + { bool const onMainGrid = true; auto const & v = m_deposit_on_main_grid; return static_cast(std::count( v.begin(), v.end(), onMainGrid )); } - int nSpeciesGatherFromMainGrid() const { + [[nodiscard]] int nSpeciesGatherFromMainGrid() const + { bool const fromMainGrid = true; auto const & v = m_gather_from_main_grid; return static_cast(std::count( v.begin(), v.end(), fromMainGrid )); @@ -273,16 +275,17 @@ public: */ void UpdateAntennaPosition(amrex::Real dt) const; - int doContinuousInjection() const; + [[nodiscard]] int doContinuousInjection() const; // Inject particles from a surface during the simulation void ContinuousFluxInjection(amrex::Real t, amrex::Real dt) const; - std::vector GetSpeciesNames() const { return species_names; } + [[nodiscard]] std::vector GetSpeciesNames() const { return species_names; } - std::vector GetLasersNames() const { return lasers_names; } + [[nodiscard]] std::vector GetLasersNames() const { return lasers_names; } - std::vector GetSpeciesAndLasersNames() const { + [[nodiscard]] std::vector GetSpeciesAndLasersNames() const + { std::vector tmp = species_names; tmp.insert(tmp.end(), lasers_names.begin(), lasers_names.end()); return tmp; @@ -326,7 +329,7 @@ public: const amrex::MultiFab& Bz); #endif - int getSpeciesID (std::string product_str) const; + [[nodiscard]] int getSpeciesID (std::string product_str) const; protected: @@ -373,6 +376,7 @@ protected: std::vector species_types; template + [[nodiscard]] amrex::MFItInfo getMFItInfo (const WarpXParticleContainer& pc_src, Args const&... pc_dsts) const noexcept { @@ -420,12 +424,12 @@ protected: /** * Returns the number of species having Quantum Synchrotron process enabled */ - int NSpeciesQuantumSync() const { return m_nspecies_quantum_sync;} + [[nodiscard]] int NSpeciesQuantumSync() const { return m_nspecies_quantum_sync;} /** * Returns the number of species having Breit Wheeler process enabled */ - int NSpeciesBreitWheeler() const { return m_nspecies_breit_wheeler;} + [[nodiscard]] int NSpeciesBreitWheeler() const { return m_nspecies_breit_wheeler;} /** * Initializes the Quantum Synchrotron engine @@ -450,7 +454,7 @@ protected: void BreitWheelerGenerateTable(); /** Whether or not to activate Schwinger process */ - bool m_do_qed_schwinger = 0; + bool m_do_qed_schwinger = false; /** Name of Schwinger electron product species */ std::string m_qed_schwinger_ele_product_name; /** Name of Schwinger positron product species */ diff --git a/Source/Particles/MultiParticleContainer.cpp b/Source/Particles/MultiParticleContainer.cpp index 58b490af10a..4eaf89c2aa4 100644 --- a/Source/Particles/MultiParticleContainer.cpp +++ b/Source/Particles/MultiParticleContainer.cpp @@ -805,13 +805,13 @@ MultiParticleContainer::getSpeciesID (std::string product_str) const { auto species_and_lasers_names = GetSpeciesAndLasersNames(); int i_product = 0; - bool found = 0; + bool found = false; // Loop over species for (int i=0; i < static_cast(species_and_lasers_names.size()); i++){ // If species name matches, store its ID // into i_product if (species_and_lasers_names[i] == product_str){ - found = 1; + found = true; i_product = i; } } @@ -832,12 +832,12 @@ MultiParticleContainer::SetDoBackTransformedParticles (const bool do_back_transf void MultiParticleContainer::SetDoBackTransformedParticles (std::string species_name, const bool do_back_transformed_particles) { auto species_names_list = GetSpeciesNames(); - bool found = 0; + bool found = false; // Loop over species for (int i = 0; i < static_cast(species_names.size()); ++i) { // If species name matches, set back-transformed particles parameters if (species_names_list[i] == species_name) { - found = 1; + found = true; auto& pc = allcontainers[i]; pc->SetDoBackTransformedParticles(do_back_transformed_particles); } diff --git a/Source/Particles/NamedComponentParticleContainer.H b/Source/Particles/NamedComponentParticleContainer.H index 6ff9d157790..c9f66e6c2b0 100644 --- a/Source/Particles/NamedComponentParticleContainer.H +++ b/Source/Particles/NamedComponentParticleContainer.H @@ -169,13 +169,13 @@ public: } /** Return the name-to-index map for the compile-time and runtime-time real components */ - std::map getParticleComps () const noexcept { return particle_comps;} + [[nodiscard]] std::map getParticleComps () const noexcept { return particle_comps;} /** Return the name-to-index map for the compile-time and runtime-time integer components */ - std::map getParticleiComps () const noexcept { return particle_icomps;} + [[nodiscard]] std::map getParticleiComps () const noexcept { return particle_icomps;} /** Return the name-to-index map for the runtime-time real components */ - std::map getParticleRuntimeComps () const noexcept { return particle_runtime_comps;} + [[nodiscard]] std::map getParticleRuntimeComps () const noexcept { return particle_runtime_comps;} /** Return the name-to-index map for the runtime-time integer components */ - std::map getParticleRuntimeiComps () const noexcept { return particle_runtime_icomps;} + [[nodiscard]] std::map getParticleRuntimeiComps () const noexcept { return particle_runtime_icomps;} protected: std::map particle_comps; diff --git a/Source/Particles/ParticleBoundaryBuffer.H b/Source/Particles/ParticleBoundaryBuffer.H index cbc8667c8d4..1e9748b2ff5 100644 --- a/Source/Particles/ParticleBoundaryBuffer.H +++ b/Source/Particles/ParticleBoundaryBuffer.H @@ -23,7 +23,7 @@ class WARPX_EXPORT ParticleBoundaryBuffer public: ParticleBoundaryBuffer (); - ~ParticleBoundaryBuffer() {} + ~ParticleBoundaryBuffer() = default; /** Copy constructor for ParticleBoundaryBuffer */ ParticleBoundaryBuffer ( const ParticleBoundaryBuffer &) = delete; diff --git a/Source/Particles/ParticleCreation/SmartCopy.H b/Source/Particles/ParticleCreation/SmartCopy.H index 94d10dfce6b..da5637a7919 100644 --- a/Source/Particles/ParticleCreation/SmartCopy.H +++ b/Source/Particles/ParticleCreation/SmartCopy.H @@ -146,7 +146,7 @@ public: m_defined{true} {} - SmartCopy getSmartCopy () const noexcept + [[nodiscard]] SmartCopy getSmartCopy () const noexcept { AMREX_ASSERT(m_defined); return SmartCopy{m_tag_real.size(), @@ -159,7 +159,7 @@ public: m_policy_int.dataPtr()}; } - bool isDefined () const noexcept { return m_defined; } + [[nodiscard]] bool isDefined () const noexcept { return m_defined; } }; #endif diff --git a/Source/Particles/ParticleCreation/SmartCreate.H b/Source/Particles/ParticleCreation/SmartCreate.H index 82e3cbce710..1be3bd79cf3 100644 --- a/Source/Particles/ParticleCreation/SmartCreate.H +++ b/Source/Particles/ParticleCreation/SmartCreate.H @@ -87,7 +87,7 @@ class SmartCreateFactory { PolicyVec m_policy_real; PolicyVec m_policy_int; - bool m_defined; + bool m_defined{false}; public: template @@ -97,14 +97,14 @@ public: m_defined{true} {} - SmartCreate getSmartCreate () const noexcept + [[nodiscard]] SmartCreate getSmartCreate () const noexcept { AMREX_ASSERT(m_defined); return SmartCreate{m_policy_real.dataPtr(), m_policy_int.dataPtr()}; } - bool isDefined () const noexcept { return m_defined; } + [[nodiscard]] bool isDefined () const noexcept { return m_defined; } }; #endif //SMART_CREATE_H_ diff --git a/Source/Particles/ParticleCreation/SmartUtils.H b/Source/Particles/ParticleCreation/SmartUtils.H index 4604ef59680..732a12bb729 100644 --- a/Source/Particles/ParticleCreation/SmartUtils.H +++ b/Source/Particles/ParticleCreation/SmartUtils.H @@ -31,7 +31,7 @@ struct SmartCopyTag amrex::Gpu::DeviceVector src_comps; amrex::Gpu::DeviceVector dst_comps; - int size () const noexcept { return static_cast(common_names.size()); } + [[nodiscard]] int size () const noexcept { return static_cast(common_names.size()); } }; PolicyVec getPolicies (const NameMap& names) noexcept; diff --git a/Source/Particles/PhotonParticleContainer.H b/Source/Particles/PhotonParticleContainer.H index ada11857b34..d4498785e49 100644 --- a/Source/Particles/PhotonParticleContainer.H +++ b/Source/Particles/PhotonParticleContainer.H @@ -36,7 +36,7 @@ public: PhotonParticleContainer (amrex::AmrCore* amr_core, int ispecies, const std::string& name); - ~PhotonParticleContainer () override {} + ~PhotonParticleContainer () override = default; PhotonParticleContainer ( PhotonParticleContainer const &) = delete; PhotonParticleContainer& operator= ( PhotonParticleContainer const & ) = delete; diff --git a/Source/Particles/PhysicalParticleContainer.H b/Source/Particles/PhysicalParticleContainer.H index 526e5e01b2d..ec9834c4e7e 100644 --- a/Source/Particles/PhysicalParticleContainer.H +++ b/Source/Particles/PhysicalParticleContainer.H @@ -56,7 +56,7 @@ public: * the run if one of them is specified. */ void BackwardCompatibility (); - ~PhysicalParticleContainer () override {} + ~PhysicalParticleContainer () override = default; PhysicalParticleContainer (PhysicalParticleContainer const &) = delete; PhysicalParticleContainer& operator= (PhysicalParticleContainer const & ) = delete; @@ -312,7 +312,7 @@ public: * * @param[in] timestep the current timestep. */ - void resample (int timestep, bool verbose) final; + void resample (int timestep, bool verbose=true) final; #ifdef WARPX_QED //Functions decleared in WarpXParticleContainer.H diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp index 70614702ca5..6df69eb4d9b 100644 --- a/Source/Particles/PhysicalParticleContainer.cpp +++ b/Source/Particles/PhysicalParticleContainer.cpp @@ -2467,7 +2467,8 @@ PhysicalParticleContainer::SplitParticles (int lev) 0, attr_int, 1, NoSplitParticleID); // Copy particles from tmp to current particle container - addParticles(pctmp_split,1); + constexpr bool local_flag = true; + addParticles(pctmp_split,local_flag); // Clear tmp container pctmp_split.clearParticles(); } diff --git a/Source/Particles/RigidInjectedParticleContainer.H b/Source/Particles/RigidInjectedParticleContainer.H index 29bf294f939..acbe3a0493d 100644 --- a/Source/Particles/RigidInjectedParticleContainer.H +++ b/Source/Particles/RigidInjectedParticleContainer.H @@ -50,7 +50,6 @@ public: const std::string& name); ~RigidInjectedParticleContainer () override = default; - RigidInjectedParticleContainer ( RigidInjectedParticleContainer const &) = delete; RigidInjectedParticleContainer& operator= ( RigidInjectedParticleContainer const & ) = delete; RigidInjectedParticleContainer ( RigidInjectedParticleContainer&& ) = default; diff --git a/Source/Particles/WarpXParticleContainer.H b/Source/Particles/WarpXParticleContainer.H index f42863d5357..8ae6e4ef64b 100644 --- a/Source/Particles/WarpXParticleContainer.H +++ b/Source/Particles/WarpXParticleContainer.H @@ -57,23 +57,28 @@ public: WarpXParIter (ContainerType& pc, int level, amrex::MFItInfo& info); - const std::array& GetAttribs () const { + [[nodiscard]] const std::array& GetAttribs () const + { return GetStructOfArrays().GetRealData(); } - std::array& GetAttribs () { + [[nodiscard]] std::array& GetAttribs () + { return GetStructOfArrays().GetRealData(); } - const RealVector& GetAttribs (int comp) const { + [[nodiscard]] const RealVector& GetAttribs (int comp) const + { return GetStructOfArrays().GetRealData(comp); } - RealVector& GetAttribs (int comp) { + [[nodiscard]] RealVector& GetAttribs (int comp) + { return GetStructOfArrays().GetRealData(comp); } - IntVector& GetiAttribs (int comp) { + [[nodiscard]] IntVector& GetiAttribs (int comp) + { return GetStructOfArrays().GetIntData(comp); } }; @@ -115,7 +120,7 @@ public: using DiagnosticParticles = amrex::Vector, DiagnosticParticleData> >; WarpXParticleContainer (amrex::AmrCore* amr_core, int ispecies); - ~WarpXParticleContainer() override {} + ~WarpXParticleContainer() override = default; // Move and copy operations WarpXParticleContainer(const WarpXParticleContainer&) = delete; diff --git a/Source/Utils/Parser/IntervalsParser.H b/Source/Utils/Parser/IntervalsParser.H index 7c4e861ca15..3e0533aaf77 100644 --- a/Source/Utils/Parser/IntervalsParser.H +++ b/Source/Utils/Parser/IntervalsParser.H @@ -42,7 +42,7 @@ namespace utils::parser * * @param[in] n the input integer */ - bool contains (int n) const; + [[nodiscard]] bool contains (int n) const; /** * \brief A method that returns the smallest integer strictly greater than n such that @@ -50,7 +50,7 @@ namespace utils::parser * * @param[in] n the input integer */ - int nextContains (int n) const; + [[nodiscard]] int nextContains (int n) const; /** * \brief A method that returns the greatest integer strictly smaller than n such that @@ -58,31 +58,31 @@ namespace utils::parser * * @param[in] n the input integer */ - int previousContains (int n) const; + [[nodiscard]] int previousContains (int n) const; /** * \brief A method that returns the slice period. * */ - int getPeriod () const; + [[nodiscard]] int getPeriod () const; /** * \brief A method that returns the slice start. * */ - int getStart () const; + [[nodiscard]] int getStart () const; /** * \brief A method that returns the slice stop. * */ - int getStop () const; + [[nodiscard]] int getStop () const; /** * @brief A method that returns the number of integers contained by the slice. * */ - int numContained() const; + [[nodiscard]] int numContained() const; private: bool m_isBTD = false; @@ -122,7 +122,7 @@ namespace utils::parser * * @param[in] n the input integer */ - bool contains (int n) const; + [[nodiscard]] bool contains (int n) const; /** * \brief A method that returns the smallest integer strictly greater than n such that @@ -130,7 +130,7 @@ namespace utils::parser * * @param[in] n the input integer */ - int nextContains (int n) const; + [[nodiscard]] int nextContains (int n) const; /** * \brief A method that returns the greatest integer strictly smaller than n such that @@ -138,7 +138,7 @@ namespace utils::parser * * @param[in] n the input integer */ - int previousContains (int n) const; + [[nodiscard]] int previousContains (int n) const; /** * \brief A method that returns the greatest integer smaller than or equal to n such that @@ -146,7 +146,7 @@ namespace utils::parser * * @param[in] n the input integer */ - int previousContainsInclusive (int n) const; + [[nodiscard]] int previousContainsInclusive (int n) const; /** * \brief A method the local period (in timesteps) of the IntervalsParser at timestep n. @@ -154,13 +154,13 @@ namespace utils::parser * * @param[in] n the input integer */ - int localPeriod (int n) const; + [[nodiscard]] int localPeriod (int n) const; /** * \brief A method that returns true if any of the slices contained by the IntervalsParser * has a strictly positive period. */ - bool isActivated () const; + [[nodiscard]] bool isActivated () const; private: std::vector m_slices; @@ -193,26 +193,26 @@ namespace utils::parser /** * @brief Return the total number of unique labframe snapshots */ - int NumSnapshots () const; + [[nodiscard]] int NumSnapshots () const; /** * @brief Return the iteration number stored at index i_buffer * * @param i_buffer buffer or iteration index, between 0 and NumSnapshots */ - int GetBTDIteration(int i_buffer) const; + [[nodiscard]] int GetBTDIteration(int i_buffer) const; /** * @brief Return the final BTD iteration * */ - int GetFinalIteration() const; + [[nodiscard]] int GetFinalIteration() const; /** * \brief A method that returns true if any of the slices contained by the IntervalsParser * has a strictly positive period. */ - bool isActivated () const; + [[nodiscard]] bool isActivated () const; private: std::vector m_btd_iterations; diff --git a/Source/Utils/Parser/ParserUtils.cpp b/Source/Utils/Parser/ParserUtils.cpp index 650d46c9e84..aeacedc5b4f 100644 --- a/Source/Utils/Parser/ParserUtils.cpp +++ b/Source/Utils/Parser/ParserUtils.cpp @@ -37,7 +37,7 @@ namespace { template< typename int_type > AMREX_FORCE_INLINE int_type safeCastTo(const amrex::Real x, const std::string& real_name) { - int_type result = int_type(0); + auto result = int_type(0); bool error_detected = false; std::string assert_msg; // (2.0*(numeric_limits::max()/2+1)) converts numeric_limits::max()+1 to a real ensuring accuracy to all digits diff --git a/Source/Utils/WarpXTagging.cpp b/Source/Utils/WarpXTagging.cpp index 807c92b2f0b..8d87dfccc9b 100644 --- a/Source/Utils/WarpXTagging.cpp +++ b/Source/Utils/WarpXTagging.cpp @@ -47,7 +47,7 @@ WarpX::ErrorEst (int lev, TagBoxArray& tags, Real /*time*/, int /*ngrow*/) const RealVect pos {AMREX_D_DECL((i+0.5_rt)*dx[0]+problo[0], (j+0.5_rt)*dx[1]+problo[1], (k+0.5_rt)*dx[2]+problo[2])}; - bool tag_val = 0; + bool tag_val = false; if (ref_parser) { #if defined (WARPX_DIM_3D) tag_val = (ref_parser(pos[0], pos[1], pos[2]) == 1); diff --git a/Source/WarpX.H b/Source/WarpX.H index 6a4d76f3c01..b2bea527069 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -97,6 +97,7 @@ public: */ static void Finalize(); + /** Destructor */ ~WarpX () override; /** Copy constructor */ @@ -112,7 +113,7 @@ public: static std::string Version (); //!< Version of WarpX executable static std::string PicsarVersion (); //!< Version of PICSAR dependency - int Verbose () const { return verbose; } + [[nodiscard]] int Verbose () const { return verbose; } void InitData (); @@ -136,7 +137,7 @@ public: * If an authors' string is specified in the inputfile, this method returns that string. * Otherwise, it returns an empty string. */ - std::string GetAuthors () const { return m_authors; } + [[nodiscard]] std::string GetAuthors () const { return m_authors; } //! Initial electric field on the grid static amrex::Vector E_external_grid; @@ -446,7 +447,7 @@ public: static std::map multifab_map; static std::map imultifab_map; - std::array + [[nodiscard]] std::array get_array_Bfield_aux (const int lev) const { return { Bfield_aux[lev][0].get(), @@ -454,7 +455,8 @@ public: Bfield_aux[lev][2].get() }; } - std::array + + [[nodiscard]] std::array get_array_Efield_aux (const int lev) const { return { Efield_aux[lev][0].get(), @@ -463,28 +465,28 @@ public: }; } - amrex::MultiFab * get_pointer_Efield_aux (int lev, int direction) const { return Efield_aux[lev][direction].get(); } - amrex::MultiFab * get_pointer_Bfield_aux (int lev, int direction) const { return Bfield_aux[lev][direction].get(); } - - amrex::MultiFab * get_pointer_Efield_fp (int lev, int direction) const { return Efield_fp[lev][direction].get(); } - amrex::MultiFab * get_pointer_Bfield_fp (int lev, int direction) const { return Bfield_fp[lev][direction].get(); } - amrex::MultiFab * get_pointer_current_fp (int lev, int direction) const { return current_fp[lev][direction].get(); } - amrex::MultiFab * get_pointer_current_fp_nodal (int lev, int direction) const { return current_fp_nodal[lev][direction].get(); } - amrex::MultiFab * get_pointer_rho_fp (int lev) const { return rho_fp[lev].get(); } - amrex::MultiFab * get_pointer_F_fp (int lev) const { return F_fp[lev].get(); } - amrex::MultiFab * get_pointer_G_fp (int lev) const { return G_fp[lev].get(); } - amrex::MultiFab * get_pointer_phi_fp (int lev) const { return phi_fp[lev].get(); } - amrex::MultiFab * get_pointer_vector_potential_fp (int lev, int direction) const { return vector_potential_fp_nodal[lev][direction].get(); } - - amrex::MultiFab * get_pointer_Efield_cp (int lev, int direction) const { return Efield_cp[lev][direction].get(); } - amrex::MultiFab * get_pointer_Bfield_cp (int lev, int direction) const { return Bfield_cp[lev][direction].get(); } - amrex::MultiFab * get_pointer_current_cp (int lev, int direction) const { return current_cp[lev][direction].get(); } - amrex::MultiFab * get_pointer_rho_cp (int lev) const { return rho_cp[lev].get(); } - amrex::MultiFab * get_pointer_F_cp (int lev) const { return F_cp[lev].get(); } - amrex::MultiFab * get_pointer_G_cp (int lev) const { return G_cp[lev].get(); } - - amrex::MultiFab * get_pointer_edge_lengths (int lev, int direction) const { return m_edge_lengths[lev][direction].get(); } - amrex::MultiFab * get_pointer_face_areas (int lev, int direction) const { return m_face_areas[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_Efield_aux (int lev, int direction) const { return Efield_aux[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_Bfield_aux (int lev, int direction) const { return Bfield_aux[lev][direction].get(); } + + [[nodiscard]] amrex::MultiFab * get_pointer_Efield_fp (int lev, int direction) const { return Efield_fp[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_Bfield_fp (int lev, int direction) const { return Bfield_fp[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_current_fp (int lev, int direction) const { return current_fp[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_current_fp_nodal (int lev, int direction) const { return current_fp_nodal[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_rho_fp (int lev) const { return rho_fp[lev].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_F_fp (int lev) const { return F_fp[lev].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_G_fp (int lev) const { return G_fp[lev].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_phi_fp (int lev) const { return phi_fp[lev].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_vector_potential_fp (int lev, int direction) const { return vector_potential_fp_nodal[lev][direction].get(); } + + [[nodiscard]] amrex::MultiFab * get_pointer_Efield_cp (int lev, int direction) const { return Efield_cp[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_Bfield_cp (int lev, int direction) const { return Bfield_cp[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_current_cp (int lev, int direction) const { return current_cp[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_rho_cp (int lev) const { return rho_cp[lev].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_F_cp (int lev) const { return F_cp[lev].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_G_cp (int lev) const { return G_cp[lev].get(); } + + [[nodiscard]] amrex::MultiFab * get_pointer_edge_lengths (int lev, int direction) const { return m_edge_lengths[lev][direction].get(); } + [[nodiscard]] amrex::MultiFab * get_pointer_face_areas (int lev, int direction) const { return m_face_areas[lev][direction].get(); } const amrex::MultiFab& getEfield (int lev, int direction) {return *Efield_aux[lev][direction];} const amrex::MultiFab& getBfield (int lev, int direction) {return *Bfield_aux[lev][direction];} @@ -509,15 +511,15 @@ public: const amrex::MultiFab& getEfield_avg_cp (int lev, int direction) {return *Efield_avg_cp[lev][direction];} const amrex::MultiFab& getBfield_avg_cp (int lev, int direction) {return *Bfield_avg_cp[lev][direction];} - bool DoPML () const {return do_pml;} - bool DoFluidSpecies () const {return do_fluid_species;} + [[nodiscard]] bool DoPML () const {return do_pml;} + [[nodiscard]] bool DoFluidSpecies () const {return do_fluid_species;} #if (defined WARPX_DIM_RZ) && (defined WARPX_USE_PSATD) const PML_RZ* getPMLRZ() {return pml_rz[0].get();} #endif /** get low-high-low-high-... vector for each direction indicating if mother grid PMLs are enabled */ - std::vector getPMLdirections() const; + [[nodiscard]] std::vector getPMLdirections() const; static amrex::LayoutData* getCosts (int lev); @@ -652,7 +654,7 @@ public: /** \brief returns the load balance interval */ - utils::parser::IntervalsParser get_load_balance_intervals () const + [[nodiscard]] utils::parser::IntervalsParser get_load_balance_intervals () const { return load_balance_intervals; } @@ -819,25 +821,25 @@ public: const amrex::Vector>& charge_cp, const amrex::Vector>& charge_buffer); - amrex::Vector getnsubsteps () const {return nsubsteps;} - int getnsubsteps (int lev) const {return nsubsteps[lev];} - amrex::Vector getistep () const {return istep;} - int getistep (int lev) const {return istep[lev];} + [[nodiscard]] amrex::Vector getnsubsteps () const {return nsubsteps;} + [[nodiscard]] int getnsubsteps (int lev) const {return nsubsteps[lev];} + [[nodiscard]] amrex::Vector getistep () const {return istep;} + [[nodiscard]] int getistep (int lev) const {return istep[lev];} void setistep (int lev, int ii) {istep[lev] = ii;} - amrex::Vector gett_old () const {return t_old;} - amrex::Real gett_old (int lev) const {return t_old[lev];} - amrex::Vector gett_new () const {return t_new;} - amrex::Real gett_new (int lev) const {return t_new[lev];} + [[nodiscard]] amrex::Vector gett_old () const {return t_old;} + [[nodiscard]] amrex::Real gett_old (int lev) const {return t_old[lev];} + [[nodiscard]] amrex::Vector gett_new () const {return t_new;} + [[nodiscard]] amrex::Real gett_new (int lev) const {return t_new[lev];} void sett_new (int lev, amrex::Real time) {t_new[lev] = time;} - amrex::Vector getdt () const {return dt;} - amrex::Real getdt (int lev) const {return dt.at(lev);} - int getdo_moving_window() const {return do_moving_window;} - amrex::Real getmoving_window_x() const {return moving_window_x;} - bool getis_synchronized() const {return is_synchronized;} + [[nodiscard]] amrex::Vector getdt () const {return dt;} + [[nodiscard]] amrex::Real getdt (int lev) const {return dt.at(lev);} + [[nodiscard]] int getdo_moving_window() const {return do_moving_window;} + [[nodiscard]] amrex::Real getmoving_window_x() const {return moving_window_x;} + [[nodiscard]] bool getis_synchronized() const {return is_synchronized;} - int maxStep () const {return max_step;} + [[nodiscard]] int maxStep () const {return max_step;} void updateMaxStep (const int new_max_step) {max_step = new_max_step;} - amrex::Real stopTime () const {return stop_time;} + [[nodiscard]] amrex::Real stopTime () const {return stop_time;} void updateStopTime (const amrex::Real new_stop_time) {stop_time = new_stop_time;} void AverageAndPackFields( amrex::Vector& varnames, @@ -911,12 +913,12 @@ public: void ComputeDivE(amrex::MultiFab& divE, int lev); - amrex::IntVect getngEB() const { return guard_cells.ng_alloc_EB; } - amrex::IntVect getngF() const { return guard_cells.ng_alloc_F; } - amrex::IntVect getngUpdateAux() const { return guard_cells.ng_UpdateAux; } - amrex::IntVect get_ng_depos_J() const {return guard_cells.ng_depos_J;} - amrex::IntVect get_ng_depos_rho() const {return guard_cells.ng_depos_rho;} - amrex::IntVect get_ng_fieldgather () const {return guard_cells.ng_FieldGather;} + [[nodiscard]] amrex::IntVect getngEB() const { return guard_cells.ng_alloc_EB; } + [[nodiscard]] amrex::IntVect getngF() const { return guard_cells.ng_alloc_F; } + [[nodiscard]] amrex::IntVect getngUpdateAux() const { return guard_cells.ng_UpdateAux; } + [[nodiscard]] amrex::IntVect get_ng_depos_J() const {return guard_cells.ng_depos_J;} + [[nodiscard]] amrex::IntVect get_ng_depos_rho() const {return guard_cells.ng_depos_rho;} + [[nodiscard]] amrex::IntVect get_ng_fieldgather () const {return guard_cells.ng_FieldGather;} /** Coarsest-level Domain Decomposition * @@ -925,7 +927,7 @@ public: * * @return the number of MPI processes per dimension if specified, otherwise a 0-vector */ - amrex::IntVect get_numprocs() const {return numprocs;} + [[nodiscard]] amrex::IntVect get_numprocs() const {return numprocs;} ElectrostaticSolver::PoissonBoundaryHandler m_poisson_boundary_handler; void ComputeSpaceChargeField (bool reset_fields); @@ -1312,7 +1314,7 @@ private: void AllocLevelData (int lev, const amrex::BoxArray& new_grids, const amrex::DistributionMapping& new_dmap); - amrex::DistributionMapping + [[nodiscard]] amrex::DistributionMapping GetRestartDMap (const std::string& chkfile, const amrex::BoxArray& ba, int lev) const; void InitFromCheckpoint (); @@ -1343,10 +1345,12 @@ private: void BuildBufferMasks (); - const amrex::iMultiFab* getCurrentBufferMasks (int lev) const { + [[nodiscard]] const amrex::iMultiFab* getCurrentBufferMasks (int lev) const { return current_buffer_masks[lev].get(); } - const amrex::iMultiFab* getGatherBufferMasks (int lev) const { + + [[nodiscard]] const amrex::iMultiFab* getGatherBufferMasks (int lev) const + { return gather_buffer_masks[lev].get(); } @@ -1419,7 +1423,7 @@ private: std::unique_ptr multi_diags; // Fluid container - bool do_fluid_species = 0; + bool do_fluid_species = false; std::unique_ptr myfl; // @@ -1597,7 +1601,7 @@ private: // Other runtime parameters int verbose = 1; - bool use_hybrid_QED = 0; + bool use_hybrid_QED = false; int max_step = std::numeric_limits::max(); amrex::Real stop_time = std::numeric_limits::max(); @@ -1665,7 +1669,9 @@ private: // Factory for field data amrex::Vector > > m_field_factory; - amrex::FabFactory const& fieldFactory (int lev) const noexcept { + [[nodiscard]] + amrex::FabFactory const& fieldFactory (int lev) const noexcept + { return *m_field_factory[lev]; } diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 240c7fbb4e0..393e3f3110e 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -201,7 +201,7 @@ int WarpX::self_fields_verbosity = 2; bool WarpX::do_subcycling = false; bool WarpX::do_multi_J = false; int WarpX::do_multi_J_n_depositions; -bool WarpX::safe_guard_cells = 0; +bool WarpX::safe_guard_cells = false; std::map WarpX::multifab_map; std::map WarpX::imultifab_map; @@ -836,7 +836,7 @@ WarpX::ReadParameters () pp_warpx.query("do_single_precision_comms", do_single_precision_comms); #ifdef AMREX_USE_FLOAT if (do_single_precision_comms) { - do_single_precision_comms = 0; + do_single_precision_comms = false; ablastr::warn_manager::WMRecordWarning( "comms", "Overwrote warpx.do_single_precision_comms to be 0, since WarpX was built in single precision.", @@ -1049,9 +1049,7 @@ WarpX::ReadParameters () const ParmParse pp_fluids("fluids"); std::vector fluid_species_names = {}; pp_fluids.queryarr("species_names", fluid_species_names); - - if (!fluid_species_names.empty()) do_fluid_species = 1; - + do_fluid_species = !fluid_species_names.empty(); if (do_fluid_species) { WARPX_ALWAYS_ASSERT_WITH_MESSAGE(max_level <= 1, "Fluid species cannot currently be used with mesh refinement."); diff --git a/Source/ablastr/parallelization/KernelTimer.H b/Source/ablastr/parallelization/KernelTimer.H index 0329948da75..9e94014822b 100644 --- a/Source/ablastr/parallelization/KernelTimer.H +++ b/Source/ablastr/parallelization/KernelTimer.H @@ -57,19 +57,26 @@ public: } //! Destructor. - AMREX_GPU_DEVICE - ~KernelTimer () { #if (defined AMREX_USE_GPU) - if (m_do_timing && m_cost) { # if defined(AMREX_USE_CUDA) || defined(AMREX_USE_HIP) - m_wt = clock64() - m_wt; - amrex::Gpu::Atomic::Add( m_cost, amrex::Real(m_wt)); + AMREX_GPU_DEVICE + ~KernelTimer () + { + if (m_do_timing && m_cost) { + m_wt = clock64() - m_wt; + amrex::Gpu::Atomic::Add( m_cost, amrex::Real(m_wt)); + } + } # elif defined(AMREX_USE_DPCPP) - // To be updated + // To be updated + AMREX_GPU_DEVICE + ~KernelTimer () = default; # endif - } + +#else +~KernelTimer () = default; #endif //AMREX_USE_GPU - } + KernelTimer ( KernelTimer const &) = default; KernelTimer& operator= ( KernelTimer const & ) = default; diff --git a/Source/ablastr/utils/SignalHandling.H b/Source/ablastr/utils/SignalHandling.H index 29ea2db9f99..d6d10b61f31 100644 --- a/Source/ablastr/utils/SignalHandling.H +++ b/Source/ablastr/utils/SignalHandling.H @@ -63,6 +63,9 @@ public: //! Check whether a given action has been requested, and reset the associated flag static bool TestAndResetActionRequestFlag (int action_to_test); + // Don't allow clients to incorrectly try to construct and use an instance of this type + SignalHandling () = delete; + private: //! Is any signal handling action configured in signal_conf_requests ? static bool m_any_signal_action_active; @@ -81,9 +84,6 @@ private: //! Boolean flags transmitted between CheckSignals() and //! HandleSignals() to indicate actions requested by signals static bool signal_actions_requested[SIGNAL_REQUESTS_SIZE]; - - // Don't allow clients to incorrectly try to construct and use an instance of this type - SignalHandling () = delete; }; } // namespace ablastr::utils diff --git a/Source/ablastr/utils/msg_logger/MsgLogger.H b/Source/ablastr/utils/msg_logger/MsgLogger.H index 2296eceac25..01a61be5c80 100644 --- a/Source/ablastr/utils/msg_logger/MsgLogger.H +++ b/Source/ablastr/utils/msg_logger/MsgLogger.H @@ -65,7 +65,7 @@ namespace ablastr::utils::msg_logger * * @return a byte vector */ - std::vector serialize() const; + [[nodiscard]] std::vector serialize() const; /** * \brief This function generates a Msg struct from a byte vector @@ -101,7 +101,7 @@ namespace ablastr::utils::msg_logger * * @return a byte vector */ - std::vector serialize() const; + [[nodiscard]] std::vector serialize() const; /** * \brief This function generates a MsgWithCounter struct from a byte vector @@ -140,7 +140,7 @@ namespace ablastr::utils::msg_logger * * @return a byte vector */ - std::vector serialize() const; + [[nodiscard]] std::vector serialize() const; /** * \brief This function generates a MsgWithCounterAndRanks struct from a byte vector @@ -203,7 +203,7 @@ namespace ablastr::utils::msg_logger * * @return a vector of the recorded messages */ - std::vector get_msgs() const; + [[nodiscard]] std::vector get_msgs() const; /** * \brief This function returns a vector containing the recorded messages @@ -211,7 +211,8 @@ namespace ablastr::utils::msg_logger * * @return a vector of the recorded messages with counters */ - std::vector get_msgs_with_counter() const; + [[nodiscard]] std::vector + get_msgs_with_counter() const; /** * \brief This collective function generates a vector containing the messages @@ -220,7 +221,7 @@ namespace ablastr::utils::msg_logger * * @return a vector of messages with counters and ranks if I/O rank, an empty vector otherwise */ - std::vector + [[nodiscard]] std::vector collective_gather_msgs_with_counter_and_ranks() const; private: @@ -231,7 +232,7 @@ namespace ablastr::utils::msg_logger * * @return a vector of messages with counters and ranks */ - std::vector + [[nodiscard]] std::vector one_rank_gather_msgs_with_counter_and_ranks() const; #ifdef AMREX_USE_MPI @@ -243,8 +244,8 @@ namespace ablastr::utils::msg_logger * @param[in] how_many_msgs the number of messages that the current rank has * @return a pair containing the ID of the "gather rank" and its number of messages */ - std::pair find_gather_rank_and_its_msgs( - int how_many_msgs) const; + [[nodiscard]] std::pair + find_gather_rank_and_its_msgs(int how_many_msgs) const; /** * \brief This function uses data gathered on the "gather rank" to generate @@ -256,7 +257,7 @@ namespace ablastr::utils::msg_logger * @param[in] gather_rank the ID of the "gather rank" * @return if gather_rank==m_rank a vector of messages with global counters and emitting rank lists, dummy data otherwise */ - std::vector + [[nodiscard]] std::vector compute_msgs_with_counter_and_ranks( const std::map& my_msg_map, const std::vector& all_data, diff --git a/Source/ablastr/utils/timer/Timer.H b/Source/ablastr/utils/timer/Timer.H index e7787f36b75..efbb7d6a2bb 100644 --- a/Source/ablastr/utils/timer/Timer.H +++ b/Source/ablastr/utils/timer/Timer.H @@ -23,7 +23,7 @@ namespace ablastr::utils::timer /** * \brief The constructor. */ - Timer(); + Timer() = default; /** diff --git a/Source/ablastr/utils/timer/Timer.cpp b/Source/ablastr/utils/timer/Timer.cpp index 90f6fcb99c5..5682a07c290 100644 --- a/Source/ablastr/utils/timer/Timer.cpp +++ b/Source/ablastr/utils/timer/Timer.cpp @@ -11,8 +11,6 @@ using namespace ablastr::utils::timer; -Timer::Timer(){} - void Timer::record_start_time() noexcept { diff --git a/Source/ablastr/warn_manager/WarnManager.H b/Source/ablastr/warn_manager/WarnManager.H index 34a8102620e..737fb8fba73 100644 --- a/Source/ablastr/warn_manager/WarnManager.H +++ b/Source/ablastr/warn_manager/WarnManager.H @@ -95,7 +95,7 @@ namespace ablastr::warn_manager * @param[in] when a string to mark when the warnings are printed out (it appears in the warning list) * @return a string containing the "local" warning list */ - std::string PrintLocalWarnings( + [[nodiscard]] std::string PrintLocalWarnings( const std::string& when) const; /** @@ -105,7 +105,7 @@ namespace ablastr::warn_manager * @param[in] when a string to mark when the warnings are printed out (it appears in the warning list) * @return a string containing the "global" warning list */ - std::string PrintGlobalWarnings( + [[nodiscard]] std::string PrintGlobalWarnings( const std::string& when) const; /** @@ -120,7 +120,7 @@ namespace ablastr::warn_manager * * @return the m_always_warn_immediately flag */ - bool GetAlwaysWarnImmediatelyFlag() const; + [[nodiscard]] bool GetAlwaysWarnImmediatelyFlag() const; /** * \brief Setter for the m_abort_on_warning_threshold flag @@ -135,7 +135,7 @@ namespace ablastr::warn_manager * * @return the m_abort_on_warning_threshold flag */ - std::optional GetAbortThreshold() const; + [[nodiscard]] std::optional GetAbortThreshold() const; /** * \brief This function reads warning messages from the inputfile. It is intended for @@ -163,7 +163,7 @@ namespace ablastr::warn_manager * @param[in] msg_with_counter a MessageWithCounter * @return a string containing the warning message */ - std::string PrintWarnMsg( + [[nodiscard]] std::string PrintWarnMsg( const ablastr::utils::msg_logger::MsgWithCounter& msg_with_counter) const; /** @@ -174,7 +174,7 @@ namespace ablastr::warn_manager * @param[in] msg_with_counter_and_ranks a MsgWithCounterAndRanks * @return a string containing the warning message */ - std::string PrintWarnMsg( + [[nodiscard]] std::string PrintWarnMsg( const ablastr::utils::msg_logger::MsgWithCounterAndRanks& msg_with_counter_and_ranks) const; /** From 5cf442f12ea7847f7d955c7104efad94cd27559e Mon Sep 17 00:00:00 2001 From: Luca Fedeli Date: Mon, 27 Nov 2023 21:54:05 +0100 Subject: [PATCH 104/110] Move AnyFFT.H and FFT wrappers into ablastr (#4005) * move AnyFFT.H into ablastr * place anyfft under ablastr/math/fft * add logic to enable fft in ablastr via flag ABLASTR_USE_FFT * ABLASTR FFT: Build System Update Co-authored-by: Axel Huebl --- CMakeLists.txt | 12 +- .../FieldSolver/SpectralSolver/CMakeLists.txt | 8 -- .../FieldSolver/SpectralSolver/Make.package | 7 -- .../SpectralSolver/SpectralFieldData.H | 5 +- .../SpectralSolver/SpectralFieldData.cpp | 24 ++-- .../SpectralSolver/SpectralFieldDataRZ.H | 5 +- .../SpectralSolver/SpectralFieldDataRZ.cpp | 26 ++-- Source/Make.WarpX | 2 +- Source/Utils/CMakeLists.txt | 1 - Source/Utils/Make.package | 1 - Source/Utils/WarpX_Complex.H | 4 +- Source/Utils/WarpXrocfftUtil.H | 24 ---- Source/Utils/WarpXrocfftUtil.cpp | 34 ----- Source/ablastr/CMakeLists.txt | 3 +- Source/ablastr/Make.package | 1 + Source/ablastr/math/CMakeLists.txt | 1 + Source/ablastr/math/Make.package | 3 + .../math/fft}/AnyFFT.H | 119 ++++++++++-------- Source/ablastr/math/fft/CMakeLists.txt | 14 +++ Source/ablastr/math/fft/Make.package | 13 ++ .../math/fft}/WrapCuFFT.cpp | 29 ++--- .../math/fft}/WrapFFTW.cpp | 19 +-- Source/ablastr/math/fft/WrapNoFFT.cpp | 17 +++ .../math/fft}/WrapRocFFT.cpp | 32 +++-- Source/main.cpp | 12 +- cmake/dependencies/FFT.cmake | 4 +- 26 files changed, 222 insertions(+), 198 deletions(-) delete mode 100644 Source/Utils/WarpXrocfftUtil.H delete mode 100644 Source/Utils/WarpXrocfftUtil.cpp create mode 100644 Source/ablastr/math/CMakeLists.txt create mode 100644 Source/ablastr/math/Make.package rename Source/{FieldSolver/SpectralSolver => ablastr/math/fft}/AnyFFT.H (58%) create mode 100644 Source/ablastr/math/fft/CMakeLists.txt create mode 100644 Source/ablastr/math/fft/Make.package rename Source/{FieldSolver/SpectralSolver => ablastr/math/fft}/WrapCuFFT.cpp (84%) rename Source/{FieldSolver/SpectralSolver => ablastr/math/fft}/WrapFFTW.cpp (91%) create mode 100644 Source/ablastr/math/fft/WrapNoFFT.cpp rename Source/{FieldSolver/SpectralSolver => ablastr/math/fft}/WrapRocFFT.cpp (89%) diff --git a/CMakeLists.txt b/CMakeLists.txt index b2744d9c768..4b3eae1663c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -141,10 +141,14 @@ option(ABLASTR_POSITION_INDEPENDENT_CODE "Build ABLASTR with position independent code" ON) mark_as_advanced(ABLASTR_POSITION_INDEPENDENT_CODE) +option(ABLASTR_FFT "compile AnyFFT wrappers" ${WarpX_PSATD}) +if(WarpX_PSATD) + set(ABLASTR_FFT ON CACHE STRING "compile AnyFFT wrappers" FORCE) +endif() + # this defined the variable BUILD_TESTING which is ON by default #include(CTest) - # Dependencies ################################################################ # @@ -404,7 +408,7 @@ foreach(D IN LISTS WarpX_DIMS) target_link_libraries(ablastr_${SD} PUBLIC WarpX::thirdparty::amrex_${D}d) endif() - if(WarpX_PSATD) + if(ABLASTR_FFT) target_link_libraries(ablastr_${SD} PUBLIC WarpX::thirdparty::FFT) if(D STREQUAL "RZ") target_link_libraries(ablastr_${SD} PUBLIC blaspp) @@ -501,6 +505,10 @@ foreach(D IN LISTS WarpX_DIMS) if(WarpX_PSATD) target_compile_definitions(ablastr_${SD} PUBLIC WARPX_USE_PSATD) endif() + if(ABLASTR_FFT) + # We need to enable FFT support in ABLASTR for PSATD solver + target_compile_definitions(ablastr_${SD} PUBLIC ABLASTR_USE_FFT) + endif() if(WarpX_PYTHON AND pyWarpX_VERSION_INFO) # for module __version__ diff --git a/Source/FieldSolver/SpectralSolver/CMakeLists.txt b/Source/FieldSolver/SpectralSolver/CMakeLists.txt index 42ebe2be947..d1fea38672b 100644 --- a/Source/FieldSolver/SpectralSolver/CMakeLists.txt +++ b/Source/FieldSolver/SpectralSolver/CMakeLists.txt @@ -7,14 +7,6 @@ foreach(D IN LISTS WarpX_DIMS) SpectralSolver.cpp ) - if(WarpX_COMPUTE STREQUAL CUDA) - target_sources(ablastr_${SD} PRIVATE WrapCuFFT.cpp) - elseif(WarpX_COMPUTE STREQUAL HIP) - target_sources(ablastr_${SD} PRIVATE WrapRocFFT.cpp) - else() - target_sources(ablastr_${SD} PRIVATE WrapFFTW.cpp) - endif() - if(D STREQUAL "RZ") target_sources(lib_${SD} PRIVATE diff --git a/Source/FieldSolver/SpectralSolver/Make.package b/Source/FieldSolver/SpectralSolver/Make.package index 8be3a6812ea..a5143abc906 100644 --- a/Source/FieldSolver/SpectralSolver/Make.package +++ b/Source/FieldSolver/SpectralSolver/Make.package @@ -1,13 +1,6 @@ CEXE_sources += SpectralSolver.cpp CEXE_sources += SpectralFieldData.cpp CEXE_sources += SpectralKSpace.cpp -ifeq ($(USE_CUDA),TRUE) - CEXE_sources += WrapCuFFT.cpp -else ifeq ($(USE_HIP),TRUE) - CEXE_sources += WrapRocFFT.cpp -else - CEXE_sources += WrapFFTW.cpp -endif ifeq ($(USE_RZ),TRUE) CEXE_sources += SpectralSolverRZ.cpp diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldData.H b/Source/FieldSolver/SpectralSolver/SpectralFieldData.H index a4ef6a39189..8ced43ced94 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldData.H +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldData.H @@ -10,10 +10,11 @@ #include "SpectralFieldData_fwd.H" -#include "AnyFFT.H" #include "SpectralKSpace.H" #include "Utils/WarpX_Complex.H" +#include + #include #include #include @@ -172,7 +173,7 @@ class SpectralFieldData // right before/after the Fourier transform SpectralField tmpSpectralField; // contains Complexs amrex::MultiFab tmpRealField; // contains Reals - AnyFFT::FFTplans forward_plan, backward_plan; + ablastr::math::anyfft::FFTplans forward_plan, backward_plan; // Correcting "shift" factors when performing FFT from/to // a cell-centered grid in real space, instead of a nodal grid SpectralShiftFactor xshift_FFTfromCell, xshift_FFTtoCell, diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp b/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp index c4d9a0dab35..f256c4d2148 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldData.cpp @@ -163,8 +163,8 @@ SpectralFieldData::SpectralFieldData( const int lev, #endif // Allocate and initialize the FFT plans - forward_plan = AnyFFT::FFTplans(spectralspace_ba, dm); - backward_plan = AnyFFT::FFTplans(spectralspace_ba, dm); + forward_plan = ablastr::math::anyfft::FFTplans(spectralspace_ba, dm); + backward_plan = ablastr::math::anyfft::FFTplans(spectralspace_ba, dm); // Loop over boxes and allocate the corresponding plan // for each box owned by the local MPI proc for ( MFIter mfi(spectralspace_ba, dm); mfi.isValid(); ++mfi ){ @@ -179,15 +179,15 @@ SpectralFieldData::SpectralFieldData( const int lev, // the FFT plan, the valid dimensions are those of the real-space box. const IntVect fft_size = realspace_ba[mfi].length(); - forward_plan[mfi] = AnyFFT::CreatePlan( + forward_plan[mfi] = ablastr::math::anyfft::CreatePlan( fft_size, tmpRealField[mfi].dataPtr(), - reinterpret_cast( tmpSpectralField[mfi].dataPtr()), - AnyFFT::direction::R2C, AMREX_SPACEDIM); + reinterpret_cast( tmpSpectralField[mfi].dataPtr()), + ablastr::math::anyfft::direction::R2C, AMREX_SPACEDIM); - backward_plan[mfi] = AnyFFT::CreatePlan( + backward_plan[mfi] = ablastr::math::anyfft::CreatePlan( fft_size, tmpRealField[mfi].dataPtr(), - reinterpret_cast( tmpSpectralField[mfi].dataPtr()), - AnyFFT::direction::C2R, AMREX_SPACEDIM); + reinterpret_cast( tmpSpectralField[mfi].dataPtr()), + ablastr::math::anyfft::direction::C2R, AMREX_SPACEDIM); if (do_costs) { @@ -203,8 +203,8 @@ SpectralFieldData::~SpectralFieldData() { if (!tmpRealField.empty()){ for ( MFIter mfi(tmpRealField); mfi.isValid(); ++mfi ){ - AnyFFT::DestroyPlan(forward_plan[mfi]); - AnyFFT::DestroyPlan(backward_plan[mfi]); + ablastr::math::anyfft::DestroyPlan(forward_plan[mfi]); + ablastr::math::anyfft::DestroyPlan(backward_plan[mfi]); } } } @@ -266,7 +266,7 @@ SpectralFieldData::ForwardTransform (const int lev, } // Perform Fourier transform from `tmpRealField` to `tmpSpectralField` - AnyFFT::Execute(forward_plan[mfi]); + ablastr::math::anyfft::Execute(forward_plan[mfi]); // Copy the spectral-space field `tmpSpectralField` to the appropriate // index of the FabArray `fields` (specified by `field_index`) @@ -406,7 +406,7 @@ SpectralFieldData::BackwardTransform (const int lev, } // Perform Fourier transform from `tmpSpectralField` to `tmpRealField` - AnyFFT::Execute(backward_plan[mfi]); + ablastr::math::anyfft::Execute(backward_plan[mfi]); // Copy the temporary field tmpRealField to the real-space field mf and // normalize, dividing by N, since (FFT + inverse FFT) results in a factor N diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H index a8282fffbed..9fa7a65be3b 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.H @@ -11,7 +11,8 @@ #include "SpectralFieldData.H" #include "SpectralHankelTransform/SpectralHankelTransformer.H" #include "SpectralKSpaceRZ.H" -#include "FieldSolver/SpectralSolver/AnyFFT.H" + +#include #include @@ -26,7 +27,7 @@ class SpectralFieldDataRZ // Define the FFTplans type, which holds one fft plan per box // (plans are only initialized for the boxes that are owned by // the local MPI rank) - using FFTplans = amrex::LayoutData; + using FFTplans = amrex::LayoutData; // Similarly, define the Hankel transformers and filter for each box. using MultiSpectralHankelTransformer = amrex::LayoutData; diff --git a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp index cc0e64fd1de..236fe3fbb29 100644 --- a/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp +++ b/Source/FieldSolver/SpectralSolver/SpectralFieldDataRZ.cpp @@ -7,9 +7,9 @@ #include "SpectralFieldDataRZ.H" #include "Utils/WarpXUtil.H" -#include "FieldSolver/SpectralSolver/AnyFFT.H" #include "WarpX.H" +#include #include #include @@ -174,8 +174,10 @@ SpectralFieldDataRZ::SpectralFieldDataRZ (const int lev, dims, 2, // int howmany_rank, howmany_dims, - reinterpret_cast(tempHTransformed[mfi].dataPtr()), // complex *in - reinterpret_cast(tmpSpectralField[mfi].dataPtr()), // complex *out + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tempHTransformed[mfi].dataPtr()), // complex *in + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tmpSpectralField[mfi].dataPtr()), // complex *out FFTW_FORWARD, // int sign FFTW_ESTIMATE); // unsigned flags backward_plan[mfi] = @@ -188,8 +190,10 @@ SpectralFieldDataRZ::SpectralFieldDataRZ (const int lev, dims, 2, // int howmany_rank, howmany_dims, - reinterpret_cast(tmpSpectralField[mfi].dataPtr()), // complex *in - reinterpret_cast(tempHTransformed[mfi].dataPtr()), // complex *out + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tmpSpectralField[mfi].dataPtr()), // complex *in + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tempHTransformed[mfi].dataPtr()), // complex *out FFTW_BACKWARD, // int sign FFTW_ESTIMATE); // unsigned flags #endif @@ -265,8 +269,10 @@ SpectralFieldDataRZ::FABZForwardTransform (amrex::MFIter const & mfi, amrex::Box # else result = cufftExecZ2Z(forward_plan[mfi], # endif - reinterpret_cast(tempHTransformed[mfi].dataPtr(mode)), // Complex *in - reinterpret_cast(tmpSpectralField[mfi].dataPtr(mode)), // Complex *out + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tempHTransformed[mfi].dataPtr(mode)), // Complex *in + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tmpSpectralField[mfi].dataPtr(mode)), // Complex *out CUFFT_FORWARD); if (result != CUFFT_SUCCESS) { ablastr::warn_manager::WMRecordWarning("Spectral solver", @@ -382,8 +388,10 @@ SpectralFieldDataRZ::FABZBackwardTransform (amrex::MFIter const & mfi, amrex::Bo # else result = cufftExecZ2Z(forward_plan[mfi], # endif - reinterpret_cast(tmpSpectralField[mfi].dataPtr(mode)), // Complex *in - reinterpret_cast(tempHTransformed[mfi].dataPtr(mode)), // Complex *out + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tmpSpectralField[mfi].dataPtr(mode)), // Complex *in + reinterpret_cast< + ablastr::math::anyfft::Complex*>(tempHTransformed[mfi].dataPtr(mode)), // Complex *out CUFFT_INVERSE); if (result != CUFFT_SUCCESS) { ablastr::warn_manager::WMRecordWarning("Spectral solver", diff --git a/Source/Make.WarpX b/Source/Make.WarpX index 06fff2052e2..6c55372d199 100644 --- a/Source/Make.WarpX +++ b/Source/Make.WarpX @@ -173,7 +173,7 @@ endif ifeq ($(USE_PSATD),TRUE) USERSuffix := $(USERSuffix).PSATD - DEFINES += -DWARPX_USE_PSATD + DEFINES += -DWARPX_USE_PSATD -DABLASTR_USE_FFT ifeq ($(USE_CUDA),TRUE) # Use cuFFT libraries += -lcufft diff --git a/Source/Utils/CMakeLists.txt b/Source/Utils/CMakeLists.txt index 77626c0681e..3d804fe9cde 100644 --- a/Source/Utils/CMakeLists.txt +++ b/Source/Utils/CMakeLists.txt @@ -10,7 +10,6 @@ foreach(D IN LISTS WarpX_DIMS) WarpXMovingWindow.cpp WarpXTagging.cpp WarpXUtil.cpp - WarpXrocfftUtil.cpp WarpXVersion.cpp ) endforeach() diff --git a/Source/Utils/Make.package b/Source/Utils/Make.package index 4b5888ef22b..dd7e61ff4fa 100644 --- a/Source/Utils/Make.package +++ b/Source/Utils/Make.package @@ -1,7 +1,6 @@ CEXE_sources += WarpXMovingWindow.cpp CEXE_sources += WarpXTagging.cpp CEXE_sources += WarpXUtil.cpp -CEXE_sources += WarpXrocfftUtil.cpp CEXE_sources += WarpXVersion.cpp CEXE_sources += WarpXAlgorithmSelection.cpp CEXE_sources += Interpolate.cpp diff --git a/Source/Utils/WarpX_Complex.H b/Source/Utils/WarpX_Complex.H index 048ee29940f..7a1f1669286 100644 --- a/Source/Utils/WarpX_Complex.H +++ b/Source/Utils/WarpX_Complex.H @@ -9,7 +9,7 @@ #define WARPX_COMPLEX_H_ #ifdef WARPX_USE_PSATD -# include "FieldSolver/SpectralSolver/AnyFFT.H" +# include #endif #include @@ -22,7 +22,7 @@ using Complex = amrex::GpuComplex; #ifdef WARPX_USE_PSATD -static_assert(sizeof(Complex) == sizeof(AnyFFT::Complex), +static_assert(sizeof(Complex) == sizeof(ablastr::math::anyfft::Complex), "The complex type in WarpX and the FFT library do not match."); #endif diff --git a/Source/Utils/WarpXrocfftUtil.H b/Source/Utils/WarpXrocfftUtil.H deleted file mode 100644 index 85332e6de3b..00000000000 --- a/Source/Utils/WarpXrocfftUtil.H +++ /dev/null @@ -1,24 +0,0 @@ -/* Copyright 2023 Luca Fedeli - * - * This file is part of WarpX. - * - * License: BSD-3-Clause-LBNL - */ - -#ifndef WARPX_ROCFFT_UTIL_H_ -#define WARPX_ROCFFT_UTIL_H_ - -namespace utils::rocfft -{ - /** This function is a wrapper around rocff_setup(). - * It is a no-op in case rocfft is not used. - */ - void setup(); - - /** This function is a wrapper around rocff_cleanup(). - * It is a no-op in case rocfft is not used. - */ - void cleanup(); -} - -#endif //WARPX_ROCFFT_UTILS_H_ diff --git a/Source/Utils/WarpXrocfftUtil.cpp b/Source/Utils/WarpXrocfftUtil.cpp deleted file mode 100644 index cc12d691449..00000000000 --- a/Source/Utils/WarpXrocfftUtil.cpp +++ /dev/null @@ -1,34 +0,0 @@ -/* Copyright 2023 Luca Fedeli - * - * This file is part of WarpX. - * - * License: BSD-3-Clause-LBNL - */ - -#include "WarpXrocfftUtil.H" - -#include - -#if defined(AMREX_USE_HIP) && defined(WARPX_USE_PSATD) -# if __has_include() // ROCm 5.3+ -# include -# else -# include -# endif -#endif - -void -utils::rocfft::setup() -{ -#if defined(AMREX_USE_HIP) && defined(WARPX_USE_PSATD) - rocfft_setup(); -#endif -} - -void -utils::rocfft::cleanup() -{ -#if defined(AMREX_USE_HIP) && defined(WARPX_USE_PSATD) - rocfft_cleanup(); -#endif -} diff --git a/Source/ablastr/CMakeLists.txt b/Source/ablastr/CMakeLists.txt index 95b2ea0e24c..a41fa4ae3a0 100644 --- a/Source/ablastr/CMakeLists.txt +++ b/Source/ablastr/CMakeLists.txt @@ -1,5 +1,6 @@ -#add_subdirectory(fields) add_subdirectory(coarsen) +add_subdirectory(math) +#add_subdirectory(fields) add_subdirectory(parallelization) #add_subdirectory(particles) #add_subdirectory(profiler) diff --git a/Source/ablastr/Make.package b/Source/ablastr/Make.package index 2d848e63a44..3426020e45d 100644 --- a/Source/ablastr/Make.package +++ b/Source/ablastr/Make.package @@ -1,6 +1,7 @@ #CEXE_sources += ParticleBoundaries.cpp include $(WARPX_HOME)/Source/ablastr/coarsen/Make.package +include $(WARPX_HOME)/Source/ablastr/math/Make.package include $(WARPX_HOME)/Source/ablastr/parallelization/Make.package include $(WARPX_HOME)/Source/ablastr/particles/Make.package include $(WARPX_HOME)/Source/ablastr/utils/Make.package diff --git a/Source/ablastr/math/CMakeLists.txt b/Source/ablastr/math/CMakeLists.txt new file mode 100644 index 00000000000..9093da83ae1 --- /dev/null +++ b/Source/ablastr/math/CMakeLists.txt @@ -0,0 +1 @@ +add_subdirectory(fft) diff --git a/Source/ablastr/math/Make.package b/Source/ablastr/math/Make.package new file mode 100644 index 00000000000..a0e95b11225 --- /dev/null +++ b/Source/ablastr/math/Make.package @@ -0,0 +1,3 @@ +include $(WARPX_HOME)/Source/ablastr/math/fft/Make.package + +VPATH_LOCATIONS += $(WARPX_HOME)/Source/ablastr diff --git a/Source/FieldSolver/SpectralSolver/AnyFFT.H b/Source/ablastr/math/fft/AnyFFT.H similarity index 58% rename from Source/FieldSolver/SpectralSolver/AnyFFT.H rename to Source/ablastr/math/fft/AnyFFT.H index 79dbdf0e03d..8a4c11c7654 100644 --- a/Source/FieldSolver/SpectralSolver/AnyFFT.H +++ b/Source/ablastr/math/fft/AnyFFT.H @@ -1,73 +1,89 @@ -/* Copyright 2019-2020 +/* Copyright 2019-2023 * - * This file is part of WarpX. + * This file is part of ABLASTR. * * License: BSD-3-Clause-LBNL */ -#ifndef ANYFFT_H_ -#define ANYFFT_H_ - -#include -#include - -#if defined(AMREX_USE_CUDA) -# include -#elif defined(AMREX_USE_HIP) -# if __has_include() // ROCm 5.3+ -# include -# else -# include -# endif -#else -# include +#ifndef ABLASTR_ANYFFT_H_ +#define ABLASTR_ANYFFT_H_ + +#ifdef ABLASTR_USE_FFT +# include +# include + +# if defined(AMREX_USE_CUDA) +# include +# elif defined(AMREX_USE_HIP) +# if __has_include() // ROCm 5.3+ +# include +# else +# include +# endif +# else +# include +# endif #endif + /** * Wrapper around FFT libraries. The header file defines the API and the base types * (Complex and VendorFFTPlan), and the implementation for different FFT libraries is * done in different cpp files. This wrapper only depends on the underlying FFT library * AND on AMReX (There is no dependence on WarpX). */ -namespace AnyFFT +namespace ablastr::math::anyfft { + + /** This function is a wrapper around rocff_setup(). + * It is a no-op in case rocfft is not used. + */ + void setup(); + + /** This function is a wrapper around rocff_cleanup(). + * It is a no-op in case rocfft is not used. + */ + void cleanup(); + +#ifdef ABLASTR_USE_FFT + // First, define library-dependent types (complex, FFT plan) /** Complex type for FFT, depends on FFT library */ -#if defined(AMREX_USE_CUDA) -# ifdef AMREX_USE_FLOAT - using Complex = cuComplex; -# else - using Complex = cuDoubleComplex; -# endif -#elif defined(AMREX_USE_HIP) -# ifdef AMREX_USE_FLOAT - using Complex = float2; -# else - using Complex = double2; -# endif -#else -# ifdef AMREX_USE_FLOAT - using Complex = fftwf_complex; -# else - using Complex = fftw_complex; -# endif -#endif +# if defined(AMREX_USE_CUDA) +# ifdef AMREX_USE_FLOAT + using Complex = cuComplex; +# else + using Complex = cuDoubleComplex; +# endif +# elif defined(AMREX_USE_HIP) +# ifdef AMREX_USE_FLOAT + using Complex = float2; +# else + using Complex = double2; +# endif +# else +# ifdef AMREX_USE_FLOAT + using Complex = fftwf_complex; +# else + using Complex = fftw_complex; +# endif +# endif /** Library-dependent FFT plans type, which holds one fft plan per box * (plans are only initialized for the boxes that are owned by the local MPI rank). */ -#if defined(AMREX_USE_CUDA) - using VendorFFTPlan = cufftHandle; -#elif defined(AMREX_USE_HIP) - using VendorFFTPlan = rocfft_plan; -#else -# ifdef AMREX_USE_FLOAT - using VendorFFTPlan = fftwf_plan; -# else - using VendorFFTPlan = fftw_plan; -# endif -#endif +# if defined(AMREX_USE_CUDA) + using VendorFFTPlan = cufftHandle; +# elif defined(AMREX_USE_HIP) + using VendorFFTPlan = rocfft_plan; +# else +# ifdef AMREX_USE_FLOAT + using VendorFFTPlan = fftwf_plan; +# else + using VendorFFTPlan = fftw_plan; +# endif +# endif // Second, define library-independent API @@ -108,6 +124,9 @@ namespace AnyFFT * \param[out] fft_plan plan for which the FFT is performed */ void Execute(FFTplan& fft_plan); + +#endif + } -#endif // ANYFFT_H_ +#endif // ABLASTR_ANYFFT_H_ diff --git a/Source/ablastr/math/fft/CMakeLists.txt b/Source/ablastr/math/fft/CMakeLists.txt new file mode 100644 index 00000000000..f7d689c98e9 --- /dev/null +++ b/Source/ablastr/math/fft/CMakeLists.txt @@ -0,0 +1,14 @@ +foreach(D IN LISTS WarpX_DIMS) + warpx_set_suffix_dims(SD ${D}) + if(WarpX_PSATD STREQUAL ON) + if(WarpX_COMPUTE STREQUAL CUDA) + target_sources(ablastr_${SD} PRIVATE WrapCuFFT.cpp) + elseif(WarpX_COMPUTE STREQUAL HIP) + target_sources(ablastr_${SD} PRIVATE WrapRocFFT.cpp) + else() + target_sources(ablastr_${SD} PRIVATE WrapFFTW.cpp) + endif() + else() + target_sources(ablastr_${SD} PRIVATE WrapNoFFT.cpp) + endif() +endforeach() diff --git a/Source/ablastr/math/fft/Make.package b/Source/ablastr/math/fft/Make.package new file mode 100644 index 00000000000..b04062bd9d5 --- /dev/null +++ b/Source/ablastr/math/fft/Make.package @@ -0,0 +1,13 @@ +ifeq ($(USE_PSATD),TRUE) + ifeq ($(USE_CUDA),TRUE) + CEXE_sources += WrapCuFFT.cpp + else ifeq ($(USE_HIP),TRUE) + CEXE_sources += WrapRocFFT.cpp + else + CEXE_sources += WrapFFTW.cpp + endif +else + CEXE_sources += WrapNoFFT.cpp +endif + +VPATH_LOCATIONS += $(WARPX_HOME)/Source/ablastr/math/fft diff --git a/Source/FieldSolver/SpectralSolver/WrapCuFFT.cpp b/Source/ablastr/math/fft/WrapCuFFT.cpp similarity index 84% rename from Source/FieldSolver/SpectralSolver/WrapCuFFT.cpp rename to Source/ablastr/math/fft/WrapCuFFT.cpp index 59b200a32fc..8ecea2a2486 100644 --- a/Source/FieldSolver/SpectralSolver/WrapCuFFT.cpp +++ b/Source/ablastr/math/fft/WrapCuFFT.cpp @@ -1,17 +1,21 @@ -/* Copyright 2019-2020 +/* Copyright 2019-2023 * - * This file is part of WarpX. + * This file is part of ABLASTR. * * License: BSD-3-Clause-LBNL */ #include "AnyFFT.H" -#include "Utils/TextMsg.H" +#include "ablastr/utils/TextMsg.H" -namespace AnyFFT +namespace ablastr::math::anyfft { + void setup(){/*nothing to do*/} + + void cleanup(){/*nothing to do*/} + #ifdef AMREX_USE_FLOAT cufftType VendorR2C = CUFFT_R2C; cufftType VendorC2R = CUFFT_C2R; @@ -37,7 +41,7 @@ namespace AnyFFT result = cufftPlan2d( &(fft_plan.m_plan), real_size[1], real_size[0], VendorR2C); } else { - WARPX_ABORT_WITH_MESSAGE("only dim=2 and dim=3 have been implemented"); + ABLASTR_ABORT_WITH_MESSAGE("only dim=2 and dim=3 have been implemented"); } } else { if (dim == 3) { @@ -47,15 +51,12 @@ namespace AnyFFT result = cufftPlan2d( &(fft_plan.m_plan), real_size[1], real_size[0], VendorC2R); } else { - WARPX_ABORT_WITH_MESSAGE("only dim=2 and dim=3 have been implemented"); + ABLASTR_ABORT_WITH_MESSAGE("only dim=2 and dim=3 have been implemented"); } } - if ( result != CUFFT_SUCCESS ) { - amrex::Print() << Utils::TextMsg::Err( - "cufftplan failed! Error: " - + cufftErrorToString(result)); - } + ABLASTR_ALWAYS_ASSERT_WITH_MESSAGE(result == CUFFT_SUCCESS, + "cufftplan failed! Error: " + cufftErrorToString(result)); // Store meta-data in fft_plan fft_plan.m_real_array = real_array; @@ -89,11 +90,11 @@ namespace AnyFFT result = cufftExecZ2D(fft_plan.m_plan, fft_plan.m_complex_array, fft_plan.m_real_array); #endif } else { - WARPX_ABORT_WITH_MESSAGE( - "direction must be AnyFFT::direction::R2C or AnyFFT::direction::C2R"); + ABLASTR_ABORT_WITH_MESSAGE( + "direction must be FFTplan::direction::R2C or FFTplan::direction::C2R"); } if ( result != CUFFT_SUCCESS ) { - WARPX_ABORT_WITH_MESSAGE( + ABLASTR_ABORT_WITH_MESSAGE( "forward transform using cufftExec failed ! Error: " +cufftErrorToString(result)); } diff --git a/Source/FieldSolver/SpectralSolver/WrapFFTW.cpp b/Source/ablastr/math/fft/WrapFFTW.cpp similarity index 91% rename from Source/FieldSolver/SpectralSolver/WrapFFTW.cpp rename to Source/ablastr/math/fft/WrapFFTW.cpp index df330c4f67c..6711bbface9 100644 --- a/Source/FieldSolver/SpectralSolver/WrapFFTW.cpp +++ b/Source/ablastr/math/fft/WrapFFTW.cpp @@ -1,22 +1,25 @@ -/* Copyright 2019-2021 +/* Copyright 2019-2023 * - * This file is part of WarpX. + * This file is part of ABLASTR. * * License: BSD-3-Clause-LBNL */ #include "AnyFFT.H" -#include "Utils/TextMsg.H" +#include "ablastr/utils/TextMsg.H" #include #include #include -#include - -namespace AnyFFT +namespace ablastr::math::anyfft { + + void setup(){/*nothing to do*/} + + void cleanup(){/*nothing to do*/} + #ifdef AMREX_USE_FLOAT const auto VendorCreatePlanR2C3D = fftwf_plan_dft_r2c_3d; const auto VendorCreatePlanC2R3D = fftwf_plan_dft_c2r_3d; @@ -54,7 +57,7 @@ namespace AnyFFT fft_plan.m_plan = VendorCreatePlanR2C2D( real_size[1], real_size[0], real_array, complex_array, FFTW_ESTIMATE); } else { - WARPX_ABORT_WITH_MESSAGE( + ABLASTR_ABORT_WITH_MESSAGE( "only dim=2 and dim=3 have been implemented"); } } else if (dir == direction::C2R){ @@ -65,7 +68,7 @@ namespace AnyFFT fft_plan.m_plan = VendorCreatePlanC2R2D( real_size[1], real_size[0], complex_array, real_array, FFTW_ESTIMATE); } else { - WARPX_ABORT_WITH_MESSAGE( + ABLASTR_ABORT_WITH_MESSAGE( "only dim=2 and dim=3 have been implemented. Should be easy to add dim=1."); } } diff --git a/Source/ablastr/math/fft/WrapNoFFT.cpp b/Source/ablastr/math/fft/WrapNoFFT.cpp new file mode 100644 index 00000000000..c6dc5cdf484 --- /dev/null +++ b/Source/ablastr/math/fft/WrapNoFFT.cpp @@ -0,0 +1,17 @@ +/* Copyright 2019-2023 + * + * This file is part of ABLASTR. + * + * License: BSD-3-Clause-LBNL + */ + +#include "AnyFFT.H" + +namespace ablastr::math::anyfft +{ + + void setup(){/*nothing to do*/} + + void cleanup(){/*nothing to do*/} + +} diff --git a/Source/FieldSolver/SpectralSolver/WrapRocFFT.cpp b/Source/ablastr/math/fft/WrapRocFFT.cpp similarity index 89% rename from Source/FieldSolver/SpectralSolver/WrapRocFFT.cpp rename to Source/ablastr/math/fft/WrapRocFFT.cpp index 5f3131dec47..4bd39efab57 100644 --- a/Source/FieldSolver/SpectralSolver/WrapRocFFT.cpp +++ b/Source/ablastr/math/fft/WrapRocFFT.cpp @@ -1,26 +1,34 @@ -/* Copyright 2019-2020 +/* Copyright 2019-2023 * - * This file is part of WarpX. + * This file is part of ABLASTR. * * License: BSD-3-Clause-LBNL */ #include "AnyFFT.H" -#include "Utils/TextMsg.H" +#include "ablastr/utils/TextMsg.H" -namespace AnyFFT +namespace ablastr::math::anyfft { + void setup() + { + rocfft_setup(); + } + + void cleanup() + { + rocfft_cleanup(); + } std::string rocfftErrorToString (const rocfft_status err); - namespace { - void assert_rocfft_status (std::string const& name, rocfft_status status) + namespace + { + void assert_rocfft_status (std::string const& name, rocfft_status const& status) { - if (status != rocfft_status_success) { - WARPX_ABORT_WITH_MESSAGE( - name + " failed! Error: " + rocfftErrorToString(status)); - } + ABLASTR_ALWAYS_ASSERT_WITH_MESSAGE(status == rocfft_status_success, + name + " failed! Error: " + rocfftErrorToString(status)); } } @@ -91,8 +99,8 @@ namespace AnyFFT (void**)&(fft_plan.m_real_array), // out execinfo); } else { - WARPX_ABORT_WITH_MESSAGE( - "direction must be AnyFFT::direction::R2C or AnyFFT::direction::C2R"); + ABLASTR_ABORT_WITH_MESSAGE( + "direction must be FFTplan::direction::R2C or FFTplan::direction::C2R"); } assert_rocfft_status("rocfft_execute", result); diff --git a/Source/main.cpp b/Source/main.cpp index cfe80534987..2a1b828c64f 100644 --- a/Source/main.cpp +++ b/Source/main.cpp @@ -10,8 +10,8 @@ #include "Initialization/WarpXAMReXInit.H" #include "Utils/WarpXProfilerWrapper.H" -#include "Utils/WarpXrocfftUtil.H" +#include #include #include #include @@ -25,7 +25,7 @@ int main(int argc, char* argv[]) warpx::initialization::amrex_init(argc, argv); - utils::rocfft::setup(); + ablastr::math::anyfft::setup(); { WARPX_PROFILE_VAR("main()", pmain); @@ -39,14 +39,14 @@ int main(int argc, char* argv[]) warpx.Evolve(); - //Print warning messages at the end of the simulation amrex::Print() << ablastr::warn_manager::GetWMInstance().PrintGlobalWarnings("THE END"); timer.record_stop_time(); - if (warpx.Verbose()) { + if (warpx.Verbose()) + { amrex::Print() << "Total Time : " - << timer.get_global_duration() << '\n'; + << timer.get_global_duration() << '\n'; } WARPX_PROFILE_VAR_STOP(pmain); @@ -54,7 +54,7 @@ int main(int argc, char* argv[]) WarpX::Finalize(); } - utils::rocfft::cleanup(); + ablastr::math::anyfft::cleanup(); amrex::Finalize(); diff --git a/cmake/dependencies/FFT.cmake b/cmake/dependencies/FFT.cmake index 56dc396b31f..571006e8530 100644 --- a/cmake/dependencies/FFT.cmake +++ b/cmake/dependencies/FFT.cmake @@ -1,4 +1,4 @@ -if(WarpX_PSATD) +if(ABLASTR_FFT) # Helper Functions ############################################################ # option(WarpX_FFTW_IGNORE_OMP "Ignore FFTW3 OpenMP support, even if found" OFF) @@ -122,4 +122,4 @@ if(WarpX_PSATD) message(STATUS "FFTW: Did NOT search for OpenMP support (WarpX_COMPUTE!=OMP)") endif() endif() -endif(WarpX_PSATD) +endif(ABLASTR_FFT) From 0c6d0097651623aa38bc0ea44094d64e42d9417a Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 27 Nov 2023 14:32:24 -0800 Subject: [PATCH 105/110] AMReX/pyAMReX/PICSAR: Weekly Update (#4447) * AMReX: Weekly Update * pyAMReX: Weekly Update --- .github/workflows/cuda.yml | 2 +- Regression/WarpX-GPU-tests.ini | 2 +- Regression/WarpX-tests.ini | 2 +- cmake/dependencies/AMReX.cmake | 2 +- cmake/dependencies/pyAMReX.cmake | 2 +- run_test.sh | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml index 051ff0ea758..aab28b74b80 100644 --- a/.github/workflows/cuda.yml +++ b/.github/workflows/cuda.yml @@ -108,7 +108,7 @@ jobs: which nvcc || echo "nvcc not in PATH!" git clone https://github.com/AMReX-Codes/amrex.git ../amrex - cd ../amrex && git checkout --detach 175b99d913dc2748e43c53192737170c770fe0e8 && cd - + cd ../amrex && git checkout --detach 9e35dc19489dc5d312e92781cb0471d282cf8370 && cd - make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2 build_nvhpc21-11-nvcc: diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini index 264106eb8ec..e11976205d1 100644 --- a/Regression/WarpX-GPU-tests.ini +++ b/Regression/WarpX-GPU-tests.ini @@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more [AMReX] dir = /home/regtester/git/amrex/ -branch = 175b99d913dc2748e43c53192737170c770fe0e8 +branch = 9e35dc19489dc5d312e92781cb0471d282cf8370 [source] dir = /home/regtester/git/WarpX diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index d67a0a03569..b73fbee77fe 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det [AMReX] dir = /home/regtester/AMReX_RegTesting/amrex/ -branch = 175b99d913dc2748e43c53192737170c770fe0e8 +branch = 9e35dc19489dc5d312e92781cb0471d282cf8370 [source] dir = /home/regtester/AMReX_RegTesting/warpx diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake index 7aea38148ea..d382cf97094 100644 --- a/cmake/dependencies/AMReX.cmake +++ b/cmake/dependencies/AMReX.cmake @@ -269,7 +269,7 @@ set(WarpX_amrex_src "" set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git" CACHE STRING "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)") -set(WarpX_amrex_branch "175b99d913dc2748e43c53192737170c770fe0e8" +set(WarpX_amrex_branch "9e35dc19489dc5d312e92781cb0471d282cf8370" CACHE STRING "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)") diff --git a/cmake/dependencies/pyAMReX.cmake b/cmake/dependencies/pyAMReX.cmake index 20232969f75..7752a82758b 100644 --- a/cmake/dependencies/pyAMReX.cmake +++ b/cmake/dependencies/pyAMReX.cmake @@ -79,7 +79,7 @@ option(WarpX_pyamrex_internal "Download & build pyAMReX" ON) set(WarpX_pyamrex_repo "https://github.com/AMReX-Codes/pyamrex.git" CACHE STRING "Repository URI to pull and build pyamrex from if(WarpX_pyamrex_internal)") -set(WarpX_pyamrex_branch "d90e390971de4144fb380ae62d5c74cdc27dd3fc" +set(WarpX_pyamrex_branch "58dc1ed58226ab683a9bdea858da2a196cd1570f" CACHE STRING "Repository branch for WarpX_pyamrex_repo if(WarpX_pyamrex_internal)") diff --git a/run_test.sh b/run_test.sh index b548f11e93f..485e07d4e2f 100755 --- a/run_test.sh +++ b/run_test.sh @@ -68,7 +68,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt # Clone AMReX and warpx-data git clone https://github.com/AMReX-Codes/amrex.git -cd amrex && git checkout --detach 175b99d913dc2748e43c53192737170c770fe0e8 && cd - +cd amrex && git checkout --detach 9e35dc19489dc5d312e92781cb0471d282cf8370 && cd - # warpx-data contains various required data sets git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git # openPMD-example-datasets contains various required data sets From cec017212c8737ad6663a8abf12128075741cf94 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 28 Nov 2023 00:02:46 -0800 Subject: [PATCH 106/110] CI: numprocs * numthreads <= 2 (#4450) We must not oversubscribe the two (2) CI cores with the number of processes times threads we use in tests. --- Regression/WarpX-tests.ini | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index b73fbee77fe..fa4082cf376 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -357,7 +357,7 @@ restartTest = 0 useMPI = 1 numprocs = 2 useOMP = 1 -numthreads = 2 +numthreads = 1 compileTest = 0 doVis = 0 analysisRoutine = Examples/Tests/nuclear_fusion/analysis_two_product_fusion.py @@ -2004,7 +2004,7 @@ restartTest = 0 useMPI = 1 numprocs = 2 useOMP = 1 -numthreads = 2 +numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 @@ -2959,7 +2959,7 @@ restartTest = 0 useMPI = 1 numprocs = 2 useOMP = 1 -numthreads = 2 +numthreads = 1 compileTest = 0 doVis = 0 analysisRoutine = Examples/Tests/nuclear_fusion/analysis_proton_boron_fusion.py @@ -2975,7 +2975,7 @@ restartTest = 0 useMPI = 1 numprocs = 2 useOMP = 1 -numthreads = 2 +numthreads = 1 compileTest = 0 doVis = 0 analysisRoutine = Examples/Tests/nuclear_fusion/analysis_proton_boron_fusion.py @@ -4435,7 +4435,7 @@ cmakeSetupOpts = -DWarpX_DIMS=3 restartTest = 0 useMPI = 1 numprocs = 1 -useOMP = 0 +useOMP = 1 numthreads = 1 compileTest = 0 doVis = 0 From c9293e5a867912bd6d666f1dd4f21ad53b233d6a Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 28 Nov 2023 09:42:37 -0800 Subject: [PATCH 107/110] Doc: Fix Typo `.integrate = true` (#4449) Typo, this should not be `==`. --- Docs/source/usage/parameters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index dd69f6087f7..0caf77ebc38 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -2748,7 +2748,7 @@ Reduced Diagnostics In RZ geometry, this only saves the 0'th azimuthal mode component of the fields. Integrated electric and magnetic field components can instead be obtained by specifying - ``.integrate == true``. + ``.integrate = true``. In a *moving window* simulation, the FieldProbe can be set to follow the moving frame by specifying ``.do_moving_window_FP = 1`` (default 0). .. warning:: From f1eb2b24d1a032a09ee8a7f2c6303486c3621652 Mon Sep 17 00:00:00 2001 From: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Date: Tue, 28 Nov 2023 09:54:52 -0800 Subject: [PATCH 108/110] Maxlevel user option for external field initialization (#4326) * add new option to initialize external fields on grid upto a max user-defined level * ensure to init level > max level to 0 * fix logic for init * Edoardo's fixes for typo Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> * Update Source/WarpX.H Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> * fix bug * adding picmi interface with doc for Analytical initial field * allocinit fields to simplify default behavior in initialization with constant and parser up to max levels * default not needed anymore * no need for a default * fix incomplete paranthesis * fix doc to remove default and include Dave's suggestion * Update doc string * fix incorrect rebase --------- Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Co-authored-by: Axel Huebl --- Docs/source/usage/parameters.rst | 18 +++++-- Docs/source/usage/python.rst | 4 ++ Python/pywarpx/picmi.py | 2 + Source/Initialization/WarpXInitData.cpp | 18 +++++-- Source/WarpX.H | 6 +++ Source/WarpX.cpp | 67 +++++++++++++------------ 6 files changed, 74 insertions(+), 41 deletions(-) diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index 0caf77ebc38..b9fedb4f050 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -1398,10 +1398,10 @@ External fields Grid initialization ^^^^^^^^^^^^^^^^^^^ -* ``warpx.B_ext_grid_init_style`` (string) optional (default is "default") +* ``warpx.B_ext_grid_init_style`` (string) optional This parameter determines the type of initialization for the external - magnetic field. The "default" style initializes the - external magnetic field (Bx,By,Bz) to (0.0, 0.0, 0.0). + magnetic field. By default, the + external magnetic field (Bx,By,Bz) is initialized to (0.0, 0.0, 0.0). The string can be set to "constant" if a constant magnetic field is required to be set at initialization. If set to "constant", then an additional parameter, namely, ``warpx.B_external_grid`` must be specified. @@ -1429,9 +1429,9 @@ Grid initialization Regarding how to prepare the openPMD data file, one can refer to the `openPMD-example-datasets `__. -* ``warpx.E_ext_grid_init_style`` (string) optional (default is "default") +* ``warpx.E_ext_grid_init_style`` (string) optional This parameter determines the type of initialization for the external - electric field. The "default" style initializes the + electric field. By default, the external electric field (Ex,Ey,Ez) to (0.0, 0.0, 0.0). The string can be set to "constant" if a constant electric field is required to be set at initialization. If set to "constant", then an @@ -1473,6 +1473,14 @@ Grid initialization the field solver. In particular, do not use any other boundary condition than periodic. +* ``warpx.maxlevel_extEMfield_init`` (default is maximum number of levels in the simulation) + With this parameter, the externally applied electric and magnetic fields + will not be applied for levels greater than ``warpx.maxlevel_extEMfield_init``. + For some mesh-refinement simulations, + the external fields are only applied to the parent grid and not the refined patches. In such cases, + ``warpx.maxlevel_extEMfield_init`` can be set to 0. + In that case, the other levels have external field values of 0. + Applied to Particles ^^^^^^^^^^^^^^^^^^^^ diff --git a/Docs/source/usage/python.rst b/Docs/source/usage/python.rst index dd44c4098fd..dee10651813 100644 --- a/Docs/source/usage/python.rst +++ b/Docs/source/usage/python.rst @@ -69,6 +69,10 @@ EmbeddedBoundary Applied fields ^^^^^^^^^^^^^^ +AnalyticInitialField +"""""""""""""""""""" +.. autoclass:: pywarpx.picmi.AnalyticInitialField + ConstantAppliedField """""""""""""""""""" .. autoclass:: pywarpx.picmi.ConstantAppliedField diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index fa7776b1def..1b0358e2e94 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -1233,9 +1233,11 @@ def initialize_inputs(self): class AnalyticInitialField(picmistandard.PICMI_AnalyticAppliedField): def init(self, kw): self.mangle_dict = None + self.maxlevel_extEMfield_init = kw.pop('warpx_maxlevel_extEMfield_init', None); def initialize_inputs(self): # Note that lower and upper_bound are not used by WarpX + pywarpx.warpx.maxlevel_extEMfield_init = self.maxlevel_extEMfield_init; if self.mangle_dict is None: # Only do this once so that the same variables are used in this distribution diff --git a/Source/Initialization/WarpXInitData.cpp b/Source/Initialization/WarpXInitData.cpp index 683955e764a..91e8b3f6319 100644 --- a/Source/Initialization/WarpXInitData.cpp +++ b/Source/Initialization/WarpXInitData.cpp @@ -764,7 +764,10 @@ WarpX::InitLevelData (int lev, Real /*time*/) for (int i = 0; i < 3; ++i) { - if (B_ext_grid_s == "constant" || B_ext_grid_s == "default") { + // Externally imposed fields are only initialized until the user-defined maxlevel_extEMfield_init. + // The default maxlevel_extEMfield_init value is the total number of levels in the simulation + if ( ( B_ext_grid_s == "constant") && (lev <= maxlevel_extEMfield_init) ) + { Bfield_fp[lev][i]->setVal(B_external_grid[i]); if (fft_do_time_averaging) { Bfield_avg_fp[lev][i]->setVal(B_external_grid[i]); @@ -778,7 +781,10 @@ WarpX::InitLevelData (int lev, Real /*time*/) } } } - if (E_ext_grid_s == "constant" || E_ext_grid_s == "default") { + // Externally imposed fields are only initialized until the user-defined maxlevel_extEMfield_init. + // The default maxlevel_extEMfield_init value is the total number of levels in the simulation + if ( ( E_ext_grid_s == "constant") && (lev <= maxlevel_extEMfield_init) ) + { Efield_fp[lev][i]->setVal(E_external_grid[i]); if (fft_do_time_averaging) { Efield_avg_fp[lev][i]->setVal(E_external_grid[i]); @@ -801,7 +807,9 @@ WarpX::InitLevelData (int lev, Real /*time*/) // if the input string for the B-field is "parse_b_ext_grid_function", // then the analytical expression or function must be // provided in the input file. - if (B_ext_grid_s == "parse_b_ext_grid_function") { + // Externally imposed fields are only initialized until the user-defined maxlevel_extEMfield_init. + // The default maxlevel_extEMfield_init value is the total number of levels in the simulation + if (B_ext_grid_s == "parse_b_ext_grid_function" && (lev <= maxlevel_extEMfield_init)) { //! Strings storing parser function to initialize the components of the magnetic field on the grid std::string str_Bx_ext_grid_function; @@ -872,7 +880,9 @@ WarpX::InitLevelData (int lev, Real /*time*/) // if the input string for the E-field is "parse_e_ext_grid_function", // then the analytical expression or function must be // provided in the input file. - if (E_ext_grid_s == "parse_e_ext_grid_function") { + // Externally imposed fields are only initialized until the user-defined maxlevel_extEMfield_init. + // The default maxlevel_extEMfield_init value is the total number of levels in the simulation + if (E_ext_grid_s == "parse_e_ext_grid_function" && (lev <= maxlevel_extEMfield_init)) { #ifdef WARPX_DIM_RZ WARPX_ABORT_WITH_MESSAGE( diff --git a/Source/WarpX.H b/Source/WarpX.H index b2bea527069..7d0ca8500cd 100644 --- a/Source/WarpX.H +++ b/Source/WarpX.H @@ -167,6 +167,12 @@ public: //! User-defined parser to initialize z-component of the electric field on the grid std::unique_ptr Ezfield_parser; + /** Maximum level up to which the externally defined electric and magnetic fields are initialized. + * The default value is set to the max levels in the simulation. + * if lev > maxlevel_extEMfield_init, the fields on those levels will have a default value of 0 + */ + int maxlevel_extEMfield_init; + // Algorithms //! Integer that corresponds to the current deposition algorithm (Esirkepov, direct, Vay) static short current_deposition_algo; diff --git a/Source/WarpX.cpp b/Source/WarpX.cpp index 393e3f3110e..9ec29b07f01 100644 --- a/Source/WarpX.cpp +++ b/Source/WarpX.cpp @@ -88,8 +88,8 @@ using namespace amrex; Vector WarpX::E_external_grid(3, 0.0); Vector WarpX::B_external_grid(3, 0.0); -std::string WarpX::B_ext_grid_s = "default"; -std::string WarpX::E_ext_grid_s = "default"; +std::string WarpX::B_ext_grid_s; +std::string WarpX::E_ext_grid_s; bool WarpX::add_external_E_field = false; bool WarpX::add_external_B_field = false; @@ -740,6 +740,9 @@ WarpX::ReadParameters () add_external_E_field = true; } + maxlevel_extEMfield_init = maxLevel(); + pp_warpx.query("maxlevel_extEMfield_init", maxlevel_extEMfield_init); + electrostatic_solver_id = GetAlgorithmInteger(pp_warpx, "do_electrostatic"); // if an electrostatic solver is used, set the Maxwell solver to None if (electrostatic_solver_id != ElectrostaticSolverAlgo::None) { @@ -2207,13 +2210,13 @@ WarpX::AllocLevelMFs (int lev, const BoxArray& ba, const DistributionMapping& dm // const std::array dx = CellSize(lev); - AllocInitMultiFab(Bfield_fp[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp[x]"); - AllocInitMultiFab(Bfield_fp[lev][1], amrex::convert(ba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp[y]"); - AllocInitMultiFab(Bfield_fp[lev][2], amrex::convert(ba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp[z]"); + AllocInitMultiFab(Bfield_fp[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp[x]", 0.0_rt); + AllocInitMultiFab(Bfield_fp[lev][1], amrex::convert(ba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp[y]", 0.0_rt); + AllocInitMultiFab(Bfield_fp[lev][2], amrex::convert(ba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp[z]", 0.0_rt); - AllocInitMultiFab(Efield_fp[lev][0], amrex::convert(ba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp[x]"); - AllocInitMultiFab(Efield_fp[lev][1], amrex::convert(ba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp[y]"); - AllocInitMultiFab(Efield_fp[lev][2], amrex::convert(ba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp[z]"); + AllocInitMultiFab(Efield_fp[lev][0], amrex::convert(ba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp[x]", 0.0_rt); + AllocInitMultiFab(Efield_fp[lev][1], amrex::convert(ba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp[y]", 0.0_rt); + AllocInitMultiFab(Efield_fp[lev][2], amrex::convert(ba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp[z]", 0.0_rt); AllocInitMultiFab(current_fp[lev][0], amrex::convert(ba, jx_nodal_flag), dm, ncomps, ngJ, lev, "current_fp[x]", 0.0_rt); AllocInitMultiFab(current_fp[lev][1], amrex::convert(ba, jy_nodal_flag), dm, ncomps, ngJ, lev, "current_fp[y]", 0.0_rt); @@ -2221,14 +2224,14 @@ WarpX::AllocLevelMFs (int lev, const BoxArray& ba, const DistributionMapping& dm // Match external field MultiFabs to fine patch if (add_external_B_field) { - AllocInitMultiFab(Bfield_fp_external[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp_external[x]"); - AllocInitMultiFab(Bfield_fp_external[lev][1], amrex::convert(ba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp_external[y]"); - AllocInitMultiFab(Bfield_fp_external[lev][2], amrex::convert(ba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp_external[z]"); + AllocInitMultiFab(Bfield_fp_external[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp_external[x]", 0.0_rt); + AllocInitMultiFab(Bfield_fp_external[lev][1], amrex::convert(ba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp_external[y]", 0.0_rt); + AllocInitMultiFab(Bfield_fp_external[lev][2], amrex::convert(ba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_fp_external[z]", 0.0_rt); } if (add_external_E_field) { - AllocInitMultiFab(Efield_fp_external[lev][0], amrex::convert(ba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp_external[x]"); - AllocInitMultiFab(Efield_fp_external[lev][1], amrex::convert(ba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp_external[y]"); - AllocInitMultiFab(Efield_fp_external[lev][2], amrex::convert(ba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp_external[z]"); + AllocInitMultiFab(Efield_fp_external[lev][0], amrex::convert(ba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp_external[x]", 0.0_rt); + AllocInitMultiFab(Efield_fp_external[lev][1], amrex::convert(ba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp_external[y]", 0.0_rt); + AllocInitMultiFab(Efield_fp_external[lev][2], amrex::convert(ba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_fp_external[z]", 0.0_rt); } if (do_current_centering) @@ -2289,13 +2292,13 @@ WarpX::AllocLevelMFs (int lev, const BoxArray& ba, const DistributionMapping& dm if (fft_do_time_averaging) { - AllocInitMultiFab(Bfield_avg_fp[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[x]"); - AllocInitMultiFab(Bfield_avg_fp[lev][1], amrex::convert(ba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[y]"); - AllocInitMultiFab(Bfield_avg_fp[lev][2], amrex::convert(ba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[z]"); + AllocInitMultiFab(Bfield_avg_fp[lev][0], amrex::convert(ba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[x]", 0.0_rt); + AllocInitMultiFab(Bfield_avg_fp[lev][1], amrex::convert(ba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[y]", 0.0_rt); + AllocInitMultiFab(Bfield_avg_fp[lev][2], amrex::convert(ba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_fp[z]", 0.0_rt); - AllocInitMultiFab(Efield_avg_fp[lev][0], amrex::convert(ba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_fp[x]"); - AllocInitMultiFab(Efield_avg_fp[lev][1], amrex::convert(ba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_fp[y]"); - AllocInitMultiFab(Efield_avg_fp[lev][2], amrex::convert(ba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_fp[z]"); + AllocInitMultiFab(Efield_avg_fp[lev][0], amrex::convert(ba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_fp[x]", 0.0_rt); + AllocInitMultiFab(Efield_avg_fp[lev][1], amrex::convert(ba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_fp[y]", 0.0_rt); + AllocInitMultiFab(Efield_avg_fp[lev][2], amrex::convert(ba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_fp[z]", 0.0_rt); } #ifdef AMREX_USE_EB @@ -2500,24 +2503,24 @@ WarpX::AllocLevelMFs (int lev, const BoxArray& ba, const DistributionMapping& dm const std::array cdx = CellSize(lev-1); // Create the MultiFabs for B - AllocInitMultiFab(Bfield_cp[lev][0], amrex::convert(cba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_cp[x]"); - AllocInitMultiFab(Bfield_cp[lev][1], amrex::convert(cba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_cp[y]"); - AllocInitMultiFab(Bfield_cp[lev][2], amrex::convert(cba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_cp[z]"); + AllocInitMultiFab(Bfield_cp[lev][0], amrex::convert(cba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_cp[x]", 0.0_rt); + AllocInitMultiFab(Bfield_cp[lev][1], amrex::convert(cba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_cp[y]", 0.0_rt); + AllocInitMultiFab(Bfield_cp[lev][2], amrex::convert(cba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_cp[z]", 0.0_rt); // Create the MultiFabs for E - AllocInitMultiFab(Efield_cp[lev][0], amrex::convert(cba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_cp[x]"); - AllocInitMultiFab(Efield_cp[lev][1], amrex::convert(cba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_cp[y]"); - AllocInitMultiFab(Efield_cp[lev][2], amrex::convert(cba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_cp[z]"); + AllocInitMultiFab(Efield_cp[lev][0], amrex::convert(cba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_cp[x]", 0.0_rt); + AllocInitMultiFab(Efield_cp[lev][1], amrex::convert(cba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_cp[y]", 0.0_rt); + AllocInitMultiFab(Efield_cp[lev][2], amrex::convert(cba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_cp[z]", 0.0_rt); if (fft_do_time_averaging) { - AllocInitMultiFab(Bfield_avg_cp[lev][0], amrex::convert(cba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_cp[x]"); - AllocInitMultiFab(Bfield_avg_cp[lev][1], amrex::convert(cba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_cp[y]"); - AllocInitMultiFab(Bfield_avg_cp[lev][2], amrex::convert(cba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_cp[z]"); + AllocInitMultiFab(Bfield_avg_cp[lev][0], amrex::convert(cba, Bx_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_cp[x]", 0.0_rt); + AllocInitMultiFab(Bfield_avg_cp[lev][1], amrex::convert(cba, By_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_cp[y]", 0.0_rt); + AllocInitMultiFab(Bfield_avg_cp[lev][2], amrex::convert(cba, Bz_nodal_flag), dm, ncomps, ngEB, lev, "Bfield_avg_cp[z]", 0.0_rt); - AllocInitMultiFab(Efield_avg_cp[lev][0], amrex::convert(cba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_cp[x]"); - AllocInitMultiFab(Efield_avg_cp[lev][1], amrex::convert(cba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_cp[y]"); - AllocInitMultiFab(Efield_avg_cp[lev][2], amrex::convert(cba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_cp[z]"); + AllocInitMultiFab(Efield_avg_cp[lev][0], amrex::convert(cba, Ex_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_cp[x]", 0.0_rt); + AllocInitMultiFab(Efield_avg_cp[lev][1], amrex::convert(cba, Ey_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_cp[y]", 0.0_rt); + AllocInitMultiFab(Efield_avg_cp[lev][2], amrex::convert(cba, Ez_nodal_flag), dm, ncomps, ngEB, lev, "Efield_avg_cp[z]", 0.0_rt); } // Create the MultiFabs for the current From b141e652d7fe1cf8431ecb9e8b6428aa1c6c40f3 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Tue, 28 Nov 2023 10:43:54 -0800 Subject: [PATCH 109/110] CPU CI: Always Serialize (#4451) Always add - `warpx.do_dynamic_scheduling=0` - `warpx.serialize_initial_conditions=1` to CPU CI jobs, so we cannot forget it when it matters (e.g., random numbers in distributions). --- Regression/WarpX-tests.ini | 114 +++++++++++++++++----------------- Regression/prepare_file_ci.py | 10 ++- 2 files changed, 66 insertions(+), 58 deletions(-) diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini index fa4082cf376..c8e9a7e1a90 100644 --- a/Regression/WarpX-tests.ini +++ b/Regression/WarpX-tests.ini @@ -317,7 +317,7 @@ analysisRoutine = Examples/analysis_default_regression.py [Deuterium_Deuterium_Fusion_3D] buildDir = . inputFile = Examples/Tests/nuclear_fusion/inputs_deuterium_deuterium_3d -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -333,7 +333,7 @@ analysisRoutine = Examples/Tests/nuclear_fusion/analysis_two_product_fusion.py [Deuterium_Deuterium_Fusion_3D_intraspecies] buildDir = . inputFile = Examples/Tests/nuclear_fusion/inputs_deuterium_deuterium_3d_intraspecies -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -349,7 +349,7 @@ analysisRoutine = Examples/Tests/nuclear_fusion/analysis_deuterium_deuterium_3d_ [Deuterium_Tritium_Fusion_3D] buildDir = . inputFile = Examples/Tests/nuclear_fusion/inputs_deuterium_tritium_3d -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -365,7 +365,7 @@ analysisRoutine = Examples/Tests/nuclear_fusion/analysis_two_product_fusion.py [Deuterium_Tritium_Fusion_RZ] buildDir = . inputFile = Examples/Tests/nuclear_fusion/inputs_deuterium_tritium_rz -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 warpx.abort_on_warning_threshold=high +runtime_params = warpx.abort_on_warning_threshold=high dim = 2 addToCompileString = USE_RZ=TRUE cmakeSetupOpts = -DWarpX_DIMS=RZ @@ -426,7 +426,7 @@ numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 -runtime_params = warpx.do_dynamic_scheduling=0 geometry.dims=2 +runtime_params = geometry.dims=2 analysisRoutine = Examples/Tests/dive_cleaning/analysis.py analysisOutputImage = Comparison.png @@ -444,7 +444,7 @@ numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = analysisRoutine = Examples/Tests/dive_cleaning/analysis.py analysisOutputImage = Comparison.png @@ -887,7 +887,7 @@ analysisRoutine = Examples/Tests/nci_psatd_stability/analysis_galilean.py [galilean_rz_psatd] buildDir = . inputFile = Examples/Tests/nci_psatd_stability/inputs_rz -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 electrons.random_theta=0 ions.random_theta=0 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium +runtime_params = electrons.random_theta=0 ions.random_theta=0 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium dim = 2 addToCompileString = USE_RZ=TRUE USE_PSATD=TRUE BLAS_LIB=-lblas LAPACK_LIB=-llapack cmakeSetupOpts = -DWarpX_DIMS=RZ -DWarpX_PSATD=ON @@ -1013,7 +1013,7 @@ analysisRoutine = Examples/Tests/AcceleratorLattice/analysis.py [initial_distribution] buildDir = . inputFile = Examples/Tests/initial_distribution/inputs -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -1080,7 +1080,7 @@ analysisRoutine = Examples/Tests/ion_stopping/analysis_ion_stopping.py [Langmuir_multi] buildDir = . inputFile = Examples/Tests/langmuir/inputs_3d -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -1099,7 +1099,7 @@ analysisOutputImage = langmuir_multi_analysis.png [Langmuir_fluid_1D] buildDir = . inputFile = Examples/Tests/langmuir_fluids/inputs_1d -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 1 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=1 @@ -1117,7 +1117,7 @@ analysisOutputImage = langmuir_fluid_multi_1d_analysis.png [Langmuir_fluid_RZ] buildDir = . inputFile = Examples/Tests/langmuir_fluids/inputs_rz -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = USE_RZ=TRUE cmakeSetupOpts = -DWarpX_DIMS=RZ @@ -1135,7 +1135,7 @@ analysisOutputImage = langmuir_fluid_rz_analysis.png [Langmuir_fluid_2D] buildDir = . inputFile = Examples/Tests/langmuir_fluids/inputs_2d -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 -DCMAKE_BUILD_TYPE=Debug @@ -1153,7 +1153,7 @@ analysisOutputImage = langmuir_fluid_multi_2d_analysis.png [Langmuir_fluid_multi] buildDir = . inputFile = Examples/Tests/langmuir_fluids/inputs_3d -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -1456,7 +1456,7 @@ analysisOutputImage = langmuir_multi_2d_analysis.png [Langmuir_multi_nodal] buildDir = . inputFile = Examples/Tests/langmuir/inputs_3d -runtime_params = warpx.do_dynamic_scheduling=0 warpx.grid_type=collocated algo.current_deposition=direct +runtime_params = warpx.grid_type=collocated algo.current_deposition=direct dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -1608,7 +1608,7 @@ analysisOutputImage = Langmuir_multi_psatd_multiJ_nodal.png [Langmuir_multi_psatd_nodal] buildDir = . inputFile = Examples/Tests/langmuir/inputs_3d -runtime_params = algo.maxwell_solver=psatd warpx.do_dynamic_scheduling=0 warpx.grid_type=collocated algo.current_deposition=direct warpx.cfl = 0.5773502691896258 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium +runtime_params = algo.maxwell_solver=psatd warpx.grid_type=collocated algo.current_deposition=direct warpx.cfl = 0.5773502691896258 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium dim = 3 addToCompileString = USE_PSATD=TRUE cmakeSetupOpts = -DWarpX_DIMS=3 -DWarpX_PSATD=ON @@ -1764,7 +1764,7 @@ aux1File = Regression/PostProcessingUtils/post_processing_utils.py [Langmuir_multi_single_precision] buildDir = . inputFile = Examples/Tests/langmuir/inputs_3d -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 3 addToCompileString = PRECISION=FLOAT USE_SINGLE_PRECISION_PARTICLES=TRUE cmakeSetupOpts = -DWarpX_DIMS=3 -DWarpX_PRECISION=SINGLE @@ -1871,7 +1871,7 @@ analysisRoutine = Examples/Physics_applications/laser_acceleration/analysis_1d_f [LaserAccelerationBoost] buildDir = . inputFile = Examples/Physics_applications/laser_acceleration/inputs_2d_boost -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 amr.n_cell=64 512 max_step=300 +runtime_params = amr.n_cell=64 512 max_step=300 dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -1996,7 +1996,7 @@ analysisOutputImage = laser_analysis.png [LaserInjection_1d] buildDir = . inputFile = Examples/Tests/laser_injection/inputs_1d_rt -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 1 addToCompileString = USE_OPENPMD=TRUE QED=FALSE cmakeSetupOpts = -DWarpX_DIMS=1 -DWarpX_OPENPMD=ON -DWarpX_QED=OFF @@ -2013,7 +2013,7 @@ analysisRoutine = Examples/Tests/laser_injection/analysis_1d.py [LaserInjection_2d] buildDir = . inputFile = Examples/Tests/laser_injection/inputs_2d_rt -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2032,7 +2032,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_2d_binary.py aux1File = Examples/Tests/laser_injection_from_file/inputs.2d_test_binary customRunCmd = ./analysis_2d_binary.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2051,7 +2051,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_3d.py aux1File = Examples/Tests/laser_injection_from_file/inputs.3d_test customRunCmd = ./analysis_3d.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -2070,7 +2070,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_1d.py aux1File = Examples/Tests/laser_injection_from_file/inputs.1d_test customRunCmd = ./analysis_1d.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 1 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=1 @@ -2089,7 +2089,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_1d_boost.py aux1File = Examples/Tests/laser_injection_from_file/inputs.1d_boost_test customRunCmd = ./analysis_1d_boost.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 1 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=1 @@ -2108,7 +2108,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_2d.py aux1File = Examples/Tests/laser_injection_from_file/inputs.2d_test customRunCmd = ./analysis_2d.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2127,7 +2127,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_RZ.py aux1File = Examples/Tests/laser_injection_from_file/inputs.RZ_test customRunCmd = ./analysis_RZ.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = USE_RZ=TRUE cmakeSetupOpts = -DWarpX_DIMS=RZ @@ -2146,7 +2146,7 @@ buildDir = . inputFile = Examples/Tests/laser_injection_from_file/analysis_from_RZ_file.py aux1File = Examples/Tests/laser_injection_from_file/inputs.from_RZ_file_test customRunCmd = ./analysis_from_RZ_file.py -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = USE_RZ=TRUE cmakeSetupOpts = -DWarpX_DIMS=RZ @@ -2163,7 +2163,7 @@ doVis = 0 [LaserIonAcc2d] buildDir = . inputFile = Examples/Physics_applications/laser_ion/inputs -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 amr.n_cell=384 512 max_step=100 +runtime_params = amr.n_cell=384 512 max_step=100 dim = 2 addToCompileString = USE_OPENPMD=TRUE cmakeSetupOpts = -DWarpX_DIMS=2 -DWarpX_OPENPMD=ON @@ -2263,7 +2263,7 @@ analysisRoutine = Examples/Tests/maxwell_hybrid_qed/analysis_Maxwell_QED_Hybrid. [momentum-conserving-gather] buildDir = . inputFile = Examples/Physics_applications/plasma_acceleration/inputs_2d -runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=400 warpx.serialize_initial_conditions=1 warpx.do_dynamic_scheduling=0 algo.field_gathering=momentum-conserving +runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=400 algo.field_gathering=momentum-conserving dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2281,7 +2281,7 @@ analysisRoutine = Examples/analysis_default_regression.py [multi_J_rz_psatd] buildDir = . inputFile = Examples/Tests/multi_j/inputs_rz -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 warpx.abort_on_warning_threshold=medium psatd.J_in_time=linear +runtime_params = warpx.abort_on_warning_threshold=medium psatd.J_in_time=linear dim = 2 addToCompileString = USE_RZ=TRUE USE_PSATD=TRUE cmakeSetupOpts = -DWarpX_DIMS=RZ -DWarpX_PSATD=ON @@ -2344,7 +2344,7 @@ numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = analysisRoutine = Examples/Tests/initial_plasma_profile/analysis.py [particle_absorption] @@ -2386,7 +2386,7 @@ analysisRoutine = Examples/Tests/boundaries/analysis.py buildDir = . inputFile = Examples/Tests/particle_fields_diags/inputs aux1File = Examples/Tests/particle_fields_diags/analysis_particle_diags_impl.py -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = USE_OPENPMD=TRUE cmakeSetupOpts = -DWarpX_DIMS=3 -DWarpX_OPENPMD=ON @@ -2404,7 +2404,7 @@ analysisRoutine = Examples/Tests/particle_fields_diags/analysis_particle_diags.p buildDir = . inputFile = Examples/Tests/particle_fields_diags/inputs aux1File = Examples/Tests/particle_fields_diags/analysis_particle_diags_impl.py -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = PRECISION=FLOAT USE_SINGLE_PRECISION_PARTICLES=TRUE USE_OPENPMD=TRUE cmakeSetupOpts = -DWarpX_DIMS=3 -DWarpX_PRECISION=SINGLE -DWarpX_OPENPMD=ON @@ -2701,7 +2701,7 @@ analysisRoutine = Examples/Tests/photon_pusher/analysis_photon_pusher.py [PlasmaAccelerationBoost2d] buildDir = . inputFile = Examples/Physics_applications/plasma_acceleration/inputs_2d_boost -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 amr.n_cell=64 256 max_step=20 +runtime_params = amr.n_cell=64 256 max_step=20 dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2717,7 +2717,7 @@ analysisRoutine = Examples/analysis_default_regression.py [PlasmaAccelerationBoost3d] buildDir = . inputFile = Examples/Physics_applications/plasma_acceleration/inputs_3d_boost -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 amr.n_cell=64 64 128 max_step=5 +runtime_params = amr.n_cell=64 64 128 max_step=5 dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -2733,7 +2733,7 @@ analysisRoutine = Examples/analysis_default_regression.py [PlasmaAccelerationBoost3d_hybrid] buildDir = . inputFile = Examples/Physics_applications/plasma_acceleration/inputs_3d_boost -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 amr.n_cell=64 64 128 max_step=25 warpx.grid_type=hybrid warpx.do_current_centering=0 +runtime_params = amr.n_cell=64 64 128 max_step=25 warpx.grid_type=hybrid warpx.do_current_centering=0 dim = 3 addToCompileString = restartTest = 0 @@ -2748,7 +2748,7 @@ analysisRoutine = Examples/analysis_default_regression.py [PlasmaAccelerationMR] buildDir = . inputFile = Examples/Physics_applications/plasma_acceleration/inputs_2d -runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=400 warpx.serialize_initial_conditions=1 warpx.do_dynamic_scheduling=0 +runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=400 dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2820,7 +2820,7 @@ analysisRoutine = Examples/Tests/plasma_lens/analysis.py [PlasmaMirror] buildDir = . inputFile = Examples/Physics_applications/plasma_mirror/inputs_2d -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 amr.n_cell=256 128 max_step=20 +runtime_params = amr.n_cell=256 128 max_step=20 dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2852,7 +2852,7 @@ analysisRoutine = Examples/analysis_default_regression.py [pml_psatd_rz] buildDir = . inputFile = Examples/Tests/pml/inputs_rz -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 warpx.cfl=0.7 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium +runtime_params = warpx.cfl=0.7 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium dim = 2 addToCompileString = USE_RZ=TRUE USE_PSATD=TRUE cmakeSetupOpts = -DWarpX_DIMS=RZ -DWarpX_PSATD=ON @@ -2868,7 +2868,7 @@ analysisRoutine = Examples/Tests/pml/analysis_pml_psatd_rz.py [pml_x_ckc] buildDir = . inputFile = Examples/Tests/pml/inputs_2d -runtime_params = warpx.do_dynamic_scheduling=0 algo.maxwell_solver=ckc +runtime_params = algo.maxwell_solver=ckc dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2884,7 +2884,7 @@ analysisRoutine = Examples/Tests/pml/analysis_pml_ckc.py [pml_x_galilean] buildDir = . inputFile = Examples/Tests/pml/inputs_2d -runtime_params = algo.maxwell_solver=psatd psatd.update_with_rho=1 warpx.do_dynamic_scheduling=0 diag1.fields_to_plot=Ex Ey Ez Bx By Bz rho divE warpx.cfl=0.7071067811865475 warpx.do_pml_dive_cleaning=1 warpx.do_pml_divb_cleaning=1 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium psatd.v_galilean=0. 0. 0.99 warpx.grid_type=collocated +runtime_params = algo.maxwell_solver=psatd psatd.update_with_rho=1 diag1.fields_to_plot=Ex Ey Ez Bx By Bz rho divE warpx.cfl=0.7071067811865475 warpx.do_pml_dive_cleaning=1 warpx.do_pml_divb_cleaning=1 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium psatd.v_galilean=0. 0. 0.99 warpx.grid_type=collocated dim = 2 addToCompileString = USE_PSATD=TRUE cmakeSetupOpts = -DWarpX_DIMS=2 -DWarpX_PSATD=ON @@ -2900,7 +2900,7 @@ analysisRoutine = Examples/Tests/pml/analysis_pml_psatd.py [pml_x_psatd] buildDir = . inputFile = Examples/Tests/pml/inputs_2d -runtime_params = algo.maxwell_solver=psatd psatd.update_with_rho=1 warpx.do_dynamic_scheduling=0 diag1.fields_to_plot = Ex Ey Ez Bx By Bz rho divE warpx.cfl = 0.7071067811865475 warpx.do_pml_dive_cleaning=0 warpx.do_pml_divb_cleaning=0 chk.file_prefix=pml_x_psatd_chk chk.file_min_digits=5 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium +runtime_params = algo.maxwell_solver=psatd psatd.update_with_rho=1 diag1.fields_to_plot = Ex Ey Ez Bx By Bz rho divE warpx.cfl = 0.7071067811865475 warpx.do_pml_dive_cleaning=0 warpx.do_pml_divb_cleaning=0 chk.file_prefix=pml_x_psatd_chk chk.file_min_digits=5 psatd.current_correction=0 warpx.abort_on_warning_threshold=medium dim = 2 addToCompileString = USE_PSATD=TRUE cmakeSetupOpts = -DWarpX_DIMS=2 -DWarpX_PSATD=ON @@ -2917,7 +2917,7 @@ analysisRoutine = Examples/Tests/pml/analysis_pml_psatd.py [pml_x_yee] buildDir = . inputFile = Examples/Tests/pml/inputs_2d -runtime_params = warpx.do_dynamic_scheduling=0 algo.maxwell_solver=yee chk.file_prefix=pml_x_yee_chk chk.file_min_digits=5 +runtime_params = algo.maxwell_solver=yee chk.file_prefix=pml_x_yee_chk chk.file_min_digits=5 dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2934,7 +2934,7 @@ analysisRoutine = Examples/Tests/pml/analysis_pml_yee.py [pml_x_yee_eb] buildDir = . inputFile = Examples/Tests/pml/inputs_2d -runtime_params = warpx.do_dynamic_scheduling=0 algo.maxwell_solver=yee chk.file_prefix=pml_x_yee_eb_chk chk.file_min_digits=5 +runtime_params = algo.maxwell_solver=yee chk.file_prefix=pml_x_yee_eb_chk chk.file_min_digits=5 dim = 2 addToCompileString = USE_EB=TRUE cmakeSetupOpts = -DWarpX_DIMS=2 -DWarpX_EB=ON @@ -2951,7 +2951,7 @@ analysisRoutine = Examples/Tests/pml/analysis_pml_yee.py [Proton_Boron_Fusion_2D] buildDir = . inputFile = Examples/Tests/nuclear_fusion/inputs_proton_boron_2d -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -2967,7 +2967,7 @@ analysisRoutine = Examples/Tests/nuclear_fusion/analysis_proton_boron_fusion.py [Proton_Boron_Fusion_3D] buildDir = . inputFile = Examples/Tests/nuclear_fusion/inputs_proton_boron_3d -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -3706,7 +3706,7 @@ analysisRoutine = Examples/analysis_default_regression.py [Python_reduced_diags_loadbalancecosts_timers] buildDir = . inputFile = Examples/Tests/reduced_diags/PICMI_inputs_loadbalancecosts.py -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 algo.load_balance_costs_update=Timers +runtime_params = algo.load_balance_costs_update=Timers customRunCmd = python3 PICMI_inputs_loadbalancecosts.py dim = 3 addToCompileString = USE_PYTHON_MAIN=TRUE @@ -3975,7 +3975,7 @@ analysisRoutine = Examples/Tests/radiation_reaction/test_const_B_analytical/ana buildDir = . inputFile = Examples/Tests/reduced_diags/inputs aux1File = Examples/Tests/reduced_diags/analysis_reduced_diags_impl.py -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -3992,7 +3992,7 @@ analysisRoutine = Examples/Tests/reduced_diags/analysis_reduced_diags.py [reduced_diags_loadbalancecosts_heuristic] buildDir = . inputFile = Examples/Tests/reduced_diags/inputs_loadbalancecosts -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 algo.load_balance_costs_update=Heuristic +runtime_params = algo.load_balance_costs_update=Heuristic dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -4009,7 +4009,7 @@ analysisRoutine = Examples/Tests/reduced_diags/analysis_reduced_diags_loadbalanc [reduced_diags_loadbalancecosts_timers] buildDir = . inputFile = Examples/Tests/reduced_diags/inputs_loadbalancecosts -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 algo.load_balance_costs_update=Timers +runtime_params = algo.load_balance_costs_update=Timers dim = 3 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=3 @@ -4026,7 +4026,7 @@ analysisRoutine = Examples/Tests/reduced_diags/analysis_reduced_diags_loadbalanc [reduced_diags_loadbalancecosts_timers_psatd] buildDir = . inputFile = Examples/Tests/reduced_diags/inputs_loadbalancecosts -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 algo.load_balance_costs_update=Timers +runtime_params = algo.load_balance_costs_update=Timers dim = 3 addToCompileString = USE_PSATD=TRUE cmakeSetupOpts = -DWarpX_DIMS=3 -DWarpX_PSATD=ON @@ -4044,7 +4044,7 @@ analysisRoutine = Examples/Tests/reduced_diags/analysis_reduced_diags_loadbalanc buildDir = . inputFile = Examples/Tests/reduced_diags/inputs aux1File = Examples/Tests/reduced_diags/analysis_reduced_diags_impl.py -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 3 addToCompileString = PRECISION=FLOAT USE_SINGLE_PRECISION_PARTICLES=TRUE cmakeSetupOpts = -DWarpX_DIMS=3 -DWarpX_PRECISION=SINGLE @@ -4090,7 +4090,7 @@ numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = analysisRoutine = Examples/Tests/relativistic_space_charge_initialization/analysis.py analysisOutputImage = Comparison.png @@ -4170,7 +4170,7 @@ analysisRoutine = Examples/Tests/restart/analysis_restart.py [RigidInjection_BTD] buildDir = . inputFile = Examples/Tests/rigid_injection/inputs_2d_BoostedFrame -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 2 addToCompileString = USE_OPENPMD=TRUE cmakeSetupOpts = -DWarpX_DIMS=2 -DWarpX_OPENPMD=ON @@ -4188,7 +4188,7 @@ analysisRoutine = Examples/Tests/rigid_injection/analysis_rigid_injection_Booste [RigidInjection_lab] buildDir = . inputFile = Examples/Tests/rigid_injection/inputs_2d_LabFrame -runtime_params = warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 @@ -4297,7 +4297,7 @@ numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 -runtime_params = warpx.do_dynamic_scheduling=0 +runtime_params = analysisRoutine = Examples/Tests/space_charge_initialization/analysis.py analysisOutputImage = Comparison.png @@ -4315,14 +4315,14 @@ numthreads = 1 compileTest = 0 doVis = 0 compareParticles = 0 -runtime_params = warpx.do_dynamic_scheduling=0 geometry.dims=2 +runtime_params = geometry.dims=2 analysisRoutine = Examples/Tests/space_charge_initialization/analysis.py analysisOutputImage = Comparison.png [subcyclingMR] buildDir = . inputFile = Examples/Tests/subcycling/inputs_2d -runtime_params = warpx.serialize_initial_conditions=1 warpx.do_dynamic_scheduling=0 +runtime_params = dim = 2 addToCompileString = cmakeSetupOpts = -DWarpX_DIMS=2 diff --git a/Regression/prepare_file_ci.py b/Regression/prepare_file_ci.py index 7ec0af427d2..1fa6b4c9a0a 100644 --- a/Regression/prepare_file_ci.py +++ b/Regression/prepare_file_ci.py @@ -62,7 +62,7 @@ text = re.sub('USE_PSATD=FALSE', '', text) -# Ccache +# CCache if ci_ccache: text = re.sub('addToCompileString =', 'addToCompileString = USE_CCACHE=TRUE ', text) @@ -77,6 +77,14 @@ 'warpx.always_warn_immediately=1 warpx.abort_on_warning_threshold=low', text) +# Add runtime options for CPU: +# > serialize initial conditions and no dynamic scheduling in OpenMP +if arch == 'CPU': + text = re.sub('runtime_params =', + 'runtime_params = '+ + 'warpx.do_dynamic_scheduling=0 warpx.serialize_initial_conditions=1', + text) + # Use less/more cores for compiling, e.g. public CI only provides 2 cores if ci_num_make_jobs is not None: text = re.sub( 'numMakeJobs = \d+', 'numMakeJobs = {}'.format(ci_num_make_jobs), text ) From 22e05bb883717b605894bd0a5aa549e67a550f31 Mon Sep 17 00:00:00 2001 From: Edward Basso Date: Tue, 28 Nov 2023 14:01:05 -0600 Subject: [PATCH 110/110] Contributing style and conventions additions (#4452) * Add class definition convention and reword function definition convention * Use code-block and modify documentation.rst to conform to conventions * Small fix to code-block * Fix typo and add new lines * Revert new lines * Indents code-blocks and words in between * Use less code blocks * Use "amrex::Print" instead of "Print" --- CONTRIBUTING.rst | 30 ++++++++++++++++++++---- Docs/source/developers/documentation.rst | 12 +++++----- 2 files changed, 32 insertions(+), 10 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 1495a98a96f..bb3a25e0daf 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -212,12 +212,34 @@ Style and conventions - Space before and after assignment operator (``=``) -- To define a function , for e.g., ``myfunction()`` use a space between the name of the function and the paranthesis - ``myfunction ()``. - To call the function, the space is not required, i.e., just use ``myfunction()``. +- To define a function, use a space between the name of the function and the paranthesis, e.g., ``myfunction ()``. + When calling a function, no space should be used, i.e., just use ``myfunction()``. + The reason this is beneficial is that when we do a ``git grep`` to search for ``myfunction ()``, we can clearly see the locations where ``myfunction ()`` is defined and where ``myfunction()`` is called. + Also, using ``git grep "myfunction ()"`` searches for files only in the git repo, which is more efficient compared to the ``grep "myfunction ()"`` command that searches through all the files in a directory, including plotfiles for example. -- The reason this is beneficial is that when we do a ``git grep`` to search for ``myfunction ()``, we can clearly see the locations where ``myfunction ()`` is defined and where ``myfunction()`` is called. +- To define a class, use ``class`` on the same line as the name of the class, e.g., ``class MyClass``. + The reason this is beneficial is that when we do a ``git grep`` to search for ``class MyClass``, we can clearly see the locations where ``class MyClass`` is defined and where ``MyClass`` is called. -- Also, using ``git grep "myfunction ()"`` searches for files only in the git repo, which is more efficient compared to the ``grep "myfunction ()"`` command that searches through all the files in a directory, including plotfiles for example. +- When defining a function or class, make sure the starting ``{`` token appears on a new line. + +- Use curly braces for single statement blocks. For example: + + .. code-block:: cpp + + for (int n = 0; n < 10; ++n) { + amrex::Print() << "Like this!"; + } + + for (int n = 0; n < 10; ++n) { amrex::Print() << "Or like this!"; } + + but not + + .. code-block:: cpp + + for (int n = 0; n < 10; ++n) + amrex::Print() << "Not like this."; + + for (int n = 0; n < 10; ++n) amrex::Print() << "Nor like this."; - It is recommended that style changes are not included in the PR where new code is added. This is to avoid any errors that may be introduced in a PR just to do style change. diff --git a/Docs/source/developers/documentation.rst b/Docs/source/developers/documentation.rst index 0f3ca9dd9cd..a5013299336 100644 --- a/Docs/source/developers/documentation.rst +++ b/Docs/source/developers/documentation.rst @@ -11,21 +11,21 @@ Whenever you create a new class, please document it where it is declared (typica .. code-block:: cpp - /** A brief title + /** \brief A brief title * - * few-line description explaining the purpose of my_class. + * few-line description explaining the purpose of MyClass. * - * If you are kind enough, also quickly explain how things in my_class work. + * If you are kind enough, also quickly explain how things in MyClass work. * (typically a few more lines) */ - class my_class + class MyClass { ... } Doxygen reads this docstring, so please be accurate with the syntax! See `Doxygen manual `__ for more information. Similarly, please document functions when you declare them (typically in a header file) like: .. code-block:: cpp - /** A brief title + /** \brief A brief title * * few-line description explaining the purpose of my_function. * @@ -33,7 +33,7 @@ Doxygen reads this docstring, so please be accurate with the syntax! See `Doxyge * my_function will operate. * \return what is the meaning and value range of the returned value */ - int my_class::my_function(int* my_int); + int MyClass::my_function (int* my_int); An online version of this documentation is :ref:`linked here `.