Skip to content

Commit

Permalink
Merge branch 'development' into plasma_chamber
Browse files Browse the repository at this point in the history
  • Loading branch information
ajnonaka committed Dec 16, 2024
2 parents ae8cb99 + 7299895 commit 3572c13
Show file tree
Hide file tree
Showing 48 changed files with 1,263 additions and 150 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/cuda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ jobs:
which nvcc || echo "nvcc not in PATH!"
git clone https://github.com/AMReX-Codes/amrex.git ../amrex
cd ../amrex && git checkout --detach 24.12 && cd -
cd ../amrex && git checkout --detach 96db0a665ff1e6bbe638490fd02d3aafb9188f6b && cd -
make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_FFT=TRUE USE_CCACHE=TRUE -j 4
ccache -s
Expand Down
2 changes: 1 addition & 1 deletion Docs/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ openpmd-viewer # for checksumAPI

# PICMI API docs
# note: keep in sync with version in ../requirements.txt
picmistandard==0.31.0
picmistandard==0.33.0
# for development against an unreleased PICMI version, use:
# picmistandard @ git+https://github.com/picmi-standard/picmi.git#subdirectory=PICMI_Python

Expand Down
2 changes: 1 addition & 1 deletion Docs/source/install/hpc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ This section documents quick-start guides for a selection of supercomputers that
hpc/perlmutter
hpc/pitzer
hpc/polaris
hpc/quartz
hpc/dane
hpc/summit
hpc/taurus
hpc/tioga
Expand Down
Original file line number Diff line number Diff line change
@@ -1,27 +1,25 @@
.. _building-quartz:
.. _building-dane:

Quartz (LLNL)
Dane (LLNL)
=============

The `Quartz Intel CPU cluster <https://hpc.llnl.gov/hardware/platforms/quartz>`_ is located at LLNL.
The `Dane Intel CPU cluster <https://hpc.llnl.gov/hardware/compute-platforms/dane>`_ is located at LLNL.


Introduction
------------

If you are new to this system, **please see the following resources**:

* `LLNL user account <https://lc.llnl.gov/lorenz/mylc/mylc.cgi>`__ (login required)
* `Quartz user guide <https://computing.llnl.gov/tutorials/linux_clusters/>`_
* Batch system: `Slurm <https://computing.llnl.gov/tutorials/moab/>`_
* `LLNL user account <https://lc.llnl.gov`__ (login required)
* `Jupyter service <https://lc.llnl.gov/jupyter>`__ (`documentation <https://lc.llnl.gov/confluence/display/LC/JupyterHub+and+Jupyter+Notebook>`__, login required)
* `Production directories <https://hpc.llnl.gov/hardware/file-systems>`_:

* ``/p/lustre1/$(whoami)`` and ``/p/lustre2/$(whoami)``: personal directory on the parallel filesystem
* Note that the ``$HOME`` directory and the ``/usr/workspace/$(whoami)`` space are NFS mounted and *not* suitable for production quality data generation.


.. _building-quartz-preparation:
.. _building-dane-preparation:

Preparation
-----------
Expand All @@ -32,23 +30,23 @@ Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use system software modules, add environment hints and further dependencies via the file ``$HOME/quartz_warpx.profile``.
We use system software modules, add environment hints and further dependencies via the file ``$HOME/dane_warpx.profile``.
Create it now:

.. code-block:: bash
cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz_warpx.profile.example $HOME/quartz_warpx.profile
cp $HOME/src/warpx/Tools/machines/dane-llnl/dane_warpx.profile.example $HOME/dane_warpx.profile
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz_warpx.profile.example
.. literalinclude:: ../../../../Tools/machines/dane-llnl/dane_warpx.profile.example
:language: bash

Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
For example, if you are member of the project ``tps``, then run ``vi $HOME/quartz_warpx.profile``.
For example, if you are member of the project ``tps``, then run ``vi $HOME/dane_warpx.profile``.
Enter the edit mode by typing ``i`` and edit line 2 to read:

.. code-block:: bash
Expand All @@ -59,29 +57,29 @@ Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit).

.. important::

Now, and as the first step on future logins to Quartz, activate these environment settings:
Now, and as the first step on future logins to Dane, activate these environment settings:

.. code-block:: bash
source $HOME/quartz_warpx.profile
source $HOME/dane_warpx.profile
Finally, since Quartz does not yet provide software modules for some of our dependencies, install them once:
Finally, since Dane does not yet provide software modules for some of our dependencies, install them once:

.. code-block:: bash
bash $HOME/src/warpx/Tools/machines/quartz-llnl/install_dependencies.sh
source /usr/workspace/${USER}/quartz/venvs/warpx-quartz/bin/activate
bash $HOME/src/warpx/Tools/machines/dane-llnl/install_dependencies.sh
source /usr/workspace/${USER}/dane/venvs/warpx-dane/bin/activate
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. literalinclude:: ../../../../Tools/machines/quartz-llnl/install_dependencies.sh
.. literalinclude:: ../../../../Tools/machines/dane-llnl/install_dependencies.sh
:language: bash


.. _building-quartz-compilation:
.. _building-dane-compilation:

Compilation
-----------
Expand All @@ -91,27 +89,27 @@ Use the following :ref:`cmake commands <building-cmake>` to compile the applicat
.. code-block:: bash
cd $HOME/src/warpx
rm -rf build_quartz
rm -rf build_dane
cmake -S . -B build_quartz -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_quartz -j 6
cmake -S . -B build_dane -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_dane -j 6
The WarpX application executables are now in ``$HOME/src/warpx/build_quartz/bin/``.
The WarpX application executables are now in ``$HOME/src/warpx/build_dane/bin/``.
Additionally, the following commands will install WarpX as a Python module:

.. code-block:: bash
rm -rf build_quartz_py
rm -rf build_dane_py
cmake -S . -B build_quartz_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_quartz_py -j 6 --target pip_install
cmake -S . -B build_dane_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_dane_py -j 6 --target pip_install
Now, you can :ref:`submit Quartz compute jobs <running-cpp-quartz>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
Or, you can use the WarpX executables to submit Quartz jobs (:ref:`example inputs <usage-examples>`).
For executables, you can reference their location in your :ref:`job script <running-cpp-quartz>` or copy them to a location in ``$PROJWORK/$proj/``.
Now, you can :ref:`submit Dane compute jobs <running-cpp-dane>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
Or, you can use the WarpX executables to submit Dane jobs (:ref:`example inputs <usage-examples>`).
For executables, you can reference their location in your :ref:`job script <running-cpp-dane>` or copy them to a location in ``$PROJWORK/$proj/``.


.. _building-quartz-update:
.. _building-dane-update:

Update WarpX & Dependencies
---------------------------
Expand All @@ -135,34 +133,34 @@ If you already installed WarpX in the past and want to update it, start by getti
And, if needed,

- :ref:`update the quartz_warpx.profile file <building-quartz-preparation>`,
- :ref:`update the dane_warpx.profile file <building-dane-preparation>`,
- log out and into the system, activate the now updated environment profile as usual,
- :ref:`execute the dependency install scripts <building-quartz-preparation>`.
- :ref:`execute the dependency install scripts <building-dane-preparation>`.

As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_quartz`` and rebuild WarpX.
As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_dane`` and rebuild WarpX.


.. _running-cpp-quartz:
.. _running-cpp-dane:

Running
-------

.. _running-cpp-quartz-CPUs:
.. _running-cpp-dane-CPUs:

Intel Xeon E5-2695 v4 CPUs
Intel Sapphire Rapids CPUs
^^^^^^^^^^^^^^^^^^^^^^^^^^

The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL.
The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Dane at LLNL.
Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``.

.. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz.sbatch
.. literalinclude:: ../../../../Tools/machines/dane-llnl/dane.sbatch
:language: bash
:caption: You can copy this file from ``Tools/machines/quartz-llnl/quartz.sbatch``.
:caption: You can copy this file from ``Tools/machines/dane-llnl/dane.sbatch``.

To run a simulation, copy the lines above to a file ``quartz.sbatch`` and run
To run a simulation, copy the lines above to a file ``dane.sbatch`` and run

.. code-block:: bash
sbatch quartz.sbatch
sbatch dane.sbatch
to submit the job.
12 changes: 12 additions & 0 deletions Docs/source/refs.bib
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,18 @@ @article{VayFELB2009
url = {https://doi.org/10.1063/1.3080930},
}

@article{Barnes2021,
title = {Improved C1 shape functions for simplex meshes},
author = {D.C. Barnes},
journal = {Journal of Computational Physics},
volume = {424},
pages = {109852},
year = {2021},
issn = {0021-9991},
doi = {https://doi.org/10.1016/j.jcp.2020.109852},
url = {https://www.sciencedirect.com/science/article/pii/S0021999120306264},
}

@article{Rhee1987,
author = {Rhee, M. J. and Schneider, R. F. and Weidman, D. J.},
title = "{Simple time‐resolving Thomson spectrometer}",
Expand Down
17 changes: 17 additions & 0 deletions Docs/source/usage/parameters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,23 @@ Overall simulation parameters
\boldsymbol{\nabla}^2 \phi = - \rho/\epsilon_0 \qquad \boldsymbol{E} = - \boldsymbol{\nabla}\phi \\
\boldsymbol{\nabla}^2 \boldsymbol{A} = - \mu_0 \boldsymbol{j} \qquad \boldsymbol{B} = \boldsymbol{\nabla}\times\boldsymbol{A}
* ``labframe-effective-potential``: Poisson's equation is solved with a modified dielectric function
(resulting in an "effective potential") to create a semi-implicit scheme which is robust to the
numerical instability seen in explicit electrostatic PIC when :math:`\Delta t \omega_{pe} > 2`.
If this option is used the additional parameter ``warpx.effective_potential_factor`` can also be
specified to set the value of :math:`C_{EP}` (default 4). The method is stable for :math:`C_{EP} \geq 1`
regardless of :math:`\Delta t`, however, the larger :math:`C_{EP}` is set, the lower the numerical plasma
frequency will be and therefore care must be taken to not set it so high that the plasma mode
hybridizes with other modes of interest.
Details of the method can be found in Appendix A of :cite:t:`param-Barnes2021` (note that in that paper
the method is referred to as "semi-implicit electrostatic" but here it has been renamed to "effective potential"
to avoid confusion with the semi-implicit method of Chen et al.).
In short, the code solves:

.. math::
\boldsymbol{\nabla}\cdot\left(1+\frac{C_{EP}}{4}\sum_{s \in \text{species}}(\omega_{ps}\Delta t)^2 \right)\boldsymbol{\nabla} \phi = - \rho/\epsilon_0 \qquad \boldsymbol{E} = - \boldsymbol{\nabla}\phi
* ``relativistic``: Poisson's equation is solved **for each species**
in their respective rest frame. The corresponding field
is mapped back to the simulation frame and will produce both E and B
Expand Down
4 changes: 4 additions & 0 deletions Docs/source/usage/python.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,10 @@ Particle distributions can be used for to initialize particles in a particle spe

.. autoclass:: pywarpx.picmi.AnalyticDistribution

.. autoclass:: pywarpx.picmi.UniformFluxDistribution

.. autoclass:: pywarpx.picmi.AnalyticFluxDistribution

.. autoclass:: pywarpx.picmi.ParticleListDistribution

Particle layouts determine how to microscopically place macro particles in a grid cell.
Expand Down
1 change: 1 addition & 0 deletions Examples/Tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ add_subdirectory(restart)
add_subdirectory(restart_eb)
add_subdirectory(rigid_injection)
add_subdirectory(scraping)
add_subdirectory(effective_potential_electrostatic)
add_subdirectory(silver_mueller)
add_subdirectory(single_particle)
add_subdirectory(space_charge_initialization)
Expand Down
12 changes: 12 additions & 0 deletions Examples/Tests/effective_potential_electrostatic/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Add tests (alphabetical order) ##############################################
#

add_warpx_test(
test_3d_effective_potential_electrostatic_picmi # name
3 # dims
2 # nprocs
inputs_test_3d_effective_potential_electrostatic_picmi.py # inputs
analysis.py # analysis
diags/field_diag/ # output
OFF # dependency
)
90 changes: 90 additions & 0 deletions Examples/Tests/effective_potential_electrostatic/analysis.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
#
# --- Analysis script for the effective potential Poisson solver test. This test
# --- is based on the adiabatic plasma expansion benchmark from Connor et al. (2021)
# --- doi.org/10.1109/TPS.2021.3072353.
# --- The electron density distribution (as a function of radius) is compared
# --- with the analytically calculated density based on the input parameters
# --- of the test simulation at each output timestep.

import os
import sys

import dill
import matplotlib.pyplot as plt
import numpy as np
from openpmd_viewer import OpenPMDTimeSeries
from scipy.interpolate import RegularGridInterpolator

from pywarpx import picmi

constants = picmi.constants

# load simulation parameters
with open("sim_parameters.dpkl", "rb") as f:
sim = dill.load(f)

# characteristic expansion time
tau = sim["sigma_0"] * np.sqrt(sim["M"] / (constants.kb * (sim["T_e"] + sim["T_i"])))


def get_analytic_density(r, t):
expansion_factor = 1.0 + t**2 / tau**2
T = sim["T_e"] / expansion_factor
sigma = sim["sigma_0"] * np.sqrt(expansion_factor)
return (
sim["n_plasma"] * (T / sim["T_e"]) ** 1.5 * np.exp(-(r**2) / (2.0 * sigma**2))
)


def get_radial_function(field, info):
"""Helper function to transform Cartesian data to polar data and average
over theta and phi."""
r_grid = np.linspace(0, np.max(info.z) - 1e-3, 30)
theta_grid = np.linspace(0, 2 * np.pi, 40, endpoint=False)
phi_grid = np.linspace(0, np.pi, 20)

r, t, p = np.meshgrid(r_grid, theta_grid, phi_grid, indexing="ij")
x_sp = r * np.sin(p) * np.cos(t)
y_sp = r * np.sin(p) * np.sin(t)
z_sp = r * np.cos(p)

interp = RegularGridInterpolator((info.x, info.y, info.z), field)
field_sp = interp((x_sp, y_sp, z_sp))
return r_grid, np.mean(field_sp, axis=(1, 2))


diag_dir = "diags/field_diag"
ts = OpenPMDTimeSeries(diag_dir, check_all_files=True)

rms_errors = np.zeros(len(ts.iterations))

for ii, it in enumerate(ts.iterations):
rho_e, info = ts.get_field(field="rho_electrons", iteration=it)
r_grid, n_e = get_radial_function(-rho_e / constants.q_e, info)

n_e_analytic = get_analytic_density(r_grid, ts.t[ii])
rms_errors[ii] = (
np.sqrt(np.mean(np.sum((n_e - n_e_analytic) ** 2))) / n_e_analytic[0]
)

plt.plot(r_grid, n_e_analytic, "k--", alpha=0.6)
plt.plot(r_grid, n_e, label=f"t = {ts.t[ii]*1e6:.2f} $\mu$s")

print("RMS error (%) in density: ", rms_errors)
assert np.all(rms_errors < 0.05)

plt.ylabel("$n_e$ (m$^{-3}$)")
plt.xlabel("r (m)")
plt.grid()
plt.legend()
plt.show()

if len(sys.argv) > 1:
sys.path.insert(1, "../../../../warpx/Regression/Checksum/")
import checksumAPI

filename = sys.argv[1]

test_name = os.path.split(os.getcwd())[1]
checksumAPI.evaluate_checksum(test_name, filename, output_format="openpmd")
Loading

0 comments on commit 3572c13

Please sign in to comment.