Home

Building QUDA

The following pages give information on QUDA's build system and dependencies

• QUDA Quick Start Guide

  1. Installation
  2. Internal Testing
  3. Kernel Autotuning
  4. Debugging
  5. Manually Running Basic Tests

• QUDA Build With CMake

  1. QUDA and CMake
  2. Obtaining CMake
  3. Building QUDA using CMake
  4. Reducing QUDA build time
  5. Building QUDA with clang-cuda
  6. Building QUDA with HIP/ROCm

• QUDA (Optional) Dependencies

  1. ARPACK
  2. OpenBLAS
  3. QIO and QMP

• QUDA Environment Variables

Multi-GPU QUDA

These pages give information on how to build QUDA using MPI, including some machine-specific instructions

• Multi GPU Support
  1. Compiling for Multi-GPU
  2. Running
  3. Running QUDA's Tests
  4. Multi-GPU Emulation
  5. Peer-to-peer Communication
  6. GPU Direct RDMA and CUDA-aware MPI
  7. Maximizing GDR Performance
  8. Dependence on CUDA_DEVICE_MAX_CONNECTIONS
  9. Low-level Details
  10. Dslash Policy Tuning
  11. Dslash Component Benchmarking
• Installing MVAPICH2 GDR
• QIO and QMP
  1. Loading and Saving Gauge Fields
  2. Loading and Saving Vector Fields
  3. Notes on PARTFILE
  4. Notes on MG
• QUDA on Perlmutter
• QUDA on Summit
• QUDA and NVSHMEM
  1. NVSHMEM on Summit

QUDA Solver Acceleration Strategies

QUDA has several features one can take advantage of to accelerate operator inversion. We list here some of the algorithms currently available.

• EigCG, GMRES-DR, and FGMRES-DR

  1. EigCG
  2. GMRES-DR
  3. FGMRES-DR

• Multigrid Solver

  1. Setting up the Multigrid solver
  2. Running the Multigrid solver
  3. Example solve using Multigrid
  4. Tuning the Multigrid solver
  5. Multigrid with coarse grid deflation

• Staggered Multigrid Solver

  1. Compiling
  2. Running without deflation on the coarsest level
  3. Running with deflation on the coarsest level
  4. Running with CG

• QUDA's eigensolvers

  1. Thick Restarted Lanczos Eigensolver
     • Chebyshev Acceleration
     • Staggered Eigensolver
  2. Implictly Restarted Arnoldi Eigensolver
  3. Eigensolver Features
     • Observing Coarse Grid Spectra
     • Initial Guess Deflation
     • Computing the Gamma5 Hermitian Spectrum
     • Memory Saving Ritz Rotations
     • ARPACK Interface

• Preconditioning

• Split Grid

QUDA also offers the option to use mixed precision in its solvers Mixed Precision

QUDA Gauge Algorithms and Utilities

QUDA offers a variety of pure SU(3) gauge algorithms and utilities that are needed for LQCD studies.

• Gauge Measurements
  1. Wilson (Plaquette) Action
  2. Clover Action
  3. Topological Charge and Density
• Gauge Smearing
  1. APE smearing
  2. Stout and Over-Improved Stout smearing
  3. Wilson and Symanzik Flow
• Gauge Fixing
  1. Over-Improved
  2. Steepest Descent FFT
• Heatbath

Using QUDA In Your LQCD applications

• Clover Fermions
• MILC with QUDA
  1. Compiling QUDA
  2. Compiling MILC
  3. Running
  4. Performance Tuning
  5. Compiling MILC with the QUDA 1.0.x Release Branch
  6. NERSC MILC Benchmarks
  7. HISQ MG for Measurements
• Chroma with QUDA
• CPS with QUDA
• OpenQxD with QUDA
• QUDA Interface
• QUDA Parameter Structures
• QUDA Data Structures And Types
• Best Practices

QUDA Development

Contributing to QUDA

• Updating the QUDA Interface
• Adding new QUDA features
• Writing QUDA Kernels
• Checking that new QUDA features preserve Dslash behavior
• Adding new Nc to Multigrid
• Coding Conventions and Style
• Validating Large Multigrid Changes

Resources

• QUDA Development model Some Rules and Best Practices for writing code and use of git
• Continuous Integration
• QUDA Release Procedure
• QUDA Release Testing
• QUDA Debugging
• Timing QUDA build
• External QUDA mods

Projects in development

These projects have branches associated with them, but have not yet made been pulled into mainline QUDA (develop branch). Some projects are in a useful state, others are in early stages. We list here some of the more prominent

• Meson/Baryon contractions
• Color agnostic LQCD
• Full (R)HMC functionality
• (Stochastic) Laplace Heaviside (s)LapH spectroscopy
• Deflation using Split Grid
• Compressed Thick Restarted Eigensolver
• Exponentiated Clover Action

Case Studies: Pedagogical Instructions On Using Some of QUDA's Key Features

Below we have included some important case studies, designed to be guides on how to get the most out of QUDA solvers for different use cases in modern LQCD.

• Clover HMC evolution with multigrid Here we use the Multigrid solver to accelerate inversions during the HMC evolution. As the HMC algorithm evolves the operator, the Multigrid preconditioner loses coherence with the target. We demonstrate here how to set up a multigrid preconditioner to accelerate solves, show the operator decohereing, and how to tune the parameters to minimise this decoherence.

• Twisted clover deflated multigrid Here we recognise that the Twisted-clover operator is pathologically ill conditioned, to the extent that even with an aggressive three level multgrid solve, these low eigenmodes remain on the coarse grid. Previous attempts to avoid these low modes involved artificially adjusting the coarse grid operator to have a large value on mu, known colloquially as mu scaling. In this case study we demonstrate that one can avoid mu scaling completely by explicitly deflating the offending low modes on the coarse grid, and thence acquire significant speed up over the previous method.

• Persistent deflation spaces for multi-mass/flavor problems In this case study we show how one can use the deflated multigrid solver (as expounded on in the Twisted-clover deflated multigrid study) for one mass/flavour of quark, keep the eigenvectors used in deflation, and recompute the eigenvalues for different masses. This gives significant speed-up over recomputing the multigrid preconditioner for each mass/flavor.

• HISQ mixed-precision deflation

• Möbius minimal-memory deflation Here we show a simple, yet powerful trick to minimise the amount of device memory needed to perform deflation of the Möbius operator when using the Thick Restarted eigensolver. We employ an LU decomposition of the dense rotation matrix used in the Krlyov space compression to break up the rotation into small chunks. This reduction allows for an approximately 50% reduction in the partition size needed for a given problem with very little degradation of performance.

• The-Exact-One-Flavor-Algorithm (EOFA)

• Multi-Splitting Preconditioned CG A proper additive Schwarz preconditioner for Mobius fermions.

Benchmarks

Miscellaneous features

• Clover
• HISQ Stencil

QUDA calls

• QUDA calls

CodeFests

*CodeFest 5

*CodeFest Jlab January 2018

New Interface Planning

Miscellaneous

• Algorithm Ideas
• Tensor Cores