Matrix-free GPU implementation of a preconditioned conjugate gradient solver for anisotropic elliptic PDEs

Abstract

Many problems in geophysical and atmospheric modelling require the fast solution of elliptic partial differential equations (PDEs) in “flat” three dimensional geometries. In particular, an anisotropic elliptic PDE for the pressure correction has to be solved at every time step in the dynamical core of many numerical weather prediction (NWP) models, and equations of a very similar structure arise in global ocean models, subsurface flow simulations and gas and oil reservoir modelling. The elliptic solve is often the bottleneck of the forecast, and to meet operational requirements an algorithmically optimal method has to be used and implemented efficiently. Graphics Processing Units (GPUs) have been shown to be highly efficient (both in terms of absolute performance and power consumption) for a wide range of applications in scientific computing, and recently iterative solvers have been parallelised on these architectures. In this article we describe the GPU implementation and optimisation of a Preconditioned Conjugate Gradient (PCG) algorithm for the solution of a three dimensional anisotropic elliptic PDE for the pressure correction in NWP. Our implementation exploits the strong vertical anisotropy of the elliptic operator in the construction of a suitable preconditioner. As the algorithm is memory bound, performance can be improved significantly by reducing the amount of global memory access. We achieve this by using a matrix-free implementation which does not require explicit storage of the matrix and instead recalculates the local stencil. Global memory access can also be reduced by rewriting the PCG algorithm using loop fusion and we show that this further reduces the runtime on the GPU. We demonstrate the performance of our matrix-free GPU code by comparing it both to a sequential CPU implementation and to a matrix-explicit GPU code which uses existing CUDA libraries. The absolute performance of the algorithm for different problem sizes is quantified in terms of floating point throughput and global memory bandwidth.

Appendix: A interleaved PCG kernels

The kernels for the interleaved PCG algorithm described in Sect. 4.3 are shown in Algorithms 2 and 3. Using similar notation as in Sect. 3.1, the index pair $(i',j')$ runs over all immediate neighbours of a horizontal cell $(i,j)$, i.e. $(i',j')\in \mathcal {N}(i,j) = \{(i-1,j),(i+1,j),(i,j-1),(i,j+1)\}$, and we define $\alpha _{ij}=\sum _{(i',j')\in \mathcal {N}(i,j)}\alpha _{i'j'}$. The coefficients $a_k'$, $b_k'$ and $c_k'$ are defined in Eq. (17).

In the GPU implementation each thread calculates dot products and norms in one column. To obtain global sums, these need to be reduced with an additional BLAS operation. However, as this operation only operates on two-dimensional (horizontal) vectors, its cost is very small (${<}1\,\%$ of the time per iteration).