Acceleration of fusion plasma turbulence simulations using the mixed-precision communication-avoiding Krylov method

Idomura, Yasuhiro; Ina, Takuya*; Ali, Y.*; Imamura, Toshiyuki*

Proceedings of International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 2020) (Internet), p.1318 - 1330, 2020/11

https://doi.org/10.1109/SC41405.2020.00097

The multi-scale full- simulation of the next generation experimental fusion reactor ITER based on a five dimensional (5D) gyrokinetic model is one of the most computationally demanding problems in fusion science. In this work, a Gyrokinetic Toroidal 5D Eulerian code (GT5D) is accelerated by a new mixed-precision communication-avoiding (CA) Krylov method. The bottleneck of global collective communication on accelerated computing platforms is resolved using a CA Krylov method. In addition, a new FP16 preconditioner, which is designed using the new support for FP16 SIMD operations on A64FX, reduces both the number of iterations (halo data communication) and the computational cost. The performance of the proposed method for ITER size simulations with 0.1 trillion grids on 1,440 CPUs/GPUs on Fugaku and Summit shows 2.8x and 1.9x speedups respectively from the conventional non-CA Krylov method, and excellent strong scaling is obtained up to 5,760 CPUs/GPUs.

GPU-acceleration of locally mesh allocated two phase flow solver for nuclear reactors

Onodera, Naoyuki; Idomura, Yasuhiro; Ali, Y.*; Yamashita, Susumu; Shimokawabe, Takashi*; Aoki, Takayuki*

Proceedings of Joint International Conference on Supercomputing in Nuclear Applications + Monte Carlo 2020 (SNA + MC 2020), p.210 - 215, 2020/10

This paper presents a GPU-based Poisson solver on a block-based adaptive mesh refinement (block-AMR) framework. The block-AMR method is essential for GPU computation and efficient description of the nuclear reactor. In this paper, we successfully implement a conjugate gradient method with a state-of-the-art multi-grid preconditioner (MG-CG) on the block-AMR framework. GPU kernel performance was measured on the GPU-based supercomputer TSUBAME3.0. The bandwidth of a vector-vector sum, a matrix-vector product, and a dot product in the CG kernel gave good performance at about 60% of the peak performance. In the MG kernel, the smoothers in a three-stage V-cycle MG method are implemented using a mixed precision RB-SOR method, which also gave good performance. For a large-scale Poisson problem with cells, the developed MG-CG method reduced the number of iterations to less than 30% and achieved 2.5 speedup compared with the original preconditioned CG method.

GPU acceleration of communication avoiding Chebyshev basis conjugate gradient solver for multiphase CFD simulations

Ali, Y.*; Onodera, Naoyuki; Idomura, Yasuhiro; Ina, Takuya*; Imamura, Toshiyuki*

Proceedings of 10th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (ScalA 2019), p.1 - 8, 2019/11

https://doi.org/10.1109/ScalA49573.2019.00006

Iterative methods for solving large linear systems are common parts of computational fluid dynamics (CFD) codes. The Preconditioned Conjugate Gradient (P-CG) method is one of the most widely used iterative methods. However, in the P-CG method, global collective communication is a crucial bottleneck especially on accelerated computing platforms. To resolve this issue, communication avoiding (CA) variants of the P-CG method are becoming increasingly important. In this paper, the P-CG and Preconditioned Chebyshev Basis CA CG (P-CBCG) solvers in the multiphase CFD code JUPITER are ported to the latest V100 GPUs. All GPU kernels are highly optimized to achieve about 90% of the roofline performance, the block Jacobi preconditioner is re-designed to extract high computing power of GPUs, and the remaining bottleneck of halo data communication is avoided by overlapping communication and computation. The overall performance of the P-CG and P-CBCG solvers is determined by the competition between the CA properties of the global collective communication and the halo data communication, indicating an importance of the inter-node interconnect bandwidth per GPU. The developed GPU solvers are accelerated up to 2x compared with the former CPU solvers on KNLs, and excellent strong scaling is achieved up to 7,680 GPUs on the Summit.

Coherent control of single spins in silicon carbide at room temperature

Widmann, M.*; Lee, S.-Y.*; Rendler, T.*; Son, N. T.*; Fedder, H.*; Paik, S.*; Yang, L.-P.*; Zhao, N.*; Yang, S.*; Booker, I.*; et al.

Nature Materials, 14(2), p.164 - 168, 2015/02

https://doi.org/10.1038/nmat4145

High-resolution water window X-ray imaging of in vivo cells and their products using LiF crystal detectors

Bonfigli, F.*; Faenov, A. Y.; Flora, F.*; Francucci, M.*; Gaudio, P.*; Lai, A.*; Martellucci, S.*; Montereali, R. M.*; Pikuz, T.*; Reale, L.*; et al.

Microscopy Research and Technique, 71(1), p.35 - 41, 2008/01

https://doi.org/10.1002/jemt.20523

XPS and XANES observations on non-stoichiometric SiN produced by low-energy ion implantation

Ali, M.; Baba, Yuji; Sekiguchi, Tetsuhiro; Li, Y.; Yamamoto, Hiroyuki

Photon Factory Activity Report 1998, P. 36, 1999/11

http://ccdb5fs.kek.jp/cgi-bin/img/tiff2gif?199923003+00066

no abstracts in English

A GPU-accelerated Poisson equation solver for multiphase CFD code JUPITER

Onodera, Naoyuki; Idomura, Yasuhiro; Ali, Y.*; Yamashita, Susumu; Ina, Takuya*; Imamura, Toshiyuki*

no journal, , 

Transient heat flow analysis of nuclear reactors is very important from the view point of efficient design and safety. We have developed the stencil-based CFD code JUPITER for simulating three-dimensional multiphase flows. We extended the JUPITER with GPU-accelerated Poisson solvers based on the P-CG and P-CBCG methods. All main kernels were implemented using CUDA, and the GPU kernel function is well tuned to achieve high performance on the latest Volta-core GPUs. The developed solvers showed good strong scaling up to 2,048 GPUs/CPUs on the Summit (NVIDIA TESLA V100), the ABCI (NVIDIA TESLA V100), the Oakforest-PACS (Intel Knights Landing). Finally, the performance gain from the Oakforest-PACS ranges from 1.2 1.6x on the Summit and 1.4 1.7x on the ABCI, respectively.

Development of exascale matrix solvers based on communication avoiding algorithms

Imamura, Toshiyuki*; Idomura, Yasuhiro; Ina, Takuya*; Yamashita, Susumu; Onodera, Naoyuki; Ali, Y.*; Yamada, Susumu

no journal, , 

Towards exascale computing on the Post-K computer, a novel matrix solvers are developed by using communication avoiding algorithms. In this talk, we review two main approaches used in the three-dimensional thermal hydraulic multi-phase CFD code, JUPITER. One is a communication avoiding Krylov sub-space method, in which multiple basis vectors are generated and orthogonalized at once to reduce global collective communications. The other is a Krylov sub-space method with multi-grid preconditioning, which dramatically improve convergence property and reduce the number of iterations, and thus, global collective communications. We compare these approaches on the latest many core platform.

GPU optimization of matrix solvers

Ali, Y.*; Onodera, Naoyuki; Idomura, Yasuhiro; Ina, Takuya*; Imamura, Toshiyuki*

no journal, , 

Krylov solvers can account for up to 90% of the total computing cost in extreme scale nuclear CFD simulations. In order to accelerate such CFD codes, we ported the conventional Preconditioned Conjugate Gradient (PCG) and the two latest communication avoiding algorithms, the Preconditioned Chebyshev Basis communication-avoiding Conjugate Gradient (P-CBCG) and the Communication-Avoiding Generalized Minimal RESidual (CA-GMRES) methods, on to GPUs. In this talk, we discuss a trade-off between the performance portability and the performance improvement for implementations using OpenACC and CUDA, and show performance tests on the latest GPU supercomputers.

Optimization of fusion plasma turbulence code GT5D on FUGAKU and SUMMIT

Idomura, Yasuhiro; Ali, Y.*; Ina, Takuya*; Imamura, Toshiyuki*

no journal, , 

Implicit finite difference solvers based on Krylov subspace methods occupy dominant computing costs in the Gyrokinetic Toroidal 5D full-f Eulerian code GT5D. Under the post-K project, advanced communication avoiding (CA) Krylov subspace methods have been developed for exascale computing platforms, which have limited inter-node communication performance compared with accelerated computation. In this work, we develop a new mixed precision CA-GMRES solver using a FP16 preconditioner, which dramatically reduces the number of iterations, and thus, halo data communications. We port the new solver on FUGAKU and Summit, and compare its performance against conventional solvers on existing muti/many-core processors.