Communication-avoiding sparse matrix solvers for extreme scale nuclear CFD simulations

Idomura, Yasuhiro   

Communication-avoiding (CA) algorithms are key technologies towards extreme scale CFD simulations on future exascale machines, which are characterized by accelerated computation and relatively low communication bandwidth. In order to resolve this communication bottleneck, we developed two types of CA-based sparse matrix solvers on extreme scale nuclear simulations such as the five dimensional (5D) fusion plasma turbulence code GT5D and the 3D multi-phase thermal-hydraulic code JUPITER. One is a CA Krylov method, in which multiple basis vectors are generated and orthogonalized at once. By using this approach, one can avoid the bottleneck of All_Reduce communication, which is required at each iteration in the conventional Krylov method. The other is a CA multigrid (MG) method, in which the number of iteration or All_Reduce is reduced by improving the convergence property. In addition, MG implementation with a mixed precision approach reduces both computation and communication. By applying these CA solvers, the performances of GT5D and JUPITER were dramatically improved, and the strong scaling was extended up to the full system size of the Oakforest-PACS, which consists of 8,208 KNLs.