Idomura, Yasuhiro; Ina, Takuya*; Ali, Y.*; Imamura, Toshiyuki*
Proceedings of International Conference on High Performance Computing, Networking, Storage, and Analysis (SC 2020) (Internet), p.1318 - 1330, 2020/11
The multi-scale full- simulation of the next generation experimental fusion reactor ITER based on a five dimensional (5D) gyrokinetic model is one of the most computationally demanding problems in fusion science. In this work, a Gyrokinetic Toroidal 5D Eulerian code (GT5D) is accelerated by a new mixed-precision communication-avoiding (CA) Krylov method. The bottleneck of global collective communication on accelerated computing platforms is resolved using a CA Krylov method. In addition, a new FP16 preconditioner, which is designed using the new support for FP16 SIMD operations on A64FX, reduces both the number of iterations (halo data communication) and the computational cost. The performance of the proposed method for ITER size simulations with 0.1 trillion grids on 1,440 CPUs/GPUs on Fugaku and Summit shows 2.8x and 1.9x speedups respectively from the conventional non-CA Krylov method, and excellent strong scaling is obtained up to 5,760 CPUs/GPUs.
Asahi, Yuichi*; Latu, G.*; Bigot, J.*; Maeyama, Shinya*; Grandgirard, V.*; Idomura, Yasuhiro
Concurrency and Computation; Practice and Experience, 32(5), p.e5551_1 - e5551_21, 2020/03
Two five-dimensional gyrokinetic codes GYSELA and GKV were ported to the modern accelerators, Xeon Phi KNL and Tesla P100 GPU. Serial computing kernels of GYSELA on KNL and GKV on P100 GPU were respectively 1.3x and 7.4x faster than those on a single Skylake processor. Scaling tests of GYSELA and GKV were respectively performed from 16 to 512 KNLs and from 32 to 256 P100 GPUs, and data transpose communications in semi-Lagrangian kernels in GYSELA and in convolution kernels in GKV were found to be main bottlenecks, respectively. In order to mitigate the communication costs, pipeline-based and task-based communication overlapping were implemented in these codes.
Asahi, Yuichi*; Grandgirard, V.*; Sarazin, Y.*; Donnel, P.*; Garbet, X.*; Idomura, Yasuhiro; Dif-Pradalier, G.*; Latu, G.*
Plasma Physics and Controlled Fusion, 61(6), p.065015_1 - 065015_15, 2019/05
The role of poloidal convective cells on transport processes is studied with the full-F gyrokinetic code GYSELA. For this purpose, we apply a numerical filter to convective cells and compare the simulation results with and without the filter. The energy flux driven by the magnetic drifts turns out to be reduced by a factor of about 2 once the numerical filter is applied. A careful analysis reveals that the frequency spectrum of the convective cells is well-correlated with that of the turbulent Reynolds stress tensor, giving credit to their turbulence-driven origin. The impact of convective cells can be interpreted as a synergy between turbulence and neoclassical dynamics.
Idomura, Yasuhiro; Ina, Takuya*; Mayumi, Akie; Yamada, Susumu; Matsumoto, Kazuya*; Asahi, Yuichi*; Imamura, Toshiyuki*
Proceedings of 8th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems (ScalA 2017), p.7_1 - 7_8, 2017/11
A communication-avoiding generalized minimal residual (CA-GMRES) method is applied to the gyrokinetic toroidal five dimensional Eulerian code GT5D, and its performance is compared against the original code with a generalized conjugate residual (GCR) method on the JAEA ICEX (Haswell), the Plasma Simulator (FX100), and the Oakforest-PACS (KNL). The CA-GMRES method has higher arithmetic intensity than the GCR method, and thus, is suitable for future Exa-scale architectures with limited memory and network bandwidths. In the performance evaluation, it is shown that compared with the GCR solver, its computing kernels are accelerated by , and the cost of data reduction communication is reduced from to of the total cost at 1,280 nodes.
Asahi, Yuichi*; Grandgirard, V.*; Idomura, Yasuhiro; Garbet, X.*; Latu, G.*; Sarazin, Y.*; Dif-Pradalier, G.*; Donnel, P.*; Ehrlacher, C.*
Physics of Plasmas, 24(10), p.102515_1 - 102515_17, 2017/10
Two full-F global gyrokinetic codes are benchmarked to compute flux-driven ion temperature gradient turbulence in tokamak plasmas. For this purpose, the Semi-Lagrangian code GYSELA and the Eulerian code GT5D are employed, which solve the full-F gyrokinetic equation with a realistic fixed flux condition. Using the appropriate settings for the boundary and initial conditions, flux-driven ITG turbulence simulations are carried out. The avalanche-like transport is assessed with a focus on spatio-temporal properties. A statistical analysis is performed to discuss this self-organized criticality (SOC) like behaviors, where we found spectra and a transition to spectra at high-frequency side in both codes. Based on these benchmarks, it is verified that the SOC-like behavior is robust and not dependent on numerics.
Physics of Plasmas, 24(8), p.080701_1 - 080701_5, 2017/08
An electron heating modulation numerical experiment based on a global full-f gyrokinetic model shows that transitions from ion temperature gradient driven (ITG) turbulence to trapped electron mode (TEM) turbulence induced by electron heating generate density peaking and rotation changes. Toroidal angular momentum balance during the rotation changes is revealed by direct observation of toroidal angular momentum conservation, in which in addition to ion turbulent stress, ion neoclassical stress, radial currents, and toroidal field stress of ions and electrons are important. Toroidal torque flipping between ITG and TEM phases is found to be related to reversal of the ion radial current that indicates coupling of particle and momentum transport channels. The ion and electron radial currents are balanced to satisfy the ambipolar condition, and the electron radial current is cancelled by the electron toroidal field stress, which indirectly affects toroidal torque.
Asahi, Yuichi*; Latu, G.*; Ina, Takuya; Idomura, Yasuhiro; Grandgirard, V.*; Garbet, X.*
IEEE Transactions on Parallel and Distributed Systems, 28(7), p.1974 - 1988, 2017/07
High-dimensional stencil computation from fusion plasma turbulence codes involving complex memory access patterns, the indirect memory access in a Semi-Lagrangian scheme and the strided memory access in a Finite-Difference scheme, are optimized on accelerators such as GPGPUs and Xeon Phi coprocessors. On both devices, the Array of Structure of Array (AoSoA) data layout is preferable for contiguous memory accesses. It is shown that the effective local cache usage by improving spatial and temporal data locality is critical on Xeon Phi. On GPGPU, the texture memory usage improves the performance of the indirect memory accesses in the Semi-Lagrangian scheme. Thanks to these optimizations, the fusion kernels on accelerators become 1.4x - 8.1x faster than those on Sandy Bridge (CPU).
Maeyama, Shinya*; Watanabe, Tomohiko*; Idomura, Yasuhiro; Nakata, Motoki*; Ishizawa, Akihiro*; Nunami, Masanori*
Nuclear Fusion, 57(6), p.066036_1 - 066036_10, 2017/05
Multi-scale plasma turbulence including electron and ion temperature gradient (ETG/ITG) modes has been investigated by means of electromagnetic gyrokinetic simulations. Triad transfer analyses on nonlinear mode coupling reveal cross-scale interactions between electron and ion scales. One of the interactions is suppression of electron-scale turbulence by ion- scale turbulence, where ITG-driven short-wavelength eddies act like shear flows and suppress ETG turbulence. Another cross-scale interaction is enhancement of ion-scale turbulence in the presence of electron-scale turbulence. This is caused via short-wavelength zonal flows, which are created by the response of passing kinetic electrons in ITG turbulence, suppress ITG turbulence by their shearing, and are damped by ETG turbulence. In both cases, sub-ion-scale structures between electron and ion scales play important roles in the cross-scale interactions.
Kawai, Chika*; Idomura, Yasuhiro; Maeyama, Shinya*; Ogawa, Yuichi*
Physics of Plasmas, 24(4), p.042303_1 - 042303_13, 2017/04
Self-organization in the slab electron temperature gradient driven (ETG) turbulence is investigated based on gyrokinetic simulations and the Hasegawa-Mima (HM) equation. The scale and the anisotropy of self-organized turbulent structures vary depending on the Rhines scale and the characteristic scale given by the adiabatic response term in the HM equation. The former is determined by competition between the linear wave dispersion and the nonlinear turbulent cascade, while the latter is given as the scale, at which the turbulent cascade is impeded. These scales are controlled by plasma parameters such as the density and temperature gradient, and the temperature ratio of ion to electron. It is found that depending on the plasma parameters, the ETG turbulence shows either isotropic turbulence or zonal flows, which give significantly different transport levels. Although the modulational instability excites zonal modes regardless of the plasma parameters, the final turbulent structure is determined by the self-organization process.
Idomura, Yasuhiro; Asahi, Yuichi; Ina, Takuya; Matsuoka, Seikichi
Proceedings of 24th International Congress of Theoretical and Applied Mechanics (ICTAM 2016), p.3106 - 3107, 2016/08
Turbulent transport in fusion plasmas is one of key issues in ITER. To address this issue via the five dimensional (5D) gyrokinetic model, a novel computing technique is developed, and strong scaling of the Gyrokinetic Toroidal 5D Eulerian code GT5D is improved up to million cores on the K-computer. The computing technique consists of multi-dimensional/multi-layer domain decomposition, overlap of communication and computation, and optimization of computing kernels for multi-core CPUs. The computing power enabled us to study ITER relevant issues such as the plasma size scaling of turbulent transport. Towards the next generation burning plasma turbulence simulations, the physics model is extended including kinetic electrons and multi-species ions, and computing kernels are further optimized for the latest many-core architectures.
Plasma and Fusion Research (Internet), 11, p.2403006_1 - 2403006_5, 2016/02
In this work, we address saturation mechanisms of decaying turbulence induced by the ion temperature gradient driven trapped electron mode. In the simulation, turbulent transport is quenched in the nonlinear quasi-steady phase, where temperature profiles exceeding linear critical temperature gradient parameters are formed. This kind of nonlinear critical temperature gradient is sustained by radial electric fields with strong shear, which is generated by corrugated density profiles. It is found that the density profile structure is related to electrons transport near low order mode rational surfaces, where non-adiabatic response of passing electrons becomes important.
Maeyama, Shinya*; Idomura, Yasuhiro; Watanabe, Tomohiko*; Nakata, Motoki*; Yagi, Masatoshi; Miyato, Naoaki; Ishizawa, Akihiro*; Nunami, Masanori*
Physical Review Letters, 114(25), p.255002_1 - 255002_5, 2015/06
Multiscale gyrokinetic turbulence simulations with the real ion-to-electron mass ratio and value are realized for the first time, where the value is given by the ratio of plasma pressure to magnetic pressure and characterizes electromagnetic effects on microinstabilities. Numerical analysis at both the electron scale and the ion scale is used to reveal the mechanism of their cross-scale interactions. Even with the real- mass scale separation, ion-scale turbulence eliminates electron-scale streamers and dominates heat transport, not only of ions but also of electrons. When the ion-scale modes are stabilized by finite- effects, the contribution of the electron-scale dynamics to the turbulent transport becomes non-negligible and turns out to enhance ion-scale turbulent transport.
Idomura, Yasuhiro; Tokuda, Shinji; Kishimoto, Yasuaki
Nuclear Fusion, 45(12), p.1571 - 1581, 2005/12
Using a global gyrokinetic toroidal particle code, the toroidal electron temperature gradient driven (ETG) turbulence is studied in positive and reversed shear tokamaks. In the nolinear turbulent state, the ETG turbulence in the positive and reversed shear configurations show quite different structure formations. In the positive shear configuration, the ETG turbulence is dominated by streamers which have a ballooning type structure, and the electron temperature profile is quickly relaxed to the marginally stable state in a turbulent time scale. In the reversed shear configuration, quasi-steady zonal flows are produced in the regative shear region, while the positive shear region is characterized by streamers. Accordingly, the electron thermal diffusivity has a gap structure across the surface, and the gradinet is sustained above the marginal value for a long time in the quasi-steady phase. The results suggest a stiffness of the profile in positive shear tokamaks, and a possibility of the Te transport barrier in reversed shear tokamaks.
Matsumoto, Taro; Naito, Hiroshi*; Tokuda, Shinji; Kishimoto, Yasuaki*
Nuclear Fusion, 45(11), p.1264 - 1270, 2005/11
A gyrokinetic particle simulation is executed to clarify the effect of the electron inertia on the MHD phenomena in the reversed shear configuration (RSC) of a cylindrical tokamak plasma. It is found that the collisionless (kinetic) double tearing modes grow up at the Alfvn time scale, and nonlinearly induce the internal collapse when the helical flux at the magnetic axis is less than that at the outer resonant surface. After the internal collapse, the secondary reconnection is induced by the current concentration due to the convective flow. It is also clarified that a nonlinear dynamics accompanied with the elementary processes caused by the flow can generate a new RSC with resonant surfaces. In the presence of the density gradient, after the full reconnection induced by the mode, the radial electric field is found to be generated due to the difference of the motion between ions and electrons. However, the intensity of the radial field is not so large as that induced by the collisionless kink mode.
Matsumoto, Taro; Naito, Hiroshi*; Tokuda, Shinji; Kishimoto, Yasuaki
Physics of Plasmas, 12(9), p.092505_1 - 092505_7, 2005/09
The behavior of the collisionless magnetohydrodynamics modes is investigated by the gyro-kinetic particle simulation in a cylindrical tokamak plasma in the parameter region where the effects of electron inertia and electron parallel compressibility are competitive for magnetic reconnection. Although the linear growth of the internal kink-tearing mode is dominated by the electron inertia, it is found that the growth rate can be nonlinearly accelerated due to the electron parallel compressibility proportional to the ion sound Larmor radius . It is also found that, as decreasing the electron skin depth , the maximum growth rate before the internal collapse saturates independently of the microscopic scales such as and . The acceleration of growth rate is also observed in the nonlinear phase of the double tearing mode.
Purazuma, Kaku Yugo Gakkai-Shi, 81(8), p.581 - 592, 2005/08
A gyrokinetic particle simulation is a powerful tool in studying tokamak microturbulence. A method, which is a standard method in recent gyrokinetic particle simulations, dramatically improved an efficiency of a particle simulation by reducing a particle noise, and full torus turbulence simulations are enabled. In this paper, the method is reviewed, and issues in full torus gyrokinetic particle simulations are discussed.
Idomura, Yasuhiro; Kishimoto, Yasuaki; Tokuda, Shinji
Europhysics Conference Abstracts (CD-ROM), 29C, 4 Pages, 2005/00
A gyrokinetic simulation is an essential tool to study anomalous turbulent transport in tokamak plasmas. Although a delta-f PIC code or a particle approach has been a standard method, it has difficulty in implementing non-conservative effects such as a heat source and collisions, which are important for a realistic long time turbulence simulation. On the other hand, a Vlasov code or a mesh approach is much more flexible to simulate these non-conservative effects. In this work, a new gyrokinetic Vlasov code is developed based on the CIP method, which is one of recent advanced CFD schemes. Numerical properties and computational costs of the gyrokinetic PIC and CIP codes are compared in ITG turbulence simulations.
Idomura, Yasuhiro; Tokuda, Shinji; Kishimoto, Yasuaki
Journal of Plasma and Fusion Research SERIES, Vol.6, p.17 - 72, 2004/00
A global gyrokinetic toroidal particle code for a 3D nonlinear simulation (GT3D) has been developed for a comprehensive study of the ion and electron anomalous transport arising from the ion temperature gradient driven - trapped electron mode (ITG-TEM) turbulence in tokamak plasmas. In the preliminary linear ITG-TEM calculations, basic properties of ITG-TEM modes are confirmed. Adding trapped electrons not only increases the growth rate of the ITG mode, but also produces another unstable electron mode, the TEM mode, which is unstable even at . The dominant mode changes from the ITG mode to the TEM mode depending on and . In linear benchmark calculations using Cyclone base case parameters, eigenfrequencies obtained from GT3D, GTC(PPPL-UCI) and FULL(PPPL) show reasonable quantitative agreement.
Jolliet, S.*; Angelino, P.*; Bottino, A.*; Idomura, Yasuhiro; Villard, L.*
Theory of Fusion Plasmas, ISPP21, p.345 - 351, 2004/00
Global particle-in-cell (PIC) simulations are a very useful tool for studying the time evolution of turbulence induced by ion-temperature-gradient (ITG) instabilities. Unfortunately, the linear code LORB5 and its non-linear version ORB5 require high computational power. In order to study more sophisticated models, we need to optimize these codes. We will focus on LORB5, which uses a cylindrical grid (r,z) for solving the Vlasov equation and a (s,) grid for the Poisson equation. The approach presented in this work consists of implementing the gyrokinetic model using a single (s,) grid. Here is the straight-field-line poloidal coordinate. A method to avoid the singularity at the magnetic axis is presented, and a benchmark with the CYCLONE case is shown.
Idomura, Yasuhiro; Tokuda, Shinji; Kishimoto, Yasuaki
Nuclear Fusion, 43(4), p.234 - 243, 2003/04
no abstracts in English