GPU-enabled ensemble data assimilation for mesh-refined lattice Boltzmann method

Hasegawa, Yuta; Idomura, Yasuhiro; Onodera, Naoyuki

EPJ Web of Conferences, 302, p.03005_1 - 03005_9, 2024/10

https://doi.org/10.1051/epjconf/202430203005

We implemented the ensemble data assimilation (DA) method, the local ensemble transform Kalman filter (LETKF), into the mesh-refined lattice Boltzmann method (LBM) for turbulent flows. Both the LETKF and the mesh-refined LBM were fully implemented on GPUs, so that they are efficiently computed on modern GPU-based supercomputers. We examined the DA accuracy against the flow around a cylinder. The result showed that our method enabled accurate DA with spatially- and temporarily-sparse observation data; the error of the assimilated velocity field with the observation interval of and the observation resolution (1.56% of the total computational grids) was smaller than the amplitude of the observation noise, where is the period of the Krmn vortex and is diameter of the square cylinder.

Structure of drifting snow simulated by Lagrangian particle dispersion model coupled with large-eddy simulation using the lattice Boltzmann method

Watanabe, Tsutomu*; Ishikawa, Shuhei*; Kawashima, Masayuki*; Shimoyama, Ko*; Onodera, Naoyuki; Hasegawa, Yuta; Inagaki, Atsushi*

Journal of Wind Engineering and Industrial Aerodynamics, 250, p.105783_1 - 105783_17, 2024/07

https://doi.org/10.1016/j.jweia.2024.105783

This paper presents simulations of drifting snow using a Lagrangian particle dispersion model coupled with a large-eddy simulation code. The model accurately replicates observed features such as mass transport rate dependency on flow velocity and variations in particle size distribution. It also shows that the saltation layer height increases monotonically with flow velocity, contrary to conventional estimates. Additionally, the study confirms the transition from saltation to suspension near the estimated saltation layer height and finds that dense snow streamers are linked to small-scale low-speed streaks in near-surface flows.

Interactive steering on in situ particle-based volume rendering framework

Kawamura, Takuma; Hasegawa, Yuta; Idomura, Yasuhiro

Journal of Visualization, 27(1), p.89 - 107, 2024/02

https://doi.org/10.1007/s12650-023-00945-z

Interactive in-situ steering is an effective tool for debugging, searching for optimal solutions, and analyzing inverse problems in fast and large-scale computational fluid dynamics (CFD) simulations. We propose an interactive in-situ steering framework for large-scale CFD simulations on GPU supercomputers. This framework employs in-situ particle-based volume rendering (PBVR), in-situ data sampling, and a file-based control that enables interactive communication of steering parameters, compressed particle data, and sampled monitoring data between supercomputers and user PCs. The parallelized PBVR is processed on the host CPU to avoid interference with CFD simulations on the GPU. We apply the proposed framework to a real-time plume dispersion analysis code CityLBM on GPU supercomputers. In the numerical experiment, we address an inverse problem to find a pollutant source from the monitoring data, and demonstrate the effectiveness of the human-in-the-loop approach.

Validation of integrated thermal power measurement using solution fuel STACY experimental data for modified STACY performance test

Araki, Shohei; Gunji, Satoshi; Arakaki, Yu; Murakami, Takahiko; Yoshikawa, Tomoki; Hasegawa, Kenta; Tada, Yuta; Izawa, Kazuhiko; Suyama, Kenya

Proceedings of 4th Reactor Physics Asia Conference (RPHA2023) (Internet), 4 Pages, 2023/10

To conduct integrated thermal power measurements for the performance test of the modified STACY, we re-analyzed the experimental data measured in the solution fuel STACY using the activation method. We validated its feasibility under the limited number of activation detectors. The re-analyzed results of the activation method by using MVP and PHITS with JENDL-4.0 indicated that the effect of the difference of the position between activation detectors was small enough, and the results agreed with that of the fission product analysis within almost 10%. It is conceivable that the activation method could be adopted instead of the fission product analysis.

Data assimilation of two-dimensional isotropic turbulence by lattice Boltzmann method and local ensemble transform Kalman filter (LBM-LETKF)

Hasegawa, Yuta; Onodera, Naoyuki; Asahi, Yuichi; Idomura, Yasuhiro

Keisan Kogaku Koenkai Rombunshu (CD-ROM), 28, 5 Pages, 2023/05

We implemented and investigated the data assimilation (DA) of two-dimensional isotropic turbulence using the lattice Boltzmann method and the local ensemble transform Kalman filter (LBM-LETKF). We carried out the numerical experiment with 256 grids, 256 or less observation points, 10% root mean square (RMS) observation noise in the velocity observation, and 4, 16, or 64 ensemble members. The numerical experiment showed that the accuracy of the LETKF was better than the nudging DA with both dense and sparse observation. The lack of observation points caused the numerical instability in the LETKF, but such a numerical instability can be suppressed by increasing the number of ensemble members. In the sparse observation case (88 observation points) with 64 ensemble members, the root means squared error (RMSE) of the velocity in the LBM-LETKF was smaller than the RMS of the observation noise, while the nudging DA required 3232 observation points to obtain the same accuracy. Overall, the LETKF with sufficiently large number of ensemble members was highly accurate and robust, thus, the LETKF was a good choice for the DA of turbulent flows using the LBM.

Microstructural evolution in tungsten binary alloys under proton and self-ion irradiations at 800C

Miyazawa, Takeshi; Kikuchi, Yuta*; Ando, Masami*; Yu, J.-H.*; Yabuuchi, Kiyohiro*; Nozawa, Takashi*; Tanigawa, Hiroyasu*; Nogami, Shuhei*; Hasegawa, Akira*

Journal of Nuclear Materials, 575, p.154239_1 - 154239_11, 2023/03

https://doi.org/10.1016/j.jnucmat.2023.154239

Parameter optimization for turbulent boundary layer generation using ensemble Kalman filter

Onodera, Naoyuki; Idomura, Yasuhiro; Hasegawa, Yuta; Nakayama, Hiromasa

Dai-36-Kai Suchi Ryutai Rikigaku Shimpojiumu Koen Rombunshu (Internet), 3 Pages, 2022/12

We have developed a wind simulation code named CityLBM to realize wind digital twins. Mesoscale wind conditions are given as boundary conditions in CityLBM by using a nudging data assimilation method. It is found that conventional approaches with constant nudging coefficients fail to reproduce turbulent intensity in long time simulations, where atmospheric stability conditions change significantly. We propose a dynamic parameter optimization method for the nudging coefficient based on an ensemble Kalman filter. CityLBM was validated against plume dispersion experiments in the complex urban environment of Oklahoma City. The nudging coefficient was updated to reduce the error of the turbulent intensity between the simulation and the observation. The mean error of velocity variance is reduced by 10% compared to the conventional nudging method with a constant nudging coefficient.

Development of local-scale high-resolution atmospheric dispersion and dose assessment system

Nakayama, Hiromasa; Onodera, Naoyuki; Satoh, Daiki; Nagai, Haruyasu; Hasegawa, Yuta; Idomura, Yasuhiro

Journal of Nuclear Science and Technology, 59(10), p.1314 - 1329, 2022/10

https://doi.org/10.1080/00223131.2022.2038302

We developed a local-scale high-resolution atmospheric dispersion and dose assessment system (LHADDAS) for safety and consequence assessment of nuclear facilities and emergency response to nuclear accidents or deliberate releases of radioactive materials in built-up urban areas. This system is composed of pre-processing of input files, main calculation by local-scale high-resolution atmospheric dispersion model using large-eddy simulation (LOHDIM-LES) and real-time urban dispersion simulation model based on a lattice Boltzmann method (CityLBM), and post-processing of dose-calculation by simulation code powered by lattice dose-response functions (SIBYL). LHADDAS has a broad utility and offers superior performance in (1) simulating turbulent flows, plume dispersion, and dry deposition under realistic meteorological conditions, (2) performing real-time tracer dispersion simulations using a locally mesh-refined lattice Boltzmann method, and (3) estimating air dose rates of radionuclides from air concentrations and surface deposition in consideration of the influence of individual buildings and structures. This system is promising for safety assessment of nuclear facilities as an alternative to wind tunnel experiments, detailed pre/post-analyses of a local-scale radioactive plume dispersion in case of nuclear accidents, and quick response to emergency situations resulting from deliberate release of radioactive materials by a terrorist attack in an urban central district area.

GPU implementation of local ensemble transform Kalman filter (LETKF) with two-dimensional lattice Boltzmann method

Hasegawa, Yuta; Onodera, Naoyuki; Asahi, Yuichi; Idomura, Yasuhiro

Keisan Kogaku Koenkai Rombunshu (CD-ROM), 27, 4 Pages, 2022/06

We developed GPU implementation of ensemble data assimilation (DA) using the local ensemble transform Kalman filter (LETKF) with the lattice Boltzmann method (LBM). The performance test was carried out upto 32 ensembles of two-dimensional isotropic turbulence simulations using the D2Q9 LBM. The computational cost of the LETKF was less than or nearly equal to that of the LBM upto eight ensembles, while the former exceeded the latter at larger ensembles. At 32 ensembles, their computational costs per cycle were respectively 28.3 msec and 5.39 msec. These results suggested that further speedup of the LETKF is needed for practical 3D LBM simulations.

GPU optimization of lattice Boltzmann method with local ensemble transform Kalman filter

Hasegawa, Yuta; Imamura, Toshiyuki*; Ina, Takuya; Onodera, Naoyuki; Asahi, Yuichi; Idomura, Yasuhiro

Proceedings of 13th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH22) (Internet), p.10 - 17, 2022/00

https://doi.org/10.1109/ScalAH56622.2022.00007

The ensemble data assimilation of computational fluid dynamics simulations based on the lattice Boltzmann method (LBM) and the local ensemble transform Kalman filter (LETKF) is implemented and optimized on a GPU supercomputer based on NVIDIA A100 GPUs. To connect the LBM and LETKF parts, data transpose communication is optimized by overlapping computation, file I/O, and communication based on data dependency in each LETKF kernel. In two dimensional forced isotropic turbulence simulations with the ensemble size of and the number of grid points of , the optimized implementation achieved speedup from the naive implementation, in which the LETKF part is not parallelized. The main computing kernel of the local problem is the eigenvalue decomposition (EVD) of real symmetric dense matrices, which is computed by a newly developed batched EVD in EigenG. The batched EVD in EigenG outperforms that in cuSolver, and speedup was achieved.