Improvement of the J-PARC operation data archiver using HBase/Hadoop

Ikeda, Hiroshi; Kikuzawa, Nobuhiro ; Yoshii, Akinobu*; Kato, Yuko

The Linac and the RCS in J-PARC provide enormous operation data and we have been stored them into PostgreSQL. We are planning to replace it with HBase, a data store specialized for big-data and is on a distributed file system provided by Hadoop, using a cluster with advantages including automatically recovering and easily extending the capacity. In the previous presentation we reported we updated HBase/Hadoop in our test system and we showed issues to fix our tools in the new system, while we also mentioned issues about the construction of our cluster itself. In this presentation we are reporting we have re-designed and re-constructed the cluster with resolving the issues, including enhancing hardware of master nodes, creating automatic scripts to construct nodes, and introducing monitoring tools for nodes. Having accordingly adjusted the configurations of HBase/Hadoop and measured the performance of our new system, we are also reporting its results and considerations.