Depth image noise reduction and super-resolution by pixel-wise multi-frame fusion

村山 真大*; 東山 豊大*; 原園 友規*; 石井 裕剛*; 下田 宏*; 大城戸 忍*; 樽田 泰宜

IEICE Transactions on Information and Systems, E105-D(6), p.1211 - 1224, 2022/06

https://doi.org/10.1587/transinf.2021EDP7226

High-quality depth images are required for stable and accurate computer vision. Depth images captured by depth cameras tend to be noisy, incomplete, and of low-resolution. Therefore, increasing the accuracy and resolution of depth images is desirable. We propose a method to accomplish pixel-by-pixel noise reduction, depth completion, and super-resolution of depth images. For each pixel in the target image, the linear space from the focal point of the camera through each pixel to the existing object is divided into equally spaced grids. In each grid, the difference from each grid to the object surface is obtained from multiple tracked depth images, which have noisy depth values of the respective image pixels. Then, the coordinates of the correct object surface are obtainable by reducing the depth random noise. The missing values are completed. The resolution can also be increased by creating new pixels between existing pixels and by then using the same process as that used for noise reduction. Evaluation results have demonstrated that the proposed method can do processing with less GPU memory. Furthermore, the proposed method was able to reduce noise more accurately, especially around edges, and was able to process more details of objects than the conventional method. The super-resolution of the proposed method also produced a high-resolution depth image with smoother and more accurate edges than the conventional methods.