Recovery of Lost Color and Depth Frames in Multiview Videos

In this paper, we consider an integrated error concealment system for lost color frames and lost depth frames in multiview videos with depths. We first proposed a pixel-based color error-concealment method with the use of depth information. Instead of assuming that the same moving object in consecutive frames has minimal depth difference, as is done in a state-of-the-art method, a more realistic situation in which the same moving object in consecutive frames can be in different depths is considered. In the derived motion vector candidate set, we consider all the candidate motion vectors in the set, and weight the reference pixels by the depth differences to obtain the final recovered pixel. Compared with the two state-of-the-art methods, the proposed method has average peak signal-to-noise ratio gains of up to 8.73 and 3.98 dB, respectively. Second, we proposed an iterative depth frame error-concealment method. The initial recovered depth frame is obtained by depth-image-based rendering from another available view. The holes in the recovered depth frame are then filled in the proposed priority order. Preprocessing methods (depth difference compensation and inconsistent pixel removal) are performed to improve the performance. Compared with a method that uses the available motion vector in a color frame to recover the lost depth pixels, the hybrid motion vector extrapolation method, the inpainting method and the proposed method have gains of up to 4.31, 10.29, and 6.04 dB, respectively. Finally, for the situation in which the color and the depth frames are lost at the same time, our two methods jointly perform better with a gain of up to 7.79 dB.