Image sequence interpolation based on optical flow, segmentation, and optimal control

When using motion fields to interpolate between two consecutive images in an image sequence, a major problem is to handle occlusions and disclusions properly. However, in most cases, one of both images contains the information that is either discluded or occluded; if the first image contains the information (i.e., the region will be occluded), forward interpolation shall be employed, while for information that is contained in the second image (i.e., the region will be discluded), one should use backward interpolation. Hence, we propose to improve an existing approach for image sequence interpolation by incorporating an automatic segmentation in the process, which decides in which region of the image forward or backward interpolation shall be used. Our approach is a combination of the optimal transport approach to image sequence interpolation and the segmentation by the Chan-Vese approach. We propose to solve the resulting optimality condition by a segregation loop, combined with a level set approach. We provide examples that illustrate the performance both in the interpolation error and in the human perception.