Occlusion-robust sensing method by using the light-field of a 3D display system toward interaction with a 3D image

The recent developments in 3D display technology are remarkable. Owing to such developments, the importance of methods for interacting with 3D images has been increasing. In particular, the ability to perform input operations by directly touching displayed images is important. Therefore, a sensing method is required for recognizing the 3D positions of fingers and determining whether the fingers are touching the 3D image displayed. Conventionally, such a sensing method generally involves position sensing of fingers through image-based active sensing. However, this does not solve the following two problems: (1) the problem of positional registration, that is, the accurate matching between the displayed image and input location, and (2) the problem of occlusion robustness, that is, the achievement of successful sensing even with the presence of an occluding object between the sensor and hands to allow free movement of the hands. Our proposed method solves these problems through the following two ideas. First, we used a method called aerial imaging by retroreflection, which focuses light rays from a wider range than other 3D display systems. Second, for capturing the reflected light as well as projecting to the sensing target in active sensing, we used the light-field formed by the 3D display system. We also propose a method for obtaining rotation information, considering that the light-field formed by the reflected light changes based on the angle of the sensing target. Simulations and experiments were performed to evaluate the proposed system, and the system setup was optimized through simulations. In the first two experiments conducted on position and rotation sensing, we evaluated the sensing errors due to occlusion and found them to be less than 1.74 mm and less than 15.4 deg, respectively. In the third experiment, we constructed an interaction system with 3D images by using the proposed method and evaluated this system.