Projection surface detection and pose selection for autonomously displaying multimedia on walls using mobile robots

Mobile robots equipped with projectors enable versatile applications such as multimedia display, interactive communication, and environmental augmentation. However, wall projection, which is required for displaying multimedia content on walls, remains challenging, because it is difficult to autonomously locate a projection space that is both flat and unobstructed. Some existing approaches address wall projection using 2D maps or by considering only large continuous surfaces, but these methods fail to capture the full potential of detailed 3D information of the operation area and often overlook wall segments that are partially occluded by objects, for example, areas blocked by pictures or whiteboards. To tackle these challenges, we propose a method based on 3D PanopticNDT maps, that uses semantic information to identify suitable wall segments. A tailored scoring function then evaluates potential robot poses to ensure optimal projection conditions, such as proper viewing angle, appropriate distance, and maximal surface area. Our experiments on both synthetic and real-world datasets demonstrate that our approach is well-suited for practical applications, effectively overcoming the limitations of earlier methods.