HandLight: Light Estimation from Hand Interaction in Mixed Reality.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Mixed Reality

Background:

• Accurate environmental illumination estimation is crucial for realistic Mixed Reality (MR) experiences.
• Traditional methods rely on static light probes, which require scene preparation and limit dynamic illumination capture.

Purpose of the Study:

• To introduce HandLight, a novel approach for estimating scene illumination using only the user's hands in MR environments.
• To overcome the limitations of static light probes by enabling dynamic and preparation-free illumination estimation.

Main Methods:

• Developed a neural network system that learns environmental lighting from images of the user's hand.
• Trained the network on a dataset of common hand gestures (pinch, fist, bloom) under diverse lighting conditions.
• Generated a dynamic atlas of light probes from hand movements, reflecting real-time illumination variations.

Main Results:

• Demonstrated that HandLight can provide believable illumination estimations for various lighting scenarios.
• Validated the approach using a dataset of real hand images captured during interaction.
• Showcased the system's ability to adapt to changing environmental lighting without manual setup.

Conclusions:

• HandLight offers a practical and efficient solution for illumination estimation in MR by leveraging gestural interaction.
• The proposed method eliminates the need for dedicated hardware and environmental preparation, enhancing user experience.
• This approach paves the way for more immersive and visually coherent MR applications.