VR-HandNet: A Visually and Physically Plausible Hand Manipulation System in Virtual Reality.

Area of Science:

• Computer Science
• Virtual Reality
• Robotics

Background:

• Dexterous hand manipulation in virtual environments is challenging.
• Current methods often lack physical realism or visual fidelity.

Purpose of the Study:

• To develop a novel deep neural network for realistic virtual hand manipulation.
• To enable users to interact with virtual objects using hand-held VR controllers.

Main Methods:

• Mapping VR controllers to virtual hands for motion synthesis.
• Utilizing a deep neural network (VR-HandNet) trained with reinforcement learning for physical plausibility.
• Employing imitation learning to enhance visual realism by mimicking motion datasets.

Main Results:

• VR-HandNet dynamically synthesizes hand motions based on user input and object interactions.
• The method produces physically plausible hand movements through physics-simulated trial-and-error learning.
• Imitation learning improves the visual fidelity of the synthesized motions.

Conclusions:

• The proposed VR-HandNet effectively enables dexterous virtual hand manipulation.
• The combination of reinforcement and imitation learning achieves both physical and visual realism.
• The method successfully meets the design goal of intuitive object interaction in VR.