SIAgent: Spatial Interaction Agent via LLM-Powered Eye-Hand Motion Intent Understanding in VR.

Area of Science:

• Human-Computer Interaction
• Virtual Reality
• Artificial Intelligence

Background:

• Multimodal interaction in VR is growing, but current methods require users to learn gestures and memorize task associations.
• The
• Operation-to-Intent
• paradigm leads to high learning costs and low error tolerance.

Purpose of the Study:

• To introduce SIAgent, a novel
• Intent-to-Operation
• framework for intuitive VR interaction.
• To reduce user learning costs and increase error tolerance in VR interfaces.

Main Methods:

• Developed a two-component framework: intent recognition (spatial data to natural language and intent inference) and agent-based execution.
• Utilized natural eye-hand motions for user intent expression, bypassing the need for gesture memorization.
• Conducted user studies comparing SIAgent with existing
• Operation-to-Intent
• techniques.

Main Results:

• SIAgent achieved higher intent recognition accuracy (97.2%) compared to gaze + pinch interaction (93.1%).
• Demonstrated reduced arm fatigue, improved usability, and increased user preference.
• Verified the contribution of eye gaze and hand motion channels to intent recognition.

Conclusions:

• SIAgent enhances VR interaction intelligence through an intent-driven design, accommodating individual habits and preferences.
• The framework offers a more natural and efficient user experience in Virtual Reality.
• Future work will involve making source code and LLM prompts publicly available.