Action recognition based on multimode fusion for VR online platform.

Area of Science:

• Human-Computer Interaction
• Virtual Reality
• Computer Vision

Background:

• Online communication lacks the richness of face-to-face interaction.
• Current virtual reality (VR) platforms use avatars but lack realistic user action tracking.
• Effective data collection methods for VR user actions are needed for decision-making.

Purpose of the Study:

• To develop a method for collecting multimodal action data from VR users.
• To create a high-accuracy action recognition model for VR environments.
• To enhance the realism of avatar-based communication in VR.

Main Methods:

• Collected three modalities of nine actions using VR head-mounted display (HMD) sensors, RGB cameras, and human pose estimation.
• Employed advanced multimodal fusion networks for action recognition.
• Designed a 2D key point augmentation scheme using 3D position data from the VR HMD.

Main Results:

• Achieved a high-accuracy action recognition model using multimodal fusion.
• Demonstrated that augmented 2D key point data and VR HMD sensor data yield accurate and stable action recognition models.
• Focused data collection and experiments on classroom scenes with potential for broader application.

Conclusions:

• The proposed method effectively collects and recognizes VR user actions, improving communication realism.
• The multimodal fusion approach and data augmentation techniques enhance model accuracy and stability.
• Findings are applicable to classroom settings and can be extended to other VR environments.