Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning.

Area of Science:

• Human-Computer Interaction
• Machine Learning
• Virtual Reality

Background:

• Traditional user behavior models are hand-crafted, limiting application in novel virtual and augmented reality (VR/AR) environments.
• User behavior in complex mid-air interactions, like virtual typing, is not well understood.
• Reinforcement learning (RL) has shown promise in modeling simple goal-oriented reaching tasks.

Purpose of the Study:

• To develop the first RL-based user model for mid-air and surface-aligned virtual keyboard typing.
• To demonstrate RL's capability in modeling complex user behavior beyond simple reaching tasks.
• To explore RL as a tool for augmenting or replacing human testing in virtual keyboard development.

Main Methods:

• Developed a novel reinforcement learning agent to model user behavior during virtual keyboard typing.
• Trained the RL agent on mid-air and surface-aligned typing interactions.
• Evaluated the model's ability to replicate high-level human typing behavior.

Main Results:

• The RL-based model successfully replicated high-level human typing behavior on a virtual keyboard.
• Demonstrated the feasibility of using RL for complex mid-air interaction tasks.
• The model showed potential for use in virtual keyboard validation and development.

Conclusions:

• Reinforcement learning can effectively model complex user behavior in virtual environments, specifically for virtual keyboard typing.
• RL-based user models offer a scalable and adaptable approach for HCI research and development.
• This work paves the way for more efficient validation and design of virtual interfaces in VR/AR.