Order Up! Multimodal Interaction Techniques for Notifications in Augmented Reality

Area of Science:

• Human-Computer Interaction
• Augmented Reality Systems
• Usability Engineering

Background:

• Augmented reality (AR) integration into professional and social settings is increasing.
• Effective management of frequent notifications is a critical challenge for AR users.
• Smartphone notification volume (nearly 200 daily) highlights the need for AR interaction translation.

Purpose of the Study:

• To explore multimodal interaction with notifications in augmented reality (AR).
• To investigate user preferences, usability, workload, and performance in AR notification management.
• To identify key issues and provide guidance for optimizing AR multimodal interaction techniques.

Main Methods:

• A virtual cooking task was used to simulate a real-world scenario.
• Participants managed customer orders while interacting with notifications.
• Evaluated interaction techniques included Point and Pinch, Gaze and Pinch, Point and Voice, Gaze and Voice, and Touch.

Main Results:

• Significant impacts on user workload were observed across different interaction methods.
• Notification interaction techniques notably affected task performance.
• Usability varied considerably depending on the chosen multimodal interaction method.

Conclusions:

• The choice of multimodal interaction technique critically influences user experience in AR.
• Key challenges in AR multimodal notification management were identified.
• Guidance is provided for optimizing interaction techniques for seamless AR integration.