Multimodal Feedback for Handheld Tool Guidance: Combining Wrist-Based Haptics with Augmented Reality

Area of Science:

• Human-Computer Interaction
• Augmented Reality
• Haptics

Background:

• Augmented reality (AR) overlays aid surgical tasks but face challenges like visual occlusion and interface ambiguity.
• Existing AR systems can compromise precision and user confidence in visually complex procedures.

Purpose of the Study:

• To investigate vibrotactile wrist feedback for enhancing spatial guidance in optical see-through AR for handheld tool movement.
• To design and evaluate a multimodal system combining AR visuals with wrist-worn haptic cues.

Main Methods:

• A formative study with surgeons identified key maneuvers and haptic cue preferences.
• A cue identification experiment (N=21) tested vibration pattern recognition under visual load.
• A guidance task experiment (N=27) compared AR alone, haptics alone, and combined AR-haptics.

Main Results:

• Participants accurately recognized vibrotactile cues, with full-actuator states performing better than directional cues.
• The combined AR and haptics group showed significantly higher spatial precision (5.8 mm) and usability (SUS=88.1).
• Haptic cues provided reassurance and reduced cognitive effort, despite minor increases in task time.

Conclusions:

• Integrating wrist-based haptics into AR systems shows promise for high-precision tasks like surgical guidance.
• Multimodal interfaces with haptic feedback can improve user confidence and efficiency in tool manipulation.
• Design implications for effective multimodal interfaces supporting complex manipulation tasks are discussed.