Multimodal human-computer interaction in interventional radiology and surgery: a systematic literature review

Area of Science:

• Medical interactive systems
• Human-computer interaction
• Surgical technology

Background:

• Multimodal interaction (MMI) offers sterile and natural interaction in surgical and interventional settings.
• Previous reviews have focused on MMI technology and robotics, not specifically surgical applications.
• Advancements in technology drive the need for touchless and MMI in medicine.

Purpose of the Study:

• To systematically review multimodal interaction (MMI) for surgical and interventional scenarios over the past decade.
• To analyze the current state, challenges, and future directions of MMI in medical contexts.
• To identify gaps in research regarding MMI application and evaluation in surgery.

Main Methods:

• Systematic literature search across two databases for publications within the last 10 years.
• Eligibility screening and forward/backward citation searching to identify relevant studies.
• Clustering of references by medical field, input/output modalities, and development/evaluation challenges.

Main Results:

• 31 references analyzed (16 journal articles, 15 conference papers).
• MMI predominantly used in laparoscopy and radiology, often combining voice and hand controls.
• Visual feedback was common (81%), but gaze, body, and facial controls were minimally applied due to ergonomics.

Conclusions:

• MMI has significant potential in surgical and interventional settings.
• Future research should focus on in-depth use-case analysis and standardized evaluation methods.
• Incorporating insights from other sectors, like the gaming industry, can advance MMI in surgery.