Towards user-centered interactive medical image segmentation in VR with an assistive AI agent

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Virtual Reality

Background:

• Manual segmentation of volumetric medical scans is time-consuming and error-prone.
• Fully automatic segmentation algorithms can be improved with user feedback.
• Integrating AI and VR offers a novel approach to medical image analysis.

Purpose of the Study:

• To develop SAMIRA, a conversational AI agent for medical VR to assist in localizing, segmenting, and visualizing 3D medical concepts.
• To evaluate the usability and effectiveness of SAMIRA in a human-in-the-loop segmentation workflow.
• To compare different interaction modes (controller, head, eye tracking) for refining segmentation masks in VR.

Main Methods:

• Development of SAMIRA, a conversational AI agent leveraging radiological AI foundation models and VR.
• Implementation of speech-based interaction for understanding radiological features and generating segmentation masks.
• Comparison of VR controller pointing, head pointing, and eye tracking for interactive segmentation refinement.
• User study to evaluate system usability, task load, and training potential.

Main Results:

• SAMIRA demonstrated high usability with a System Usability Scale (SUS) score of 90.0 ± 9.0.
• The system achieved low overall task load for users.
• Evaluations showed strong support for SAMIRA's guidance, training potential, and AI integration in radiological segmentation.

Conclusions:

• SAMIRA offers an effective and intuitive AI-powered VR solution for medical image segmentation and visualization.
• The conversational agent enhances understanding of radiological features and patient-specific anatomy.
• The proposed human-in-the-loop VR system shows significant promise for improving radiological segmentation tasks.