Mixed Reality and Artificial Intelligence: A Holistic Approach to Multimodal Visualization and Extended Interaction...

Area of Science:

• Orthopedic surgery
• Medical technology
• Human-computer interaction

Background:

• Augmented reality (AR) has advanced orthopedic surgical planning and navigation.
• Mixed reality (MR) applications in orthopedics are less explored.
• Artificial intelligence (AI) can enhance MR capabilities through automation and personalization.

Purpose of the Study:

• To assess the Holoknee prototype, an AI and MR-based system for multimodal data visualization and surgical planning in knee osteotomy.
• To evaluate its usability on the HoloLens 2 headset.

Main Methods:

• Two preclinical test sessions with 11 participants (surgeons, residents, medical student).
• Participants performed six tasks related to holographic data interaction and preoperative planning.
• User perception and usability were assessed via questionnaires.

Main Results:

• Participants showed improved task execution times across sessions.
• High agreement (90.9%) on MR's effectiveness for preoperative planning.
• Positive feedback on MR for surgical training and potential for intraoperative guidance.

Conclusions:

• Holoknee is the first holistic AI and MR application for knee osteotomy surgical planning.
• Demonstrated promising potential for surgical training, preoperative planning, and intraoperative guidance.
• Aims to support surgeons and enhance resident training.