OSAIRIS: Lessons Learned From the Hospital-Based Implementation and Evaluation of an Open-Source Deep-Learning Model...

Area of Science:

• Medical Physics
• Radiotherapy
• Artificial Intelligence

Background:

• Autosegmentation for organs at risk (OAR) outlining in radiotherapy is widely studied for accuracy.
• Existing evaluations often overlook perceived utility and safety, focusing solely on metrics.
• The OSAIRIS open-source AI model was developed as a medical device for OAR contouring.

Purpose of the Study:

• To clinically evaluate the OSAIRIS AI model for radiotherapy treatment planning.
• To assess tangible benefits, quantify utility, and evaluate risks within a clinical workflow.
• To compare OSAIRIS's performance against traditional methods and commercial AI devices.

Main Methods:

• Clinical evaluation of OSAIRIS for head and neck and male pelvis (prostate) OAR contouring.
• Quantification of time-saving benefits for clinicians.
• Correlation analysis between accuracy metrics (Dice, Hausdorff) and editing time.
• Safety evaluation focusing on error detection and correction by clinicians.
• Blinded clinician contour rating exercise and 'Mystery Shopping' experiment for risk assessment.

Main Results:

• OSAIRIS achieved significant time savings: 36% for prostate and 67% for head and neck contouring.
• Hausdorff distance showed the best correlation with editing time (Spearman 0.70, Kendall 0.52), outperforming Dice.
• Clinicians corrected most errors (72% head and neck, 81% prostate).
• Anchoring bias was observed during AI contour editing.
• Blinded ratings indicated AI contours were not rated lower than clinician contours.

Conclusions:

• Clinical evaluation of AI in radiotherapy must extend beyond accuracy metrics to include workflow, utility, and safety.
• Human-AI interaction effects are crucial for assessing practical usability and technology adoption.
• Comprehensive risk assessment and mitigation strategies are essential for AI medical devices.