Development and validation of a checklist for use with automatically generated radiotherapy plans

Area of Science:

• Medical Physics
• Radiotherapy
• Artificial Intelligence in Healthcare

Background:

• Automated treatment planning systems generate radiotherapy plans rapidly.
• Manual review of these plans is crucial for patient safety but can be time-consuming.
• Ensuring accuracy and quality in AI-generated plans requires efficient error detection methods.

Purpose of the Study:

• To develop and evaluate a custom checklist for improving error detection in automatically generated radiotherapy plans.
• To enhance the safety and quality of radiotherapy treatment planning through systematic review.

Main Methods:

• A custom checklist was developed based on established guidelines and failure modes and effects analysis of an automated planning tool.
• Two studies were conducted: the first with experienced physicists and the second with senior residents, comparing error detection rates with and without the checklist.
• Participants reviewed AI-generated plans, recorded errors, and rated clinical acceptability, followed by usability feedback for checklist refinement.

Main Results:

• The checklist significantly increased error detection rates in the first study (3.4 to 4.4 errors per participant, p=0.02), representing a 20% improvement.
• In the second study, error detection increased by 18% with the checklist (from 2.9 to 3.5 errors per participant, p=0.08), though not statistically significant.
• The checklist demonstrated a consistent trend towards improved error identification in both cohorts.

Conclusions:

• The use of a customized checklist enhances the review process for automated radiotherapy treatment plans.
• Implementing this checklist can lead to improved patient safety by increasing the detection of errors in AI-generated plans.