Evaluation of AI-based auto-contouring tools in radiotherapy: A single-institution study

Accurate delineation of organs at risk (OARs) is crucial yet time-consuming in the radiotherapy treatment planning workflow. Modern artificial intelligence (AI) technologies had made automation of OAR contouring feasible. This report details a single institution's experience in evaluating two commercial auto-contouring software tools and making well-informed decisions about their clinical adoption.

A cohort of 36 patients previously treated at our institution were selected for the software performance assessment. Fifty-eight OAR structures from seven disease sites were automatically segmented with each tool. Five radiation oncologists with different specialties qualitatively scored the automatic OAR contours' clinical usability by a 4-level scale (0-3), termed as quality score (QS), representing from "0: not usable" to "3: directly usable for a clinic." Additionally, quantitative comparison with clinically approved contours using Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95) was performed in complement to QS from physicians.

Software A achieved an average QS of 2.17 ± 0.69, comparable to Software B's average QS of 2.17 ± 0.72. Software B performed better with more OARs (42 vs. 37) that required minor or no modification than Software A. Major modifications were needed for 13 out of 58 automated contours from both tools. Both DSC and HD95 scores for the two tools were comparable to each other, with DSC: 0.67 ± 0.23 versus 0.66 ± 0.21 and HD95: 13.07 ± 15.84 versus 15.55 ± 18.45 for Software A and Software B, respectively. Correlation coefficients between the physician score and the quantitative metrics suggested that the contouring results from Software A aligned more closely with the physician's evaluations.

Based on our study, either software tool could produce clinically acceptable contours for about 65% of the OAR structures. However, further refinement is necessary for several challenging OARs to improve model performance.