Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems

This study quantifies variation in radiation treatment plan quality for plans generated by a population of treatment planners given very specific plan objectives.

A "Plan Quality Metric" (PQM) with 14 submetrics, each with a unique value function, was defined for a prostate treatment plan, serving as specific goals of a hypothetical "virtual physician." The exact PQM logic was distributed to a population of treatment planners (to remove ambiguity of plan goals or plan assessment methodology) as was a predefined computed tomographic image set and anatomic structure set (to remove anatomy delineation as a variable). Treatment planners used their clinical treatment planning system (TPS) to generate their best plan based on the specified goals and submitted their results for analysis.

One hundred forty datasets were received and 125 plans accepted and analyzed. There was wide variability in treatment plan quality (defined as the ability of the planners and plans to meet the specified goals) quantified by the PQM. Despite the variability, the resulting PQM distributions showed no statistically significant difference between TPS employed, modality (intensity modulated radiation therapy versus arc), or education and certification status of the planner. The PQM results showed negligible correlation to number of beam angles, total monitor units, years of experience of the planner, or planner confidence.

The ability of the treatment planners to meet the specified plan objectives (as quantified by the PQM) exhibited no statistical dependence on technologic parameters (TPS, modality, plan complexity), nor was the plan quality statistically different based on planner demographics (years of experience, confidence, certification, and education). Therefore, the wide variation in plan quality could be attributed to a general "planner skill" category that would lend itself to processes of continual improvement where best practices could be derived and disseminated to improve the mean quality and minimize the variation in any population of treatment planners.