Development of robustness evaluation strategies for enabling statistically consistent reporting

Robustness evaluation of proton therapy treatment plans is essential for ensuring safe treatment delivery. However, available evaluation procedures feature a limited exploration of the actual robustness of the plan and generally do not provide confidence levels. This study compared established and more sophisticated robustness evaluation procedures, with quantified confidence levels. We have evaluated several robustness evaluation methods for 5 bilateral head-and-neck patients optimized considering spot scanning delivery and with a conventional CTV-to-PTV margin of 4 mm. Method (1) good practice scenario selection (GPSS) (e.g. +/- 4 mm setup error 3% range uncertainty); (2) statistically sound scenario selection (SSSS) either only on or both on and inside isoprobability hypersurface encompassing 90% of the possible errors; (3) statistically sound dosimetric selection (SSDS). In the last method, the 90% best plans were selected according to either target coverage quantified by D ₉₅ (SSDS_D ₉₅) or to an approximation of the final objective function (OF) used during treatment optimization (SSDS_OF). For all methods, we have considered systematic setup and systematic range errors. A mix of systematic and random setup errors were also simulated for SSDS, but keeping the same conventional margin of 4 mm. All robustness evaluations have been performed using the fast Monte Carlo dose engine MCsquare. Both SSSS strategies yielded on average very similar results. SSSS and GPSS yield comparable values for target coverage (within 0.5 Gy). The most noticeable differences were found for the CTV between GPSS, on the one hand, and SSDS_D ₉₅ and SSDS_OF, on the other hand (average worst-case D ₉₈ were 2.8 and 2.0 Gy larger than for GPSS, respectively). Simulating explicitly random errors in SSDS improved almost all DVH metrics. We have observed that the width of DVH-bands and the confidence levels depend on the method chosen to sample the scenarios. Statistically sound estimation of the robustness of the plan in the dosimetric space may provide an improved insight on the actual robustness of the plan for a given confidence level.