Benchmarking and performance evaluation of a novel deformable image registration software for radiotherapy CT images

We evaluated and benchmarked a novel deformable image registration (DIR) software functionality (DirOne, Cosylab d.d., Ljubljana, Slovenia) by comparing it to two commercial systems, MIM and VelocityAI, following AAPM task group 132 (TG-132) guidelines.

Three publicly available datasets were used for evaluation. The first dataset includes primary and deformed phantom images for a male pelvis. The second, from DIR-Lab, contains ten sets of 4D CT thoracic scans. The third dataset, from the DIR Evaluation Project (DIREP), includes ten head and neck CTs. VelocityAI and MIM served as benchmarks to assess DirOne's performance. Target registration error (TRE), dice similarity coefficient (DSC), and mean distance to agreement (MDA) were the evaluation metrics.

For TRE, the average results for DirOne, MIM, and VelocityAI were 3.3 ± 3.1 mm, 2.7 ± 3.7 mm, and 3.4 ± 2.4 mm, respectively. For DSC, DirOne achieved 0.96 ± 0.02, MIM 0.98 ± 0.02, and VelocityAI 0.98 ± 0.01 across the first and second datasets. In the DIREP dataset, DirOne achieved 0.73 ± 0.34 for MDA and 0.91 ± 0.03 for DSC; MIM achieved 0.54 ± 0.36 and 0.93 ± 0.02, and VelocityAI 0.93 ± 0.38 and 0.90 ± 0.03.

The novel DIR software demonstrated clinically acceptable accuracy compared to other commercial systems, supporting its potential use in radiotherapy treatment planning applications such as automatic image segmentation, 4D segmentation propagation, and dose warping.