Thresholds for human detection of patient setup errors in digitally reconstructed portal images of prostate fields

Computer-assisted methods to analyze electronic portal images for the presence of treatment setup errors should be studied in controlled experiments before use in the clinical setting. Validation experiments using images that contain known errors usually report the smallest errors that can be detected by the image analysis algorithm. This paper offers human error-detection thresholds as one benchmark for evaluating the smallest errors detected by algorithms. Unfortunately, reliable data are lacking describing human performance. The most rigorous benchmarks for human performance are obtained under conditions that favor error detection. To establish such benchmarks, controlled observer studies were carried out to determine the thresholds of detectability for in-plane and out-of-plane translation and rotation setup errors introduced into digitally reconstructed portal radiographs (DRPRs) of prostate fields.

Seventeen observers comprising radiation oncologists, radiation oncology residents, physicists, and therapy students participated in a two-alternative forced choice experiment involving 378 DRPRs computed using the National Library of Medicine Visible Human data sets. An observer viewed three images at a time displayed on adjacent computer monitors. Each image triplet included a reference digitally reconstructed radiograph displayed on the central monitor and two DRPRs displayed on the flanking monitors. One DRPR was error free. The other DRPR contained a known in-plane or out-of-plane error in the placement of the treatment field over a target region in the pelvis. The range for each type of error was determined from pilot observer studies based on a Probit model for error detection. The smallest errors approached the limit of human visual capability. The observer was told what kind of error was introduced, and was asked to choose the DRPR that contained the error. Observer decisions were recorded and analyzed using repeated-measures analysis of variance.

The thresholds of detectability averaged over all observers were approximately 2.5 mm for in-plane translations, 1.6 degrees for in-plane rotations, 1 degrees for out-of-plane rotations, and 8% change in magnification for out-of-plane translations along the central axis. When one inexperienced observer is excluded, the average threshold for change in magnification is 5%. Experienced observers tended to perform better, but differences between groups were not statistically significant. Thresholds were computed as averages over all observers. Because of the broad range of observer capabilities, some detection tasks were too difficult for some observers, leading to missing threshold values in our data analysis. The missing values were excluded from computation of the average thresholds reported above. The effect of the missing values is to bias the average values toward the best human performance.

Under favorable conditions, humans can detect small errors in setup geometry. The thresholds for error detection reported in this study are believed to represent rigorous but reasonable benchmarks that can be incorporated into studies evaluating algorithms for computer-assisted detection of setup errors in electronic portal images.