Comparison of typical evaluation methods for computer-aided diagnostic schemes: Monte Carlo simulation study

Computer-aided diagnostic (CAD) schemes have been developed for assisting radiologists in the detection of various lesions in medical images. Many evaluation approaches, such as the resubstitution, leave-one-out, cross-validation, and hold-out methods, have been employed for the assessment of the performance of various CAD schemes. For these evaluation methods, some investigators have studied their bias in the estimated performance levels of CAD schemes trained with finite samples. However, systematical study has not been conducted for the comparison of these common evaluation methods in terms of multiple important characteristics such as the bias of the estimated performance, the generalization performance, and the uniqueness of the trained CAD scheme. Therefore, in this study, we examined and compared these important characteristics for various evaluation methods and attempted to provide a guideline for investigators to select appropriate evaluation methods for the assessment of CAD schemes in typical practical situations.