Correcting for optimistic prediction in small data sets.

Area of Science:

• Biostatistics
• Medical Screening
• Statistical Modeling

Background:

• The C statistic is a key metric for evaluating screening test accuracy.
• Overfitting in small datasets often leads to overestimation (optimism) of the C statistic.
• Existing methods to correct for optimism are diverse and some introduce bias.

Purpose of the Study:

• To evaluate and compare different methods for adjusting the optimism of the C statistic.
• To identify reliable methods for obtaining unbiased C statistic estimates in clinical screening.

Main Methods:

• Analysis of UK Down syndrome and Scottish national pregnancy discharge clinical datasets.
• Comparison of sample splitting, various cross-validation techniques (leave-1-out, with replication), and bootstrapping.
• Evaluation of a novel method: leave-pair-out cross-validation.

Main Results:

• Sample splitting, leave-1-out, and cross-validation without replication yielded biased C statistic estimates with higher errors.
• Cross-validation with replication, bootstrapping, and leave-pair-out cross-validation produced unbiased estimates with comparable errors.
• In simulations, these three methods performed similarly, though bootstrapping showed lower errors with limited data.

Conclusions:

• Cross-validation with replication, bootstrapping, and leave-pair-out cross-validation are recommended for unbiased C statistic adjustment.
• These methods offer reliable performance across different dataset sizes and C statistic values.
• Careful selection of optimism adjustment methods is crucial for accurate screening test evaluation.