Why do so many prognostic factors fail to pan out?

Area of Science:

• Biostatistics
• Oncology
• Medical Informatics

Background:

• Failure to replicate study findings can stem from overlooked data exploration consequences.
• Spurious significance due to extensive data analysis is a common pitfall in research.

Purpose of the Study:

• To investigate the impact of testing multiple prognostic factor cutpoints on Type I error rates.
• To assess the reliability of validation sets in confirming prognostic factors for node-negative breast cancer.

Main Methods:

• Simulation experiments using relapse-free survival data from node-negative breast cancer patients.
• Datasets (250 or 500 cases) were split into training (cutpoint selection) and validation (cutpoint confirmation) sets.
• Analysis focused on Type I error rates and statistical power across varying numbers of tested cutpoints.

Main Results:

• Testing numerous cutpoints significantly increased the risk of Type I errors (false positives).
• Statistical power to detect true differences was substantial and rose with more cutpoints.
• Type I error rates remained stable on independent validation sets, irrespective of training set analyses.
• Validation set power to detect true differences was unaffected by the number of cutpoints tested.

Conclusions:

• Exploratory prognostic factor analyses require methods to adjust for increased Type I errors.
• Independent validation sets are recommended to confirm findings and mitigate spurious significance.
• Statistical adjustments like ad hoc factors or other estimation methods should be employed to accurately assess risk.