Using big data to retrospectively validate the COMPASS-CAT risk assessment model: considerations on methodology

Area of Science:

• Medical Informatics
• Clinical Epidemiology
• Health Services Research

Background:

• External validation is crucial for clinical prediction model adoption.
• Methodologically sound external validation studies are infrequent.
• Large datasets can mitigate validation failures but require transparent reporting.

Purpose of the Study:

• To describe methodological challenges in the first retrospective external validation of the COMPASS-CAT Risk Assessment Model using multiple big datasets.
• To present a structured plan for overcoming challenges and reducing bias in external validation.
• To ensure the validation study adheres to prediction model reporting guidelines.

Main Methods:

• Utilized multiple large, real-world datasets for retrospective external validation.
• Addressed challenges in defining time-sensitive variables and outcome measures.
• Developed validated definitions from administrative codes in non-research databases.

Main Results:

• Successfully performed the first retrospective external validation of the COMPASS-CAT model.
• Identified and addressed key methodological hurdles in big data validation.
• Established a framework for bias reduction in external validation studies.

Conclusions:

• External validation of prediction models using big data presents unique methodological challenges.
• A structured approach is essential for rigorous validation and reliable reporting.
• This study provides a roadmap for conducting robust external validation of cancer-associated thrombosis risk models.