A permutation method to assess heterogeneity in external validation for risk prediction models.

Area of Science:

• Biostatistics
• Clinical Epidemiology
• Bioinformatics

Background:

• The reliability of risk prediction models hinges on their performance in external datasets.
• External validation is essential to ensure generalizability and clinical utility.
• Assessing heterogeneity between development and validation data is critical for model transportability.

Purpose of the Study:

• To introduce a novel permutation method for assessing heterogeneity in external validation of risk prediction models.
• To provide a quantitative measure (permutation p-value) of homology between development and external validation datasets.
• To evaluate the statistical properties and practical utility of the proposed permutation method.

Main Methods:

• A permutation-based approach was developed to quantify the similarity between model development and external validation datasets.
• Monte-Carlo simulations were performed to assess the statistical performance of the permutation method.
• The method was demonstrated using two independent microarray breast cancer datasets.

Main Results:

• The permutation p-value effectively measures the degree of homology between datasets.
• A p-value less than 0.05 suggests that the prediction model may require revision or updating for the external population.
• Simulations confirmed the method's statistical validity, and dataset analyses showed its practical applicability.

Conclusions:

• The proposed permutation method offers a straightforward and effective way to assess heterogeneity in external validation.
• This method aids in determining the suitability of risk prediction models for new populations.
• Routine implementation of this permutation method is recommended for robust external validation practices.