Identify the underlying true model from other models for clinical practice using model performance measures.

Area of Science:

• Clinical modeling
• Statistical performance evaluation
• Biostatistics

Background:

• Identifying the true data-generating model is crucial for reliable clinical practice.
• Current performance measures may not adequately distinguish true models from suboptimal candidates.

Purpose of the Study:

• To evaluate if conventional performance measures can identify the true data-generating model in clinical settings.
• To assess model identification across various simulation scenarios and a cardiovascular disease (CVD) risk prediction case.

Main Methods:

• Simulated clinical data from thousands of true model scenarios.
• Trained and compared numerous candidate and true models using 25 conventional performance measures.
• Evaluated univariate, multivariate simulations, and a CVD risk prediction case analysis.

Main Results:

• True models showed distinct performance compared to random (flip-coin) models.
• Minimal performance differences were observed between true models and candidates with added noise.
• Slight differences were found between true models and proxy models lacking causal predictors.

Conclusions:

• Conventional measures do not consistently identify the true model for binary outcomes, even when present.
• There is a need for novel statistical approaches to identify true causal models from proxy models, especially those with noise or missing predictors.