Expected Value of Sample Information Calculations for Risk Prediction Model Development

Area of Science:

• Biostatistics
• Clinical Epidemiology
• Decision Analysis

Background:

• Risk prediction models are often presented as deterministic, but finite sample sizes introduce inherent uncertainty.
• Traditional methods address performance metric uncertainty and prediction stability, but not clinical utility directly.
• Statistical inference is less relevant for evaluating clinical utility using metrics like net benefit.

Purpose of the Study:

• To define and quantify the Expected Value of Sample Information (EVSI) for risk prediction model development.
• To evaluate the expected gain in clinical utility from acquiring additional development data.
• To propose a decision-theoretic approach complementing classical inferential methods in study design.

Main Methods:

• Defined EVSI as the expected gain in net benefit (NB) from an additional development sample.
• Proposed a bootstrap-based algorithm for computing EVSI.
• Demonstrated the feasibility and face validity of the EVSI computation algorithm in a case study.

Main Results:

• The Expected Value of Sample Information (EVSI) was defined and computed using a novel bootstrap algorithm.
• The study showed that procuring more development data is associated with an expected gain in model utility.
• The proposed method demonstrated feasibility and face validity in a practical case study.

Conclusions:

• Decision-theoretic metrics, such as EVSI, can effectively complement classical inferential methods for designing risk prediction model studies.
• EVSI provides a framework to quantify the value of additional data in terms of clinical utility.
• This approach aids in optimizing sample size and resource allocation for developing robust risk prediction models.