A multiple testing framework for diagnostic accuracy studies with co-primary endpoints.

Area of Science:

• Medical Informatics
• Statistical Methodology
• Machine Learning in Healthcare

Background:

• Machine learning (ML) shows promise for disease diagnosis and prognosis using complex data.
• Overoptimistic performance assessments are common, necessitating rigorous evaluation of ML models before clinical use.

Purpose of the Study:

• To propose a multiple testing framework for phase III diagnostic accuracy studies.
• To evaluate multiple ML diagnostic models concurrently within a single study.

Main Methods:

• Developed a parametric simultaneous test procedure for co-primary endpoints (sensitivity and specificity).
• Implemented a multiplicity correction for point and interval estimates.
• Utilized a Bayesian approach to determine the optimal number of models for simultaneous evaluation.

Main Results:

• The proposed framework provides asymptotic control of the family-wise error rate.
• Simulations show the multiple testing strategy improves final diagnostic model performance and statistical power.
• The Bayesian approach optimizes expected model performance based on prior data.

Conclusions:

• Assessing multiple diagnostic models in one study offers advantages when multiple comparison adjustments are employed.
• This framework supports more robust and powerful evaluation of ML diagnostic tools.
• The methodology aids in selecting optimal ML models for clinical practice.