A model-free framework for evaluating the reliability of a new device with multiple imperfect reference standards

Area of Science:

• Medical Imaging
• Biostatistics
• Diagnostic Accuracy

Background:

• Evaluating computer-aided diagnostic (CAD) devices typically relies on gold standard tests.
• Gold standards are often unavailable in clinical studies, necessitating the use of multiple imperfect reference standards.
• Heterogeneity in diagnostic accuracy across reference standards can bias device evaluation.

Purpose of the Study:

• To develop a statistical framework for evaluating CAD devices using multiple imperfect reference standards.
• To address the challenge of heterogeneous diagnostic accuracy among reference standards.
• To provide an intuitive and easy-to-use method for device reliability assessment.

Main Methods:

• A statistical framework assessing agreement between a CAD device and a weighted sum of multiple imperfect reference standards.
• A model-free, unsupervised inductive procedure to determine weights for reference standards based on their relative reliability.
• Recursive weight assignment favoring more consistent and majority-opinion reference standards.

Main Results:

• The proposed framework effectively evaluates CAD devices by accounting for varying reference standard accuracy.
• Weights are assigned to reference standards without requiring modeling assumptions or external data.
• The method demonstrated its utility in evaluating a CAD device for kidney obstruction against multiple physician assessments.

Conclusions:

• The developed statistical framework offers a robust method for evaluating CAD devices when gold standards are absent.
• The unsupervised weighting procedure handles accuracy heterogeneity among reference standards effectively.
• This approach provides a reliable tool for assessing diagnostic device performance in real-world clinical settings.