Diagnostic accuracy analysis for multiple raters using probit hierarchical model for ordinal ratings.

Area of Science:

• Statistics
• Medical Informatics
• Biostatistics

Background:

• Ordinal classification is crucial in medical diagnosis.
• Accurate assessment of rater performance and disease severity is challenging with multiple raters.
• Existing methods often lack closed-form solutions for diagnostic accuracy metrics.

Purpose of the Study:

• To develop a hierarchical model for ordinal classification with multiple raters.
• To provide closed-form expressions for receiver operator characteristic (ROC) curves and area under the ROC curves (AUC).
• To extend the model with covariates for enhanced diagnostic accuracy analysis.

Main Methods:

• A Probit hierarchical model is proposed, linking rater ratings to diagnostic skills (bias, magnifier) and latent disease severity.
• Latent disease severity is modeled using a latent class normal mixture distribution.
• Covariate information is incorporated via a regression layer for diagnostic skills and/or disease severity.

Main Results:

• The proposed model yields closed-form expressions for overall and individual rater ROC curves and AUC.
• Extended covariate models provide closed-form solutions for covariate-specific ROCs and AUCs.
• The methods are demonstrated effectively using a mammography dataset.

Conclusions:

• The developed analytical tools significantly simplify traditional diagnostic accuracy analysis.
• The hierarchical model offers a robust framework for understanding rater performance and disease severity.
• This approach facilitates more precise evaluation in multi-rater ordinal classification tasks.