Measuring agreement in medical informatics reliability studies.

Area of Science:

• Statistics
• Psychometrics
• Data Analysis

Background:

• Agreement measures are crucial for reliability studies involving categorical data.
• Simple measures like observed agreement provide initial insights.
• Chance-corrected agreement, such as the kappa statistic, is widely used but sensitive to experimental factors.

Purpose of the Study:

• To highlight the importance of analyzing disagreement components for improving reliability.
• To explore advanced modeling approaches for understanding agreement metrics.
• To demonstrate how decision-making models clarify the behavior of agreement measures.

Main Methods:

• Review of agreement measures for categorical data.
• Analysis of disagreement components.
• Application of decision-making models (tetrachoric, polychoric, latent trait, latent class).

Main Results:

• Simple agreement measures offer sample insights but can be misleading.
• Kappa statistic's magnitude is context-dependent.
• Decision-making models help dissect disagreement and understand metric behavior.
• Low response prevalence can bias agreement estimates (e.g., kappa underestimation, observed agreement overestimation).

Conclusions:

• Separating disagreement components is key for enhancing instrument and rater reliability.
• Advanced statistical models offer deeper insights into agreement metrics.
• Understanding the influence of response prevalence is vital for accurate reliability assessment.