A cautionary note on the use of probability values to evaluate interobserver agreement

Area of Science:

• Statistics
• Biostatistics
• Medical Informatics

Background:

• Interobserver agreement is crucial for reliable data collection in various scientific fields.
• Existing statistical methods for assessing interobserver agreement may be inadequate when data exhibit serial correlations.

Purpose of the Study:

• To identify limitations in current statistical approaches for interobserver agreement.
• To propose alternative methods for accurate statistical significance testing with serially correlated data.

Main Methods:

• Analysis of proposed statistical methods for interobserver agreement.
• Simulation or theoretical evaluation of methods on serially correlated data.
• Comparison of results from standard methods versus proposed interval sampling or Markovian techniques.

Main Results:

• Standard methods produce erroneous probability values for interobserver agreement on serially correlated data.
• Limiting analysis to every k(th) data interval provides a viable alternative.
• Markovian techniques effectively accommodate serial correlations for accurate assessment.

Conclusions:

• Traditional statistical significance testing for interobserver agreement is unreliable with serially correlated data.
• Interval sampling and Markovian techniques offer robust solutions for accurate statistical evaluation.
• Adoption of these methods is recommended for reliable interobserver agreement assessment.