Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data.

Area of Science:

• Behavioral Science
• Applied Behavior Analysis

Background:

• Observer agreement is crucial for data reliability in applied behavior analysis.
• Current methods for representing observer agreement include graphical and statistical indices, termed 'judgmental aids'.

Purpose of the Study:

• To evaluate the utility of graphical and statistical indices for representing observer agreement data.
• To discuss the advantages and limitations of these indices for both researchers and research consumers.

Main Methods:

• Analysis of graphical plotting of observer agreement data.
• Review of conventional statistical aids for observer agreement.
• Examination of alternative agreement estimates (e.g., percent occurrence, nonoccurrence) and correlational statistics (e.g., Kappa, Phi).

Main Results:

• Plotting observer agreement data serves as a useful supplement for researchers.
• Conventional statistical aids have limitations for research consumers.
• Special agreement estimates and correlational statistics offer improved utility for research consumers.

Conclusions:

• Graphical data presentation is valuable for researchers but less so for consumers.
• Alternative statistical methods are recommended for enhancing the interpretability of observer agreement for research consumers in applied behavior analysis.