Assessing observer accuracy in continuous recording of rate and duration: three algorithms compared.

Area of Science:

• Behavior Analysis
• Psychology
• Research Methodology

Background:

• Interobserver agreement (IOA) is crucial for ensuring data reliability in behavior analysis.
• Continuous recording is a common data collection method, often analyzed using specific IOA algorithms.
• Previous research highlights the importance of selecting appropriate IOA methods for accurate data assessment.

Purpose of the Study:

• To evaluate the accuracy of data recorded by observers using handheld computers.
• To assess the performance of three common interobserver agreement algorithms with continuous recording data.
• To identify potential biases in accuracy estimates produced by different IOA algorithms under varying response rates and durations.

Main Methods:

• Twelve observers recorded rate and duration of responding from video samples on handheld computers.
• Three frequently used interobserver agreement algorithms (block-by-block, exact agreement, time-window analysis) were applied.
• Recorded data files were compared against criterion records to determine observer accuracy.

Main Results:

• Block-by-block and exact agreement algorithms showed inflated accuracy estimates at lower response rates and durations.
• The exact agreement method proved too stringent for high response rates (23.5 responses/min) and high relative durations (72% of session).
• Time-window analysis inflated accuracy at relatively high rates/durations but not at low rates/durations (4.8 responses/min, 8% of session).

Conclusions:

• The choice of interobserver agreement algorithm significantly impacts the accuracy assessment of continuous recording data.
• Block-by-block and exact agreement methods may overestimate accuracy, particularly under low-rate conditions.
• Time-window analysis can also lead to inflated accuracy estimates, especially at higher response rates and durations.