Reliability analysis of observational data: problems, solutions, and software implementation

In the reliability analysis literature, little attention has been given to the various possible ways of creating a basis for the comparison required to compute observer agreement. One needs this comparison to turn a sequential list of behavioral records into a confusion matrix. It is shown that the way to do this depends on the research question one needs to answer. Four methods for creating a basis for comparison for the computation of observer agreement in observational data are presented. Guidelines are given for computing observer agreement in a way that fits one's goals. Finally, we discuss how these methods have been implemented in The Observer software. The Observer 4.1 supports all the methods that have been discussed. Most of these methods are not present in any other software package.