Observation and judgment in psychology: assessing agreement among markings of behavioral events

In a sequence from a videotaped dialogue between two persons (e.g., psychotherapist and patient), observers are given the task of searching for, marking, and commenting on conspicuous events. The observers receive no information on where to look for conspicuous events and no information on what events are worth marking. Moreover, nothing is known about the observers' marking preferences. Marking is therefore very spontaneous and subjective. This marking freedom leads to the problem that a superimposition of all observers' markings yields complicated configurations of mutually overlapping intervals. To address this difficulty, a new definition of the concept of marking agreement is proposed that is based upon relational considerations. According to this definition, a set of agreeing marking subsets of various sizes can be found in marking configurations. A significance concept is proposed to allow comparison of different configurations, and standardization on the basis of random configurations is performed using Monte Carlo simulations.