Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Reliability analysis of observational data: problems, solutions, and software implementation.

Roland G Jansen1, Leon F Wiertz, Erik S Meyer

  • 1Noldus Information Technology, Wageningen, The Netherlands. r.jansen@noldus.nl

Behavior Research Methods, Instruments, & Computers : a Journal of the Psychonomic Society, Inc
|November 1, 2003
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Neurophysiological and multimodal sensory evaluation of baijiu and food pairing liking.

Food research international (Ottawa, Ont.)·2025
Same author

Validation framework for <i>in vivo</i> digital measures.

Frontiers in toxicology·2025
Same author

Disentangling rodent behaviors to improve automated behavior recognition.

Frontiers in neuroscience·2023
Same author

CatWalk XT gait parameters: a review of reported parameters in pre-clinical studies of multiple central nervous system and peripheral nervous system disease models.

Frontiers in behavioral neuroscience·2023
Same author

Emerging Role of Translational Digital Biomarkers Within Home Cage Monitoring Technologies in Preclinical Drug Discovery and Development.

Frontiers in behavioral neuroscience·2022
Same author

Measuring Behavior in the Home Cage: Study Design, Applications, Challenges, and Perspectives.

Frontiers in behavioral neuroscience·2021
Same journal

Real-time language comprehension research using the Apple-Psych system.

Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc·2014
Same journal

Constraints on the perception of synthetic speech generated by rule.

Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc·2014
Same journal

Perception of synthetic speech produced automatically by rule: Intelligibility of eight text-to-speech systems.

Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc·2012
Same journal

Ninety-three pictures and 108 questions for the elicitation of homophones.

Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc·2011
Same journal

Free-association norms for the Spanish names of the Snodgrass and Vanderwart pictures.

Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc·2005
Same journal

French normative data and naming times for action pictures.

Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc·2005
See all related articles

This study introduces four methods for calculating observer agreement in observational data, crucial for reliability analysis. The choice of method depends on the specific research question, ensuring accurate behavioral record comparison.

Area of Science:

  • Behavioral Science
  • Psychometrics
  • Data Analysis

Background:

  • Observer agreement is critical for the reliability of observational studies.
  • Existing literature lacks comprehensive methods for establishing a comparison basis for observer agreement calculations.
  • Turning sequential behavioral records into a confusion matrix requires a defined comparison framework.

Purpose of the Study:

  • To address the gap in reliability analysis literature regarding methods for computing observer agreement.
  • To present four distinct methods for creating a comparison basis for observer agreement.
  • To provide guidelines for selecting the appropriate method based on research objectives.

Main Methods:

  • Development and presentation of four novel methods for generating a comparison basis for observer agreement.

Related Experiment Videos

  • Application of these methods to observational data to create confusion matrices.
  • Implementation and discussion of these methods within The Observer software (version 4.1).

    • Main Results:

    • The study demonstrates that the optimal method for creating a comparison basis is contingent upon the specific research question.
    • Four distinct methods for computing observer agreement are detailed and illustrated.
    • The Observer software version 4.1 incorporates all presented methods, offering advanced functionality.

    Conclusions:

    • The selection of a method for observer agreement computation must align with the research question to ensure valid reliability analysis.
    • The presented methods offer a flexible framework for enhancing the rigor of observational research.
    • The Observer software provides unique capabilities by supporting multiple observer agreement computation methods, exceeding the offerings of other packages.