Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Observer agreement for event sequences: methods and software for sequence alignment and reliability estimates.

Vicenç Quera1, Roger Bakeman, Augusto Gnisci

  • 1Departamento de Metodología de las Ciencias del Comportamiento, Universidad de Barcelona, Barcelona, Spain. vquera@ub.edu

Behavior Research Methods
|June 8, 2007
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

The psychology of offensive and defensive intergroup violence: Preregistered insights from 58 countries.

Proceedings of the National Academy of Sciences of the United States of America·2026
Same author

The mental health toll of the Russian-Ukraine war across 11 countries: Cross-sectional data on war-related stressors, PTSD and CPTSD symptoms.

Psychiatry research·2024
Same author

Insights from a Developmental Psychologist: Going Against the Grain.

The Journal of genetic psychology·2024
Same author

How HEXACO personality traits are involved in school performance of middle school adolescents (10-14 years).

Acta psychologica·2024
Same author

Active Direction: A new observational measure of African American parenting.

Infant behavior & development·2024
Same author

Targeting Peer Contagion Dynamics in Children with ADHD: Effects from a Two-Site Randomized Controlled Trial.

Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53·2024
Same journal

Planned missingness in intensive longitudinal studies: Extensions and comparisons of multiform designs.

Behavior research methods·2026
Same journal

A validity-guided workflow for robust large language model research in psychology.

Behavior research methods·2026
Same journal

Are 7-point Likert scales preferable to 5-point scales in language research?

Behavior research methods·2026
Same journal

Generative psychometrics via AI-GENIE: Automatic item generation and validation with network-integrated evaluation.

Behavior research methods·2026
Same journal

Exploring psychological tradeoffs: Developing and demonstrating an R Shiny app for Pareto optimization.

Behavior research methods·2026
Same journal

The performance of Bayesian fit measures in detecting misspecified multilevel structural equation modeling.

Behavior research methods·2026
See all related articles

This study introduces an optimal event sequence alignment method for calculating inter-rater reliability using Cohen's kappa. High observer accuracy (≥90%) generally ensures reliable agreement (kappa ≥0.60) in event coding.

Area of Science:

  • Quantitative Psychology
  • Behavioral Science
  • Computational Linguistics

Background:

  • Event sequence data often lacks precise onset times, complicating inter-rater reliability assessment.
  • Traditional agreement statistics like Cohen's kappa require aligned event sequences.
  • Optimal alignment is crucial for accurate measurement of observer agreement.

Purpose of the Study:

  • To develop and evaluate a method for optimally aligning event sequences when onset times are not preserved.
  • To assess the impact of various factors on the reliability of alignment-based Cohen's kappa.
  • To provide computational tools for sequence alignment and agreement analysis.

Main Methods:

  • Adaptation of the Needleman-Wunsch algorithm for nucleotide sequence alignment to event sequence alignment.

Related Experiment Videos

  • Simulation studies varying coder accuracy, number of codes, sequence length, and code variability.
  • Development of Global Sequence Alignment (GSA) for simulations and Event Alignment (ELign) for practical use.

    • Main Results:

    • Observer accuracies of 90% or higher typically yield alignment kappas of 0.60 or better.
    • Alignment kappa is generally robust to sequence length, number of codes, and code probability variability.
    • Missed events and false alarms negatively impact alignment kappa values.

    Conclusions:

    • The proposed alignment method provides reliable Cohen's kappa values under typical conditions with high observer accuracy.
    • Alignment algorithms should prioritize substitutions over insertions/deletions to mitigate the impact of missed events and false alarms.
    • The developed software (GSA and ELign) facilitates the application of this alignment method.