Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Naturalistic Observations02:30

Naturalistic Observations

If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
Reliability and Validity01:29

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
Statistical Analysis: Overview01:11

Statistical Analysis: Overview

When we take repeated measurements on the same or replicated samples, we will observe inconsistencies in the magnitude. These inconsistencies are called errors. To categorize and characterize these results and their errors, the researcher can use statistical analysis to determine the quality of the measurements and/or suitability of the methods.
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
Observational Studies01:11

Observational Studies

Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One example of...
Kendall's Coefficient of Concordance01:20

Kendall's Coefficient of Concordance

Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects or...
Data Collection by Observations01:08

Data Collection by Observations

Data collection refers to a systematic way of obtaining, observing, measuring, and analyzing accurate information. Observational studies are one of the most widely used methods of data collection. It involves collecting data by observing the behavior and physical characteristics of a sample without making any modifications to the sample.
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Synchrony in therapist's and patient's vocally encoded arousal and its association with the quality of the therapeutic relationship.

Psychotherapy research : journal of the Society for Psychotherapy Research·2026
Same author

Association of unhealthy alcohol use reported in routine outpatient screening with 30-day hospital readmission risk.

Journal of substance use and addiction treatment·2026
Same author

"You're Hoping for the Best, but Preparing for the Worst": Discussions of Starting Buprenorphine in the Context of Fentanyl Use with Clinicians and People Who Use Fentanyl.

Journal of general internal medicine·2026
Same author

Recognition of depression by nurses in primary healthcare in Zimbabwe: Cross-sectional study.

Global mental health (Cambridge, England)·2026
Same author

Smartphone-Based Contingency Management for Patients Who Use Methamphetamine: Qualitative Analysis of Patient and Clinician Perspectives.

JMIR formative research·2026
Same author

Addressing substance use and mental illness among Quinault Indian Nation adolescents and young adults: community perspectives on community and cultural connection.

Addiction science & clinical practice·2026
Same journal

Conducting Simulation Studies in the R Programming Environment.

Tutorials in quantitative methods for psychology·2014
See all related articles

Related Experiment Video

Updated: May 20, 2026

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
08:40

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity

Published on: June 12, 2019

Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.

Kevin A Hallgren1

  • 1University of New Mexico, Department of Psychology.

Tutorials in Quantitative Methods for Psychology
|July 27, 2012
PubMed
Summary
This summary is machine-generated.

This study highlights common errors in assessing inter-rater reliability (IRR), offering guidance on proper statistical methods and reporting. It emphasizes the importance of accurate IRR for robust research findings and statistical power.

More Related Videos

Measuring Light-Switching Behavior Using an Occupancy and Light Data Logger
05:50

Measuring Light-Switching Behavior Using an Occupancy and Light Data Logger

Published on: January 16, 2020

Related Experiment Videos

Last Updated: May 20, 2026

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
08:40

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity

Published on: June 12, 2019

Measuring Light-Switching Behavior Using an Occupancy and Light Data Logger
05:50

Measuring Light-Switching Behavior Using an Occupancy and Light Data Logger

Published on: January 16, 2020

Area of Science:

  • Methodology
  • Statistics
  • Observational Research

Background:

  • Inter-rater reliability (IRR) is crucial for validating observational data.
  • Many studies incorrectly assess or report IRR, impacting result interpretation.
  • The influence of IRR on statistical power for hypothesis testing is often overlooked.

Purpose of the Study:

  • To provide an overview of methodological issues in IRR assessment.
  • To guide researchers in selecting, computing, interpreting, and reporting IRR statistics.
  • To address the impact of IRR on subsequent statistical analyses.

Main Methods:

  • Review of methodological issues in IRR assessment.
  • Focus on study design, statistical selection, computation, and interpretation.
  • Inclusion of SPSS and R syntax for common IRR statistics (Cohen's kappa, ICC).

Main Results:

  • Identified common statistical and reporting errors in IRR assessment.
  • Provided practical guidance on appropriate IRR statistical procedures.
  • Demonstrated computation of Cohen's kappa and intra-class correlations using statistical software.

Conclusions:

  • Accurate IRR assessment is vital for research integrity.
  • Proper reporting of IRR enhances study transparency and replicability.
  • Addressing IRR is essential for maintaining adequate statistical power in research.