Jove
Visualize
Contact Us

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

13.4K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.4K
One-Way ANOVA: Equal Sample Sizes01:15

One-Way ANOVA: Equal Sample Sizes

3.7K
One-Way ANOVA can be performed on three or more samples with equal or unequal sample sizes. When one-way ANOVA is performed on two datasets with samples of equal sizes, it can be easily observed that the computed F statistic is highly sensitive to the sample mean.
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
3.7K
Theory of Attribution II: Kelley's Covariation Theory01:29

Theory of Attribution II: Kelley's Covariation Theory

165
Attribution theory plays a crucial role in social psychology, helping to explain how individuals interpret the causes of behavior. One prominent model within this field is Harold Kelley's covariation theory, which provides a systematic approach to determining whether internal traits or external circumstances drive a person's actions. The model posits that individuals rely on three key types of information—consensus, consistency, and distinctiveness—to make these judgments.Consensus:...
165
Friedman Two-way Analysis of Variance by Ranks01:21

Friedman Two-way Analysis of Variance by Ranks

357
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
357
Two-Way ANOVA01:17

Two-Way ANOVA

2.9K
The two-way ANOVA is an extension of the one-way ANOVA. It is a statistical test performed on three or more samples categorized by two factors - a row factor and a column factor. Ronald Fischer mentioned it in 1925 in his book 'Statistical Methods for Researchers.'
The two-way ANOVA analysis initially begins by stating the null hypothesis that there is an interaction effect between the two factors of a dataset. This effect can be visualized using line segments formed by joining the...
2.9K
Variability: Analysis01:11

Variability: Analysis

254
Measures of variability are statistical metrics that reveal the dispersion pattern within a dataset. They are pivotal in biostatistics, providing insights into the heterogeneity within health and biological data. Variability signifies the degree to which data points diverge from one another, helping researchers understand the potential range of values and associated uncertainty within the data.
The range is a simple measure of variability, indicating the difference between the highest and...
254

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Bias and precision in true-score estimation.

The British journal of mathematical and statistical psychology·2026
Same author

Reducing Attenuation Bias in Regression Analyses Involving Rating Scale Data via Psychometric Modeling.

Psychometrika·2026
Same author

Standard Errors for Reliability Coefficients.

Psychometrika·2025
Same author

Empirical Bayes Priors for MCMC Estimation of the Multivariate Social Relations Model.

Multivariate behavioral research·2025
Same author

How to Estimate Intraclass Correlation Coefficients for Interrater Reliability from Planned Incomplete Data.

Multivariate behavioral research·2025
Same author

A tutorial on estimating dynamic treatment regimes from observational longitudinal data using lavaan.

Psychological methods·2025
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Nov 10, 2025

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
08:27

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Published on: September 27, 2019

7.1K

Interrater reliability for multilevel data: A generalizability theory approach.

Debby Ten Hove1, Terrence D Jorgensen1, L Andries van der Ark1

  • 1Research Institute of Child Development and Education.

Psychological Methods
|April 5, 2021
PubMed
Summary
This summary is machine-generated.

New methods using generalizability theory improve interrater reliability (IRR) estimates for multilevel data. This approach accounts for nested structures, offering more accurate subject- and cluster-level reliability in research.

More Related Videos

Basics of Multivariate Analysis in Neuroimaging Data
06:35

Basics of Multivariate Analysis in Neuroimaging Data

Published on: July 24, 2010

17.1K
The Innovation Arena: A Method for Comparing Innovative Problem-Solving Across Groups
14:14

The Innovation Arena: A Method for Comparing Innovative Problem-Solving Across Groups

Published on: May 13, 2022

6.1K

Related Experiment Videos

Last Updated: Nov 10, 2025

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
08:27

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Published on: September 27, 2019

7.1K
Basics of Multivariate Analysis in Neuroimaging Data
06:35

Basics of Multivariate Analysis in Neuroimaging Data

Published on: July 24, 2010

17.1K
The Innovation Arena: A Method for Comparing Innovative Problem-Solving Across Groups
14:14

The Innovation Arena: A Method for Comparing Innovative Problem-Solving Across Groups

Published on: May 13, 2022

6.1K

Area of Science:

  • Psychometrics
  • Multilevel Modeling
  • Statistical Methods

Background:

  • Current interrater reliability (IRR) coefficients fail to account for the nested structure inherent in multilevel observational data.
  • This oversight leads to biased estimations of both subject- and cluster-level IRR, compromising data integrity.

Purpose of the Study:

  • To introduce a novel conceptualization and estimation method for IRR in continuous multilevel observational data using generalizability theory.
  • To address the limitations of existing IRR coefficients by incorporating the multilevel data structure.

Main Methods:

  • Employed generalizability theory to decompose multilevel data variance into subject, cluster, and rater components.
  • Utilized Markov chain Monte Carlo (MCMC) estimation to derive IRR coefficients (intraclass correlation coefficients - ICCs) from these variance components.
  • Conducted simulation studies to assess the quality of MCMC estimates and the impact of rater numbers on ICC accuracy.

Main Results:

  • Identified small numbers of raters as a primary source of bias and inefficiency in ICC estimations.
  • Demonstrated that a planned missing data design significantly mitigates estimation difficulties in multilevel IRR.
  • Validated the proposed method using real-world data on student-teacher relationships.

Conclusions:

  • The generalizability theory approach provides a robust framework for estimating multilevel interrater reliability.
  • Planned missing data designs offer a practical solution for improving the accuracy and feasibility of multilevel IRR estimation in social and behavioral research.
  • The developed methods and software enhance the reliability assessment of complex observational data.