Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

One-Way ANOVA: Equal Sample Sizes01:15

One-Way ANOVA: Equal Sample Sizes

One-Way ANOVA can be performed on three or more samples with equal or unequal sample sizes. When one-way ANOVA is performed on two datasets with samples of equal sizes, it can be easily observed that the computed F statistic is highly sensitive to the sample mean.
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
Reliability and Validity01:29

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
One-Way ANOVA: Unequal Sample Sizes01:15

One-Way ANOVA: Unequal Sample Sizes

One-way ANOVA can be performed on three or more samples of unequal sizes. However, calculations get complicated when sample sizes are not always the same. So, while performing ANOVA with unequal samples size, the following equation is used:
Comparing Experimental Results: Student's t-Test01:09

Comparing Experimental Results: Student's t-Test

The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
Friedman Two-way Analysis of Variance by Ranks01:21

Friedman Two-way Analysis of Variance by Ranks

Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures from...
Significance Testing: Overview01:04

Significance Testing: Overview

Significance testing is a set of statistical methods used to test whether a claim about a parameter is valid. In analytical chemistry, significance testing is used primarily to determine whether the difference between two values comes from determinate or random errors. The effect of a particular change in the measurement protocol, analyst, or sample itself can cause a deviation from the expected result. In the case of a suspected deviation/outlier, we need to be able to confirm mathematically...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Neural Network Copulas for Generating Synthetic Test Data Preserving Psychometric Properties.

Journal of Intelligence·2026
Same author

Conceptual Issues in CSEM Estimation and Interpretation: Commentary on Pfadt et al.

Psychometrika·2026
Same author

Estimation of Conditional Standard Errors of Measurement for MLE Scores in MST.

Educational and psychological measurement·2026
Same author

Interval estimation procedures for true scores of a test composed of polytomous items: An application of the multinomial error model.

Psychological methods·2020
Same author

Defining and describing medical learning communities: results of a national survey.

Academic medicine : journal of the Association of American Medical Colleges·2009

Related Experiment Video

Updated: Jul 12, 2026

Basics of Multivariate Analysis in Neuroimaging Data
06:35

Basics of Multivariate Analysis in Neuroimaging Data

Published on: July 24, 2010

Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework.

Rabia Karatoprak Ersen1, Won-Chan Lee2, Donald B Yarbrough2

  • 1GESIS-Leibniz Institute for the Social Sciences, Cologne, Germany.

Educational and Psychological Measurement
|July 10, 2026
PubMed
Summary
This summary is machine-generated.

This study examined the reliability and dependability of intervention difference scores using multivariate generalizability theory. Results show that omitting nested factors like groups or persons can inflate reliability estimates, with persons yielding the highest coefficients.

Keywords:
dependabilitydifference scoreseducational standardsmultilevel datamultivariate generalizability theoryreliability

More Related Videos

Nest Building Behavior as an Early Indicator of Behavioral Deficits in Mice
06:11

Nest Building Behavior as an Early Indicator of Behavioral Deficits in Mice

Published on: October 19, 2019

Using Cholesky Decomposition to Explore Individual Differences in Longitudinal Relations between Reading Skills
06:52

Using Cholesky Decomposition to Explore Individual Differences in Longitudinal Relations between Reading Skills

Published on: September 17, 2019

Related Experiment Videos

Last Updated: Jul 12, 2026

Basics of Multivariate Analysis in Neuroimaging Data
06:35

Basics of Multivariate Analysis in Neuroimaging Data

Published on: July 24, 2010

Nest Building Behavior as an Early Indicator of Behavioral Deficits in Mice
06:11

Nest Building Behavior as an Early Indicator of Behavioral Deficits in Mice

Published on: October 19, 2019

Using Cholesky Decomposition to Explore Individual Differences in Longitudinal Relations between Reading Skills
06:52

Using Cholesky Decomposition to Explore Individual Differences in Longitudinal Relations between Reading Skills

Published on: September 17, 2019

Area of Science:

  • Psychometrics
  • Statistical modeling

Background:

  • Assessing intervention effectiveness often relies on difference scores from pretest and posttest measures.
  • Nested data structures (persons within groups, groups within sites) are common in intervention studies.
  • Reliability and dependability of difference scores are crucial for accurate intervention evaluation.

Purpose of the Study:

  • To apply multivariate generalizability theory to estimate the reliability and dependability of difference scores.
  • To investigate the impact of nested structures (persons, groups, sites) on these estimates.
  • To compare generalizability and dependability across different levels of measurement objects.

Main Methods:

  • Employed multivariate generalizability theory (G study) with nested designs.
  • Included persons (p), groups (g), and sites (s) as nested factors.
  • Utilized pretest and posttest as the two levels of the multivariate facet (i).

Main Results:

  • Omitting nested factors (e.g., groups within sites) led to underestimated error variances and inflated generalizability/dependability coefficients.
  • Relative error correlations increased, while absolute error and universe score correlations decreased when specific nested factors were omitted.
  • Generalizability and dependability coefficients were similar in magnitude across designs due to small item variance.
  • Coefficients were highest when persons were the object of measurement and lowest when sites were.

Conclusions:

  • The inclusion of all relevant nested factors is essential for accurate reliability and dependability estimates of difference scores.
  • Intervention effectiveness assessments are most reliable when focused on individual persons.
  • Multivariate generalizability theory provides a robust framework for evaluating measurement precision in complex nested designs.