Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

13.6K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.6K
Comparing Experimental Results: Student's t-Test01:09

Comparing Experimental Results: Student's t-Test

4.5K
The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
4.5K
Friedman Two-way Analysis of Variance by Ranks01:21

Friedman Two-way Analysis of Variance by Ranks

421
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
421
Group Design02:01

Group Design

10.0K
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between...
10.0K
Behrens–Fisher Test00:57

Behrens–Fisher Test

209
The Behrens-Fisher test is a statistical method designed to address the Behrens-Fisher problem, which arises when comparing the means of two normally distributed populations with unequal variances. Unlike the Student's t-test, which assumes equal variances, the Behrens-Fisher test allows for mean comparison without this restrictive assumption. This flexibility makes it particularly valuable in scenarios where two independent samples exhibit normality but lack variance homogeneity.
This test...
209
Bonferroni Test01:10

Bonferroni Test

3.2K
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
3.2K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Examining Parameter Estimation when Treating Semi-Mixed Multidimensional Constructs as Unidimensional.

Journal of applied measurement·2019
Same author

Using a Multidimensional IRT Framework to Better Understand Differential Item Functioning (DIF): A Tale of Three DIF Detection Procedures.

Educational and psychological measurement·2018
See all related articles

Related Experiment Video

Updated: Dec 16, 2025

Computerized Adaptive Testing System of Functional Assessment of Stroke
05:21

Computerized Adaptive Testing System of Functional Assessment of Stroke

Published on: January 7, 2019

6.1K

Using Differential Item Functioning to Test for Interrater Reliability in Constructed Response Items.

Cindy M Walker1, Sakine Göçer Şahin2

  • 1Duquesne University, Pittsburgh, PA, USA.

Educational and Psychological Measurement
|July 4, 2020
PubMed
Summary
This summary is machine-generated.

Differential item functioning (DIF) analysis offers a promising new method for evaluating interrater reliability on rating scales. This approach can identify if raters are consistently more lenient or severe in their scoring.

Keywords:
classical test theoryconstructed response itemsdifferential item functioninginterrater reliabilityrater severity

More Related Videos

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
08:40

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity

Published on: June 12, 2019

7.7K
Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
09:00

Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education

Published on: August 16, 2024

1.1K

Related Experiment Videos

Last Updated: Dec 16, 2025

Computerized Adaptive Testing System of Functional Assessment of Stroke
05:21

Computerized Adaptive Testing System of Functional Assessment of Stroke

Published on: January 7, 2019

6.1K
Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
08:40

Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity

Published on: June 12, 2019

7.7K
Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education
09:00

Author Spotlight: Validation of SICOLE-R for Assessing Cognitive and Reading Skills in Spanish-Speaking Children and Its Role in Personalized Education

Published on: August 16, 2024

1.1K

Area of Science:

  • Psychometrics
  • Educational Measurement
  • Statistical Analysis

Background:

  • Interrater reliability is crucial for ensuring consistent scoring of subjective assessments.
  • Traditional methods may not fully capture rater discrepancies on polytomous scales.
  • A need exists for advanced methods to evaluate rater agreement and bias.

Purpose of the Study:

  • To investigate differential item functioning (DIF) as a novel approach for assessing interrater reliability.
  • To compare DIF-based methods with traditional interrater reliability statistics.
  • To determine if DIF analysis can identify rater severity or leniency.

Main Methods:

  • Employed differential item functioning (DIF) analyses to evaluate interrater reliability.
  • Compared DIF statistics with intraclass correlation coefficient (ICC) and Cohen's kappa.
  • Utilized the Poly-SIBTEST procedure for DIF analysis on polytomous items.

Main Results:

  • DIF procedures demonstrated potential as an effective alternative for assessing interrater reliability.
  • DIF analysis successfully identified differences in rater severity or leniency.
  • The DIF approach does not necessitate a fully crossed rater design.

Conclusions:

  • Differential item functioning (DIF) is a viable and promising method for evaluating interrater reliability.
  • DIF analysis provides nuanced insights into rater performance on polytomous scales.
  • This method enhances the assessment of constructed response items and rating scales.