Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Kendall's Coefficient of Concordance01:20

Kendall's Coefficient of Concordance

Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects or...
Introduction to R01:11

Introduction to R

R is a powerful software environment for statistical computing and graphics. Originating as an implementation of the S language, developed at Bell Laboratories, R has evolved into a robust, open-source statistical software favored by statisticians and data scientists worldwide. Its comprehensive suite includes data manipulation, calculation, and graphical display capabilities, making it versatile for data analysis and visualization. Its programming language is at the core of R's functionality,...
Friedman Two-way Analysis of Variance by Ranks01:21

Friedman Two-way Analysis of Variance by Ranks

Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures from...
Ranks01:02

Ranks

Unlike parametric methods, nonparametric statistics are ideal for nominal and ordinal data, requiring fewer assumptions about the population's nature or distribution. This makes nonparametric methods easier to apply and interpret, as they do not depend on parameters like mean or standard deviation. One common approach in nonparametric analysis is to sort data according to a specific criterion. For instance, we might arrange weather data from hottest to coldest days in a month or rank cities...
Interpreting R Charts01:22

Interpreting R Charts

R chart, or range chart, is a fundamental tool in statistical process control used to monitor the variability within a process. It complements the X-bar (x̄) chart by focusing on the range of the data, rather than individual values, providing a clear picture of the process dispersion over time.
An R chart plots the range of subsets of measurements collected from a process. Each point on the chart represents the range—defined as the difference between the maximum and minimum values—of a sample...
Wilcoxon Signed-Ranks Test for Matched Pairs01:09

Wilcoxon Signed-Ranks Test for Matched Pairs

The Wilcoxon signed-rank test for matched pairs evaluates the null hypothesis by combining the ranks of differences with their signs. It essentially tests whether the median of the differences in a population of matched pairs is zero. Since the test incorporates more information than the sign test, it generally yields more trustable conclusions. This test also does not require the data to follow a normal distribution, but two conditions must be met for it to be applicable: (1) the data must...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same authorSame journal

babebi: An R Package for Bayesian Estimation and Validation in Small-N Two-Rater Pre-Post Designs.

Applied psychological measurement·2026
Same author

Agreement and Alignment in Binary Rating Tasks: Strategic Convergence as an Equilibrium Outcome.

Educational and psychological measurement·2026
Same author

From Agreement to Epistemic Alignment: A Signal Detection-Theoretic Model of Inter-Rater Reliability.

Educational and psychological measurement·2026
Same journal

The EM Algorithm and Its Variants in Cognitive Diagnostic Models: Comparing Their Propensity for Boundaries, Extremes, Convergence, and Suboptimal Solutions.

Applied psychological measurement·2026
Same journal

When Perceptions of Social Desirability Differ: Implications for the Multidimensional Nominal Response Model of Faking.

Applied psychological measurement·2026
Same journal

Confirmatory Factor Analysis with Adaptive Quadrature Estimator Using Four Link Functions.

Applied psychological measurement·2026
Same journal

Automatic Item Generation Measurement Models Respecting the Stochastic Sampling Space for Cross-Classified and Two-Level Sampling of Subjects and Incidentals.

Applied psychological measurement·2026
Same journal

Multistage Testing for Cognitive Diagnosis Based on Skill-Space Partitioning.

Applied psychological measurement·2026
See all related articles

Related Experiment Video

Updated: Jul 3, 2026

Constructing and Visualizing Models using Mime-based Machine-learning Framework
06:19

Constructing and Visualizing Models using Mime-based Machine-learning Framework

Published on: July 22, 2025

A Tool for Agreement and Alignment Analysis in Binary Rating Tasks: The R Package scindex.

Irene Gianeselli1

  • 1Faculty of Education, Free University of Bolzano-Bozen, Bressanone-Brixen (BZ), Italy.

Applied Psychological Measurement
|July 2, 2026
PubMed
Summary
This summary is machine-generated.

The scindex R package enhances inter-rater reliability analysis for binary classification. It calculates key metrics like Cohen's κ, Fleiss' κ, and signal detection parameters for robust assessment.

Keywords:
Ragreement analysisbinary classificationdecision thresholdsinter-rater reliabilitysignal detection theory

More Related Videos

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances
07:35

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances

Published on: October 11, 2018

An R-Based Landscape Validation of a Competing Risk Model
05:37

An R-Based Landscape Validation of a Competing Risk Model

Published on: September 16, 2022

Related Experiment Videos

Last Updated: Jul 3, 2026

Constructing and Visualizing Models using Mime-based Machine-learning Framework
06:19

Constructing and Visualizing Models using Mime-based Machine-learning Framework

Published on: July 22, 2025

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances
07:35

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances

Published on: October 11, 2018

An R-Based Landscape Validation of a Competing Risk Model
05:37

An R-Based Landscape Validation of a Competing Risk Model

Published on: September 16, 2022

Area of Science:

  • Statistics
  • Biostatistics
  • Psychometrics

Background:

  • Inter-rater reliability is crucial for consistent diagnostic and classification tasks.
  • Existing methods may not fully capture the nuances of rater agreement and response criteria.

Purpose of the Study:

  • To introduce scindex, an R package for comprehensive inter-rater reliability analysis.
  • To provide tools for calculating established reliability metrics and novel convergence measures.

Main Methods:

  • The scindex package computes Cohen's kappa and Fleiss' kappa.
  • It estimates signal detection theory parameters (sensitivity, specificity, decision thresholds) using ground-truth labels.
  • The Strategic Convergence Index (SCI) is implemented to assess rater response criteria convergence.

Main Results:

  • scindex offers a unified framework for analyzing inter-rater reliability in binary classification.
  • The package facilitates the estimation of both agreement levels and underlying decision-making processes.
  • The Strategic Convergence Index provides a novel approach to evaluating rater consistency.

Conclusions:

  • scindex is a valuable tool for researchers and practitioners needing to assess inter-rater reliability.
  • The package enhances the analysis of binary classification tasks by integrating multiple statistical measures.
  • Utilizing scindex can lead to more reliable and interpretable results in classification studies.