Related Experiment Video
Updated: Jul 9, 2025

Modeling Verbal Behavior Deficits with the Stimulus Control Ratio Equation, SCoRE
Published on: May 14, 2019
Evaluating and Tracking Qualitative Content Coder Performance Using Item Response Theory
Michael Hennessy1, Amy Bleakley1, Morgan E Ellithorpe1
1University of Delaware, Department of Communication, 250 Pearson Hall, Newark, DE 19716.
Abstract:
Content analysis of traditional and social media has a central role in investigating features of media content, measuring media exposure, and calculating calculation of media effects. The reliability of content coding is usually evaluated using "Kappa-like" agreement measures, but these measures produce results that aggregate individual coder decisions, which obscure the performance of individual coders. Using a data set of 105 advertisements for sports and energy drinks media content coded by five coders, we demonstrate that Item Response Theory can track coder performance over time and give coder-specific information on the consistency of decisions over qualitatively coded objects. We conclude that IRT should be added to content analysts' tool kit of useful methodologies to track and measure content coders' performance.
Related Concept Videos
Cochran's Q Test
Qualitative Analysis
For instance, group IV...
Reaction Quotient
Goodness-of-Fit Test
Response Surface Methodology
The process of RSM involves several key steps:
Self-Evaluation: Self-Enhancement and Self-Verification

