Reporting Diagnostic Scores in Educational Testing: Temptations, Pitfalls, and Some Solutions.
Sandip Sinharay1, Gautam Puhan1, Shelby J Haberman1
1a Educational Testing Service.
Multivariate Behavioral Research
|January 14, 2016
Summary
Diagnostic scores in educational testing offer instructional benefits but require sufficient items and distinctiveness to be valuable. Many reported subscores lack these qualities, making them not worth reporting.
Area of Science:
- Educational Measurement
- Psychometrics
- Educational Psychology
Background:
- Diagnostic scores are increasingly used in educational testing for their remedial and instructional benefits.
- The proliferation of diagnostic scores in tests and research necessitates a critical evaluation of their reporting methods.
Purpose of the Study:
- To critically evaluate the reporting of diagnostic scores in educational testing.
- To discuss existing methods and review a recent approach for assessing the value of diagnostic scores.
Main Methods:
- Review of existing diagnostic score reporting methods.
- Evaluation of a recent method (Haberman, 2008a) for determining the worth of diagnostic scores.
- Analysis of operational and simulated data to assess score reporting criteria.
Main Results:
- Diagnostic scores require a sufficient number of items to be meaningful.
- Subscores must be sufficiently distinct from each other to warrant reporting.
- Several commonly reported subscores in operational educational tests do not meet these criteria and are not worth reporting.
Conclusions:
- The reporting of diagnostic scores in educational testing should be approached with caution.
- Recommendations are provided for improving the reporting of diagnostic scores to ensure their utility.
- Ensuring sufficient item count and score distinctiveness is crucial for the validity of diagnostic score reporting.
