Constructing Subscores That Add Validity: A Case Study of Identifying Students at Risk

Area of Science:

• Educational Psychology
• Psychometrics
• Reading Comprehension Research

Background:

• Traditional achievement tests often yield limited value from subscores compared to total scores.
• Existing subscores based on content areas may not accurately reflect underlying student abilities.
• The utility of novel subscore generation methods requires further investigation.

Purpose of the Study:

• To explore new methods for creating reliable and valid subscores for inferential reading comprehension tests.
• To assess the added value of these novel subscores over a single total score in predicting academic success.
• To determine the impact of subscores based on error patterns and response efficiency on predictive models.

Main Methods:

• Developed four new subscores for an inferential reading comprehension test administered to 625 students (grades 3-5).
• Three subscores were derived from patterns of incorrect answers; a fourth measured temporal efficiency in correct responses.
• Employed logistic regression and receiver operating characteristic (ROC) curve analyses (Area Under the Curve - AUC) to evaluate predictive validity.

Main Results:

• All four developed subscores demonstrated reliability.
• Subscores based on incorrect answer patterns significantly enhanced predictive validity and model fit across all tested grades.
• The comprehension efficiency subscore showed modest improvements in predictive fit for fourth and fifth grades only.
• Models incorporating subscores consistently outperformed models with only a single total score, as indicated by higher AUC statistics.

Conclusions:

• Subscores derived from patterns of student errors in reading comprehension offer significant added value and validity.
• These error-pattern-based subscores improve the prediction of academic proficiency more effectively than traditional total scores or efficiency-based subscores.
• The findings suggest a re-evaluation of how reading comprehension is assessed, advocating for the use of nuanced subscores to better understand student performance and inform instruction.