A new method for estimating the true score of subjects in the assessment based on the binary checklist

Assessments of subjects by multiple raters are completed in several health research and practice arenas. As observed in interrater comparisons of checklists during assessment, raters generally make mistakes in assessment. Despite these errors, the average scores on all the checklists have been historically used to represent the subjects' true scores. The effect of the assessment rater error on these average scores has been seldom studied. In the present study, the effect of random rater error on subjects' average scores was investigated mathematically. The analysis based on a mathematical model revealed that average scores were truly representative only in special cases where raters made no assessment errors. In addition, average scores were poor estimates of the true score due to random assessment error. A new, more precise formula to estimate subjects' true scores was proposed. The method to apply this formula to actual checklist assessment data was demonstrated.