A Psychometric Replication of Fan (1998) Item Response Theory and Classical Test Theory: An Empirical Comparison of...

Streiner, Norman and Cairney (2015) "Health Measurement Scales: A practical guide to their development and use", now in its fifth edition, is one of the foundational texts of the health outcomes movement. It states that "the differences between scales constructed with IRT and CTT are trivial." (Streiner, Norman and Cairney, 2015, p. 299) This statement is representative of the view which emphasizes the equivalence of True-Score Theory (TST) (also known as Classical Test Theory [CTT]) and the Rasch Measurement Model [RMM]). This view is widely held and has been one factor in limiting the application of RMM in the development of health outcome measures. However, this equivalence view relies heavily on a paper by Fan (1998) which examined the item statistics derived from TST, IRT (Item Response Theory) and the RMM for a large educational dataset. While subject to a number of theoretical and practical criticisms from a RMM perspective this paper has not been replicated with a large sample. This paper by replicating and extending the paper by Fan (1998) challenges the finding that item difficulty indexes derived from high and low ability samples using TST techniques are invariant. They are not. On the other hand, item locations derived from the RMM have a high degree of invariance. This secondary data analysis, by working through the methods used by Fan (1998) also demonstrates that a reliance on the magnitude of correlational coefficients cannot be used to determine the invariance of item difficulty indexes. An investigation into the linearity of the correlations using scatter plots is also required. Finally, an item analysis derived from the item difficulty indexes which displays a picture of the test as a whole shows that, for this large sample, the differences between scales constructed with TST and the RMM are not trivial.