A Comparison of Paper and Online Tests Using a Within-Subjects Design and Propensity Score Matching Study

This inquiry had 2 components: (1) the first was substantive and focused on the comparability of paper-based and computer-based test forms and (2) the second was a within-study comparison wherein a quasi-experimental method, propensity score matching, was compared with a credible benchmark method, a within-subjects design. The tests used in the comparison of online tests and paper-based tests were End-of-Course tests in Algebra and English, in a statewide high school testing program. Students were tested in Grades 8 and 9. In general, the substantive studies suggested that the online and paper tests appeared to be measuring the same underlying constructs with the same level of reliability. The within-study portion of the investigation indicated that propensity score matching study yielded results that were virtually identical to the outcome of the more conventional within-subjects experimental design. Both the methodological and substantive aspects of this investigation yielded outcomes that should be of interest to investigators in both of these areas.