Comparison of two Bayesian methods to detect mode effects between paper-based and computerized adaptive assessments:...

Area of Science:

• Psychometrics
• Health outcomes research
• Statistical modeling

Background:

• Computerized adaptive testing (CAT) is increasingly used for health outcome measures originally developed as paper-and-pencil (P&P) instruments.
• Differences in respondent item endorsement between CAT and P&P modes can introduce measurement error if not addressed.
• Accurate estimation of health outcomes requires identifying and correcting for mode effects.

Purpose of the Study:

• To propose and evaluate methods for detecting item-level mode effects in health outcome measures administered via CAT versus P&P.
• To assess the performance of a modified robust Z (RZ) test and 95% credible intervals (CrI) in identifying differential item functioning (DIF) due to mode of administration.
• To investigate the impact of various simulation conditions on the accuracy and power of these DIF detection methods.

Main Methods:

• Employed Bayesian estimation to derive posterior distributions of item parameters for detecting mode effects.
• Proposed two methods: a modified robust Z (RZ) test and 95% credible intervals (CrI) for item difficulty differences.
• Conducted a simulation study with varying data-generating models (1- & 2-parameter IRT), DIF sizes, percentage of DIF items, and mean trait level differences.

Main Results:

• Both RZ test and CrI demonstrated good to excellent control of false positives.
• The RZ test provided superior false positive control compared to CrI, particularly when items were easy to endorse or when mean trait levels differed between modes.
• Power to detect true positives was influenced by CAT item usage, item difficulty, and item discrimination, though overall detection power was suboptimal.

Conclusions:

• The RZ test is a robust method for controlling false positives in detecting mode-based DIF, outperforming CrI.
• While false positives were managed, the power to detect true DIF remains suboptimal, necessitating further research.
• Investigating the impact of prior assumptions and data conformity is crucial for refining these DIF detection methods, especially concerning easy-to-endorse items.