Measurement equivalence in mixed mode surveys

Area of Science:

• Social Sciences
• Survey Methodology
• Psychometrics

Background:

• Mixed mode surveys combine different data collection methods to manage costs and response rates.
• However, combining survey modes can introduce measurement errors due to variations in question presentation and interpretation, known as measurement mode effects.

Purpose of the Study:

• To develop methodological and statistical tools for identifying and quantifying measurement mode effects in mixed mode surveys.
• To assess the significance of measurement mode effects in established instruments within a specific mixed mode panel survey.

Main Methods:

• Utilized confirmatory factor analysis (CFA) as the primary analytical technique for multi-item scales.
• Employed propensity score methods to address and correct for potential selection biases in respondent data.
• Analyzed data from the Netherlands Kinship Panel Study (NKPS), a mixed mode panel survey.

Main Results:

• The study found that most measurement instruments in the NKPS exhibit partial measurement equivalence across different survey modes.
• Controlling for demographic variables and prior survey responses partially improved measurement equivalence for some scales.
• Not all scales demonstrated improved measurement equivalence even after statistical adjustments.

Conclusions:

• While mixed mode surveys offer practical advantages, measurement mode effects pose a challenge to data comparability.
• The findings highlight the need for careful consideration of measurement equivalence when analyzing data from mixed mode surveys.
• Recommendations are provided for addressing the implications of these mode effects in future survey research and data analysis.