Comparison of self-report data validity in undergraduate samples using remote versus in-person administration methods

Area of Science:

• Psychology
• Research Methodology
• Survey Research

Background:

• Internet-based research allows remote recruitment, participation, and compensation for surveys.
• Remote data collection offers benefits like diverse samples and reduced costs.
• Concerns exist regarding increased inattentive or invalid response patterns in unsupervised remote settings.

Purpose of the Study:

• To compare the validity of self-report survey data collected remotely versus in a laboratory setting.
• To investigate the impact of administration mode on response patterns and data quality.

Main Methods:

• Undergraduate students (N=678) were randomly assigned to complete surveys in-person or remotely.
• Data validation employed multiple methods: embedded/stand-alone validity scales, response timing, and outlier identification.
• A preregistered analytic plan guided data screening and analysis.

Main Results:

• Null-to-small differences were observed between in-person and remote survey administration conditions regarding data validity.
• Significant discrepancies emerged in the proportion of data flagged as 'invalid' across different screening methods.
• Response patterns did not substantially differ based on the administration setting.

Conclusions:

• Remote survey administration is a viable method with comparable data validity to in-person settings.
• The choice of data validation method significantly impacts the classification of survey data quality.
• Implications for study design and data analysis strategies in survey research are discussed.