Predicting survey responses: how and why semantics shape survey statistics on organizational behaviour

Area of Science:

• Social Sciences
• Computational Social Science
• Organizational Behavior Research

Background:

• Social sciences often depend on surveys to identify variable relationships.
• The predictability of these relationships from survey item semantics was unexplored.

Purpose of the Study:

• To investigate if semantic properties of survey items can predict empirical relationships.
• To assess the utility of language processing algorithms as a novel research method in social sciences.

Main Methods:

• Applied language processing algorithms to compute semantic similarity among survey items.
• Utilized these algorithms to explain and predict human response patterns in organizational behavior surveys.
• Examined surveys on transformational leadership, work motivation, and work outcomes.

Main Results:

• Semantic algorithms explained 60-86% of the variance in survey response patterns.
• Accurately predicted human survey responses, except for one personality test.
• Successfully predicted relationships between independent and dependent variables.

Conclusions:

• Survey response patterns appear significantly influenced by item semantics, potentially predicting results before data collection.
• Language processing algorithms offer powerful new tools for social science research.
• Findings challenge the empirical basis of some survey data and extend beyond current psychometric explanations.