Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons

Area of Science:

• Social Sciences
• Computational Social Science
• Survey Methodology

Background:

• Survey research often uses closed-ended questions for efficiency, but they limit response depth.
• Open-ended questions provide richer data but are resource-intensive to code.
• Existing large language model (LLM) applications for survey analysis have limitations.

Purpose of the Study:

• To propose and evaluate a new method for scaling open-ended survey responses using LLMs.
• To address the limitations of current LLM approaches in survey data analysis.
• To enhance the depth and variability of survey response measurement.

Main Methods:

• Developed a pairwise comparison method leveraging LLMs to compare open-ended survey statements.
• Employed a Bayesian Bradley-Terry model to derive latent scale scores from LLM pairwise comparisons.
• Tested the approach on an open-ended question about US interest rate knowledge.

Main Results:

• The LLM-based pairwise comparison method demonstrated greater consistency than zero-shot ratings across various LLM sizes.
• The derived scores showed finer discrimination, reduced anchoring bias, and better uncertainty measurement.
• Results were consistent with knowledgeable crowdsourced worker assessments.

Conclusions:

• The pairwise comparison method offers a flexible and effective way to scale open-ended survey responses using LLMs.
• This approach improves upon traditional methods and existing LLM applications for survey analysis.
• The method shows promise for more nuanced measurement in social science research.