Turning words into numbers: Assessing work attitudes using natural language processing

Area of Science:

• Organizational Psychology
• Natural Language Processing
• Computational Social Science

Background:

• Organizations collect vast qualitative employee data, like open-ended survey comments.
• Traditional analysis of qualitative data for employee perceptions faces practical challenges.
• Existing methods often rely on numerical ratings, missing rich contextual insights.

Purpose of the Study:

• To investigate the development of natural language processing (NLP) algorithms for scoring qualitative employee feedback.
• To create algorithms that automatically assess employee attitudes and work perceptions from text.
• To develop the Text-Based Attitude and Perception Scoring (TAPS) dictionaries.

Main Methods:

• Developed NLP algorithms using a large employee dataset.
• Algorithms translate text comments into theme scores and valence scores for 28 work constructs.
• Utilized a combination of count-based scoring and transformer neural networks.

Main Results:

• Theme scores effectively differentiated responses based on construct relevance.
• Valence scores demonstrated strong reliability and validity, especially for construct-relevant text.
• The Text-Based Attitude and Perception Scoring (TAPS) dictionaries were created and made publicly available.

Conclusions:

• NLP algorithms can reliably score qualitative employee feedback for attitudes and perceptions.
• Explicitly designing text prompts to elicit construct-specific information enhances NLP assessment accuracy.
• The TAPS dictionaries provide a valuable tool for researchers and practitioners analyzing employee work perceptions.