SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks

Area of Science:

• Natural Language Processing (NLP)
• Social Science Research

Background:

• Social science NLP tasks require understanding implicit pragmatic cues in text.
• Limited training data is a common challenge in this domain.
• Instruction tuning has improved LLM capabilities in other areas, but its effectiveness in the social domain is underexplored.

Purpose of the Study:

• To investigate the effectiveness of instruction tuning for social science NLP tasks.
• To introduce Socialite-Llama, an open-source instruction-tuned Llama2 model.
• To evaluate the model's performance on a diverse set of social science tasks.

Main Methods:

• Developed Socialite-Llama by applying instruction tuning to the Llama2 model.
• Evaluated Socialite-Llama on a suite of 20 social science NLP tasks.
• Compared Socialite-Llama's performance against the base Llama2 model and a state-of-the-art multi-task fine-tuned model.

Main Results:

• Socialite-Llama demonstrated improved performance over the base Llama2 model on the majority of the 20 social science tasks.
• The model matched or surpassed the performance of a state-of-the-art multi-task fine-tuned model on most tasks.
• Significant improvements were observed in generalized social understanding, with gains on 5 out of 6 related social tasks.

Conclusions:

• Instruction tuning is effective for enhancing LLM performance on social science NLP tasks.
• Socialite-Llama represents a valuable open-source resource for advancing research in computational social science.
• The findings suggest that instruction tuning can foster broader social understanding in LLMs.