Automated scoring of the Ambiguous Intentions Hostility Questionnaire with fine-tuned large language models

Area of Science:

• Psychology
• Artificial Intelligence
• Computational Linguistics

Background:

• Hostile attribution bias, the tendency to perceive social interactions as hostile, is measured by the Ambiguous Intentions Hostility Questionnaire (AIHQ).
• AIHQ open-ended responses offer deep insights but require laborious human scoring.
• Automating AIHQ scoring could significantly enhance research and clinical efficiency.

Purpose of the Study:

• To evaluate the efficacy of large language models (LLMs) in automating the scoring of AIHQ open-ended responses.
• To assess the alignment of LLM-generated scores with human expert ratings.
• To explore the potential of LLMs in facilitating psychological assessments.

Main Methods:

• Utilized a dataset of AIHQ responses from individuals with and without traumatic brain injury (TBI).
• Fine-tuned two LLMs on human-generated ratings for half of the dataset.
• Tested the fine-tuned models on the remaining AIHQ responses and an independent dataset.

Main Results:

• LLM-generated ratings demonstrated strong alignment with human ratings for hostility and aggression attributions.
• Fine-tuned models exhibited higher alignment compared to baseline models.
• The models successfully replicated known group differences in attributional styles between TBI and non-TBI groups.

Conclusions:

• LLMs can effectively and accurately automate the scoring of AIHQ responses, streamlining psychological assessments.
• This automated approach holds promise for both research and clinical applications, particularly for populations like those with TBI.
• Accessible scoring interfaces were developed to encourage broader adoption of LLM-based AIHQ scoring.