Deductively coding psychosocial autopsy interview data using a few-shot learning large language model.

Area of Science:

• Computational Social Science
• Psychiatry
• Artificial Intelligence in Research

Background:

• Psychosocial autopsy is a qualitative method to identify suicide risk factors.
• Traditional qualitative research is time-intensive, costly, and prone to bias.
• Investigating Large Language Models (LLMs) for integration into qualitative research procedures.

Purpose of the Study:

• To evaluate the feasibility of integrating LLMs into psychosocial autopsy research.
• To assess LLM performance in deductively coding and summarizing qualitative interview data.
• To compare LLM performance against human qualitative researchers.

Main Methods:

• 38 semi-structured interviews from individuals bereaved by suicide were analyzed.
• Data was deductively coded by qualitative researchers and a LLAMA3 LLM.
• LLM performance was evaluated on classification tasks and data summarization using Cohen's Kappa and Constant Comparative Method.

Main Results:

• LLM achieved substantial agreement (accuracy: 0.84 for binary classification, 0.67 for sliding window).
• LLM summaries were generally adequate (80% rated 'adequate' or 'good').
• LLM performance varied across codes, with issues like elaboration and hallucination noted.

Conclusions:

• LLMs show potential to support researchers in coding complex interview data, saving time and resources.
• A collaborative model, combining LLM coding with researcher review and interpretation, is recommended.
• Future research should explore LLM performance in diverse contexts and with larger context sizes.