Enhancing Psychological Assessments with Open-Ended Questionnaires and Large Language Models: An ASD Case Study | JoVE Visualize

Area of Science:

Natural Language Processing
Computational Psychiatry
Machine Learning in Healthcare

Background:

Open-ended questionnaires provide rich data but are difficult to analyze manually.
Advancements in Natural Language Processing (NLP) offer potential for automated analysis of qualitative data.
The application of NLP, specifically large language models (LLMs), in psychological classification remains underexplored.

Purpose of the Study:

To propose and evaluate a methodology using pre-trained LLMs for the automatic classification of open-ended questionnaire responses.
To apply this methodology to classify autism spectrum disorder (ASD) using parental reports.
To compare different LLM training strategies and assess their performance and interpretability.

Main Methods:

Utilized transcribed parental reports from 51 participants (26 typically developing, 25 with ASD).
Compared various LLM fine-tuning strategies, input representations, and specificity levels.
Derived subject-level predictions by aggregating responses from 12 individual questions, employing a voting system for final classification.
Conducted zero-shot evaluation using GPT-4o and assessed interpretability of different LLM approaches.

Main Results:

The best-performing approach achieved 84% subject-wise accuracy and 1.0 ROC-AUC.
This optimal strategy involved an OpenAI embedding model, per-question training, inclusion of questions in the input, and a voting system for aggregation.
Zero-shot evaluation with GPT-4o yielded comparable results, demonstrating the efficacy of both compact and large LLMs.
Interpretability varied, with locally deployable LLMs offering greater transparency than proprietary models.

Conclusions:

LLMs provide a validated, scalable, and cost-effective method for automatically classifying open-ended questionnaires in psychological assessments.
This methodology shows significant promise for autism spectrum disorder (ASD) assessment and has broader applicability for analyzing qualitative data in mental health research.
A trade-off exists between the performance of proprietary LLMs and the interpretability offered by locally deployable models.