Observer-Independent Assessment of Content Overlap in Mental Health Questionnaires: Large Language Model-Based Study.

Area of Science:

• Psychiatry and Mental Health
• Computational Linguistics
• Health Informatics

Background:

• Mental health assessments frequently use questionnaires, but significant content divergence exists between tools measuring the same psychopathology.
• Previous content overlap studies relied on time-consuming, observer-dependent manual symptom labeling.

Purpose of the Study:

• To employ large language models (LLMs) for observer-independent analysis of content overlap in mental health questionnaires.
• To compare LLM-derived results with clinical expertise.

Main Methods:

• Analysis of questionnaires across depression, psychosis risk, mania, OCD, and sleep disorders using two LLM approaches: sentence Bidirectional Encoder Representations from Transformers (sBERT) and Generative Pretrained Transformer (GPT).
• LLM-based symptom clustering was compared to expert manual categorization using the adjusted rand index.
• Content overlap within diagnostic domains was assessed using LLM-derived clusters.

Main Results:

• Varying degrees of similarity were observed between expert and LLM-based clustering across domains, with GPT showing better alignment than sBERT.
• GPT-based clustering revealed weak to moderate content overlap for questionnaires across analyzed diagnostic domains (e.g., CHR-P: 0.344 to adult depression: 0.574).
• LLM analysis results showed variations but were not substantially different from previous manual investigations of content overlap.

Conclusions:

• Large language models (LLMs) offer a feasible method for objective assessment of content overlap in diagnostic questionnaires.
• The Generative Pretrained Transformer (GPT) approach demonstrates significant promise for aligning with expert-derived symptom structures in mental health assessments.