Using large language models for extracting and pre-annotating texts on mental health from noisy data in a...

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Mental Healthcare
• Computational Linguistics

Background:

• Advancements in Large Language Models (LLMs) offer potential for mental healthcare conversational agents (CAs).
• Limited high-quality training data, privacy concerns, and high annotation costs hinder LLM development, especially for low-resource languages.
• Publicly available social media discussions are noisy, requiring LLM adaptation for data cleaning and pre-classification to reduce manual annotation.

Purpose of the Study:

• To explore the potential of zero-shot classification using LLMs for pre-classifying mental health texts into psychiatric disorder topics.
• To facilitate the development of CAs for disorder-specific mental health counseling.
• To introduce the first openly available Russian-language dataset and model for mental health counseling CA development.

Main Methods:

• Utilized zero-shot classification with four LLMs to select and pre-classify Russian-language texts into seven psychiatric disorder categories.
• Compared zero-shot pre-filtering with LLM fine-tuning (standard and Natural Language Inference modes) for classification accuracy.
• Evaluated the impact of lemmatization and the performance of multilingual versus English-only models on translated texts.

Main Results:

• LLM fine-tuning significantly improved classification accuracy (more than threefold) compared to non-fine-tuned methods with preliminary filtering.
• Natural Language Inference (NLI) fine-tuning achieved slightly higher accuracy (0.64) but was six times slower than standard fine-tuning.
• Lemmatization did not affect classification quality; multilingual models performed slightly better than English-only models on translated data.

Conclusions:

• LLM fine-tuning is crucial for enhancing text classification quality in mental health applications, substantially improving upon zero-shot approaches.
• While NLI fine-tuning shows promise, its computational cost necessitates further research into hypothesis engineering for efficiency.
• The developed Russian-language dataset and model represent a valuable, openly accessible resource for advancing mental health CAs.