Comparing traditional natural language processing and large language models for mental health status classification:...

Area of Science:

• Computational psychiatry
• Digital mental health
• Natural Language Processing

Background:

• Global rise in mental health disorders necessitates scalable detection tools.
• Digital environments offer platforms for automated mental health classification.
• Existing computational methods require rigorous evaluation for accuracy and reliability.

Purpose of the Study:

• To compare the efficacy of traditional NLP, prompt-engineered LLMs, and fine-tuned LLMs for automated mental health classification.
• To evaluate classification accuracy, precision, recall, and F1-score across seven mental health conditions.
• To identify the most effective computational approach for reliable mental health detection in digital text.

Main Methods:

• Utilized a dataset of over 51,000 social media text statements tagged with seven mental health conditions.
• Implemented and compared three approaches: traditional NLP with feature engineering, prompt-engineered LLMs, and fine-tuned LLMs.
• Monitored overfitting in fine-tuned LLMs using validation loss across training epochs.

Main Results:

• Traditional NLP with advanced feature engineering achieved 95% accuracy, significantly outperforming prompt-engineered LLMs (65%) and fine-tuned LLMs (91%).
• The specialized NLP model demonstrated superior accuracy and precision.
• Fine-tuning LLMs for three epochs yielded optimal results, with further training leading to overfitting.

Conclusions:

• Advanced text preprocessing and feature engineering in traditional NLP models offer significant benefits for mental health classification.
• Off-the-shelf LLMs using prompt engineering are insufficient for accurate mental health classification.
• Specialized computational approaches, particularly enhanced traditional NLP, remain superior to general-purpose LLMs for critical healthcare applications like mental health classification.