Supervised Learning and Large Language Model Benchmarks on Mental Health Datasets: Cognitive Distortions and Suicidal...

Area of Science:

• Computational linguistics
• Psychological informatics
• Artificial intelligence in mental health

Background:

• Social media users express emotions, potentially indicating psychological distress.
• Early identification of suicidal ideation and cognitive distortions is crucial for intervention.

Purpose of the Study:

• Introduce two novel Chinese social media datasets: SOS-HL-1K (suicidal risk) and SocialCD-3K (cognitive distortion).
• Evaluate supervised learning methods and large language models (LLMs) for psychological state detection.
• Compare prompt engineering strategies and fine-tuning for LLMs.

Main Methods:

• Developed and utilized SOS-HL-1K (1249 posts) and SocialCD-3K (3407 posts) datasets.
• Evaluated two supervised learning models and eight LLMs.
• Implemented zero-shot and few-shot prompt strategies, alongside fine-tuning.

Main Results:

• Supervised learning achieved high performance (82.76% F1 for suicide risk, 76.10% micro-F1 for cognitive distortion).
• Prompted LLMs underperformed supervised methods without fine-tuning.
• Fine-tuning significantly improved LLM performance, reducing the gap to 4.31% and 3.14%.

Conclusions:

• Supervised learning remains essential for complex psychological tasks on social media.
• LLMs show promise but require fine-tuning to match supervised methods.
• The developed datasets facilitate further research in computational psychology.