MentalQLM: A Lightweight Large Language Model for Mental Healthcare Based on Instruction Tuning and Dual LoRA Modules

Area of Science:

• Artificial Intelligence
• Computational Psychiatry
• Natural Language Processing

Background:

• Mental disorders present significant healthcare challenges and social implications.
• Large language models (LLMs) offer potential for mental healthcare improvement.
• Current LLM approaches incur high computational costs due to large datasets and models.

Purpose of the Study:

• To develop a novel, lightweight LLM for efficient mental disorder diagnosis.
• To reduce computational costs associated with LLM fine-tuning for mental healthcare.
• To enhance LLM performance on complex multi-class mental health classification tasks.

Main Methods:

• Proposed MentalQLM, a lightweight LLM using a dual Low-Rank Adaptation (LoRA) strategy.
• Implemented dataset pruning based on perplexity and diversity analysis.
• Employed a two-stage fine-tuning process with two LoRA modules and a dense layer augmentation.

Main Results:

• MentalQLM, with 0.5 billion parameters, achieved a 0.778 weighted F1-score on mental disorder diagnosis.
• Outperformed MentaLLaMA-Chat-13B by 3.2% and GPT-4 by 17.7% on benchmark datasets.
• Demonstrated significantly lower resource requirements compared to existing state-of-the-art models.

Conclusions:

• MentalQLM provides a cost-effective and efficient solution for mental healthcare applications.
• The developed model is particularly suitable for computationally constrained environments.
• The approach offers a promising direction for advancing AI in mental health diagnosis.