Comparative analysis of BERT-based and generative large language models for detecting suicidal ideation: a...

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Mental Health
• Computational Linguistics

Background:

• Artificial intelligence (AI) shows potential for detecting suicidal ideation in text.
• BERT-based models excel in text classification tasks.
• Large Language Models (LLMs) can address queries without specific training.

Purpose of the Study:

• To compare the performance of three BERT model variations and three LLMs (Google Bard, Microsoft Bing/GPT-4, OpenAI ChatGPT-3.5) in identifying suicidal ideation in Brazilian Portuguese texts.
• To evaluate the effectiveness of AI models in a nonclinical text setting.

Main Methods:

• A dataset of 2,691 non-suicidal and 1,097 suicidal ideation sentences was used, with 100 selected for testing.
• BERT models underwent preprocessing, hyperparameter optimization, and cross-validation.
• LLMs were evaluated using zero-shot prompting engineering.

Main Results:

• Microsoft Bing/GPT-4 achieved the highest performance with 98% accuracy.
• Fine-tuned BERT models showed strong results: BERTimbau-Large (96%), BERTimbau-Base (94%), and BERT-Multilingual (87%).
• Google Bard (62%) and OpenAI ChatGPT-3.5 (81%) had lower accuracy compared to Bing/GPT-4 and fine-tuned BERT models.

Conclusions:

• The high recall capacity of these AI models suggests a potential for reducing misclassification of at-risk individuals.
• While promising for supporting suicidal ideation detection, these models lack clinical validation for patient monitoring.
• Caution is advised when using these AI tools to assist healthcare professionals in detecting suicidal ideation.