Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors.

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Natural Language Processing
• Mental Health Informatics

Background:

• Suicide is a major global health issue, necessitating early identification of risk factors.
• Existing AI methods often simplify suicidality detection, missing complex co-occurring factors.
• Psychiatric electronic health records (EHRs) contain valuable but unstructured data on suicidality.

Purpose of the Study:

• To evaluate generative large language models (LLMs) for multi-label classification (MLC) of suicidality-related factors (SrFs) in psychiatric EHRs.
• To develop and assess a novel end-to-end generative MLC pipeline.
• To introduce advanced evaluation metrics for complex clinical classification tasks.

Main Methods:

• Utilized GPT-3.5 and GPT-4.5 for MLC of SrFs (suicide ideation, suicide attempts, exposure to suicide, non-suicidal self-injury) from EHRs.
• Developed a novel end-to-end generative MLC pipeline.
• Employed advanced evaluation metrics, including label-set-level metrics and a multi-label confusion matrix.

Main Results:

• Fine-tuned GPT-3.5 achieved high performance (0.94 partial-match accuracy, 0.91 F1 score).
• GPT-4.5 with guided prompting showed superior, more balanced performance across all label sets, including rare ones.
• Identified systematic errors like conflation of suicide ideation and suicide attempts, and a tendency toward over-labeling.

Conclusions:

• Generative LLMs are feasible for complex clinical classification tasks like identifying multiple SrFs.
• GPT-4.5 demonstrates robust performance, particularly for underrepresented clinical factors.
• This approach offers a blueprint for structuring EHR data to advance clinical research and evidence-based medicine.