Optimizing Entity Recognition in Psychiatric Treatment Data with Large Language Models.

Area of Science:

• Natural Language Processing
• Pharmacovigilance
• Artificial Intelligence in Healthcare

Background:

• Extracting adverse drug reactions (ADRs) from patient self-reported messages is crucial for pharmacovigilance.
• HIPAA constraints and data privacy challenges complicate ADR extraction from sensitive patient data.
• The need for efficient and privacy-preserving methods for real-time ADR monitoring is increasing.

Purpose of the Study:

• To evaluate the efficacy of locally deployable small language models (LLMs) for extracting adverse drug reactions (ADRs) from patient self-reported messages.
• To assess the performance of Mistral-7B, Llama-3-8B, and Gemma-7B LLMs in ADR extraction under data privacy constraints.
• To explore in-context learning, demonstration selection, and fine-tuning with QLoRA for optimizing ADR extraction.

Main Methods:

• Utilized the PsyTAR dataset comprising patient self-reported messages.
• Implemented and compared three small LLMs: Mistral-7B, Llama-3-8B, and Gemma-7B.
• Applied techniques including in-context learning, demonstration selection, and fine-tuning with QLoRA.

Main Results:

• Mistral-7B demonstrated superior performance in few-shot learning scenarios.
• Fine-tuning achieved a high F1 score of 86% for ADR extraction.
• The developed pipeline enables real-time ADR monitoring while maintaining data privacy.

Conclusions:

• Small, locally deployable LLMs are effective for ADR extraction, even under strict data privacy regulations.
• These resource-efficient solutions empower healthcare organizations to enhance patient safety through rapid ADR identification.
• The findings support the use of accessible AI technologies for real-time pharmacovigilance and improved patient outcomes.