Detecting Stigmatizing Language in Clinical Notes with Large Language Models for Addiction Care

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Medical Ethics

Background:

• Stigmatizing language in clinical notes can negatively influence patient care and perpetuate bias.
• Patients with substance use disorders (SUD) are particularly vulnerable to stigma in healthcare settings.
• Intensive care unit (ICU) progress notes are a potential source of stigmatizing language.

Purpose of the Study:

• To evaluate the performance of Large Language Models (LLMs) in detecting stigmatizing language within ICU progress notes.
• To compare different LLM approaches, including Zero-Shot, in-context learning, Retrieval Augmented Generation (RAG), and supervised fine-tuning (SFT).
• To assess the LLMs' ability to provide reasoning for their stigma detection and identify novel stigmatizing terms.

Main Methods:

• A dataset of 77,104 ICU notes from MIMIC-III was created, balanced for stigmatizing and non-stigmatizing encounters.
• Four LLM approaches (Zero-Shot, in-context learning, RAG, SFT) and a keyword search baseline were tested.
• Models were evaluated on accuracy and macro F1 score using train/development/test splits and an external validation set.

Main Results:

• Supervised fine-tuning (SFT) achieved the highest accuracy (97.2%), followed by in-context learning.
• LLMs provided coherent reasoning for their predictions, aiding in the review of false positives.
• Both SFT and in-context learning identified stigmatizing language missed during manual annotation.

Conclusions:

• LLMs, particularly SFT and in-context learning, are highly effective in identifying stigmatizing language in ICU notes.
• These models offer an efficient alternative to manual review, reducing time and effort.
• LLMs can help mitigate stigma in clinical documentation, improving care for patients with SUD.