Detecting stigmatizing language in clinical notes with large language models for addiction care

Area of Science:

• Medical Informatics
• Natural Language Processing
• Clinical Documentation

Background:

• Intensive care unit (ICU) progress notes may contain stigmatizing language, perpetuating negative patient biases.
• Patients with substance use disorders are especially vulnerable to such stigma.
• Automated detection of stigmatizing language is crucial for improving patient care and reducing bias.

Purpose of the Study:

• To evaluate the performance of Large Language Models (LLMs) in identifying stigmatizing language within ICU progress notes.
• To compare various LLM approaches, including zero-shot, in-context learning, and supervised fine-tuning (SFT), for stigma detection.
• To assess the LLMs' ability to provide reasoning for their classifications and identify novel stigmatizing language.

Main Methods:

• A dataset of over 77,000 ICU notes from MIMIC-III was annotated for stigmatizing content.
• Meta's Llama-3 8B Instruct LLM was employed for zero-shot, in-context learning, selective retrieval, SFT, and keyword search experiments.
• Performance was evaluated on held-out test sets and an external validation dataset from the University of Wisconsin Health System.

Main Results:

• Supervised fine-tuning (SFT) demonstrated the highest accuracy at 97.2%, followed by in-context learning.
• LLMs with SFT and in-context learning provided coherent reasoning for false positives during human review.
• Both SFT and in-context learning identified stigmatizing language missed during the initial annotation process.
• SFT achieved 97.9% accuracy on the external validation dataset.

Conclusions:

• LLMs, particularly SFT and in-context learning, are highly effective tools for identifying stigmatizing language in clinical notes.
• These models can accurately detect bias, explain their reasoning, and identify previously unrecognized stigmatizing language.
• The findings support the use of LLMs to improve the quality and reduce bias in clinical documentation.