Detecting stigmatizing language with large language models: mind the settings.

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Documentation Improvement

Background:

• Stigmatizing language in clinical documentation can exacerbate healthcare disparities and harm patient-provider relationships.
• Large language models (LLMs) possess advanced contextual understanding capabilities, making them suitable for identifying and mitigating biased language.
• This study investigates the efficacy of LLMs in detecting stigmatizing language within medical records.

Purpose of the Study:

• To evaluate the accuracy of open-source LLMs in detecting stigmatizing language in clinical notes.
• To determine the impact of model size, temperature settings, and the inclusion of example prompts on LLM performance.
• To benchmark LLM performance against human annotators.

Main Methods:

• Two Llama-based LLMs (3B and 8B) were configured with different temperature settings (0.25, 0.5, 0.75) and evaluated with and without example prompts.
• A dataset of 3643 de-identified clinical notes from a tertiary care teaching hospital was used for evaluation.
• Performance metrics included accuracy, True Positive Rate (TPR), and True Negative Rate (TNR).

Main Results:

• The 8B model with a temperature of 0.25 and examples achieved the highest accuracy (70.2%) and TPR (94.1%), but the lowest TNR (47.4%).
• The 3B model without examples had the highest TNR (99.7%) but a very low TPR (2%).
• Including examples improved accuracy across configurations; temperature impact varied by model size, and note type affected performance (e.g., ED notes higher accuracy than plan of care).

Conclusions:

• Model size, temperature, and example inclusion are critical for optimizing LLM performance in detecting stigmatizing language.
• Tailoring LLM parameters to specific note types can enhance effectiveness.
• Further research is needed to refine LLMs for clinical use and assess their potential in reducing healthcare documentation bias.