Large language models propagate race-based medicine.

Area of Science:

• Medical Informatics
• Artificial Intelligence Ethics
• Health Equity

Background:

• Large language models (LLMs) are increasingly integrated into healthcare.
• Concerns exist regarding their potential to perpetuate harmful, race-based medicine.
• Existing medical misconceptions about race require careful examination within AI systems.

Purpose of the Study:

• To evaluate four commercial large language models for propagating inaccurate, race-based medical content.
• To assess LLM responses to scenarios designed to detect race-based medicine and racial misconceptions.
• To identify potential risks associated with LLM deployment in clinical settings.

Main Methods:

• LLMs were tested using nine distinct questions across eight scenarios related to race-based medicine.
• Each question was posed five times per model, yielding 45 responses per LLM.
• Questions were developed by physician experts and based on prior research on medical trainees' misconceptions.

Main Results:

• All four evaluated large language models demonstrated instances of perpetuating race-based medicine.
• Model responses were inconsistent when the same questions were repeated.
• Harmful, inaccurate, race-based content was identified across multiple LLM responses.

Conclusions:

• Current large language models pose a risk of perpetuating debunked, racist medical ideas.
• The integration of LLMs into healthcare requires rigorous evaluation to prevent harm.
• Further research is needed to mitigate biases in LLMs used in medical applications.