When Artificial Intelligence Fails the Future: How Unchecked Artificial Intelligence Could Amplify Inequality in...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Health Equity Research

Background:

• Artificial intelligence (AI) is increasingly used in healthcare, but historical data can introduce biases.
• This study is the first to investigate racial and gender bias in large language models (LLMs) used for medical student career advising.
• Understanding AI bias is crucial for equitable medical education and practice.

Purpose of the Study:

• To examine racial and gender bias in LLM-generated career advice for medical students.
• To quantify the impact of demographic factors on specialty recommendations and perceived competitiveness.
• To assess the potential for AI to perpetuate or exacerbate existing inequities in medicine.

Main Methods:

• Generated 200 synthetic medical student profiles with varied demographics and academic data.
• Used a standardized prompt with an LLM to obtain top 3 specialty recommendations for each profile.
• Analyzed recommendations and competitiveness scores using National Resident Matching Program data and statistical methods, including compensatory scoring for USMLE Step 2 Clinical Knowledge points.

Main Results:

• LLMs rated male applicants as more competitive than female applicants (P < 0.0001).
• Black and Hispanic applicants were rated less competitive than White applicants (P = 0.0019 and P = 0.0108, respectively).
• Male applicants received surgical specialty recommendations 10 times more often than females (OR = 10.23, P < 0.0001); racial minorities also received fewer surgical recommendations. Compensatory scoring showed significant point deficits for underrepresented groups, especially women.

Conclusions:

• LLMs, like ChatGPT, can perpetuate racial and gender biases in medical student advising.
• Unchecked AI in medical education risks amplifying historical inequities in physician training and specialty distribution.
• Mitigation strategies are essential to ensure AI tools promote fairness and equity in medicine.