Sociodemographic Bias in Large Language Model Clinical Trial Screening

Area of Science:

• Artificial Intelligence in Clinical Research
• Health Equity and Bias in AI
• Randomized Clinical Trial Design and Screening

Background:

• Large language models (LLMs) are increasingly adopted for screening in randomized clinical trials (RCTs).
• The potential for sociodemographic bias in LLM-driven RCT screening remains an underexplored area.
• Understanding LLM behavior across diverse patient populations is crucial for equitable trial access.

Purpose of the Study:

• To investigate whether large language model (LLM) screening judgments for clinical trial eligibility differ based on patient sociodemographic characteristics.
• To assess these differences while maintaining consistent clinical details and eligibility criteria across evaluated patient profiles.
• To identify specific sociodemographic factors that may introduce bias in LLM-assisted trial screening.

Main Methods:

• A cross-sectional study evaluated Phase II-III RCT protocols from ClinicalTrials.gov.
• Physician-validated clinical vignettes were created in 34 versions, including a control and 33 identity variants (gender, race, socioeconomic status, homelessness, etc.).
• Nine contemporary LLMs evaluated these vignettes, assessing eligibility and secondary domains using Likert scales.

Main Results:

• LLM eligibility judgments were largely stable across most sociodemographic variations.
• Homelessness was the only factor significantly exceeding the trivial threshold for bias in eligibility judgments (-0.121).
• Secondary domains showed socioeconomic gradients, with homelessness negatively impacting adherence (-0.595) and resources (-0.715).

Conclusions:

• Bias in LLM-assisted trial screening is conditional, with models performing consistently within fixed criteria.
• LLMs can inherit and amplify existing societal inequities when applied to data outside of strict parameters.
• Responsible deployment requires maintaining clear boundaries to ensure AI strengthens fairness in trial access rather than perpetuating bias.