Evaluation of an Artificial Intelligence Communication Platform for Racial Biases in Pretest Education About Prostate...

Guidelines recommend germline testing in advanced prostate cancer (PCa) to inform treatment and personal/familial cancer risk, yet Black patients are less likely than White patients to complete testing. Artificial intelligence (AI) tools are increasingly used in pretest education to improve access but remain unevaluated for racial bias.

We developed ProGene, a secure, generative AI chatbot for PCa germline testing education in Black patients. We prompted ProGene with seven questions about testing types, personal benefits, family benefits, drawbacks, logistics, costs, and privacy. Each question was asked nine times across three patient vignettes (Black, non-Hispanic White, race-agnostic), in triplicate. Two blinded reviewers assessed responses across five domains: (1) comprehensiveness (0%-100%), (2) accuracy (presence/absence of inaccuracies), (3) readability (grade level via Simple Measure of Gobbledygook [SMOG] and Flesch-Kincaid), (4) actionability (0%-100% via Patient Education Materials Assessment Tool), and (5) quality (1-18 via DISCERN-AI). Outcomes were compared by race and question using two-sample t-tests or Wilcoxon rank-sum tests for continuous measures and chi-square or proportion tests for categorical measures; ANOVA was used for question-level comparisons.

The mean comprehensiveness was 67% and did not vary by race, although responses for question-1 (genetic testing types) were less comprehensive for Black versus race-agnostic vignette (60% v 93%, P < .01). Inaccuracies appeared in 32% of responses, primarily related to sample collection and cost/insurance, and did not vary by race. The mean readability were 10th (SMOG) and 13th (Flesch-Kincaid) grades, actionability 92%, and DISCERN score 14/18 (good quality); none varied by race.

We identified no statistically significant racial disparities across the five evaluation domains in ProGene. Responses were generally good quality and actionable. Although AI may facilitate equity in PCa genetic education, deficiencies in comprehensiveness, accuracy, and readability highlight the need for refinement and/or human oversight with implementation.