Not all artificial intelligence large language models are created equal: heterogeneous performance in therapeutic...

To evaluate the concordance between therapeutic recommendations generated by 3 widely used consumer large language models (Gemini, Microsoft Copilot, and ChatGPT) and the decisions of a multidisciplinary tumor board in the management of the first recurrence of high-grade serous ovarian cancer. Secondary objectives were to characterize recurrent error patterns and to assess the safety of model-generated recommendations using a 3-tier taxonomy.

A retrospective observational analysis was conducted on 44 consecutive patients with a first recurrence of high-grade serous ovarian cancer treated at a tertiary referral center. For each case, an identical, anonymized, structured zero-shot prompt was submitted to the 3 models. The multidisciplinary tumor board decision served as the reference comparator. Overall concordance and concordance for the chemotherapy backbone, maintenance or targeted-agent class, and surgical indication were compared between models using Cochran's Q test and pairwise McNemar tests (Holm-adjusted). Discordant recommendations were categorized as true hallucinations, unsafe, or suboptimal but guideline-concordant.

Concordance varied substantially across models (Cochran's Q p<.001): Gemini 77.3% (95% confidence interval 62.2 to 88.5), Microsoft Copilot 36.4% (95% confidence interval 22.4 to 52.2), and ChatGPT 18.2% (95% confidence interval 8.2 to 32.7); Gemini differed significantly from the other 2 models (Holm-adjusted p<.001), whereas the difference between Microsoft Copilot and ChatGPT did not reach significance after adjustment (p=.057). Component-level analysis localized the divergence to systemic therapy: surgical-indication agreement was uniformly high (93.2% to 95.5%, p=.85). All 3 models generated unsafe recommendations for the re-use of poly (ADP-ribose) polymerase) inhibitors in patients with documented prior progression on poly (ADP-ribose) polymerase inhibitor maintenance, with marked differences in frequency (Microsoft Copilot 10, ChatGPT 3, and Gemini 2; 15 of 74 total discordances). The remaining discordances were suboptimal but guideline concordant.

Consumer large language models showed heterogeneous and clinically meaningful differences in reproducing Multidisciplinary Tumor Board decisions in recurrent ovarian cancer; none were free of unsafe recommendations. Their current performance does not support independent clinical use, and expert supervision is required to ensure patient safety.