Study of comparative performance of general-purpose LLM-based systems in predicting IVF outcomes

Area of Science:

• Reproductive Medicine
• Artificial Intelligence
• Clinical Decision Support

Background:

• General-purpose large language models (LLMs) show promise for clinical decision support.
• LLM performance in predicting in vitro fertilization (IVF) outcomes is not well-characterized.
• This study evaluates three accessible LLMs: ChatGPT, DeepSeek, and Gemini.

Purpose of the Study:

• To compare the out-of-the-box performance of ChatGPT, DeepSeek, and Gemini.
• To assess LLM capabilities in forecasting key IVF clinical and laboratory outcomes.
• To evaluate AI's potential for clinical decision support in reproductive medicine.

Main Methods:

• Retrospective analysis of 1473 autologous IVF/ICSI cycles.
• Standardized patient vignettes submitted to LLMs via web interfaces.
• Performance evaluated using accuracy, mean absolute error, and ROC AUC for various IVF outcomes.

Main Results:

• Gemini showed highest accuracy in predicting stimulation protocols and embryo counts.
• DeepSeek had the lowest numerical error for oocyte count predictions.
• Clinical pregnancy prediction was challenging; Gemini achieved the highest AUC (0.711).

Conclusions:

• General-purpose AI systems demonstrate variable and suboptimal performance in predicting IVF outcomes.
• Current LLMs lack the reliability for safe clinical implementation in IVF.
• Further validation in controlled research settings is needed before clinical use.