Assessing the performance of generative AI chatbots in preimplantation genetic testing: a comparative study of expert...

How reliable are generative artificial intelligence (AI) chatbots in responding to patient-relevant questions about preimplantation genetic testing (PGT), as evaluated by reproductive medicine specialists?

A prospective evaluation was conducted comparing three publicly available generative AI models: ChatGPT-3.5, Gemini-1.5 and Llama-2. Twelve reproductive medicine specialists from different clinics assessed the chatbot-generated responses to 13 PGT-related questions, divided into simple and controversial categories. Each response was scored from 0 to 5 using predefined criteria. Assuming all answers were excellent, the maximum score was 25 points for simple questions and 40 points for controversial questions.

In total, 156 evaluations were completed for each chatbot. Among the simple questions, 'What are the types and techniques used for PGT?' scored lowest (mean ± SD 2.83 ± 0.94). For the controversial questions, 'What is the percentage of aneuploidy that allows an embryo to be defined as mosaic?' scored lowest (mean ± SD 2.67 ± 1.22). ChatGPT performed best across both categories (simple 16.83 ± 1.80; controversial 27.75 ± 4.49), followed by Gemini (simple 14.92 ± 2.02; controversial 26.08 ± 3.99) and Llama (simple 13.58 ± 3.60; controversial 16.92 ± 4.96). Significant differences were observed, particularly between ChatGPT and Llama for both simple and controversial questions (P = 0.027 for simple, P < 0.001 for controversial), and between Gemini and Llama for controversial questions (P < 0.001). No significant performance differences were noted between participating specialists.

Generative AI shows moderate reliability in addressing PGT-related enquiries, with ChatGPT and Gemini outperforming Llama. While performance was higher for simple questions than for controversial questions, the variability underscores the need for clinical oversight. Further refinement and validation are essential before widespread integration of AI tools in reproductive medicine.