Artificial intelligence in endometriosis care: A comparative analysis of large language model and human specialist...

Many women with endometriosis turn to digital sources for information. Meanwhile, large language models (LLMs) appear capable of offering health advice. This study aimed to evaluate the potential in answering endometriosis-related queries.

This comparative study used 150 anonymized endometriosis Q&As from online forums (2021-2023), answered by human specialists. Another 150 responses were generated using an LLM (ChatGPT-4o). Eight expert reviewers, split into two groups, blindly evaluated either the human or artificial intelligence (AI) response for each question. The primary endpoint was whether responses could be correctly identified as human or AI. Secondary endpoints included incorrect information, harm, and suitability for patient communication.

Each reviewer group assessed 600 responses; human and AI responses were differentiated significantly (χ² = 246.162, p < 0.001, slight interrater agreement). Most responses were accurate (84.8 %), harmless (87.2 %), and patient-suitable (73.8 %). There were no significant differences regarding incorrect information (p = 0.308), harm likelihood (p = 0.944), harm extent (p = 0.892), medical consensus alignment (p = 0.235), or suitability for patients (p = 0.544).

This study found that AI was not inferior to human specialists in answering endometriosis-related queries. While reviewers were able to distinguish AI- from human-generated responses significantly, interrater agreement was only slight. No substantial differences were observed in medical content. As AI continues to evolve, patients increasingly turn to it for medical guidance, highlighting the need for greater specialization in endometriosis care. Future research should further investigate the risks, benefits, and patient acceptance of AI in clinical practice.