Performance of artificial intelligence large language models (LLMs) in answering frequently asked questions about...

In this study, we aimed to examine the responses given by ChatGPT (OpenAI), Copilot (Microsoft), and Gemini (Bard) artificial intelligence applications to questions about the active ingredient isotretinoin in terms of accuracy, readability, applicability, and understandability.

The readability of the answers given by the artificial intelligence programs was evaluated using the Flesch-Kincaid ease score, and the applicability and understandability levels were evaluated using the Patient Education Materials Evaluation Tool scales. The accuracy of the answers was compared by two dermatologists who scored them between 1 and 5.

No significant difference was found between the groups in terms of Flesch Kincaid reading ease scores (p = 0.671), and all three programs were found to be at a difficult level of reading. In the Patient Education Materials Evaluation Tool scales, it was observed that Gemini and ChatGPT rates were >70% and there was a significant difference in favor of these programs between the groups (p < 0.001). In the accuracy scores of the answers, Gemini (4.90 ± 0.31) and ChatGPT (4.60 ± 0.69) had high scores and there was a significant difference between the groups (p < 0.001).

While the AI chatbots we used in the study demonstrated reasonable accuracy in answering questions about isotretinoin, they performed limited in terms of readability and usability. These findings suggest that AI programs alone are not sufficient for patient education and need to be improved to simplify responses.