Comparative evaluation of ChatGPT and Gemini artificial intelligence in patient education on clear aligner therapy

To compare the performance of ChatGPT-4 Turbo and Gemini 1.5 Pro in the domain of Clear Aligner Therapy (CAT).

A total of 36 standardized questions on CAT were created based on consent forms from aligner companies and frequently asked patient questions. Responses were generated by ChatGPT-4 Turbo and Gemini 1.5 Pro. A reference answer key was developed from current literature. Two orthodontic professors independently evaluated the responses using a six-point accuracy scale and a three-point completeness scale. Questions were also categorized by topic. Readability was assessed using Flesch-Kincaid Grade Level (FKGL) and Simplified Measure of Gobbledygook (SMOG) scores. Independent samples t-tests were used to compare readability, whereas the Mann-Whitney U test was used for accuracy and completeness. Interrater reliability was assessed with intraclass correlation coefficient (ICC). Significance was set at P < .05.

Interrater reliability was high for both models. ChatGPT showed excellent agreement for accuracy (ICC = 0.91) and good for completeness (ICC = 0.89). Gemini also showed excellent accuracy (ICC = 0.93) and moderate-to-good completeness (ICC = 0.78). No significant differences were found between models in accuracy, completeness, or readability. Both produced content with FKGL scores indicating university-level reading and SMOG scores suggesting high school-level comprehension.

The readability of the content may present challenges for general audiences due to its complexity. The models used in this study may assist in patient education; however, the results implied the importance of professional consultation and careful interpretation of artificial intelligence-generated information.