New generative artificial intelligence model: ScholarGPT's performance on dental avulsion

This study aims to evaluate the performance of ScholarGPT, a Large Language Model (LLM) developed for academic purposes, on questions related to dental avulsion. In addition, to analyze and compare it with the results of the previous study evaluating the performance of ChatGPT and Gemini.

A total of 22 technical questions (11 multiple-choice questions (MCQs), 11 true/false (T/F)) were posed to the ScholarGPT. ScholarGPT responses were assessed using a modified Global Quality Scale (GQS). Responses were randomized using an online randomizer (www.randomizer.org) before scoring. A single researcher carried out the assessments at three different times, two weeks apart, and a new randomization was performed before each scoring.

When the answers given by ScholarGPT according to the question groups were analyzed by the Mann-Whitney U test, the mean value was found to be 4.64 for MCQ questions and 4.82 for T/F questions. ScholarGPT provided similarly high-quality and consistent answers in both question types (p = 0.590). When the performance of ScholarGPT was compared with Gemini and ChatGPT via the Friedman test, the mean score of ScholarGPT responses was significantly higher than both ChatGPT and Gemini (mean difference with Gemini = 0.75; mean difference with ChatGPT = 1.62, p < 0.001). ScholarGPT produced statistically significantly more consistent and higher-quality responses than ChatGPT and Gemini.

ScholarGPT showed high performance on technical questions related to dental avulsion and produced more consistent and higher-quality answers than ChatGPT and Gemini. According to the findings, LLMs based on academic databases can provide more accurate and reliable information. In the future, through the development of LLMs specific to the branches of dentistry, artificial intelligence systems can produce higher quality and consistent information.