Evaluating Retrieval-Augmented Large Language Models on External Cervical Resorption: A Comparative Study of Gemini...

Area of Science:

• Artificial Intelligence in Dentistry
• Clinical Decision Support Systems
• Natural Language Processing in Healthcare

Background:

• This study assessed the accuracy and consistency of two Alphabet Inc. large language models: Google Gemini (GG) and NotebookLM (NLM).
• The evaluation focused on answering clinical questions related to external cervical resorption using a retrieval-augmented framework.
• NotebookLM is a document-grounded configuration, while Google Gemini was used in its base configuration.

Purpose of the Study:

• To evaluate the accuracy and consistency of Google Gemini and NotebookLM in responding to clinical questions about external cervical resorption.
• To compare the performance of a base large language model against a document-grounded configuration.
• To determine if retrieval augmentation significantly impacts the quality of responses for structured clinical tasks.

Main Methods:

• Forty-six dichotomous clinical questions on external cervical resorption were created by three endodontic experts.
• Each question was posed to Google Gemini and NotebookLM via three independent user accounts, generating 276 total responses.
• Responses were independently assessed by three endodontic experts against gold standard answers for accuracy and consistency.

Main Results:

• Google Gemini achieved 89% accuracy and 93% consistency.
• NotebookLM achieved 96% accuracy and 90% consistency.
• No statistically significant differences were found between the two models regarding accuracy and consistency.

Conclusions:

• Both NotebookLM and Google Gemini demonstrated high accuracy and consistency in answering clinical questions.
• NotebookLM exhibited a slightly superior performance compared to Google Gemini.
• Retrieval augmentation did not yield significant improvements for these specific structured clinical questions.