An artificial intelligence-based dental semantic search engine as a reliable tool for dental students and educators

This study proposes the utilization of a Natural Language Processing tool to create a semantic search engine for dental education while addressing the increasing concerns of accuracy, bias, and hallucination in outputs generated by AI tools. The paper focuses on developing and evaluating DentQA, a specialized question-answering tool that makes it easy for students to seek information to access information located in handouts or study material distributed by an institution.

DentQA is structured upon the GPT3.5 language model, utilizing prompt engineering to extract information from external dental documents that experts have verified. Evaluation involves non-human metrics (BLEU scores) and human metrics for the tool's performance, relevance, accuracy, and functionality.

Non-human metrics confirm DentQA's linguistic proficiency, achieving a Unigram BLEU score of 0.85. Human metrics reveal DentQA's superiority over GPT3.5 in terms of accuracy (p = 0.00004) and absence of hallucination (p = 0.026). Additional metrics confirmed consistent performance across different question types (X² (4, N = 200) = 13.0378, p = 0.012). User satisfaction and performance metrics support DentQA's usability and effectiveness, with a response time of 3.5 s and over 70% satisfaction across all evaluated parameters.

The study advocates using a semantic search engine in dental education, mitigating concerns of misinformation and hallucination. By outlining the workflow and the utilization of open-source tools and methods, the study encourages the utilization of similar tools for dental education while underscoring the importance of customizing AI models for dentistry. Further optimizations, testing, and utilization of recent advances can contribute to dental education significantly.