ChatGpt's accuracy in the diagnosis of oral lesions

ChatGPT, a large language model (LLM) developed by OpenAI, is designed to generate human-like responses through the analysis of textual data. This study aimed to assess the accuracy and diagnostic capability of ChatGPT-4 in answering clinical scenario-based questions regarding oral lesions.

The study included 133 multiple-choice questions (MCQs), each consisting of five possible answers, randomly selected from the Clinical Guide to Oral Disease. Two oral medicine specialists reviewed the answers in the book to ensure accuracy. A general dentist categorized the questions into three levels of difficulty, and two oral medicine specialists validated these categorizations. At each level of difficulty, 37 questions were randomly selected. Consequently, the final questionnaire, consisting of a total of 111 questions categorized by difficulty level, was prepared. The process of asking questions began using the ''new message'' command, to minimize potential bias (influence of prior answers), the researchers manually cleared the chat history before presenting each new question.

ChatGPT-4.0 demonstrated an accuracy rate of 97% for easy questions, 86.5% ± 34.6% for medium-level questions, and 78.4% ± 41.7% for difficult questions, with an overall accuracy rate of 87.4% ± 33.3.

Although ChatGPT-4.0 demonstrated satisfactory accuracy in answering clinical questions, its responses should not be exclusively relied upon for diagnostic purposes. Instead, the model should be utilized as a complementary tool under the supervision of clinicians in the diagnosis of oral lesions.