Utilizing GPT-4 to interpret oral mucosal disease photographs for structured report generation

The aim of this study is to evaluate GPT-4's reasoning ability to interpret oral mucosal disease photos and generate structured reports from free-text inputs, while exploring the role of prompt engineering in enhancing its performance. Prompt received by utilizing automatic prompt engineering and knowledge of oral physicians, was provided to GPT-4 for generating structured reports based on cases of oral mucosal disease. The structured reports included 7 fine-grained items: "location", "shape", "number", "size", "clinical manifestation", "the border of the lesion" and "diagnosis". 120 cases were used for testing, which were divided into two datasets, textbook dataset and internet dataset. Oral physicians evaluated GPT-4's responses by confusion matrices, receiving recall and accuracy. ANOVA and Wald χ2 tests with Bonferroni correction were used to statistical analysis. A total of 120 cases of oral mucosal diseases were included, encompassing the following two datasets: textbook dataset (n = 60), internet dataset (n = 60). GPT-4 had higher recall with the textbook dataset compared to the internet dataset (90.73% vs 89.12%; P = .462, χ² = 0.54) and higher accuracy (87.05% vs 84.87%; P = .393, χ² = 0.73). Performance varied by items of structured reports within each dataset, with "size" achieving the highest accuracy in the textbook dataset (98.90%) and "the border the lesion" in the internet dataset (95.00%). GPT-4 can transform incomplete descriptive text corresponding to oral mucosal disease photographs into structured reports with the assistance of carefully designed prompts. This study highlights GPT-4's potential in complex and multimodal medical tasks and underscores the importance of prompt engineering in optimizing its capabilities. Nevertheless, achieving further improvements in the model may require more comprehensive and focused efforts. This article demonstrated the capabilities of large multimodal models, represented by GPT-4, in medical photographs interpretation and medical report generation. GPT-4 was capable of recognizing photographs of oral mucosal diseases and generating structured reports, which facilitates telemedicine and peer-to-peer communication.