Utilizing GPT-4 to interpret oral mucosal disease photographs for structured report generation

Area of Science:

• Artificial Intelligence in Medicine
• Medical Image Analysis
• Natural Language Processing

Background:

• Interpreting oral mucosal disease images and generating reports is crucial for diagnosis and communication.
• Current methods may be time-consuming or lack standardization.
• Large multimodal models offer potential for automating such tasks.

Purpose of the Study:

• To evaluate GPT-4's ability to interpret oral mucosal disease photographs.
• To assess GPT-4's performance in generating structured reports from free-text inputs.
• To explore the impact of prompt engineering on GPT-4's accuracy and recall.

Main Methods:

• GPT-4 was prompted using automatic prompt engineering and oral physician expertise.
• 120 oral mucosal disease cases (textbook and internet datasets) were used for testing.
• Structured reports included 7 fine-grained items: location, shape, number, size, clinical manifestation, lesion border, and diagnosis.

Main Results:

• GPT-4 demonstrated higher recall (90.73%) and accuracy (87.05%) with the textbook dataset compared to the internet dataset.
• Performance varied by report item, with 'size' (98.90%) and 'lesion border' (95.00%) showing high accuracy.
• GPT-4 successfully transformed descriptive text into structured reports, indicating potential for medical tasks.

Conclusions:

• GPT-4 shows significant potential in interpreting medical images and generating structured reports for oral mucosal diseases.
• Prompt engineering is vital for optimizing GPT-4's performance in complex medical applications.
• Further research is needed to enhance GPT-4's capabilities for comprehensive medical image interpretation and report generation.