Fine-Tuned Bidirectional Encoder Representations From Transformers Versus ChatGPT for Text-Based Outpatient...

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Medical Informatics

Background:

• Patients face challenges in selecting appropriate outpatient specialists based on symptoms.
• Natural Language Processing (NLP) models offer potential solutions for pre-hospital decision-making.

Purpose of the Study:

• To evaluate the performance of ChatGPT in recommending medical specialties for patient inquiries.
• To compare ChatGPT's recommendations against a fine-tuned Korean Medical bidirectional encoder representations from transformers (KM-BERT) model.

Main Methods:

• Utilized a dataset of 31,482 medical questions with doctor-assigned specialty labels (27 distinct categories).
• Compared ChatGPT and KM-BERT using accuracy, precision, recall, and F1-score metrics.
• Categorized ChatGPT responses into matching and non-matching predefined specialties.

Main Results:

• ChatGPT achieved 0.939 accuracy, 0.219 precision, 0.168 recall, and 0.134 F1-score, with a 6.2% answer avoidance rate.
• KM-BERT outperformed ChatGPT with 0.977 accuracy, 0.570 precision, 0.652 recall, and 0.587 F1-score.
• ChatGPT provided explanations elucidating symptom pathophysiology.

Conclusions:

• While KM-BERT demonstrated superior accuracy in specialty recommendation, ChatGPT offers valuable explanatory capabilities.
• ChatGPT's conversational nature and detailed explanations can improve patient comprehension and the medical referral process.