Evaluating Plastic Surgery Chatbot Performance: Insights into Medical Triage, Classification Accuracy, and Escalation...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Plastic Surgery

Background:

• AI chatbots are common on plastic surgery websites for patient engagement.
• Their effectiveness in improving clinical workflow and patient satisfaction is not well-understood.

Purpose of the Study:

• To evaluate AI chatbot accuracy in triaging plastic surgery patients.
• To assess chatbot performance in urgency classification and patient interaction quality.

Main Methods:

• Analyzed chatbot responses to standardized clinical scenarios from top plastic surgery websites.
• Evaluated triage accuracy, classification, escalation, and patient experience using usability questionnaires.
• Conducted subgroup and thematic analysis of chatbot language.

Main Results:

• Significant variation in chatbot performance, especially in urgency classification (e.g., 80% false negatives for emergent cases).
• Moderate agreement with physician triage (Cohen's kappa = 0.47); over 50% of interactions required human escalation.
• Misclassified interactions correlated with lower patient usability scores.

Conclusions:

• AI chatbots are suitable for elective plastic surgery inquiries but inadequate for urgent/emergent cases.
• Current chatbots function primarily as administrative tools.
• Enhanced clinical capabilities are necessary for advanced AI chatbot deployment in healthcare.