Development and Assessment of a Custom GPT-4o Chatbot for Postoperative Counseling and Frequently Asked Questions in...

Chatbots have an impressive ability to answer patient inquiries and assist medical personnel, but they are limited by specialty-specific knowledge deficits and a lack of informational integrity. These issues can be addressed by training chatbots for specific work functions. The purpose of this study was to develop a custom-trained chatbot to answer common patient questions following breast surgery.

A custom chatbot was developed using an advanced language platform trained with internal and public patient information resources. A comparison study was conducted between the chatbot and nurse responses to common patient questions posed after breast surgery. Response tone, comprehension, accuracy, effectiveness, and clarity were evaluated using a Likert scale.

The chatbot significantly outperformed the comparison group's responses across all 5 measured metrics with large effect sizes in each (P < 0.001; rank-biserial correlation >0.5). The chatbot achieved a mean overall score of 4.30 out of 5, which was significantly higher than the comparison group's overall mean of 2.96 (P < 0.001; rank-biserial correlation 0.656). The chatbot achieved the largest effect sizes under the comprehension and effectiveness metrics (respective rank-biserial correlation 0.722 and 0.668), demonstrating highest improvement over the comparison's performance in these areas.

The chatbot shows potential to enhance postoperative patient care in breast surgery. The chatbot provided responses that were ranked higher than the comparison group across all measured communication metrics. These findings highlight the potential for chatbots to improve patient counseling, alleviate patient anxiety, increase postoperative patient compliance, and improve patient-provider communication.