Utility of an LLM-powered experts-in-the-loop chatbot for pre- and post-operative care of cataract surgery patients

PurposeTo evaluate the utility of CataractBot, an LLM (Large Language Model)-powered chatbot that provides doctor-verified answers to patient questions about cataract surgery. We examine its use by both end-users (patients and attendants) and medical experts.MethodsA 24-week study was conducted to evaluate CataractBot among patients, their attendants, doctors, and patient coordinators. The bot responded instantly to questions by querying a knowledge base curated by medical professionals. Each response was asynchronously verified by an ophthalmologist (for medical questions) or a patient coordinator (for logistical questions), and their edits contributed to updating the knowledge base, thereby minimizing future expert intervention. A mixed-methods analysis was conducted on interaction logs, including patient and attendant questions, chatbot answers, and expert verifications.ResultsA total of 318 patients and attendants sent 1,992 messages, and LLM-generated answers were verified by five doctors and two coordinators. Questions asked pre-surgery were significantly more than post-surgery $(p<0.001)$. Participants asked significantly more medical than logistical questions $(t_{309}=7.3,p<0.001)$. Doctors rated 84.5% of CataractBot's answers to medical questions as accurate and complete. Their edits, which mainly involved adding information, increased the acceptance of the bot's answers by 19.0% over time.ConclusionCataractBot was predominantly used to address medical questions. It incorporated expert corrections to improve its answers and reduce the experts' bot-related workload over time. This study highlights the potential of LLM-powered chatbots to support patient-provider communication in ophthalmology.