Chatbot Demonstrates Moderate Interrater Reliability in Billing for Hand Surgery Clinic Encounters

Artificial intelligence offers opportunities to improve the burden of health care administrative tasks. Application of machine learning to coding and billing for clinic encounters may represent time- and cost-saving benefits with low risk to patient outcomes.

Gemini, a publicly available large language model chatbot, was queried with 139 de-identified patient encounters from a single surgeon and asked to provide the Current Procedural Terminology code based on the criteria for different encounter types. Percent agreement and Cohen's kappa coefficient were calculated.

Gemini demonstrated 68% agreement for all encounter types, with a kappa coefficient of 0.586 corresponding to moderate interrater reliability. Agreement was highest for postoperative encounters (n = 43) with 98% agreement and lowest for new encounters (n = 27) with 48% agreement. Gemini recommended billing levels greater than the surgeon's billing level 31 times and lower billing levels 10 times, with 4 wrong encounter type codes.

A publicly available chatbot without specific programming for health care billing demonstrated moderate interrater reliability with a hand surgeon in billing clinic encounters. Future integration of artificial intelligence tools in physician workflow may improve the accuracy and speed of billing encounters and lower administrative costs.