Assessing AI Accuracy in Generating CPT Codes From Surgical Operative Notes

Accurate and efficient medical coding is essential for proper reimbursement and health care management. Current Procedural Terminology (CPT) codes, derived from operative notes, standardize medical billing but are often prone to variability and errors due to the complexity of surgical procedures. With advancements in artificial intelligence (AI), tools like ChatGPT and other large language models (LLMs) are being explored for their potential to automate coding tasks. This study evaluates the ability of LLMs to generate accurate CPT codes for craniofacial surgical procedures based on operative notes.

Operative notes for 10 craniofacial surgical cases were collected from a single surgeon at Nemours Children's Health. The notes were provided to AI tools (ChatGPT 4.0 and Gemini) to generate corresponding CPT codes. These AI-generated codes were compared against manually coded results by expert reviewers. Responses were evaluated for accuracy against manually generated CPT codes and classified as correct, partially correct, or incorrect.

ChatGPT and Gemini demonstrated similar performance in generating CPT codes, with no statistically significant differences in accuracy or correctness between the models ( P > 0.999). Gemini produced a slightly higher proportion of correct responses (30% versus 20%), whereas ChatGPT had more partially correct responses (50% versus 40%).

This study demonstrates that AI may be a clinically valuable resource for craniofacial CPT coding, reducing administrative burden and increasing coding accuracy. Findings from this research could inform the integration of AI into medical billing practices, promoting efficiency in surgical specialties. Future research will explore generalizability to other surgical domains and refinement of AI models for coding tasks.