Evaluating Large Language Model's accuracy in current procedural terminology coding given operative note templates...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Plastic Surgery Coding

Background:

• Manual Current Procedural Terminology (CPT) coding is time-consuming and increases administrative burden in healthcare.
• Large Language Models (LLMs) present a potential solution for automating CPT code generation.
• The accuracy of LLMs in assigning CPT codes from operative notes is not well-established.

Purpose of the Study:

• To evaluate the accuracy of three LLMs (GPT-4, Gemini, Copilot) in generating CPT codes from plastic surgery operative note templates.
• To compare the performance of these LLMs across different plastic surgery subspecialties.

Main Methods:

• Twenty-six deidentified operative note templates from six plastic surgery subspecialties were used.
• A standardized prompt was employed to request CPT codes from each LLM.
• Model outputs were compared against surgeon-verified codes and categorized as correct, partially correct, or incorrect.

Main Results:

• A significant difference in overall coding accuracy was observed between the LLMs (p=0.02176).
• Gemini and Copilot demonstrated the highest accuracy rates (19.2% each), with Copilot yielding more partially correct outputs (53.8%). GPT-4 had the lowest accuracy (7.7%).
• Gemini excelled in aesthetic surgery (60% accuracy), while Copilot was most accurate in general reconstruction (42.9%). No models accurately coded breast reconstruction or craniofacial trauma.

Conclusions:

• LLMs demonstrate potential for automating CPT coding but currently lack the necessary contextual understanding for consistent accuracy.
• Human oversight and ongoing model refinement are essential for the successful implementation of LLM-based CPT coding systems.