An Open-architecture AI Model for CPT Coding in Breast Surgery: Development, Validation, and Prospective Testing

To develop, validate, and prospectively test an open-architecture, transformer-based artificial Intelligence (AI) model to extract procedure codes from free-text breast surgery operative notes.

Operative note coding is time-intensive and error-prone, leading to lost revenue and compliance risks. Although AI offers potential solutions, adoption has been limited due to proprietary, closed-source systems lacking transparency and standardized validation.

We included all institutional breast surgery operative notes from July 2017 to December 2023. Expert medical coders manually reviewed and validated surgeon-assigned Current Procedural Terminology (CPT) codes, establishing a reference standard. We developed and validated an AI model to predict CPT codes from operative notes using 2 versions of the pretrained GatorTron clinical language model: a compact 345 million-parameter model and a larger 3.9 billion-parameter model, each fine-tuned on our labeled data-set. Performance was evaluated using the area under the precision-recall curve (AUPRC). Prospective testing was conducted on operative notes from May to October 2024.

Our data set included 3259 operative notes with 8036 CPT codes. Surgeon coding discrepancies were present in 12% of cases (overcoding: 8%, undercoding: 10%). The AI model showed strong alignment with the reference standard [compact version AUPRC: 0.976 (0.970, 0.983), large version AUPRC: 0.981 (0.977, 0.986)] on cross-validation, outperforming surgeons (AUPRC: 0.937). Prospective testing on 268 notes confirmed strong real-world performance.

Our open-architecture AI model demonstrated high performance in automating CPT code extraction, offering a scalable and transparent solution to improve surgical coding efficiency. Future work will assess whether AI can surpass human coders in accuracy and reliability.