Development of a Natural Language Processing Algorithm to Aid Creation of a Surgical Spine Database

Manual effort required for feature extraction from operative reports can be time-intensive, inaccurate, and costly.

Develop a natural language processing (NLP) algorithm for surgical spine database creation and validate performance against human reviewers with varying clinical experience.

Retrospective study at a tertiary academic institution from 2008-2024.

Operative spinal cases on adult patients from 22,951 spinal operative reports.

Primary outcome was accuracy in operated spinal level classification. Secondary outcomes include clinical diagnoses, operative procedures, and additional surgical detail classification.

A rule-based NLP algorithm was developed and validated on a random sample of 100 operative reports. Performance of the NLP algorithm and human reviewers was assessed against a reference standard. An a priori acceptability threshold was set at 0.90 for agreement, kappa, and F1. Time required for chart review/processing were also assessed for the NLP algorithm and human reviewers.

The NLP algorithm matched the reference standard on operated spinal levels (Percent Agreement=93%; Kappa=0.98; F1=0.98), which outperformed human reviewers in clinical training. Performance on procedural and diagnostic feature extraction did not meet the accepted threshold but largely outperformed human validators in clinical training. NLP algorithm computation time averaged speed of 1,039 reports per minute, compared to the fastest median time of 1 minute per operative report for human chart reviewers.

The NLP algorithm can accurately and reliably extract operated spinal levels from high volumes of operative reports. However, classification of diagnosis, operative procedures, and additional surgical procedures were not sufficient for practical implementation.