Comparison of Large Language Models with Rules-Based Natural Language Processing Algorithms for Extracting Data from...

We aimed to develop automated data extraction pipelines with large language models (LLMs) to extract registry data from total hip arthroplasty (THA) operative notes and compare the performance with that of existing natural language processing (NLP) algorithms.

We randomly sampled 1,000 primary THA cases from our institutional registry. Two human annotators manually reviewed each operative note for 3 data points: surgical approach, bearing surface, and fixation technique. All labeled THA notes were split into the development set (n = 239) and the testing set (n = 719). We developed a custom data extraction pipeline for each data point by combining an iteratively customized prompt with an LLM. The performance was compared with that of existing rules-based NLP algorithms.

The accuracy of LLMs was superior to that of NLP algorithms for all data points: surgical approach (96% compared with 94%), bearing surface (89% compared with 74%), and fixation technique (96% compared with 95%). Furthermore, the LLM accurately inferred the bearing surface for 80% of the notes that were ambiguous about the bearing surface.

We developed LLM pipelines for extracting 3 registry-relevant data points from THA operative notes, demonstrating superior performance to existing NLP algorithms.

LLMs have the potential to impact clinical care, including the evaluation of electronic medical record free-text data. As registries serve as a cornerstone of orthopaedic evidence, this work demonstrates promise for LLMs to simplify, improve, and democratize the construction of registry databases from operative notes.