Comparison of Large Language Models with Rules-Based Natural Language Processing Algorithms for Extracting Data from...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Data Informatics
• Natural Language Processing

Background:

• Developing automated data extraction pipelines is crucial for leveraging registry data in total hip arthroplasty (THA).
• Existing natural language processing (NLP) algorithms have limitations in accurately extracting specific data points from clinical notes.

Purpose of the Study:

• To develop and evaluate large language model (LLM) pipelines for automated data extraction from THA operative notes.
• To compare the performance of LLM-based pipelines against established NLP algorithms for registry data extraction.

Main Methods:

• A dataset of 1,000 primary THA operative notes was manually annotated for surgical approach, bearing surface, and fixation technique.
• Custom LLM data extraction pipelines were developed using iteratively customized prompts.
• Performance was evaluated by comparing LLM accuracy against rules-based NLP algorithms on a held-out test set.

Main Results:

• LLM pipelines achieved superior accuracy over NLP algorithms for all three data points: surgical approach (96% vs. 94%), bearing surface (89% vs. 74%), and fixation technique (96% vs. 95%).
• LLMs successfully inferred the bearing surface in 80% of ambiguous cases, demonstrating enhanced data interpretation capabilities.

Conclusions:

• LLM pipelines offer a more accurate and efficient method for extracting key registry data from THA operative notes.
• This technology has the potential to significantly improve the quality and accessibility of orthopaedic registry data, impacting clinical care and evidence generation.