The Transformative Potential of Large Language Models in Mining Electronic Health Records Data: Content Analysis

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Data Mining

Background:

• Evaluating the accuracy, efficiency, and cost-effectiveness of large language models (LLMs) for clinical information extraction.
• Focus on identifying and classifying patient comorbidities in oncology electronic health records (EHRs).

Purpose of the Study:

• Compare the performance of GPT-3.5 and GPT-4 models against specialized human evaluators.
• Assess LLM accuracy, efficiency, and cost-effectiveness in structuring clinical data.

Main Methods:

• Utilized OpenAI API to extract structured comorbidity data (JSON) from 250 personal history reports.
• Compared LLM results against manual reviews by 5 radiation oncology specialists.
• Employed metrics including sensitivity, specificity, precision, accuracy, F-value, and McNemar test.

Main Results:

• GPT-4 significantly outperformed GPT-3.5 and showed comparable or superior performance to physicians in key metrics (McNemar test, P<.001).
• GPT-4 achieved higher sensitivity (96.8%) than GPT-3.5 (88.2%) and physicians (88.8%).
• Physicians had marginally higher precision (97.7%) than GPT-4 (96.8%); GPT-4 demonstrated greater consistency and reliability.

Conclusions:

• LLMs, with effective prompting, can match or surpass medical specialists in clinical report data extraction.
• LLMs offer superior efficiency in time and cost for large-scale data mining.
• AI tools are valuable for real-world evidence generation and integration with databases.