Large language models outperform traditional natural language processing methods in extracting patient-reported...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Gastroenterology

Background:

• Patient-reported outcomes (PROs) are crucial for assessing inflammatory bowel disease (IBD) activity and treatment effectiveness.
• Manual extraction of PROs from clinical notes is time-consuming and labor-intensive.
• Improving data curation from electronic health records (EHRs) is essential for research and quality improvement.

Purpose of the Study:

• To compare traditional natural language processing (tNLP) with large language models (LLMs) for extracting IBD PROs.
• To evaluate the accuracy and generalizability of LLMs in extracting abdominal pain, diarrhea, and fecal blood from clinical notes.
• To assess the potential of LLMs in enhancing IBD research and patient care.

Main Methods:

• Annotation of clinic notes for three IBD PROs using predefined protocols.
• Development and internal testing of tNLP and LLM models at UCSF.
• External validation of models at Stanford University, comparing accuracy, sensitivity, specificity, and predictive values.
• Fairness and error assessments were conducted for all models.

Main Results:

• Inter-rater reliability for annotation exceeded 90%.
• On internal testing, LLMs (GPT-4) achieved accuracies of 96% (abdominal pain), 88% (diarrhea), and 90% (fecal blood), comparable to top tNLP models.
• External validation showed tNLP models failing to generalize (61-62% accuracy), while GPT-4 maintained >90% accuracy; PaLM-2 and GPT-4 performed similarly. No demographic biases were detected.

Conclusions:

• Large language models offer accurate and generalizable methods for extracting PROs from IBD clinical notes.
• LLMs maintain high accuracy across different institutions, irrespective of variations in note templates and authors.
• Widespread adoption of LLMs can significantly advance IBD research and improve patient care.