Related Experiment Video
Updated: May 26, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Validating large language model-assisted data extraction from clinical notes
J W van Koevorden1,2, N Aben3, V Struben3
1Department of Head and Neck Surgery, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, the Netherlands.
Background:
Health care professionals face increasing documentation burdens, which can compromise efficiency and patient safety. Large language models (LLMs) may offer a scalable solution by automating data extraction from unstructured clinical notes. This study evaluates the accuracy and clinical impact of structured data extraction by an LLM compared with manual extraction by physicians in the context of head and neck oncology consultations.
Patients And Methods:
This was a prospective validation study comparing LLM-powered data extraction with manual extraction. Clinical documentation from 60 patients (1482 pages) was analyzed. Data were extracted for 29 clinically relevant categories by two physicians and a pretrained open-source LLM. Six clinical experts evaluated 2555 extracted values in a two-step procedure: first, a blinded binary Match/Non-Match assessment between LLM output and human consensus; and second, categorization of all Non-Matches into predefined error types (Incorrect, Incomplete, Missing, Hallucination, or Overcomplete). These expert-assigned labels formed the basis for accuracy, precision, recall, and F1 scores. Extraction times were compared using a paired Student's t-test, and error impact was scored on a 5-point Likert scale.
Results:
LLM-powered extraction achieved accuracies between 74% [pathology: 95% confidence interval (CI) 65% to 82%] and 90% (patient characteristics: 95% CI 87% to 94%). Manual extraction showed 29% interobserver disagreement (95% CI 25.19% to 32.74%). Of 2555 extracted values, 68 were rated as high-impact errors, although evaluator assessments varied widely. Hallucinations were rare (0.16%) and low impact. LLM extraction reduced average time per case from 8.6 minutes to 1.9 minutes (P < 0.001).
Conclusion:
LLMs can support clinical workflows by reducing documentation time and maintaining acceptable accuracy, provided that human oversight is ensured. These findings support further exploration of AI-assisted documentation tools in clinical practice.
