Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: May 5, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.3K

Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for

Anoeska Schipper1,2, Peter Belgers3, Rory David O'Connor4

  • 1Diagnostic Image Analysis Group, Medical Imaging Department, Radboud University Medical Center, Geert Grooteplein Zuid 10, Nijmegen, 6525 GA, The Netherlands, 31 614021323.

JMIR Medical Informatics
|May 4, 2026
PubMed
Summary

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Evaluating an AI-driven Triaging Workflow for MRI-based Clinically Significant Prostate Cancer Diagnosis: A Simulation Study.

Radiology. Imaging cancer·2026
Same author

Effect of ultrasound-guided fascia iliaca compartment block on pain management in hip fracture patients: A double-blind placebo-controlled randomized clinical trial.

Medicine·2025
Same author

Leveraging open-source large language models for clinical information extraction in resource-constrained settings.

JAMIA open·2025
Same author

Artificial intelligence-based identification of thin-cap fibroatheromas and clinical outcomes: the PECTUS-AI study.

European heart journal·2025
Same author

Characterizing the Impact of Training Data on Generalizability: Application in Deep Learning to Estimate Lung Nodule Malignancy Risk.

Radiology. Artificial intelligence·2025
Same author

Deep Learning Model for Breast Shear Wave Elastography to Improve Breast Cancer Diagnosis (INSPiRED 006): An International, Multicenter Analysis.

Journal of clinical oncology : official journal of the American Society of Clinical Oncology·2025

A small language model (LLM) effectively extracts clinical data from Dutch emergency department reports, approaching physician accuracy for appendicitis prediction. This technology supports decision-making without compromising privacy or transparency.

Area of Science:

  • Natural Language Processing
  • Machine Learning in Healthcare
  • Clinical Informatics

Background:

  • Emergency department (ED) free-text notes contain vital clinical information, but are difficult to reuse for research and decision support.
  • Large language models (LLMs) show promise for extracting features from clinical text, yet studies on ED reports, especially in non-English languages like Dutch, are limited.
  • Locally deployable LLMs offer a path to automated feature extraction for decision support without increasing physician workload.

Purpose of the Study:

  • To evaluate a small, open-source LLM (Qwen 2.5:14B) for extracting 16 clinical signs and symptoms from Dutch ED reports.
  • To assess the LLM's performance using minimal and optimized 0-shot prompts against researcher and physician annotations.
  • To determine the impact of LLM-extracted features on an appendicitis prediction model (HIVE).
Keywords:
electronic health recordsemergency medicinehealth informaticslarge language modelsmachine learningnatural language processingpredictive modeling

More Related Videos

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
07:31

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Published on: May 15, 2020

8.3K

Related Experiment Videos

Last Updated: May 5, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.3K
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
07:31

Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

Published on: May 15, 2020

8.3K

Main Methods:

  • Retrospective analysis of 336 ED reports from patients with acute abdominal pain at a Dutch teaching hospital.
  • Development of minimal and optimized 0-shot prompts using 100 reports; evaluation on remaining 236 reports.
  • Extraction of 16 signs/symptoms (binary, multiclass, multilabel) by LLM and 2 ED physicians; comparison against researcher annotations.
  • Inputting LLM-extracted and physician-annotated features into the HIVE appendicitis prediction model.

Main Results:

  • The LLM achieved weighted average accuracies of 0.910 (minimal prompts) and 0.929 (optimized prompts), compared to physician accuracies of 0.961 and 0.951.
  • The HIVE model demonstrated AUCs of 0.871 (minimal prompts) and 0.911 (optimized prompts) with LLM inputs, versus 0.917 and 0.924 with physician inputs.
  • The LLM demonstrated strong performance across binary, multiclass, and multilabel extraction tasks.

Conclusions:

  • A small, locally deployable multilingual LLM can achieve near physician-level accuracy in extracting structured clinical data from Dutch ED free-text reports.
  • This approach supports downstream diagnostic modeling while maintaining patient privacy, interpretability, and statistical transparency.
  • The findings suggest LLMs can be valuable tools for leveraging unstructured clinical data in resource-limited settings.