Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Jul 13, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Development of a benchmarking dataset for symptom detection using large language models.

Joshua Davis1,2, Brigitte N Durieux1, Chloe Van Dongen1

  • 1Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States.

JAMIA Open
|July 12, 2026
PubMed
Summary

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Buprenorphine for the treatment of pain in patients with central nervous system tumors: A case series.

Neuro-oncology practice·2026
Same author

Characteristics, Diagnosis, and Treatment of Aphasia in Patients With Brain Tumors: A Scoping Review.

Neurology·2026
Same author

Pathways to Advance Targeted and Helpful Serious Illness Conversations (PATH-SIC): A Randomized Clinical Trial.

Journal of the National Comprehensive Cancer Network : JNCCN·2026
Same author

Palliative Care for Older Adults With Hip Fracture: An Explanatory Sequential Mixed-Methods Study.

Journal of pain and symptom management·2026
Same author

Patterns of Goals-of-Care Documentation in the Outpatient Oncology Setting: A Retrospective Cohort Study of Older Adults with Advanced Cancer.

Journal of palliative medicine·2026
Same author

Using large language models to identify geriatric assessment domains in patients with advanced cancer: A feasibility study.

Journal of geriatric oncology·2026

We developed an evaluation pipeline to assess how well large language models (LLMs) capture patient symptoms from clinical notes. GPT-4.1 showed the best performance in this symptom-detection task.

Area of Science:

  • Natural Language Processing
  • Clinical Informatics
  • Artificial Intelligence in Healthcare

Background:

  • Accurate symptom capture from clinical encounters is crucial for diagnosis and treatment.
  • Large Language Models (LLMs) show potential in processing clinical text but require robust evaluation.
  • Existing methods for evaluating LLM performance on clinical data are limited.

Purpose of the Study:

  • To establish a standardized pipeline for evaluating LLMs in extracting symptoms from clinical encounters.
  • To benchmark the performance of various LLMs on a curated dataset of simulated clinical interactions.
  • To provide a publicly available resource for assessing LLM capabilities in clinical symptom recognition.

Main Methods:

  • A gold-standard dataset was created with symptom annotations from 264 simulated doctor-patient encounters.
Keywords:
artificial intelligencecomputing methodologiesnatural language processingsigns and symptomssymptom assessment

Related Experiment Videos

Last Updated: Jul 13, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

  • Nine LLMs from four different vendors were tested using the developed evaluation pipeline.
  • Model outputs were assessed for structural correctness and accurate symptom information extraction.

    • Main Results:

    • Symptom information was present in 68% of 3085 evaluated text excerpts.
    • Pain, cough, and shortness of breath were the most frequently identified symptoms.
    • LLMs achieved F1 scores between 0.66-0.88 for common symptoms, with GPT-4.1 exhibiting the highest overall performance.

    Conclusions:

    • The developed evaluation pipeline and dataset are effective for benchmarking LLMs on clinical symptom extraction.
    • The findings support the ongoing development of LLMs for improved patient symptom understanding in healthcare.
    • The publicly available resources facilitate further research and optimization of LLMs in clinical applications.