Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

The survival double descent: generalization dynamics of deep neural networks in time-to-event analysis.

BMC medical research methodology·2026
Same author

The CARE framework for AI dataset documentation in clinical laboratories: a comprehensive checklist and data lineage methodology.

American journal of clinical pathology·2026
Same author

Genomic and Immune Landscape of Pancreatic Ductal Adenocarcinoma Associated with Germline Pathogenic Variants in ATM.

Clinical cancer research : an official journal of the American Association for Cancer Research·2025
Same author

Establishing a comprehensive artificial intelligence lifecycle framework for laboratory medicine and pathology: A series introduction.

American journal of clinical pathology·2025
Same author

Analysis of more than 400,000 women provides case-control evidence for BRCA1 and BRCA2 variant classification.

Nature communications·2025
Same author

Functional evaluation and clinical classification of BRCA2 variants.

Nature·2025

Related Experiment Video

Updated: Feb 28, 2026

Performing Data Mining And Integrative Analysis Of Biomarker in Breast Cancer Using Multiple Publicly Accessible Databases
07:41

Performing Data Mining And Integrative Analysis Of Biomarker in Breast Cancer Using Multiple Publicly Accessible Databases

Published on: May 17, 2019

9.7K

Agent-based large language model system for extracting structured data from breast cancer synoptic reports: a

Steven N Hart1, Teya S Bergamaschi2

  • 1Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN 55901, United States.

JAMIA Open
|February 27, 2026
PubMed
Summary
This summary is machine-generated.

Large Language Models (LLMs) show promise for extracting breast cancer pathology data, but real-world performance lags behind synthetic tests. Human oversight is crucial for clinical deployment.

Keywords:
artificial intelligenceclinical data extractionlarge language modelsnatural language processingpathology reports

More Related Videos

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.3K
A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
07:50

A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts

Published on: September 20, 2018

16.6K

Related Experiment Videos

Last Updated: Feb 28, 2026

Performing Data Mining And Integrative Analysis Of Biomarker in Breast Cancer Using Multiple Publicly Accessible Databases
07:41

Performing Data Mining And Integrative Analysis Of Biomarker in Breast Cancer Using Multiple Publicly Accessible Databases

Published on: May 17, 2019

9.7K
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.3K
A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
07:50

A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts

Published on: September 20, 2018

16.6K

Area of Science:

  • Artificial Intelligence in Medicine
  • Computational Pathology
  • Natural Language Processing in Healthcare

Background:

  • Breast cancer pathology reports contain crucial structured data for clinical decision-making.
  • Manual data extraction from pathology reports is time-consuming and prone to errors.
  • Large Language Models (LLMs) offer potential for automating structured data extraction from clinical text.

Purpose of the Study:

  • To develop and validate an agent-based LLM system for extracting structured data from breast cancer synoptic pathology reports.
  • To assess the performance gap between synthetic and real-world validation of LLM-based data extraction.
  • To compare the performance of seven leading LLMs on this task.

Main Methods:

  • Developed a modular AI agent-based framework using sequential specialized LLMs.
  • Normalized College of American Pathologists (CAP) cancer protocols into 8 sections, 86 subsections, and 229 discrete fields.
  • Validated seven LLMs using both synthetic (864 cases) and real-world (90 reports, 6651 fields) datasets.

Main Results:

  • Synthetic validation showed high accuracy (93.8%-99.0%).
  • Real-world evaluation revealed a significant performance drop (recall: 61.8%-87.7%), indicating a "reality gap".
  • Gemini-2.5-pro achieved the highest real-world recall (87.7%); smaller models (e.g., 14B-parameter Deepseek-R1) performed competitively.

Conclusions:

  • Synthetic validation alone provides misleadingly high confidence.
  • Real-world performance degradation highlights the complexity of clinical documentation.
  • Mandatory human verification is essential due to performance gaps; LLMs are best as screening tools.