Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: May 22, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Benchmarking Large Language Models from Open and Closed Source Models to Apply Data Annotation for Free-Text Criteria

Ali Nemati1, Mohammad Assadi Shalmani1, Qiang Lu2

  • 1Health Informatics Department, Zilber College of Public Health, University of Wisconsin, Milwaukee, WI 53211, USA.

Future Internet
|May 21, 2026
PubMed
Summary

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Psychological Impact of AI-Simplified Brain MRI Reports: A Randomized Trial of Patient Understanding, Anxiety, and Health Literacy.

Journal of clinical medicine·2026
Same author

SymRefine: A symbolic regression approach for refining and compressing neural networks.

Neurocomputing·2026
Same author

Explainable AI reveals temporal risk pathways in fall prediction: Extracting clinical insights from multi-horizon machine learning models.

GeroScience·2026
Same author

The impact of socioeconomic factors on pancreatic cancer care utilization.

PloS one·2025
Same author

Advancing personalized healthcare: leveraging explainable AI for BPPV risk assessment.

Health information science and systems·2024
Same author

Predicting Risk of Stroke From Lab Tests Using Machine Learning Algorithms: Development and Evaluation of Prediction Models.

JMIR formative research·2021
Same journal

Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era.

Future internet·2025
Same journal

Dynamic Cost-Aware Routing of Web Requests.

Future internet·2019
See all related articles
This summary is machine-generated.

This study introduces a novel framework to evaluate large language models (LLMs) for healthcare data annotation. The framework benchmarks LLM accuracy in extracting patient characteristics, revealing performance strengths and limitations.

Area of Science:

  • Artificial Intelligence in Medicine
  • Clinical Data Management
  • Natural Language Processing

Background:

  • Large language models (LLMs) show promise for annotating free-text healthcare records.
  • Ensuring LLM accuracy is crucial for clinical research and patient characteristic extraction.
  • Current LLM performance in healthcare data annotation requires rigorous evaluation.

Purpose of the Study:

  • To introduce a novel evaluation framework for benchmarking LLM annotation quality in healthcare.
  • To assess the accuracy of leading LLMs in extracting key patient characteristics.
  • To provide a method for responsible adoption of LLMs in clinical applications.

Main Methods:

  • Developed a novel evaluation framework using Multi-Criteria Decision Analysis (MCDA) and TOPSIS.
Keywords:
closed source and open source modelsdecision-making in healthcareevaluation metricshealthcare data annotationhuman and LLM evaluationlarge language modelsmulti-criteria decision analysis

More Related Videos

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

Related Experiment Videos

Last Updated: May 22, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

  • Defined ten evaluation metrics covering age, gender, BMI, disease presence, and blood markers.
  • Assessed leading open-source and commercial large language models.

    • Main Results:

    • Achieved accuracy scores for specific criteria: 0.59 (age), 1 (gender), 0.84 (BMI), 0.56 (disease presence), and 0.92 (blood markers).
    • Demonstrated varying performance levels across different LLMs and evaluation metrics.
    • Highlighted specific strengths and limitations of current LLM capabilities.

    Conclusions:

    • The MCDA-TOPSIS framework offers a rigorous method for evaluating LLM performance in healthcare data annotation.
    • LLMs demonstrate potential but also current limitations in accurately extracting patient characteristics.
    • This benchmarking approach supports informed decision-making for integrating LLMs into clinical research.