Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Improving Translational Accuracy02:07

Improving Translational Accuracy

2.7K
2.7K
Classification of Illness01:17

Classification of Illness

8.0K
The meaning of illness is individualized to each person who experiences an alteration in health. In contrast, disease is a medical term indicating a pathological change in the structure and function of the body or mind. It is a condition that has specific symptoms and boundaries.
An illness is a response to a disease in which the person's level of functioning is changed compared with a previous level. The general classification of illness includes acute and chronic.
Acute illness is severe...
8.0K
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving01:29

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

102
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
102

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Can AI Say "I Don't Know"?

The New England journal of medicine·2026
Same author

Performance of a large language model on the reasoning tasks of a physician.

Science (New York, N.Y.)·2026
Same author

Drug or Pokémon? An analysis of the ability of large language models to discern fabricated medications.

medRxiv : the preprint server for health sciences·2026
Same author

Siglec-F<sup>+</sup> neutrophils promote the resolution of acute lung injury through ALOX15 induction.

Science immunology·2025
Same author

Machine learning versus traditional formulas for fetal weight estimation: An international multicenter study evaluating prediction accuracy across birth weight percentiles.

International journal of gynaecology and obstetrics: the official organ of the International Federation of Gynaecology and Obstetrics·2025
Same author

Deciphering the Diagnostic Potential of Small Non-Coding RNAs for the Detection of Pancreatic Ductal Adenocarcinoma Through Liquid Biopsies.

International journal of molecular sciences·2025
Same journal

Correction: Confidence Measurement Metrics in Multimodal Large Language Models for Ultrasound-Based Radiology Cases: Comparative Evaluation Study of Self-Reported, Consistency-Based, and Hybrid Methods.

Journal of medical Internet research·2026
Same journal

Centering Equity During Health Technology Innovation: Scoping Review of Methods and Research Adjustments to Promote Inclusive Coproduction.

Journal of medical Internet research·2026
Same journal

Comparative Effectiveness of AI-Assisted Telerehabilitation, Telerehabilitation, In-Person Care, and Usual Care for Chronic Nonspecific Low Back Pain: Bayesian Network Meta-Analysis.

Journal of medical Internet research·2026
Same journal

Effectiveness of WeChat Public Account Intervention Based on the Information-Motivation-Behavioral Skills Model Among College Students With Internet Addiction: Randomized Controlled Trial.

Journal of medical Internet research·2026
Same journal

Are Traditional Registries Becoming Obsolete in the Modern Digital Health Ecosystem?

Journal of medical Internet research·2026
Same journal

Detecting and Preventing Fraudulent Participation in Qualitative Research: Content Analysis of Two Multisite Studies.

Journal of medical Internet research·2026
See all related articles

Related Experiment Video

Updated: Sep 15, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

691

Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking

Eden Avnat1,2, Michal Levy3,4, Daniel Herstain1

  • 1Faculty of Medicine, Tel Aviv University, Chaim Levanon St 55, Tel Aviv, 6997801, Israel, 972 545299622.

Journal of Medical Internet Research
|July 14, 2025
PubMed
Summary
This summary is machine-generated.

Large language models (LLMs) show varied performance in answering medical questions, excelling in semantic tasks but lagging in numerical ones compared to human experts. Claude 3 and GPT-4 demonstrate different strengths and weaknesses across medical disciplines.

Keywords:
benchmarkdatasetevidence-based medicinelarge language modelsquestions and answers

More Related Videos

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

587
Constructing and Visualizing Models using Mime-based Machine-learning Framework
06:19

Constructing and Visualizing Models using Mime-based Machine-learning Framework

Published on: July 22, 2025

726

Related Experiment Videos

Last Updated: Sep 15, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

691
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

587
Constructing and Visualizing Models using Mime-based Machine-learning Framework
06:19

Constructing and Visualizing Models using Mime-based Machine-learning Framework

Published on: July 22, 2025

726

Area of Science:

  • Artificial Intelligence in Medicine
  • Natural Language Processing
  • Clinical Decision Support

Background:

  • Clinical problem-solving relies on semantic and numerical medical knowledge.
  • Large language models (LLMs) show potential in clinical practice but face limitations in non-language-based tasks.
  • Tokenization can inherently limit LLMs' ability to generate evidence-based answers.

Purpose of the Study:

  • Evaluate LLM performance on numeric and semantic medical questions.
  • Examine intra- and inter-model differences in LLM capabilities across medical topics.
  • Compare LLM performance against human medical experts.

Main Methods:

  • Developed a comprehensive medical knowledge graph to create evidence-based medicine questions and answers (EBMQAs).
  • Benchmarked GPT-4 and Claude 3 Opus on 24,000 EBMQAs, assessing accuracy on numerical and semantic question types.
  • Validated performance by comparing LLMs against human medical experts on 100 numerical questions.

Main Results:

  • Claude 3 and GPT-4 achieved higher accuracy on semantic (68.7%, 68.4%) than numerical questions (61.3%, 56.7%).
  • Claude 3 outperformed GPT-4 in numerical question accuracy (P<.001), with significant performance variations across medical sublabels.
  • Human experts significantly surpassed both LLMs in accuracy (82.3% vs. 64.3% and 55.8%).

Conclusions:

  • LLMs perform better on semantic than numerical medical questions, with Claude 3 showing an advantage in numerical accuracy.
  • Both LLMs exhibit performance gaps across medical disciplines and are inferior to human experts.
  • LLM's decision to answer or abstain does not reliably predict accuracy, necessitating caution with their medical advice.