Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
Language and Cognition01:27

Language and Cognition

323
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
323

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Skilled Nursing Facility-to-Home Transitions After Heart Failure Hospitalization: A Mixed-Methods Study of Communication, Self-Care, Medication, and Follow-Up.

Circulation. Heart failure·2026
Same author

Engagement With Mobile Health Cardiac Rehabilitation Varies Widely Among Older Adults With Ischemic Heart Disease.

Journal of cardiopulmonary rehabilitation and prevention·2026
Same author

General-purpose large language models outperform specialized clinical AI tools on medical benchmarks.

Nature medicine·2026
Same author

What is the role of formulation in contemporary mental health practice?

Australasian psychiatry : bulletin of Royal Australian and New Zealand College of Psychiatrists·2026
Same author

Author Correction: Physiology and immunology of a pig-to-human decedent kidney xenotransplant.

Nature·2026
Same author

Amygdala microstructural changes in subjective cognitive decline: A diffusion kurtosis and neurite orientation dispersion and density imaging study.

Journal of Alzheimer's disease : JAD·2026

Related Experiment Video

Updated: Jun 5, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

502

Evaluating Large Language Models in extracting cognitive exam dates and scores.

Hao Zhang1, Neil Jethani1, Simon Jones1

  • 1NYU Grossman School of Medicine, New York, New York, United States of America.

PLOS Digital Health
|December 11, 2024
PubMed
Summary

ChatGPT and LlaMA-2 were evaluated for extracting cognitive test scores from clinical notes. ChatGPT demonstrated superior accuracy in extracting MMSE and CDR data, showing higher reliability for clinical applications.

More Related Videos

Highlighting and Reducing the Impact of Negative Aging Stereotypes During Older Adults' Cognitive Testing
06:58

Highlighting and Reducing the Impact of Negative Aging Stereotypes During Older Adults' Cognitive Testing

Published on: January 24, 2020

7.3K
Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
06:48

Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment

Published on: June 25, 2019

9.1K

Related Experiment Videos

Last Updated: Jun 5, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

502
Highlighting and Reducing the Impact of Negative Aging Stereotypes During Older Adults' Cognitive Testing
06:58

Highlighting and Reducing the Impact of Negative Aging Stereotypes During Older Adults' Cognitive Testing

Published on: January 24, 2020

7.3K
Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
06:48

Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment

Published on: June 25, 2019

9.1K

Area of Science:

  • Artificial Intelligence in Medicine
  • Natural Language Processing
  • Clinical Informatics

Background:

  • Reliability of Large Language Models (LLMs) is critical for clinical tasks.
  • Assessing LLMs for extracting specific clinical information, such as cognitive test results, is essential for their safe implementation.

Purpose of the Study:

  • To evaluate the performance of two state-of-the-art LLMs, ChatGPT (GPT-4) and LlaMA-2, in extracting clinical information related to cognitive tests (MMSE, CDR, MoCA) from electronic health records.
  • To compare the accuracy, sensitivity, and precision of ChatGPT and LlaMA-2 in identifying and extracting cognitive test scores and dates.

Main Methods:

  • A dataset of 135,307 clinical notes was curated, with 34,465 meeting inclusion criteria for MMSE, CDR, or MoCA mentions.
  • 765 notes were processed by ChatGPT and LlaMA-2, with expert review of the LLM outputs.
  • Performance metrics including accuracy, sensitivity, precision, and inter-rater agreement (Fleiss' Kappa) were calculated following TRIPOD guidelines.

Main Results:

  • ChatGPT achieved higher accuracy for MMSE extraction (83% vs. 66.4% for LlaMA-2) and CDR extraction (87.1% vs. 74.5% for LlaMA-2).
  • ChatGPT demonstrated significantly better sensitivity and precision compared to LlaMA-2 for both MMSE and CDR data extraction.
  • Qualitative analysis revealed fewer instances of hallucination and incorrect date reporting with ChatGPT compared to LlaMA-2.

Conclusions:

  • ChatGPT exhibits superior performance over LlaMA-2 in extracting cognitive test information from clinical notes, indicating its potential for clinical use.
  • LLMs like ChatGPT can aid dementia research and patient identification for treatment or clinical trials.
  • Rigorous validation is essential to understand LLM capabilities and limitations in healthcare settings.