Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Search research articles

Related Experiment Video

Updated: May 28, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study.

Romain Lombardi^1,2, Alexandre Destere^3,4, Jean Dellamonica^1,2,5

¹Critical Care Unit, Pasteur 2 University Hospital, 30 Voie Romaine, Nice, 06100, France, 33 0669032616.

JMIR Medical Education

|May 26, 2026

Summary

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

External Evaluation of Population Pharmacokinetic Models of Cabotegravir, During Its Oral and Intramuscular Administration in HIV-Infected Patients.

CPT: pharmacometrics & systems pharmacology·2026

Same author

Effects of prone positioning on recruitment-to-inflation ratio in patients with acute respiratory distress syndrome: a physiological study.

Intensive care medicine·2026

Same author

Machine learning in ARDS: an intensivist's guide to artificial intelligence applications.

Critical care (London, England)·2026

Same author

Heart-lungs interactions in mechanically ventilated patients: physiology and clinical implications.

Intensive care medicine·2026

Same author

Therapeutic Drug Monitoring of Long-Acting Cabotegravir and Rilpivirine in a National Cohort of People With HIV-1: First Results From the ANRS-MIE CARLAPOP Study.

Clinical infectious diseases : an official publication of the Infectious Diseases Society of America·2025

Same author

Safety profile of belatacept in a real-life setting: disproportionality analysis of the WHO pharmacovigilance database.

BMC pharmacology & toxicology·2025

This summary is machine-generated.

Large language models (LLMs) can identify ambiguous medical exam questions, offering a new tool for quality control in assessments. This technology may improve the fairness and clarity of medical evaluations by detecting issues before exams are finalized.

Area of Science:

Medical Education
Artificial Intelligence in Healthcare
Assessment Design

Background:

Large language models (LLMs) show potential in medical education for simulating expert reasoning and aiding assessment quality control.
This study evaluated the efficacy of LLMs in identifying poorly constructed or ambiguous questions within critical care academic assessments.

Purpose of the Study:

To develop automated ambiguity and quality scores for objective assessment of individual exam questions and entire exam components.
To evaluate the performance of multiple LLMs in detecting ambiguity and assessing the quality of medical assessment items.

Main Methods:

Analysis of 264 academic exam questions from Université Côte d'Azur across 4 docimological formats (PCC, mini-PCC, key feature problems, IQS) over 3 academic years.

Keywords:

LLM ambiguity detection artificial intelligence automated scoring critical care docimology emergency exams large language model medical education quality assessment

More Related Videos

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

Related Experiment Videos

Last Updated: May 28, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

Evaluation of 4 LLMs (ChatGPT, Gemini Pro, Le Chat, DeepSeek) without prompt engineering, using official correction keys for performance assessment.

Application of binary diagnostic tags (ambiguity, low performance, incoherence, subjective ambiguity) to generate composite ambiguity and weighted quality scores.

Main Results:

LLMs achieved scores comparable to students, with no significant difference across academic years.
Significantly higher LLM performance was observed on mini-PCC and isolated question sequences (IQS) formats (P=.049 and P=.04).
IQS items exhibited the highest ambiguity scores, and tag patterns frequently indicated ambiguity and inconsistency, with quality scores varying by academic year.

Conclusions:

LLMs provide a preliminary framework for proactively detecting ambiguous exam questions and estimating exam quality.
Integrating LLMs into assessment design can potentially reduce post-exam corrections, enhancing fairness and clarity in medical evaluations.