Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Data Validation01:03

Data Validation

6.0K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
6.0K
Radiological Investigation I: X-ray and CT01:30

Radiological Investigation I: X-ray and CT

613
Radiological investigations, including X-rays and computed tomography (CT) scans, are critical for diagnosing and evaluating various medical conditions. These imaging techniques provide valuable insights into the body's internal structures, aiding in the detection of abnormalities, assessment of disease progression, and development of treatment strategies. This article delves into two primary radiological investigations, chest X-rays and CT scans, outlining their purpose, procedures, and...
613
Radiological Investigation II: MRI and Ventilation Perfusion Scan01:30

Radiological Investigation II: MRI and Ventilation Perfusion Scan

284
Description
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
284

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Generalization of AI-Based Gestational Age Assessment Using Blind Sweep Ultrasonography.

JAMA network open·2026
Same author

Estimating high-resolution albedo for urban applications.

Nature communications·2026
Same author

Inter-facility travel time and referral for emergency obstetric care in the 15 most populated cities of Nigeria: a spatial analysis.

BMC global and public health·2026
Same author

Impact of using artificial intelligence as a second reader in breast screening including arbitration.

Nature cancer·2026
Same author

Diagnostic accuracy, fairness and clinical implementation of AI for breast cancer screening: results of multicenter retrospective and prospective technical feasibility studies.

Nature cancer·2026
Same author

The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence.

Nature medicine·2025

Related Experiment Video

Updated: Nov 1, 2025

Accuracy in Dental Medicine, A New Way to Measure Trueness and Precision
07:57

Accuracy in Dental Medicine, A New Way to Measure Trueness and Precision

Published on: April 29, 2014

13.6K

Improving reference standards for validation of AI-based radiography.

Gavin E Duggan1, Joshua J Reicher1, Yun Liu1

  • 1Google Health (G.E.D., Y.L., D.T., S.S.), Stanford Health Care and Palo Alto Veterans Affairs (J.J.R.), California, California, USA.

The British Journal of Radiology
|June 18, 2021
PubMed
Summary
This summary is machine-generated.

Combining radiologist opinions through majority vote or discussion significantly improves agreement for AI validation reference standards. This enhances label reproducibility for chest radiograph analysis, crucial for developing reliable artificial intelligence tools.

More Related Videos

Author Spotlight: Advancing CBCT and Digital Dental Image Integration with AI-Assisted Digitization
05:49

Author Spotlight: Advancing CBCT and Digital Dental Image Integration with AI-Assisted Digitization

Published on: February 23, 2024

1.1K
Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System
05:33

Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System

Published on: July 11, 2025

472

Related Experiment Videos

Last Updated: Nov 1, 2025

Accuracy in Dental Medicine, A New Way to Measure Trueness and Precision
07:57

Accuracy in Dental Medicine, A New Way to Measure Trueness and Precision

Published on: April 29, 2014

13.6K
Author Spotlight: Advancing CBCT and Digital Dental Image Integration with AI-Assisted Digitization
05:49

Author Spotlight: Advancing CBCT and Digital Dental Image Integration with AI-Assisted Digitization

Published on: February 23, 2024

1.1K
Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System
05:33

Introduction of an Integrated Pathology Image Management, Artificial Intelligence, and Reporting System

Published on: July 11, 2025

472

Area of Science:

  • Radiology and Medical Imaging
  • Artificial Intelligence in Healthcare
  • Medical Informatics

Background:

  • Establishing a reliable reference standard is critical for validating artificial intelligence (AI) applications in medical imaging, particularly for chest radiographs.
  • Individual reader interpretations can exhibit significant variability, impacting the accuracy of AI model performance assessment.
  • Current methods for creating reference standards, such as expert panels, require careful consideration of how reader disagreements are resolved.

Purpose of the Study:

  • To evaluate the importance of combining multiple readers' opinions in a context-aware manner for establishing reference standards in AI validation.
  • To compare the effectiveness of individual reader assessments, majority vote, and panel-based discussion in maximizing interobserver agreement and label reproducibility.
  • To identify optimal methods for resolving disagreements among radiologists to improve the quality of reference standards for AI development.

Main Methods:

  • 1100 frontal chest radiographs were analyzed by six radiologists for six specific findings.
  • Radiologists initially provided individual assessments, followed by asynchronous adjudication in two panels of three readers to resolve disagreements.
  • Interreader agreement was quantified to measure the reproducibility of different assessment methods.

Main Results:

  • Panel-based majority vote demonstrated improved agreement across all findings compared to individual reader assessments.
  • Asynchronous adjudication, particularly after two rounds, further enhanced reproducibility for certain findings, notably reducing missed diagnoses.
  • Improvements in agreement varied by finding category, with adjudication showing significant benefits for cardiomegaly, fractures, and pneumothorax.

Conclusions:

  • Interreader agreement, even among board-certified radiologists, must be carefully considered when using reads as a reference standard for AI tool validation.
  • Techniques like majority vote, maximum sensitivity, or asynchronous adjudication can be applied to improve agreement and reproducibility for different findings.
  • These methods support the development of higher quality clinical research and more reliable AI applications in medical imaging.