Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Multimodal Learning with Privileged Report Supervision for Generalizable Tuberculosis Detection on Chest Radiographs.

Journal of medical systems·2026
Same author

Oral Cancer Detection By Using Tabular Data Synthesis and Classification.

Proceedings ... ICDM workshops. IEEE International Conference on Data Mining·2026
Same author

Artificial Intelligence-Based Diagnosis of Kaposi Sarcoma Using Digital Photographs in Dark-Skinned Patients in Uganda.

JCO global oncology·2026
Same author

Glaucoma Classification Using a NFNet-Based Deep Learning Model with a Customized Hybrid Attention Mechanism.

Diagnostics (Basel, Switzerland)·2026
Same author

Evaluating Explainability: A Framework for Systematic Assessment of Explainable AI Features in Medical Imaging.

Bioengineering (Basel, Switzerland)·2026
Same author

Automating the Detection of Acetowhite Lesions by Classifying the Temporal Behavior of Cervical Regions.

Journal of lower genital tract disease·2025

Related Experiment Video

Updated: Aug 4, 2025

Author Spotlight: Bridging Gaps in Anatomy and Establishing a Foundation for Algorithmic Studies
04:25

Author Spotlight: Bridging Gaps in Anatomy and Establishing a Foundation for Algorithmic Studies

Published on: December 15, 2023

2.5K

Assessing Inter-Annotator Agreement for Medical Image Segmentation.

Feng Yang1, Ghada Zamzmi1, Sandeep Angara1

  • 1National Library of Medicine, National Institutes of Health, Bethesda, MD 20894, USA.

IEEE Access : Practical Innovations, Open Solutions
|April 3, 2023
PubMed
Summary

Expert variability in medical image annotation can harm AI performance. This study assesses inter-annotator agreement using heatmaps, kappa coefficients, and the STAPLE algorithm to improve AI model training and reliability.

Keywords:
Cohen’s kappaFleiss’ kappaReliabilitySTAPLEagreementheatmapinter-annotator

More Related Videos

Lesion Explorer: A Video-guided, Standardized Protocol for Accurate and Reliable MRI-derived Volumetrics in Alzheimer's Disease and Normal Elderly
12:50

Lesion Explorer: A Video-guided, Standardized Protocol for Accurate and Reliable MRI-derived Volumetrics in Alzheimer's Disease and Normal Elderly

Published on: April 14, 2014

40.3K
Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography
04:48

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Published on: November 30, 2022

2.8K

Related Experiment Videos

Last Updated: Aug 4, 2025

Author Spotlight: Bridging Gaps in Anatomy and Establishing a Foundation for Algorithmic Studies
04:25

Author Spotlight: Bridging Gaps in Anatomy and Establishing a Foundation for Algorithmic Studies

Published on: December 15, 2023

2.5K
Lesion Explorer: A Video-guided, Standardized Protocol for Accurate and Reliable MRI-derived Volumetrics in Alzheimer's Disease and Normal Elderly
12:50

Lesion Explorer: A Video-guided, Standardized Protocol for Accurate and Reliable MRI-derived Volumetrics in Alzheimer's Disease and Normal Elderly

Published on: April 14, 2014

40.3K
Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography
04:48

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Published on: November 30, 2022

2.8K

Area of Science:

  • Medical imaging
  • Artificial Intelligence
  • Computer Vision

Background:

  • AI-driven medical computer vision relies on accurate data annotations.
  • Variability among expert annotators introduces noise, potentially degrading AI algorithm performance.

Purpose of the Study:

  • To assess, illustrate, and interpret inter-annotator agreement in medical image segmentation.
  • To evaluate the impact of annotator variability on AI training data quality.

Main Methods:

  • Utilized common and ranking agreement heatmaps for qualitative assessment.
  • Employed extended Cohen's kappa and Fleiss' kappa for quantitative reliability evaluation.
  • Applied the STAPLE algorithm to generate ground truth and compute Intersection over Union (IoU), sensitivity, and specificity.

Main Results:

  • Experiments on cervical colposcopy and chest X-ray datasets demonstrated consistent inter-annotator reliability assessment.
  • Combining multiple metrics is crucial for avoiding assessment bias.
  • The proposed methods provide a robust framework for evaluating annotator agreement.

Conclusions:

  • Accurate inter-annotator agreement assessment is vital for reliable AI model development in medical imaging.
  • The combination of qualitative and quantitative metrics offers a comprehensive approach to understanding annotator variability.
  • Findings highlight the importance of addressing annotator consistency for high-performing medical AI.