Related Experiment Video
Updated: Apr 7, 2026
Author Spotlight: Bridging Gaps in Anatomy and Establishing a Foundation for Algorithmic Studies
Published on: December 15, 2023
Mind the Gap: Human and AI Uncertainties in Cardiac MRI Segmentation
Qiao Lin1, Xin Chen2, Chao Chen2
1School of Computer Science, University of Nottingham, Ningbo, China. qiao.lin@nottingham.edu.cn.
Purpose:
This study conducts quantitative and qualitative analyses to investigate the relationship between human-annotated and AI-derived uncertainty in cardiac MRI segmentation, aiming to enhance the reliability of AI-based cardiac MRI segmentation models and foster better human-AI collaboration.
Methods:
The CMRI dataset used in the experiments consists of 483 scans, each with two types of labels: an annotated segmentation mask and an uncertainty score per CMRI slice, both provided by clinicians. First, the AI-derived uncertainty estimated by the fuzzy-based algorithm is utilized to indicate the quality of segmentation. Multiple levels of uncertainties are derived from the method, including class-wise, slice-wise, subject-wise, etc. Subsequently, they are compared to the human-annotated uncertainty scores. Finally, qualitative analyses are conducted with clinicians to investigate all uncertainty measures potentially impacting real clinical applications.
Results:
Experimental results show a strong inverse correlation between AI-derived uncertainty and Dice score, a standard metric for segmentation quality, indicating that lower uncertainty predicts higher segmentation quality. Additionally, it is found that human-annotated uncertainty coincides with AI-derived uncertainty for some anatomical structures (e.g., papillary muscle). However, high human-annotated uncertainty does not necessarily correlate with low AI segmentation quality, and there is no obvious association between human-annotated uncertainty and the size of the segmented structure. Concluded from qualitative analysis with clinicians, humans are better at utilizing prior knowledge (e.g. cardiac structural and contextual information) for uncertainty scoring, while the current AI method lacks this capability and is mainly data-driven for decision-making and uncertainty estimation.
Conclusion:
AI-derived uncertainty could be utilized as quality control for CMRI segmentation. Humans utilize structural and contextual information to formulate uncertainty, while AI models currently lack this capability.
