Related Experiment Video
Updated: Aug 20, 2025
13:19
Deep Neural Networks for Image-Based Dietary Assessment
Published on: March 13, 2021
9.3K
Improving the repeatability of deep learning models with Monte Carlo dropout
Andreanne Lemay1,2, Katharina Hoebel1,3, Christopher P Bridge1,4
1Martinos Center for Biomedical Imaging, Boston, MA, USA.
NPJ Digital Medicine
|November 18, 2022
Summary
Improving artificial intelligence (AI) in healthcare requires robust models. Using Monte Carlo dropout predictions significantly enhances AI model repeatability in medical imaging tasks, ensuring more reliable clinical applications.
Area of Science:
- Medical imaging analysis
- Artificial intelligence in healthcare
- Machine learning model evaluation
Background:
- Clinical integration of artificial intelligence (AI) necessitates reliable and robust models.
- Model repeatability, a key attribute of robustness, is often overlooked during development and evaluation.
- Lack of focus on repeatability leads to AI models that are unusable in clinical practice.
Purpose of the Study:
- To evaluate the repeatability of various AI model types (binary, multi-class, ordinal classification, and regression) in medical image analysis.
- To assess the impact of Monte Carlo dropout predictions on model performance and repeatability across different medical imaging tasks.
- To determine the optimal number of Monte Carlo iterations for improving repeatability.
Main Methods:
- Evaluated four model types on four medical image classification tasks (knee osteoarthritis, cervical cancer screening, breast density estimation, retinopathy of prematurity).
- Measured and compared repeatability on ResNet and DenseNet architectures.
- Assessed the effect of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability.
Main Results:
- Leveraging Monte Carlo predictions significantly increased repeatability for all model types, reducing 95% limits of agreement by 16% and class disagreement by 7%.
- Classification accuracy generally improved alongside repeatability.
- No further significant gain in repeatability was observed beyond approximately 20 Monte Carlo iterations.
- Monte Carlo predictions demonstrated better calibration, providing more accurate output probabilities.
Conclusions:
- Monte Carlo dropout predictions are a valuable technique for enhancing the repeatability and reliability of AI models in medical imaging.
- The findings suggest that AI models with improved repeatability and calibration are more suitable for clinical workflows.
- The study provides practical insights into optimizing AI model evaluation for clinical deployment.
Related Concept Videos
01:19Survival Tree
132
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
132
02:07Improving Translational Accuracy
11.8K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.8K
01:10Propagation of Uncertainty from Systematic Error
606
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
606
01:12Observational Learning
265
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
265
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
98
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
98
00:59Propagation of Uncertainty from Random Error
793
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
793