Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Survival Tree01:19

Survival Tree

131
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
 Building a Survival Tree
Constructing a...
131
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving01:29

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

92
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
92
Mechanistic Models: Compartment Models in Individual and Population Analysis01:23

Mechanistic Models: Compartment Models in Individual and Population Analysis

74
Mechanistic models are utilized in individual analysis using single-source data, but imperfections arise due to data collection errors, preventing perfect prediction of observed data. The mathematical equation involves known values (Xi), observed concentrations (Ci), measurement errors (εi), model parameters (ϕj), and the related function (ƒi) for i number of values. Different least-squares metrics quantify differences between predicted and observed values. The ordinary least...
74
Systematic Error: Methodological and Sampling Errors01:15

Systematic Error: Methodological and Sampling Errors

1.8K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
1.8K
Data Validation01:15

Data Validation

206
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
206
Generalization, Discrimination, and Extinction01:24

Generalization, Discrimination, and Extinction

710
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
710

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Physics Insights into Liver MRI: Educational Guidance for Protocol Optimization.

Radiographics : a review publication of the Radiological Society of North America, Inc·2026
Same author

Impact of Baydur Ratio Correction on the Reliability of Esophageal Pressure Measurement.

Journal of applied physiology (Bethesda, Md. : 1985)·2026
Same author

Strengthening Biomarker Research in Canadian Cancer Clinical Trials: A Pathology-Focused White Paper.

Current oncology (Toronto, Ont.)·2026
Same author

CT-based deep learning prediction of complete response in intermediate-stage hepatocellular carcinoma treated with drug-eluting beads transarterial chemoembolization.

BJR artificial intelligence·2026
Same author

Brain MRI Analysis of Cumulative Blast Exposure and Intracranial Aneurysms in Special Operations Forces.

Radiology·2026
Same author

The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation.

Journal of medical Internet research·2026

Related Experiment Video

Updated: Aug 12, 2025

Constructing and Visualizing Models using Mime-based Machine-learning Framework
06:19

Constructing and Visualizing Models using Mime-based Machine-learning Framework

Published on: July 22, 2025

115

Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls.

Farhad Maleki1, Katie Ovens1, Rajiv Gupta1

  • 1Department of Computer Science, University of Calgary, Calgary, Canada (F.M., K.O.); Department of Radiology, Massachusetts General Hospital, Boston, Mass (R.G.); Augmented Intelligence & Precision Health Laboratory (AIPHL), Department of Radiology and the Research Institute of the McGill University Health Centre, McGill University, Montreal, Canada (C.R., R.F.); Montreal Imaging Experts, Montreal, Canada (C.R., R.F.); Division of Pathology, Jewish General Hospital, Montreal, Canada (A.S.); and Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurologic Diseases, University of Florida College of Medicine, UF Health Shands Hospital, 1600 SW Archer Rd, Gainesville, FL 32610-0374 (R.F.).

Radiology. Artificial Intelligence
|February 1, 2023
PubMed
Summary

Methodological pitfalls in machine learning, such as violating independence assumptions and using incorrect evaluation metrics, can lead to inaccurate medical image analysis models. Avoiding these issues is crucial for developing generalizable and reliable diagnostic and prognostic tools.

Keywords:
Convolutional Neural Network (CNN)Deep LearningDiagnosisGeneralizabilityMachine LearningMedical Image AnalysisModel EvaluationPrognosisRandom Forest

More Related Videos

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

445
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

659

Related Experiment Videos

Last Updated: Aug 12, 2025

Constructing and Visualizing Models using Mime-based Machine-learning Framework
06:19

Constructing and Visualizing Models using Mime-based Machine-learning Framework

Published on: July 22, 2025

115
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

445
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

659

Area of Science:

  • Medical Image Analysis
  • Machine Learning in Healthcare
  • Computational Pathology

Background:

  • Machine learning models are increasingly used in medical image analysis for diagnosis and prognosis.
  • However, methodological pitfalls can compromise model generalizability and lead to inaccurate predictions.
  • Common pitfalls include violating independence assumptions, inappropriate performance evaluation, and batch effects.

Purpose of the Study:

  • To investigate the impact of three key methodological pitfalls on the generalizability of machine learning models.
  • To quantitatively illustrate how these pitfalls affect model performance and reliability.
  • To emphasize the importance of avoiding these pitfalls for developing robust medical AI.

Main Methods:

  • Retrospective datasets including CT, histopathologic analysis, and radiography were utilized.
  • Machine learning models were developed with and without the identified methodological pitfalls.
  • Performance was measured using the F1 score, with statistical comparisons using the Wilcoxon rank sum test where applicable.

Main Results:

  • Violating independence assumptions (e.g., oversampling before splitting data) inflated F1 scores by up to 71.2% in cancer prediction tasks.
  • Inappropriate data splitting improved F1 scores superficially by 21.8% but did not guarantee high-quality segmentation.
  • Batch effects severely impacted model performance, with a pneumonia detection model misclassifying 96.14% of new healthy patient samples.

Conclusions:

  • Methodological pitfalls, often undetectable during internal validation, lead to overoptimistic performance metrics and inaccurate real-world predictions.
  • Understanding and actively avoiding these pitfalls is essential for developing trustworthy and generalizable machine learning models in medical imaging.
  • This study underscores the need for rigorous methodology in developing AI for diagnosis and prognosis.