Evaluating the Quality of Brain MRI Generators.

Area of Science:

• Neuroimaging
• Artificial Intelligence
• Medical Image Analysis

Background:

• Deep learning models can generate structural brain MRIs, potentially accelerating neuroscience research.
• Current quality evaluation metrics for generative models, often adapted from natural image processing, are insufficient for assessing brain MRI anatomical plausibility.
• This inadequacy leads to inconclusive findings regarding the quality of generated brain MRIs.

Purpose of the Study:

• To propose and validate a novel framework for evaluating the quality of generative models producing structural brain MRIs.
• To address the limitations of existing metrics in assessing anatomical plausibility and reliability.
• To provide a standardized and rigorous method for comparing different generative models for brain MRI synthesis.

Main Methods:

• A framework was developed involving uniform processing of real MRIs, standardized model implementation, and automated segmentation of generated MRIs.
• The framework quantifies anatomical plausibility by analyzing segmented brain structures.
• Reliability checks for segmentation accuracy were rigorously implemented, a critical step often overlooked.
• Six state-of-the-art generative models were trained and tested on over 3000 MRIs.

Main Results:

• Existing metrics (e.g., SSIM, FID) were shown to be sensitive to experimental setup and poor indicators of anatomical plausibility.
• Only 3 out of 6 generative models produced MRIs that met the framework's criteria for high-quality output.
• At least 95% of the generated MRIs from these 3 models had highly reliable segmentations.
• The framework's assessments aligned with qualitative evaluations, confirming its validity.

Conclusions:

• Standard image metrics are inadequate for evaluating generative models of brain MRIs.
• The proposed framework offers a reliable and valid method for assessing the anatomical plausibility and quality of synthetic brain MRIs.
• This approach is crucial for advancing the use of generative models in neuroscience discovery.
• The framework's code is publicly available for broader adoption and research.