When does more data help? Spectral Geometry and Scaling Laws in MRI Transformers

Area of Science:

• Artificial Intelligence
• Neuroimaging
• Machine Learning

Background:

• Scaling laws predict model performance growth with data.
• Eigenspectrum of latent representations influences scaling behavior.
• Investigating spectral distribution of discriminative signals in MRI transformers.

Purpose of the Study:

• Evaluate if spectral mode signal distribution predicts future scaling behavior.
• Compare supervised and self-supervised transformers for Alzheimer's disease classification.
• Analyze how representation concentration affects data scaling.

Main Methods:

• Trained 3 supervised 3D vision transformers (ViT3D, MINiT, NIT) for Alzheimer's disease classification using 2,822 MRI scans (ADNI).
• Compared encoder spectra of supervised models with a frozen self-supervised DINO ViT-B/16 encoder.
• Analyzed Mahalanobis signal via spectral expansion to assess disease information distribution.

Main Results:

• Supervised models concentrated 90-96% of CLS-token variance in a single principal component.
• Self-supervised DINO distributed signals across many latent directions.
• Supervised training concentrated disease information into one mode; self-supervised training yielded richer spectral geometry, higher effective rank, and discoverability.
• Supervised models showed flatter AUC(N) curves; DINO improved by 11.0 percentage points from N=50 to N=2,822.

Conclusions:

• Spectral distribution of discriminative signals influences discoverable performance with increasing sample size.
• Distributed representations in self-supervised models allow for continued improvement with more data.
• Concentrated representations in supervised models exhaust discoverable signal at lower sample sizes.