Hyperelastic Regularization for Near-Diffeomorphic Transformer-Based Brain MRI Registration

Transformer-based deformable brain MRI registration achieves high overlap accuracy, but predicted displacement fields can contain voxels with a non-positive Jacobian determinant-local foldings that violate the diffeomorphism assumption required by tensor-based morphometry and atlas-fusion segmentation workflows. We introduce HypEReg, a non-linear hyperelastic regularizer that acts directly on the Jacobian determinant of the predicted displacement field. HypEReg couples a clamped-rational volume-distortion penalty (detJϕ-1)2/max(detJϕ,ϵ) with an explicit per-voxel anti-folding hinge [max(0,ϵ-detJϕ)]2, integrated as a purely loss-side module into a TransMorph backbone with no inference-graph modifications. On the IXI atlas-to-subject benchmark (115 test subjects), HypEReg-TransMorph maintains grouped Dice (0.7537) while reducing the det(Jϕ)≤0 voxel ratio from 1.502×10-2 (TransMorph) to 1.5×10-5, with identical per-case runtime and parameter count to the unregularized baseline. In strict zero-shot transfer to OASIS Learn2Reg test pairs (no fine-tuning), HypEReg-TransMorph achieves Dice 0.7756 with a det(Jϕ)≤0 ratio of 7.6×10-5, roughly two orders of magnitude below plain TransMorph zero-shot (Dice 0.7691; ratio 9.6×10-3); downstream multi-atlas label fusion further confirms the practical benefit of fold suppression (fused Dice 0.8271 vs. 0.8201 for TransMorph). OASIS-2 longitudinal and ROI analyses support deformation plausibility (lower folding/SDlogJ and stronger ventricular ROI agreement), while clinical-covariate associations remain exploratory rather than biomarker-validating. Determinant-level, non-linear hyperelastic regularization substantially suppresses folding in Transformer dense-flow brain MRI registration while preserving alignment accuracy and adding zero inference cost, providing a practical drop-in regularization strategy that improves the reliability of deformation fields for morphometry-oriented deformable registration.