Multimodal neural operators for real-time biomechanical modelling of traumatic brain injury

Traumatic brain injury (TBI) remains a major public health concern, with over 69 million cases annually worldwide. Accurate patient-specific biomechanical modeling is critical for injury risk assessment, but it requires integrating heterogeneous data sources such as volumetric neuroimaging, scalar demographic parameters, and acquisition metadata. Conventional finite element solvers can perform this modeling, yet they remain too computationally expensive for time-sensitive clinical settings. Neural operators have emerged as a promising alternative by learning resolution-invariant mappings between function spaces at orders-of-magnitude faster inference. However, while recent work has introduced parameter-conditioned and multi-input operator architectures for incorporating auxiliary scalar or geometric inputs, the systematic integration of volumetric medical imaging with heterogeneous scalar metadata within an operator learning framework remains underexplored, particularly for biomechanical prediction tasks involving patient-specific anatomical and demographic variability.

This study presents a systematic investigation of multimodal neural operator architectures for brain biomechanics, evaluating strategies for fusing heterogeneous input modalities such as volumetric anatomical imaging, scalar demographic features, and acquisition parameters to predict full-field brain displacement fields from MRE data.

We reformulated TBI modeling as a multimodal operator learning problem and proposed two fusion strategies: field projection for Fourier Neural Operator (FNO) based architectures (broadcasting scalars onto spatial grids) and branch decomposition for Deep Operator Networks (DeepONet) (separate encoding with multiplicative fusion). Four architectures (FNO, Factorized FNO (F-FNO), Multi-Grid FNO (MG-FNO), DeepONet) were extended with multimodal fusion mechanisms and evaluated on 249 in vivo Magnetic Resonance Elastography (MRE) datasets across physiologically relevant frequencies (20 to 90 Hz).

DeepONet achieved the highest accuracy on real displacement fields (MSE = 0.0039, 90.0% accuracy) with the fastest inference (3.83 it/s) and fewest parameters (2.09M), while MG-FNO achieved the best performance on imaginary fields (MSE = 0.0058, 88.3% accuracy) with the lowest GPU memory among FNO variants (7.12 GB). No single architecture dominated across all criteria, revealing distinct trade-offs between accuracy, spatial fidelity, and computational cost.

The results demonstrate that neural operators augmented with multimodal fusion mechanisms can accurately predict full-field brain displacement from heterogeneous biomedical inputs, with inference times orders of magnitude faster than finite element solvers. The systematic comparison of fusion strategies and architectures provides practical guidance for selecting operator learning approaches in biomedical settings where heterogeneous data integration is required.