A multimodal fusion network for heart sound abnormality detection and classification

Objective.Accurate physiological assessment of cardiac function from heart sounds remains challenging due to background noise, variable heart rates, and the need for reliable cardiac-cycle segmentation. This study aimed to develop a fully E2E deep learning framework that extracts diagnostic information directly from raw heart sound recordings for cardiac abnormality detection and classification.Approach.We propose HS-MMNet, an E2E multi-modal deep learning framework designed for physiological heart sound analysis. Recordings are preprocessed (normalization and 25-400 Hz bandpass filtering) and divided into fixed-length 2.5 s segments. A Convolution Head with multi-atrous spatial pyramid and channel-spatial attention extracts fine-grained local temporal patterns from the filtered 1-D waveform. A Transformer Head captures long-range spectro-temporal dependencies from Log-Mel spectrograms. These hypotheses are iteratively fused by a novel multi-hypothesis cross-attention module with cyclic query-key-value assignment and a hypothesis-mixing MLP, enabling rich cross-site interaction and effective suppression of noise and non-informative regions. Recording-level classification is obtained via a fully connected layer.Main results.On the PhysioNet/CinC Challenge 2016 dataset, HS-MMNet achieved 94.80% accuracy, 92.10% sensitivity, 96.85% specificity, 87.50% precision, and 89.74%F1-score, outperforming all previously reported methods. On the balanced five-class Yaseen dataset (normal, aortic stenosis, mitral regurgitation, mitral stenosis, mitral valve prolapse), it attained 99.60% macro-averaged precision, recall, andF1-score with only four misclassifications in 1000 recordings, establishing new state-of-the-art (SOTA) benchmarks.Significance.HS-MMNet represents an advance in automated physiological measurement from heart sounds. By eliminating cardiac cycle detection and multi-channel requirements while achieving SOTA diagnostic performance, it provides a practical, scalable solution for accurate cardiovascular screening with primary-care and low-resource settings.