A transformer and 3D CNN-based feature fusion network with interpretable ability for Raman spectra analysis:...

Accurate differentiation of benign and malignant thyroid lesions continues to pose a significant clinical challenge. Raman spectroscopy offers label-free molecular fingerprints of cells, yet the identification of diagnostic spectral patterns remains challenging. While artificial intelligence has been applied to analyze Raman data as one-dimensional (1D) signals, such approaches may overlook subtle nonlinear relationships across wavenumbers, particularly in cases involving spectrally similar constituents. Converting 1D spectral data into two-dimensional (2D) representations can preserve both amplitude and positional correlations, thereby uncovering latent temporal and structural features. However, such transformations risk incurring information loss, the extent of which is contingent upon the encoding strategy employed. To address this, we propose a novel multimodal deep learning framework that synergistically integrates 1D spectral and 2D spatiotemporal features, representing the first application in Raman-based thyroid cancer detection. Our model uniquely combines a Transformer to capture global dependencies in 1D spectra and a 3D-CNN to extract local spatial patterns from multiple 2D spectral transformations. These dual-modality features are adaptively fused through a multi-head cross-attention mechanism, enabling dynamic feature integration. The multimodal model ultimately achieves an accuracy of 94.7% in the identification of thyroid lesions, outperforming the unimodal Transformer and 3D-CNN models, which achieve accuracies of 91.0% and 89.4%, respectively. Notably, the multimodal model enhances interpretability by identifying contributions of key Raman peaks to the classification decision. Thus, the integration of SERS with explainable deep learning establishes a novel method for thyroid cancer diagnosis, achieving both exceptional diagnostic performance and significantly enhanced model interpretability.