Interpretable CNN-Transformer Multimodal Hierarchical Fusion Network in Multivariate Calibration

This study proposed a novel multimodal hierarchical fusion framework integrating a convolutional neural network (CNN) and a transformer. The approach enhanced model performance by fusing spectral features with some auxiliary factors of the samples, such as the locality of growth (region), type of produce (cultivar), and sample temperature (temp). Spectral data were extracted using one-dimensional CNN to capture local spectral features, while auxiliary factors underwent sine-cosine or label encoding before being embedded into the same feature space as spectral data via a fully connected network. Ultimately, a transformer was employed to achieve global interaction and fusion between spectral features and auxiliary factors rather than merely concatenating different feature types. The fusion strategy was validated using the ultraviolet (UV)-visible (vis)-near-infrared (NIR) spectra of mango and tobacco data sets. Compared to single-modal models using spectra only, the multimodal model using spectra coupled with the auxiliary factors achieved improved prediction performance on both validation and test sets for the mango dry matter content (DMC). The RMSE decreased from 0.984 and 1.03 to 0.577 and 0.613, respectively. These results outperformed those of the other 11 machine learning models. SHAP analysis revealed that the CNN-transformer framework successfully captured the underlying relationships between auxiliary factors (region, temp, and cultivar) and spectral features near 960 nm (due to the O-H absorption signal) with DMC, with the former contributing more significantly to the model than the latter. Similar observations were obtained in the tobacco data set. The results demonstrated the advantages of the CNN-transformer multimodal model in overcoming the limitations of single-modal information, providing novel technical support for quantitative analysis.