Interpretable CNN-Transformer Multimodal Hierarchical Fusion Network in Multivariate Calibration

Area of Science:

• Agricultural Science
• Spectroscopy
• Machine Learning

Background:

• Accurate quantitative analysis of agricultural produce is crucial for quality control and resource management.
• Traditional methods often rely solely on spectral data, potentially missing valuable information from other sample characteristics.
• Integrating diverse data sources can enhance the predictive power of analytical models.

Purpose of the Study:

• To develop and validate a novel multimodal hierarchical fusion framework for enhanced quantitative analysis of agricultural produce.
• To investigate the efficacy of integrating spectral features with auxiliary factors (region, cultivar, temperature) using a CNN-transformer architecture.
• To compare the performance of the multimodal model against single-modal approaches and existing machine learning models.

Main Methods:

• A hierarchical fusion framework was proposed, utilizing a 1D convolutional neural network (CNN) for spectral feature extraction.
• Auxiliary factors underwent encoding (sine-cosine or label) and were embedded into the feature space via a fully connected network.
• A transformer model was employed for global interaction and fusion of spectral and auxiliary features, followed by validation on mango and tobacco datasets using UV-Vis-NIR spectra.

Main Results:

• The multimodal CNN-transformer model significantly improved prediction performance for mango dry matter content (DMC), reducing RMSE from ~1.0 to ~0.6.
• The proposed model outperformed 11 other machine learning models and single-modal approaches relying only on spectral data.
• SHAP analysis confirmed the significant contribution of auxiliary factors (region, temperature, cultivar) alongside spectral features (near 960 nm) to DMC prediction in both mango and tobacco datasets.

Conclusions:

• The CNN-transformer multimodal model effectively overcomes the limitations of single-modal analysis by integrating spectral and auxiliary data.
• This approach provides a robust framework for quantitative analysis, offering novel technical support for agricultural applications.
• The findings highlight the potential of multimodal fusion strategies in enhancing the accuracy and reliability of predictive models in agricultural science.