High-throughput and nondestructive multimodal deep fusion for variety identification of sorghum kernels

Accurate, nondestructive, and high-throughput identification of sorghum varieties is essential for intelligent agriculture and industrial quality control. However, single-modality approaches based on either RGB imaging or near-infrared spectroscopy (NIRS) often suffer from limited robustness when inter-variety differences are subtle. In this study, a synchronized NIRS-RGB multimodal dataset was established for six representative sorghum varieties, including 14,400 paired kernel samples (2400 samples per variety). Each sample contained a NIR spectrum covering 901.467-1706.76 nm with 256 variables and a high-resolution RGB image. An end-to-end Multi-modal Attention Network (MAN) was proposed to perform deep feature fusion through bi-directional cross-attention, enabling adaptive interactions between spectral physicochemical signatures and visual appearance cues. Under an 8:2 split with five-fold cross-validation on the training set, MAN with a Swin Transformer visual branch achieved an accuracy of 0.9888 and an F1-score of 0.9889, outperforming the best NIRS-only baseline by 8.40 percentage points in accuracy and the best RGB-only baseline by 6.09 percentage points. Ablation results showed that cross-attention was the major contributor to performance improvement, while bi-directional interaction and spectral SE attention further enhanced discriminability. Furthermore, independent three-day replicate validation achieved a pooled accuracy of 0.9758 and an F1-score of 0.9755, indicating good generalizability across acquisition days. The results demonstrate that the proposed MAN effectively exploits complementary NIRS and RGB information, providing a robust and nondestructive solution for fine-grained sorghum variety identification in high-throughput inspection scenarios.