Deep learning-enabled multi-modal fusion of smartphone vision and portable spectroscopy for apple non-destructive...

Soluble solids content (SSC) is a crucial indicator for determining optimal harvest timing and assessing apple internal quality. Traditional destructive SSC measurement methods are time-consuming, labor-intensive, and unsuitable for large-scale commercial applications. This study presents a non-destructive SSC prediction approach for Venus Golden apples (VG) by integrating smartphone-based machine vision and portable visible/near-infrared (Vis/NIR) spectroscopy (650-950 nm) with deep learning algorithms. A total of 356 apple samples with SSC values ranging from 10.3% to 16.5% were used for model development and validation. Four multi-modal fusion strategies, including concatenation, bilinear interaction, element-wise addition, and cross-attention, were systematically evaluated. The results showed that the cross-attention fusion strategy achieved the best prediction performance. Specifically, the optimal ResNet1D-ResNet34 fusion model obtained an R²p of 0.915, RMSEP of 0.326%, and RPD of 3.894. Compared with the best spectroscopy-only model based on SNV-PLSR (R_P² = 0.867, RMSEP = 0.531%, RPD = 2.389), the proposed fusion model increased R_p² by 5.5%, reduced RMSEP by 38.6%, and improved RPD by 63.0%. It also outperformed the best image-only ViT_base model (R_p² = 0.816, RMSEP = 0.526%, RPD = 2.413), increasing R_p² by 12.1%, reducing RMSEP by 38.0%, and improving RPD by 61.4%. These findings demonstrate that combining low-cost portable spectroscopy, smartphone imaging, and cross-attention-based multi-modal fusion can provide an accurate and practical solution for non-destructive fruit quality assessment.