Calibrated Uncertainty Estimation for Soil Organic Carbon from Raman Spectra.

Area of Science:

• Chemometrics
• Spectroscopy
• Machine Learning

Background:

• Machine learning (ML) models analyze Raman spectra for soil organic carbon (SOC) estimation.
• Current ML models for SOC lack reliable uncertainty estimates, limiting their practical use.

Purpose of the Study:

• To develop a framework for quantifying predictive uncertainty in SOC estimation using Shifted Excitation Raman Difference Spectroscopy (SERDS).
• To integrate conformal prediction (CP) with various uncertainty quantification (UQ) methods for statistically valid prediction intervals.

Main Methods:

• Employed conformal prediction (CP) with a calibration dataset to generate prediction intervals.
• Integrated CP with multiple UQ strategies (Deep Ensembles, Bayesian neural networks, etc.) for SOC estimation.
• Addressed both aleatoric and epistemic uncertainty sources in a field-relevant context.

Main Results:

• Conformalized UQ methods produced well-calibrated uncertainty estimates with narrow prediction intervals.
• Achieved reliable empirical coverage across various confidence levels.
• Demonstrated that uncertainty is predominantly aleatoric, emphasizing spectral quality over model complexity.

Conclusions:

• The proposed framework offers a practical and generalizable solution for trustworthy, sample-specific uncertainty estimates in Raman-based chemometrics.
• Conformalization is crucial for accurate uncertainty calibration in SOC estimation.
• Improvements in spectral data quality and preprocessing are key for enhancing predictive performance.