Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction.

Area of Science:

• Machine Learning
• Statistical Learning Theory
• Uncertainty Quantification

Background:

• Frequentist models often lack reliable uncertainty quantification and calibration.
• Bayesian methods offer calibration but require strict model assumptions.
• Conformal prediction (CP) provides general calibration guarantees but can yield large, uninformative sets with limited data.

Purpose of the Study:

• To introduce a novel meta-learning scheme, meta-XB, for reducing the size of prediction sets in conformal prediction.
• To improve the informativeness of set predictions while maintaining formal calibration guarantees.
• To enhance per-input calibration through adaptive non-conformal scores.

Main Methods:

• Developed meta-XB, a meta-learning approach based on cross-validation conformal prediction (CVCP).
• Ensured meta-XB preserves formal per-task calibration guarantees, unlike prior methods.
• Extended meta-XB with adaptive non-conformal scores for improved calibration.

Main Results:

• Meta-XB effectively reduces the size of predicted sets compared to standard CP.
• The proposed method maintains formal per-task calibration guarantees.
• Adaptive non-conformal scores empirically improve marginal per-input calibration.

Conclusions:

• Meta-XB offers a general and effective solution for enhancing conformal prediction with limited data.
• The approach balances prediction set informativeness with rigorous calibration guarantees.
• Further improvements in calibration are achievable through adaptive scoring mechanisms.