Calibrated Variant Effect Prediction at the Residue Level Using Conditional Score Distributions.

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning in Genomics

Background:

• Accurate and well-calibrated variant effect prediction (VEP) models are crucial for effective clinical use.
• Current VEP models often lack reliable probability estimates, hindering their practical application.
• Global or per-protein calibration schemes do not adequately address miscalibration within specific variant subgroups.

Purpose of the Study:

• To develop a practical and robust method for calibrating VEP models at the residue level.
• To identify variant subgroups that require targeted calibration for improved VEP performance.
• To enhance the interpretability and reliability of VEP predictions across diverse variant types.

Main Methods:

• Proposed a residue-level calibration strategy, contrasting with global or per-protein methods.
• Developed RaCoon (Residue-aware Calibration via Conditional distributions), implemented on the ESM1b model.
• Analyzed model-specific feature distributions to guide the calibration strategy.

Main Results:

• Identified specific variant subgroups where VEP models exhibit significant miscalibration, despite appearing well-calibrated on average.
• RaCoon demonstrated multicalibrated and interpretable predictions across diverse variant subgroups.
• Achieved significant performance improvements across multiple benchmarks, increasing AUCROC from 0.912 to 0.924.

Conclusions:

• Targeted residue-level calibration is essential for robust VEP model performance and reliability.
• RaCoon provides a transferable and effective strategy for improving VEP calibration and accuracy.
• The developed approach enhances clinical utility by providing more meaningful probability estimates for variant effects.