Benchmarking uncertainty quantification for protein engineering.

Area of Science:

• Computational biology
• Machine learning
• Protein engineering

Background:

• Machine learning sequence-function models are crucial for protein engineering.
• Effective protein design requires methods for sequence selection and model improvement.
• These methods rely on calibrated model uncertainty estimation.

Purpose of the Study:

• To benchmark deep learning uncertainty quantification (UQ) methods on protein sequence-function regression tasks.
• To evaluate UQ method performance across varying distributional shifts and data representations.
• To assess the utility of UQ methods in active learning and Bayesian optimization for protein design.

Main Methods:

• Implemented and compared various deep learning UQ methods on the Fitness Landscape Inference for Proteins (FLIP) benchmark.
• Assessed UQ performance using metrics like accuracy, calibration, coverage, width, and rank correlation.
• Evaluated UQ methods with one-hot encoding and pre-trained language model representations in retrospective active learning and Bayesian optimization.

Main Results:

• No single UQ method demonstrated superior performance across all datasets, splits, and metrics.
• Uncertainty-based sampling in Bayesian optimization often did not outperform greedy sampling.
• Performance varied depending on the specific UQ method, data representation, and distributional shift.

Conclusions:

• The choice of UQ method is critical and context-dependent for protein machine learning applications.
• Current UQ methods may not consistently improve active learning or Bayesian optimization in protein engineering.
• Recommendations are provided for optimizing biological sequence design using machine learning and UQ.