Updated benchmarking of variant effect predictors using deep mutational scanning.

Area of Science:

• Genomics
• Computational Biology
• Protein Science

Background:

• Assessing variant effect predictor (VEP) performance is challenged by biases from clinical data benchmarking.
• Previous work established deep mutational scanning (DMS) as an independent data source for VEP evaluation.

Purpose of the Study:

• To benchmark 55 variant effect predictors (VEPs) using independent deep mutational scanning (DMS) data for 26 human proteins.
• To evaluate the performance of VEPs and DMS datasets in discriminating between pathogenic and benign missense variants.
• To assess the impact of data circularity and bias on VEP performance.

Main Methods:

• Benchmarking 55 VEPs using DMS data from 26 human proteins, minimizing data circularity.
• Utilizing unsupervised methods (EVE, DeepSequence, ESM-1v) and supervised methods (VARITY).
• Assessing VEP and DMS performance in classifying known pathogenic versus putatively benign missense variants.

Main Results:

• Unsupervised VEPs, particularly the protein language model ESM-1v, and supervised VEPs like VARITY showed strong performance.
• Performance of DMS datasets in variant classification varied significantly.
• A strong correlation was observed between VEP agreement with DMS data and their ability to identify clinically relevant variants.

Conclusions:

• Independent benchmarking using DMS data is crucial for unbiased VEP assessment.
• Protein language models and supervised VEPs are advancing, addressing data circularity concerns.
• DMS data shows utility for validating VEP performance and identifying clinically significant variants.