ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS...

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Accurate variant calling is crucial for clinical and research applications of next-generation sequencing (NGS).
• Existing methods for evaluating variant caller performance lack consistency, transparency, and reproducibility.
• Independent, orthogonally generated data are essential for robust assessment.

Purpose of the Study:

• To introduce ICR142 Benchmarker, a novel tool for assessing germline base substitution and indel calling performance.
• To provide a standardized and reproducible method for benchmarking variant callers using the ICR142 NGS validation series.
• To facilitate the optimization and transparent evaluation of various variant calling pipelines.

Main Methods:

• Utilized the ICR142 NGS validation series, comprising exome sequence data from 142 samples and Sanger sequencing data at 704 sites.
• Developed ICR142 Benchmarker to process variant call format (VCF) files and generate performance metrics (sensitivity, specificity, false detection rates).
• Applied the tool to benchmark GATK, OpEx, DeepVariant, and commercial variant calling providers, including regression testing of DeepVariant versions.

Main Results:

• ICR142 Benchmarker successfully generated summary and detailed performance reports, highlighting pipeline-specific differences and common challenges (e.g., indel calling in repetitive regions).
• DeepVariant v0.6.1 demonstrated improved variant calling performance over v0.5.2, with minor changes in indel calling behavior noted.
• Recommended a QUAL threshold of 30 for base substitution calls using DeepVariant v0.6.1 and facilitated optimization for commercial pipelines.

Conclusions:

• ICR142 Benchmarker offers a consistent, transparent, and reproducible approach to analyzing variant calling performance.
• The tool enables users to understand pipeline performance through informative metrics and comparisons with established tools.
• ICR142 Benchmarker is freely available and supports the optimization of both open-source and commercial variant calling pipelines.