Best practices for benchmarking germline small-variant calls in human genomes

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Accurate variant calling from sequence data is crucial for genomic analysis.
• Existing variant-calling tools and performance metrics require standardization for reliable benchmarking.
• Challenges persist in assessing variant-calling accuracy across diverse genomic contexts.

Purpose of the Study:

• To present a standardized benchmarking framework for evaluating variant-calling accuracy.
• To provide guidance on matching variant calls, defining performance metrics, and stratifying results.
• To facilitate the identification of superior variant-calling methods.

Main Methods:

• Development of a benchmarking framework by the Global Alliance for Genomics and Health (GA4GH).
• Guidance on variant call matching, standard performance metric definition, and stratification by variant type and genome context.
• Utilizing a web-based application for comparing variant calls against truth sets.

Main Results:

• Demonstrated significant differences in concordance rates inside and outside high-confidence regions (e.g., 99.7% vs. 76.5% for single-nucleotide variants).
• Piloted framework in PrecisionFDA variant-calling challenges to identify best-in-class methods.
• Highlighted limitations of high-confidence regions for variant calling evaluation.

Conclusions:

• The proposed framework offers standardized performance evaluation for variant calling.
• The framework and associated tools enable reproducible and comparable assessment of variant-calling accuracy.
• Recommendations for best practices in tool usage and result evaluation are provided.