NCBench: providing an open, reproducible, transparent, adaptable, and continuous benchmark approach for...

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Accurate small variant calling is crucial for human genomics research.
• Existing benchmarking methods are often paper-specific and not continuously updated.
• The German Research Foundation (DFG) funded initiative aimed to establish a standardized benchmarking process.

Purpose of the Study:

• To develop and present NCBench, a continuous benchmarking platform for human genomic small variant callsets.
• To evaluate the performance of various variant calling pipelines using NCBench.
• To provide an open-source, continuously updated resource for variant callset evaluation.

Main Methods:

• Development of NCBench, an open-source, continuously re-evaluated benchmarking platform.
• Utilizing public infrastructure (Github, Github Actions, Zenodo) and established open-source tools.
• Evaluation of over 40 callsets from different pipelines across three exome datasets with varying coverages.

Main Results:

• NCBench successfully evaluates arbitrary callsets, reporting results visually and interactively.
• All evaluated pipelines demonstrated high overall quality.
• Subtle, systematic differences between variant callers and datasets were identified and highlighted by NCBench.

Conclusions:

• NCBench provides a robust, continuously updated, and open-source solution for benchmarking human genomic small variant callsets.
• The platform aids in improving existing variant calling pipelines and developing new workflows.
• NCBench facilitates reproducible benchmarking, saving authors effort and providing readers with up-to-date performance insights.