Set-theory based benchmarking of three different variant callers for targeted sequencing

Area of Science:

• Genomics
• Bioinformatics
• Medical Genetics

Background:

• Next-generation sequencing (NGS) advances hereditary disease research.
• Evaluating bioinformatics pipelines for NGS data is complex, necessitating robust analysis strategies for clinical applications.
• Inherited Cardiac Conditions (ICCs) are a significant cause of morbidity and mortality.

Purpose of the Study:

• To implement and evaluate a user-friendly, set-theory based method for assessing variant caller performance.
• To benchmark three variant calling pipelines (Isaac, Freebayes, VarScan) using gold standard data for Inherited Cardiac Conditions.
• To ensure reliable variant identification for clinical decision-making.

Main Methods:

• Utilized a set-theory approach for performance metrics, adhering to Global Alliance for Genomics and Health (GA4GH) benchmarking standards.
• Employed TruSight Cardio kit sequencing data from the NA12878 reference genome.
• Compared variant callers against a gold standard variant set and high-confidence regions.

Main Results:

• All three pipelines demonstrated perfect recall (1.000).
• High precision values were observed: Isaac (1.000), Freebayes (0.987), and VarScan (0.928).
• Receiver Operating Characteristic (ROC) curve analysis showed Area Under the Curve (AUC) > 0.94 for all pipelines, indicating significant differences in performance.

Conclusions:

• The set-theory method successfully identified ICC-related variants, confirming pipeline capabilities.
• Pipeline performance varied based on underlying algorithms, highlighting the need for algorithm-specific assessments.
• Emphasized the critical importance of using gold standard materials for validating variant callers in clinical settings.