Comprehensive benchmark and architectural analysis of deep learning models for nanopore sequencing basecalling

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Nanopore DNA sequencing accuracy depends on basecalling electric current signals using neural networks.
• Current benchmarking practices lack standardization in datasets and metrics, hindering progress and model comparison.
• Distinguishing data-driven from model-driven improvements in basecalling is challenging due to inconsistent evaluations.

Purpose of the Study:

• To establish a standardized benchmarking process for nanopore DNA sequencing basecalling models.
• To rigorously evaluate and compare the performance of leading basecaller models.
• To identify key architectural features driving high-performing basecalling models.

Main Methods:

• Unified existing benchmarking datasets and defined a rigorous set of evaluation metrics.
• Recreated and analyzed the neural network architectures of seven state-of-the-art basecaller models.
• Evaluated 90 novel architectures to identify performance drivers and error reduction strategies.

Main Results:

• Bonito's architecture demonstrated superior performance for basecalling among the evaluated models.
• Species bias in training data significantly impacts basecalling model performance.
• Recurrent neural networks (long short-term memory) and conditional random field decoders are critical for high-performing models.

Conclusions:

• The developed standardized benchmarking framework facilitates the evaluation of new basecalling tools.
• This work provides insights into architectural components that enhance nanopore sequencing accuracy.
• The findings encourage further community-driven expansion and refinement of basecalling benchmarking.