Duplicates, redundancies and inconsistencies in the primary nucleotide databases: a descriptive study

Area of Science:

• Bioinformatics
• Genomics
• Database Management

Background:

• The International Nucleotide Sequence Database Collaboration (INSDC) comprises major nucleotide sequence databases: GenBank, EMBL, and DDBJ.
• These databases accumulate records over decades, leading to inherent duplicates, redundancies, and inconsistencies.
• Current duplicate detection methods lack comprehensive assessment and consistent assumptions, hindering effective management.

Purpose of the Study:

• To rigorously assess the scale, characteristics, and impact of duplicates within INSDC databases.
• To provide a quantitative analysis of duplicate types at both sequence and annotation levels.
• To demonstrate the practical consequences of data duplication on downstream analyses.

Main Methods:

• Retrospective analysis of merged groups within the INSDC databases.
• Utilized a benchmark dataset of manually identified duplicates (67,888 merged groups, 111,823 duplicate pairs across 21 organisms).
• Categorized duplicates by type (sequence and annotation) and assessed their impact through a case study on GC content and melting temperature.

Main Results:

• Different organisms exhibit varying prevalence of distinct duplicate types.
• Duplicates were categorized at both sequence and annotation levels, with quantitative statistics provided.
• A case study demonstrated that duplicates can lead to inconsistent results in tasks like GC content and melting temperature calculations.

Conclusions:

• The presence of duplicates in biological databases introduces significant redundancy and can compromise the accuracy of analyses.
• Understanding the prevalence and types of duplicates is crucial for improving data quality and reliability.
• This work provides a foundation for developing more effective strategies to identify and mitigate data duplication in large-scale biological sequence repositories.