Rule-based deduplication of article records from bibliographic databases.

Area of Science:

• Biomedical Informatics
• Information Science
• Evidence-Based Medicine

Background:

• Metasearch engines aggregate results from multiple biomedical databases, necessitating duplicate record removal.
• Inconsistent data entry and varying indexing practices across databases complicate deduplication.
• Systematic reviews require high recall, making accurate and efficient deduplication crucial.

Purpose of the Study:

• To describe a rule-based method for deduplicating article records retrieved by the Metta metasearch engine.
• To develop an open-source script module for real-time, online deduplication.
• To ensure high recall while minimizing erroneous duplicate assignments for systematic review users.

Main Methods:

• Developed Metta, a metasearch engine querying PubMed, EMBASE, CINAHL, PsycINFO, and Cochrane Central.
• Implemented a rule-based deduplication module comparing records based on publication year, PubMed ID, DOI, journal, title, and authors.
• Utilized text approximation techniques for robust record comparison.

Main Results:

• The deduplication module demonstrated higher effectiveness in identifying duplicates compared to EndNote.
• No erroneous assignments (i.e., distinct articles marked as duplicates) were reported in user reviews.
• The system performs deduplication online in real-time to meet systematic review needs.

Conclusions:

• The developed rule-based deduplication method is effective for biomedical metasearch engines like Metta.
• The open-source module facilitates improved data management for systematic reviews.
• This approach balances high recall with accurate duplicate identification in complex biomedical literature searches.