Intrinsic evaluation of text mining tools may not predict performance on realistic tasks

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Automated techniques like biomedical text mining show potential for aiding database curators.
• However, their practical application in tasks like annotating mutations in the Protein Data Bank (PDB) requires thorough evaluation.
• Few studies have assessed the real-world performance of these systems for database curation.

Purpose of the Study:

• To evaluate the performance of automated mutation annotation techniques in the context of PDB database curation.
• To compare intrinsic (gold standard) versus extrinsic (database annotation) evaluation metrics for these systems.
• To assess the accuracy and completeness of existing manually curated mutation data in the PDB.

Main Methods:

• Utilized a text-mining-based approach (MutationFinder) and an alignment-based approach for mutation annotation.
• Conducted intrinsic evaluations using gold standard data.
• Performed extrinsic evaluations simulating database annotation tasks.

Main Results:

• High performance in intrinsic evaluations did not consistently translate to effective performance in extrinsic database annotation.
• Limited access to full-text journal articles significantly impacts the comprehensiveness of text-mining-based annotation.
• Manually curated mutation data in the PDB exhibits imperfections in both accuracy and completeness.

Conclusions:

• Currently, a hybrid approach combining manual and automatic annotation methods is likely the most cost-effective and reliable strategy for PDB database annotation.
• Improving access to full-text scientific literature is crucial for enhancing automated text-mining capabilities.
• Further efforts are needed to improve the quality of manually curated biological databases.