The gene normalization task in BioCreative III.

Area of Science:

• Bioinformatics
• Computational Biology
• Natural Language Processing

Background:

• The Gene Normalization (GN) challenge in BioCreative III focused on identifying gene identifiers in full-text articles.
• High costs of manual annotation necessitated alternative methods for evaluating large test sets.

Purpose of the Study:

• To develop and validate an Expectation Maximization (EM) algorithm for inferring ground truth in gene normalization tasks.
• To evaluate team performance in gene detection using both gold-standard and inferred ground truth.
• To assess the effectiveness of the inferred ground truth for differentiating team performance.

Main Methods:

• Utilized 32 fully and 500 partially annotated articles for training, and 507 articles for testing.
• Developed an Expectation Maximization (EM) algorithm to select articles for manual annotation and infer ground truth from team submissions.
• Introduced Threshold Average Precision (TAP-k) as a novel metric for evaluating team performance.

Main Results:

• 14 teams submitted 37 runs for the GN task.
• The highest TAP-k scores on gold-standard annotations were 0.3297 (k=5), 0.3538 (k=10), and 0.3535 (k=20).
• Inferred ground truth yielded higher TAP-k scores (0.4916 for k=5, 10, 20).
• A composite system combining team results achieved significant improvements over the best individual team performance.

Conclusions:

• The GN task in BioCreative III advanced gene normalization closer to real-world literature curation.
• The EM algorithm effectively differentiated team performance while managing annotation costs.
• Inferred ground truth proved as effective as gold-standard annotations for evaluating team performance in gene normalization.