Cost sensitive hierarchical document classification to triage PubMed abstracts for manual curation

Area of Science:

• Bioinformatics
• Computational Biology
• Immunoinformatics

Background:

• The Immune Epitope Database (IEDB) project manually curates scientific literature on immune epitopes.
• Previously, abstracts were classified using a Naïve Bayes classifier, with curatable abstracts further categorized by disease domain.
• Improvements were sought to enhance classification performance and minimize manual effort.

Purpose of the Study:

• To improve the classification performance of scientific abstracts for the IEDB.
• To reduce the manual workload involved in abstract categorization.
• To develop and evaluate advanced machine learning models for automated classification.

Main Methods:

• A Support Vector Machine (SVM) classifier was compared against the existing Naïve Bayes classifier for predicting abstract curatability.
• Hierarchical SVM classifiers were developed and compared to non-hierarchical approaches for disease domain categorization.
• Cost-sensitive functions were implemented to optimize the SVM classifiers' error profiles for the curation process.

Main Results:

• The SVM classifier achieved higher curatability prediction accuracy (AUC 0.899) than Naïve Bayes (AUC 0.854).
• Hierarchical SVM classifiers demonstrated superior performance in categorizing abstracts into disease domains compared to non-hierarchical SVMs.
• The optimized hierarchical SVM system achieved high prediction accuracies: 94.4% (level 1), 93.9% (level 2), and 82.1% (level 3).

Conclusions:

• Hierarchical SVM algorithms with cost-sensitive weighting enable high-quality, accurate reference classification.
• The developed system significantly reduces the manual effort required for abstract categorization.
• The findings and datasets are valuable for other databases developing document classification systems.