Building a challenging medical dataset for comparative evaluation of classifier capabilities

Area of Science:

• Computational biology
• Medical informatics
• Natural Language Processing

Background:

• Digitalization generates vast unstructured text data, including medical literature.
• Classifying documents within a single domain, like medicine, is challenging due to contextual similarities.

Purpose of the Study:

• To develop and compare machine learning and deep learning models for classifying medical articles on four common cancer types.
• To address the complexity of single-domain document classification using a large dataset.

Main Methods:

• Utilized 383,914 medical articles from PubMed API, covering Leukemia, Non-Hodgkin Lymphoma, Bladder Cancer, and Thyroid Cancer.
• Implemented and evaluated machine learning (Logistic Regression, XGBoost, CatBoost, Random Forest) and deep learning (CNN, LSTM, GRU) models.
• Employed a 70/20/10 data split for training, testing, and validation, with performance assessed via average precision, recall, and F-score over ten splits.

Main Results:

• Deep learning models achieved a top F1 score of 98% for classifying cancer-related medical articles.
• Traditional machine learning models also demonstrated strong performance, with the lowest F1 score reaching 95%.

Conclusions:

• Multiple machine learning and deep learning models effectively classify medical articles within the challenging single-domain context of common cancers.
• The study provides robust computational tools for organizing and analyzing large volumes of medical literature.