Over- and Under-sampling Approach for Extremely Imbalanced and Small Minority Data Problem in Health Record Analysis

Area of Science:

• Health Informatics
• Machine Learning
• Data Science

Background:

• Digitalization of medical systems generates vast health record (HR) data.
• Analyzing HR data is challenging due to imbalanced datasets, where target disease populations are small.
• Existing over-sampling and under-sampling methods fail with extremely imbalanced and small minority (EISM) data.

Purpose of the Study:

• To propose a novel algorithm, HUSDOS-Boost, for solving the EISM data problem in HR data analysis.
• To develop a method that effectively handles datasets with a very small number of positive cases.
• To improve the accuracy of detecting rare diseases within large health record datasets.

Main Methods:

• Developed HUSDOS-Boost, an algorithm combining boosting with heuristic under-sampling and distribution-based sampling.
• HUSDOS-Boost uses under-sampling to remove redundant majority examples and over-sampling to generate synthetic minority examples.
• The algorithm was tested on eight imbalanced datasets and applied to clinical HR data for stomach cancer detection.

Main Results:

• HUSDOS-Boost demonstrated superior performance compared to current imbalanced data handling methods.
• The algorithm was particularly effective on extremely imbalanced and small minority (EISM) datasets.
• Successful application in detecting stomach cancer patients from original clinical HR data.

Conclusions:

• HUSDOS-Boost is a valuable and effective methodology for analyzing imbalanced health record data, especially EISM datasets.
• The proposed algorithm offers a significant improvement for identifying rare conditions in large-scale health data.
• HUSDOS-Boost enhances the utility of digitalization in medical systems for clinical applications.