flowSim: Near duplicate detection for flow cytometry data.

Area of Science:

• Computational Biology
• Machine Learning
• Data Science

Background:

• Large datasets are crucial for developing effective machine learning (ML) models.
• Flow cytometry (FCM) generates high-dimensional data, often containing significant redundancy.
• Redundant data increases computational time and can lead to overfitting in ML algorithms.

Purpose of the Study:

• To introduce flowSim, the first algorithm designed for visualizing, detecting, and removing redundant information in FCM datasets.
• To decrease computational training time for ML models.
• To enhance ML algorithm performance by reducing overfitting through data optimization.

Main Methods:

• flowSim employs a combination of community detection algorithms and density analysis of marker expression values for near-duplicate detection.
• The algorithm clusters FCM data to identify and quantify similar patterns.
• Near-duplicate files are selectively discarded from training sets.

Main Results:

• flowSim demonstrated high efficiency in identifying similar patterns, achieving a mean Adjusted Rand Index of 0.90 when compared to manual clustering on a bivariate FCM dataset.
• The algorithm successfully identified and removed near-duplicate files in datasets with known redundancy.
• In a large-scale test, flowSim removed 92.6% of FCM images from a dataset exceeding 500,000 entries sourced from public repositories.

Conclusions:

• flowSim is an effective tool for optimizing flow cytometry data by removing redundancy.
• The algorithm significantly reduces dataset size, leading to decreased computational costs and faster ML model training.
• By mitigating overfitting, flowSim contributes to the development of more accurate and robust ML models in FCM analysis.