Federated Learning on Clinical Benchmark Data: Performance Assessment | JoVE Visualize

Area of Science:

Machine Learning
Artificial Intelligence
Data Science

Background:

Federated learning (FL) is an emerging machine learning technique utilizing decentralized datasets.
FL enhances data privacy by eliminating the need for data transfer during the learning process.
Active research explores FL applications across various domains due to its privacy benefits.

Purpose of the Study:

To assess the reliability and performance of federated learning (FL).
To evaluate FL using three distinct benchmark datasets, including a clinical dataset.
To test FL in realistic scenarios with varied data distributions.

Main Methods:

Implemented FL using a Python-based client-server architecture deployed on Amazon Web Services.
Evaluated FL performance on Modified National Institute of Standards and Technology (MNIST), Medical Information Mart for Intensive Care-III (MIMIC-III), and electrocardiogram (ECG) datasets.
Conducted experiments with basic, imbalanced, skewed, and combined imbalanced/skewed data distributions across datasets.

Main Results:

FL achieved high performance on MNIST, with AUROC of 0.997 and F1-score of 0.946 in basic settings, and maintained strong results (AUROC 0.990, F1 0.891) in combined imbalanced/skewed scenarios.
For MIMIC-III in-hospital mortality prediction, FL achieved an AUROC of 0.850 and F1-score of 0.944 (basic) and 0.943 (imbalanced).
FL demonstrated robust performance on ECG classification, with AUROC of 0.938 (basic) and 0.943 (imbalanced), and F1-scores of 0.807 for both.

Conclusions:

Federated learning exhibits comparable and reliable performance across diverse benchmark datasets, including those with imbalanced and skewed distributions.
FL's ability to maintain high performance while preserving data privacy makes it suitable for real-world applications with heterogeneous data sources, such as multi-hospital collaborations.
The study confirms FL's effectiveness in scenarios mimicking real-world data variations, highlighting its potential for privacy-preserving machine learning.