Foundational models and federated learning: survey, taxonomy, challenges and practical insights

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Federated learning (FL) allows collaborative model training without data sharing.
• Foundational models (FMs) are increasingly used, necessitating expanded training resources and private data integration.
• Integrating FL and FMs is crucial for leveraging distributed, private data.

Purpose of the Study:

• To survey and categorize technical methods integrating federated learning and foundational models.
• To provide a structured overview and technical comparison of existing methods.
• To offer practical guidelines for implementation, focusing on the healthcare domain.

Main Methods:

• Literature survey of over 4,200 articles, narrowed to 250+ reviewed papers.
• Development of a novel taxonomy based on the model development life-cycle.
• Categorization and technical comparison of 42 unique integration methods.

Main Results:

• Identification and categorization of methods for integrating federated learning with foundational models.
• Technical comparison of methods based on complexity, efficiency, and scalability.
• A structured overview of the state-of-the-art and practical implementation insights.

Conclusions:

• Federated learning and foundational models offer significant potential, particularly in data-sensitive domains like healthcare.
• The developed taxonomy provides a framework for understanding and selecting integration methods.
• This survey serves as a comprehensive resource for researchers and practitioners.