Heterogeneous Multi-Task Learning With Expert Diversity.

Area of Science:

• Computational biology
• Machine learning
• Bioinformatics

Background:

• Predicting multiple biological and medical targets is difficult for traditional deep learning.
• Multi-task learning (MTL) trains a single model for multiple related targets, unlike single-task learning.
• Heterogeneous MTL, with tasks of varying characteristics, poses significant challenges for existing approaches.

Purpose of the Study:

• To propose a novel deep learning method, Multi-gate Mixture-of-Experts with Exclusivity (MMoEEx), to address heterogeneous multi-task learning.
• To improve the balancing of shared and task-specific representations in MTL.
• To optimize models for tasks with competing optimization paths.

Main Methods:

• Developed the Multi-gate Mixture-of-Experts with Exclusivity (MMoEEx) model.
• Introduced an approach to increase expert diversity for improved representation learning in imbalanced and heterogeneous MTL.
• Implemented a two-step optimization strategy to balance tasks at the gradient level.

Main Results:

• MMoEEx demonstrates effectiveness in handling heterogeneous MTL settings.
• The method successfully balances competing optimization paths and improves representation learning.
• Validation was performed on diverse benchmark datasets: UCI-Census-income, MIMIC-III, and PCBA.

Conclusions:

• MMoEEx offers a promising solution for complex biological and medical target prediction using heterogeneous MTL.
• The proposed method enhances model performance by inducing expert diversity and employing gradient-level task balancing.
• MMoEEx advances the capabilities of deep learning in multi-target prediction scenarios.