MLink: Linking Black-Box Models From Multiple Domains for Collaborative Inference.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Cost-efficient model inference is crucial for real-world machine learning (ML) applications, particularly for delay-sensitive tasks and resource-constrained devices.
• Deploying multiple ML models for complex services often exceeds budget limitations, such as GPU memory, creating a significant challenge.
• Existing methods struggle to efficiently manage multiple ML models under strict resource budgets.

Purpose of the Study:

• To propose a novel learning task, model linking, to bridge knowledge between different black-box ML models.
• To develop a scheduling algorithm, MLink, that enables collaborative multi-model inference.
• To improve inference accuracy and computational efficiency under budget constraints.

Main Methods:

• Introduced 'model links' to learn mappings between the output spaces of heterogeneous black-box ML models.
• Developed adaptation and aggregation methods to address distribution discrepancies in model links.
• Designed the MLink scheduling algorithm utilizing model links for collaborative inference.

Main Results:

• Demonstrated the effectiveness of model links across various black-box ML models in multi-modal and video analytics datasets.
• MLink achieved significant cost savings, reducing inference computations by 66.7% while preserving 94% of inference accuracy under GPU memory budget.
• MLink outperformed baseline methods including multi-task learning, deep reinforcement learning-based scheduling, and frame filtering.

Conclusions:

• Model linking provides an effective approach to connect diverse black-box ML models.
• The MLink algorithm successfully leverages model links to enhance cost-efficiency and accuracy in multi-model inference scenarios.
• This work offers a practical solution for deploying complex ML applications within resource limitations.