Multi-Head Encoding for Extreme Label Classification

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Real-world data often involves a massive number of categories and multiple labels per instance.
• eXtreme Label Classification (XLC) addresses this challenge but suffers from Classifier Computational Overload Problem (CCOP) due to increased parameters and operations.
• CCOP hinders the scalability and efficiency of XLC models.

Purpose of the Study:

• To propose a novel mechanism, Multi-Head Encoding (MHE), to alleviate the Classifier Computational Overload Problem in eXtreme Label Classification.
• To develop efficient implementations of MHE tailored for various XLC task characteristics.
• To theoretically and experimentally validate the effectiveness and performance of MHE.

Main Methods:

• Introduced a Multi-Head Encoding (MHE) mechanism, replacing the vanilla classifier with a multi-head architecture.
• MHE decomposes extreme labels into a product of local labels for training, enabling geometric reduction in computational load.
• Proposed three MHE implementations: Multi-Head Product, Multi-Head Cascade, and Multi-Head Sampling, for single-label, multi-label, and pretraining tasks.

Main Results:

• MHE significantly reduces computational load during training and inference in XLC tasks.
• The proposed MHE-based methods achieve state-of-the-art performance across various XLC benchmarks.
• Theoretical analysis demonstrates MHE's performance equivalence to vanilla classifiers via a generalized low-rank approximation.

Conclusions:

• Multi-Head Encoding (MHE) effectively addresses the Classifier Computational Overload Problem in eXtreme Label Classification.
• MHE offers a scalable and efficient solution for handling massive label spaces in machine learning.
• The proposed methods provide state-of-the-art results while streamlining computational processes.