Distributed Complementary Binary Quantization for Joint Hash Table Learning.

Area of Science:

• Computer Science
• Data Science
• Machine Learning

Background:

• Multiple hash tables are effective for large-scale data indexing, ensuring search accuracy and efficiency.
• Existing multi-table indexing methods often suffer from redundancy due to uninformative hash codes and poor table complementarity.

Purpose of the Study:

• To propose a novel method for jointly learning multiple hash tables and informative hash functions.
• To address the table redundancy issue in existing multi-table indexing solutions.

Main Methods:

• A complementary binary quantization (CBQ) method is proposed for centralized learning of hash tables and functions.
• A distributed learning algorithm (D-CBQ) is designed to accelerate training on large-scale distributed datasets.
• The method utilizes prototype-based incomplete binary coding and multi-index search to align data distributions and reduce quantization loss.

Main Results:

• The (D-)CBQ method demonstrates extensibility for generating long hash codes and scalability with linear training time.
• Experiments on Euclidean and semantic nearest neighbor search tasks show efficient computation and informative binary quantization.
• The proposed method significantly outperforms state-of-the-art techniques, achieving up to 57.76% performance gains.

Conclusions:

• The (D-)CBQ method effectively reduces table redundancy and improves search accuracy and efficiency.
• It offers a scalable and efficient solution for large-scale data indexing.
• The approach shows strong table complementarity and informative binary quantization capabilities.