Scalable Distributed Hashing for Approximate Nearest Neighbor Search.

Area of Science:

• Computer Science
• Data Science
• Machine Learning

Background:

• Hashing is crucial for large-scale approximate nearest neighbor search due to efficiency and low storage needs.
• Centralized hashing methods are insufficient for distributed big data systems, where data resides across multiple nodes.
• Traditional distributed methods face challenges with scalability and accuracy, often prioritizing distributed solutions over performance gains.

Purpose of the Study:

• To propose a Scalable Distributed Hashing (SDisH) model for efficient and accurate approximate nearest neighbor search in distributed big data environments.
• To enable seamless integration of existing hashing methods into a distributed framework without modification.
• To enhance search accuracy by introducing a global search radius and a multi-iteration voting algorithm.

Main Methods:

• Developed the Scalable Distributed Hashing (SDisH) model, allowing existing hashing techniques to process distributed data.
• Implemented a global search radius as a shared variable across nodes to optimize search results iteratively.
• Introduced a voting algorithm based on multiple iteration results to minimize search errors.

Main Results:

• The SDisH model demonstrated significant accuracy improvements, achieving up to 44.75% gain over the aggregating method.
• Compared to state-of-the-art distributed hashing methods, SDisH showed a 10.23% accuracy increase.
• Theoretical analysis confirmed the model's superiority in communication, computation, and accuracy.

Conclusions:

• The SDisH model offers a scalable and accurate solution for approximate nearest neighbor search in distributed big data.
• It effectively addresses the limitations of centralized and existing distributed hashing approaches.
• The proposed model provides substantial accuracy gains and improved efficiency for distributed data search problems.