An evaluation of multi-probe locality sensitive hashing for computing similarities over web-scale query logs

Area of Science:

• Artificial Intelligence
• Computer Science
• Data Science

Background:

• Modern AI applications require efficient similarity searches in massive datasets.
• The computational cost of nearest neighbor search grows significantly with data scale.
• Existing methods struggle with the demands of large-scale object similarity retrieval.

Purpose of the Study:

• To adapt and optimize Locality Sensitive Hashing (LSH) for large-scale distributed computing environments.
• To evaluate the performance of four LSH variants on Hadoop for near-neighbor search.
• To identify optimizations for enhancing LSH performance in real-world AI applications.

Main Methods:

• Implementation and evaluation of four variants of Locality Sensitive Hashing (LSH).
• Utilized a distributed computing environment, specifically Hadoop, for large-scale data processing.
• Introduced and tested several performance optimizations tailored for very large datasets.

Main Results:

• Optimized LSH variants achieved robust performance in distributed settings.
• Demonstrated superior recall compared to the standard 'vanilla' LSH.
• Performance gains were observed without increasing memory footprint.

Conclusions:

• Optimized LSH variants are effective for large-scale near-neighbor search in AI.
• The proposed optimizations enhance LSH efficiency and recall in distributed systems.
• LSH provides a scalable solution for similarity search in big data applications.