Adaptive optimization for hybrid vector-relational queries via cost-aware plan switching | JoVE Visualize

Area of Science:

Database Systems
Information Retrieval
Machine Learning

Background:

Modern analytics and recommendation systems require efficient querying of combined structured and unstructured data.
Current vector-relational database systems often use fixed strategies, solely relying on Approximate Nearest Neighbor (ANN) indices.
This fixed approach is suboptimal, leading to over-fetching and resource waste when structured predicates are highly selective or vector distributions are sparse.

Purpose of the Study:

To develop a novel system, DYNAMICANN, that overcomes the limitations of fixed execution strategies in vector-relational databases.
To introduce a cost-aware optimizer that dynamically selects the most efficient query execution path.
To improve the performance of hybrid queries involving both vector similarity search and structured data filtering.

Main Methods:

DYNAMICANN employs a cost-aware optimizer that evaluates a runtime cost ratio to compare ANN graph traversal with relational scan.
The system adaptively selects the ANN path if its estimated cost is lower, a scan if the relational scan is cheaper, or prioritizes ANN for recall preservation.
Evaluation was performed on four hybrid query templates using the LAION-400M dataset, measuring Recall@10 accuracy and speedups.

Main Results:

DYNAMICANN achieved measured speedups of 2.1-9.8× over the CHASE system for low-selectivity queries.
The system maintained high accuracy, with Recall@10 values close to brute-force ground truth across all tested query types.
The overhead of adaptive plan switching was minimal, constituting only a small fraction of the total execution time.

Conclusions:

DYNAMICANN's adaptive approach, combining vector search and relational optimization, significantly enhances the efficiency of hybrid queries.
The cost-aware strategy effectively mitigates performance degradation caused by suboptimal ANN index traversal.
This adaptive method offers a more resource-efficient and performant solution for modern data analytics and recommendation systems.