An adaptive spark-based framework for querying large-scale NoSQL and relational databases.

Area of Science:

• Computer Science
• Data Management
• Database Systems

Background:

• Big data and cloud computing necessitate new data management standards.
• Programmers face challenges querying heterogeneous SQL and NoSQL data stores.
• Existing solutions for complex multi-store queries are often inefficient or complex.

Purpose of the Study:

• To propose an automated, fast, and unified architecture (CQNS) for managing simple and complex queries across heterogeneous data stores.
• To enhance developer efficiency in big data applications and cloud environments.

Main Methods:

• Developed a three-layer architecture: matching selector, processing, and query execution.
• Implemented a matching algorithm to direct queries to appropriate SQL or NoSQL engines.
• Utilized a Spark framework to handle diverse databases including MongoDB, Cassandra, and Neo4j.
• Evaluated performance using four benchmark datasets across various NoSQL databases.

Main Results:

• The CQNS architecture demonstrated superior performance in terms of latency and throughput.
• Achieved optimal query execution times compared to existing systems.
• Successfully queried multiple NoSQL databases including MongoDB, Cassandra, Riak, CouchDB, and Neo4j.

Conclusions:

• CQNS provides an effective solution for unified querying across heterogeneous SQL and NoSQL databases.
• The framework offers significant improvements in performance and efficiency for big data management.
• CQNS is a valuable tool for developers working with complex, multi-database environments.