SparkGIS: Resource Aware Efficient In-Memory Spatial Query Processing.

Area of Science:

• Computer Science
• Data Engineering
• Geographic Information Systems

Background:

• Distributed spatial computing systems, particularly MapReduce frameworks, face challenges with large spatial datasets.
• In-memory systems like Apache Spark offer performance gains but require substantial memory and configuration, risking fallback to disk IO or job failure.
• Existing in-memory systems lack awareness of spatial data characteristics, hindering efficient spatial processing.

Purpose of the Study:

• To introduce SparkGIS, an in-memory spatial data querying system designed for high throughput and low latency.
• To adapt Apache Spark's capabilities for efficient handling of spatial queries.
• To address performance limitations of current distributed spatial processing systems.

Main Methods:

• Developed SparkGIS, an in-memory oriented system leveraging Apache Spark.
• Implemented support for basic spatial queries (containment, spatial join, k-nearest neighbor) and complex query pipelines.
• Integrated dynamic partitioning algorithms to mitigate processing skew.
• Utilized multilevel global and local, pre-generated and on-demand in-memory indexes for data pruning.
• Employed dynamic query rewriting to manage large spatial query workflows.

Main Results:

• SparkGIS demonstrates performance on par with contemporary Spark-based platforms for smaller queries.
• Outperforms existing platforms for larger, memory-intensive spatial data workflows.
• Efficiently handles spatial data through dynamic rewriting and optimized indexing.

Conclusions:

• SparkGIS offers an effective solution for high-performance spatial querying in distributed environments.
• The system's dynamic partitioning, indexing, and query rewriting strategies enhance scalability and efficiency.
• Provides a robust alternative for handling large-scale spatial data processing with limited resources.