Efficient Management of High-Frequency Sensor Data Streams Using a Read-Optimized Learned Index

Area of Science:

• Computer Science
• Data Management
• Database Systems

Background:

• The Internet of Things (IoT) and Digital Twins generate vast amounts of sensor data, demanding efficient spatial indexing.
• Traditional spatial indexes (e.g., R-trees) have high storage overhead.
• Learned indexes like GLIN face a 'Refinement Bottleneck' due to coarse Minimum Bounding Rectangle (MBR) filtering and struggle with dynamic workloads.

Purpose of the Study:

• To develop a dynamic, read-optimized learned spatial index for high-frequency sensor streams.
• To address the limitations of existing spatial indexes in handling concurrent reads and writes in dynamic IoT environments.
• To improve both query accuracy and update throughput.

Main Methods:

• Proposed DyGLIN (Dynamic Generate Learning-Based Index) with a decoupled leaf architecture for separate query processing and data maintenance.
• Implemented a hierarchical filtering pipeline using hierarchical MBRs (HMBR) and Cuckoo Filters for aggressive false positive pruning.
• Utilized a Delta Buffer mechanism for amortizing update costs and logical deletion for high throughput.

Main Results:

• DyGLIN reduced query latency by 26.4% compared to GLIN.
• Achieved 30.0% higher insertion throughput than existing methods.
• Demonstrated superior deletion performance with only an 18.5% increase in memory overhead.

Conclusions:

• DyGLIN offers a significant performance improvement for spatial indexing in dynamic IoT environments.
• The proposed architecture and filtering mechanisms effectively overcome the limitations of previous learned indexes.
• DyGLIN provides a robust solution for high-frequency sensor data streams, balancing query performance and update efficiency.