Real-Time Exploration of Large Spatiotemporal Datasets Based on Order Statistics

Area of Science:

• Data Science
• Information Visualization
• Database Systems

Background:

• Traditional datacube structures aggregate data, potentially obscuring true distributions and leading to biased analyses.
• Interactive visual exploration of large datasets is crucial but often hindered by aggregation limitations.

Purpose of the Study:

• To introduce the Quantile Datacube Structure (QDS) for accurate, interactive visual exploration of large datasets.
• To address the limitations of standard datacubes by incorporating order statistics and distribution approximations.

Main Methods:

• Utilizing an efficient non-parametric distribution approximation scheme (p-digest).
• Employing a novel datacube indexing scheme to reduce memory footprint.
• Implementing order statistics for data distribution analysis.

Main Results:

• QDS enables interactive visualization based on order statistics, accurately approximating data distributions.
• Demonstrated effectiveness in event detection on very large datasets through case studies.
• Experimental validation confirms QDS's efficiency in memory usage and accuracy.

Conclusions:

• QDS provides a robust solution for interactive visual exploration of large datasets by accurately representing data distributions.
• The proposed methods improve upon existing datacube approaches by mitigating bias and enhancing feature discovery.
• QDS facilitates advanced analytics, including order statistics visualization and event detection, on massive datasets.