Efficient and Flexible Hierarchical Data Layouts for a Unified Encoding of Scalar Field Precision and Resolution

Area of Science:

• Data Science
• Scientific Computing
• Information Retrieval

Background:

• Growing scientific data sizes pose significant challenges for data movement and analysis.
• Efficiently managing and querying large datasets is crucial for scientific discovery.

Purpose of the Study:

• To introduce a novel encoding for scalar fields that unifies resolution and precision.
• To develop a flexible data hierarchy for efficient approximate queries.
• To demonstrate the effectiveness of this approach on large, real-world datasets.

Main Methods:

• Introduced a unified tree encoding for scalar fields, enabling sensible approximations via valid cuts.
• Developed a parameterized family of data hierarchies for flexible data representation.
• Empirically evaluated the system-level performance on datasets up to one terabyte.

Main Results:

• The proposed encoding facilitates approximate queries with minimal data movement and time.
• Specific parameter choices in the hierarchy yield known data representation schemes (zfp, idx, jpeg2000).
• The new strategy is competitive in data quality, significantly more flexible, faster, and resource-efficient than state-of-the-art compression.

Conclusions:

• The unified tree encoding offers a flexible and efficient solution for managing large scientific data.
• This approach significantly reduces data movement and accelerates analysis for approximate queries.
• The method provides a competitive alternative to existing compression techniques with enhanced performance and flexibility.