HPTMT Parallel Operators for High Performance Data Science and Data Engineering

Area of Science:

• Computer Science
• Data Science
• High-Performance Computing

Background:

• Data-intensive applications are prevalent across scientific disciplines, encompassing data engineering, deep learning, and machine learning.
• A lack of standardized data structures and operators hinders interoperability between different implementations.
• Existing big data processing systems often face challenges in efficiently integrating diverse data science and engineering tasks.

Purpose of the Study:

• To introduce and elaborate on the HPTMT architecture, designed to unify data engineering and data science.
• To demonstrate the architecture's capability in linking various aspects of data engineering and data science.
• To showcase an end-to-end application integrating deep learning and data engineering using the proposed architecture.

Main Methods:

• Proposed the HPTMT architecture, defining a set of data structures, operators, and an execution model.
• Developed and illustrated an end-to-end application integrating deep learning and data engineering components.
• Emphasized the use of efficient, compact data structures, specifically Apache Arrow, for high-performance data representation.

Main Results:

• The HPTMT architecture demonstrates superior suitability for high-performance computing environments compared to existing big data systems.
• The system effectively scales sequential computations to distributed environments while maintaining optimal performance.
• Integration of compact data structures like Apache Arrow enhances processing efficiency.

Conclusions:

• The HPTMT architecture provides an efficient and unified framework for data-intensive scientific applications.
• The proposed system facilitates seamless integration of data engineering and data science tasks, improving performance and usability.
• The architecture's emphasis on efficient data structures is crucial for advancing high-performance computing in data science.