DDS:将数据分析转换集成到基于任务的工作流中
Nihad Mammadli1, Jorge Ejarque1, Javier Alvarez2
1Workflows and Distributed Computing, Barcelona Supercomputing Center, Barcelona, Catalunya, 08034, Spain.
Open research Europe
|August 30, 2023
概括
本研究引入了开发高性能数据分析 (HPDA) 应用程序的新方法,通过将传统的高性能计算 (HPC) 工作流与使用基于任务的编程模型 (TBPM) 的数据分析框架集成. 与Spark.com等现有解决方案相比,开发的方法提供了无集成和更好的性能.
科学领域:
- 计算机科学 计算机科学
- 计算科学 计算科学
背景情况:
- 高性能数据分析 (HPDA) 将高性能计算 (HPC) 与数据分析框架相结合.
- 目前的HPDA研究主要集中在优化HPC技术的数据分析框架上,忽视了应用程序开发集成.
- 从开发人员的角度来看,在无结合HPC工作流程和数据分析算法方面存在一个差距.
研究的目的:
- 通过利用基于任务的编程模型 (TBPMs) 提出开发HPDA应用程序的新方法.
- 在统一的开发方法中实现传统HPC工作流程和数据分析转换的无集成.
- 与现有框架相比,证明拟议方法的有效性和性能优势.
主要方法:
- 开发了一种基于TBPMs的HPDA应用程序开发方法.
- 使用PyCOMPS的基于任务的编程模型实现了一个原型.
- 通过开发和评估HPDA应用程序来验证该方法,并将性能与Spark框架进行比较.
主要成果:
- 可以使用基于TBPM的方法来无开发HPDA应用程序.
- 与Spark框架相比,实现的原型显示了针对特定HPDA任务的优越性能.
- 该研究验证了将数据分析直接集成到HPC应用程序中的可行性.
结论:
- 基于任务的编程模型为统一的HPDA应用程序开发提供了有效的基础.
- 拟议的方法促进了HPC和数据分析的无结合,提供了性能优势.
- 这项工作通过合并HPC和数据分析,为更加综合和高效的科学计算铺平了道路.
相关概念视频
01:15Source Transformation
6.6K
Source transformation is a fundamental technique employed in circuit analysis, offering a valuable tool for simplifying complex electrical circuits. This technique involves the replacement of either a voltage source in series with a resistor by a current source in parallel with a resistor, or vice versa. The key concept here is that when the original sources are deactivated (turned off), the equivalent resistance at the circuit's end terminals remains the same.
It is essential to note that when...
It is essential to note that when...
6.6K
01:17Performing a Simple Data Analysis using MS-Excel Function
312
Microsoft Excel offers a suite of functions and tools ideal for statistical analysis, making it accessible to students and researchers. This article outlines fundamental Excel functions pivotal for data analysis.
SUM: This function calculates the total sum of a range of values. It's the foundation for aggregating data, essential for determining overall trends and totals in datasets.
AVERAGE: It computes the mean value of a given set of numbers, providing a quick insight into the central...
SUM: This function calculates the total sum of a range of values. It's the foundation for aggregating data, essential for determining overall trends and totals in datasets.
AVERAGE: It computes the mean value of a given set of numbers, providing a quick insight into the central...
312
01:13Overview of Microsoft Excel as a Data Analysis Tool
556
Microsoft Excel is a cornerstone tool for data analysis and statistical operations, offering a wide array of functionalities to manage, analyze, and visualize data efficiently. Recognized for its versatility, Excel facilitates the performance of basic to complex statistical operations, serving as an indispensable asset for analysts, researchers, and students alike. Excel's significance in data analysis emanates from its spreadsheet environment, where data can be organized in rows and...
556
01:12Run Charts
80
Run charts serve as an essential instrument for visualizing the performance of various processes over time, enabling the identification of trends and patterns crucial for quality improvement. These charts map out a series of data points chronologically, offering insights into the stability and efficiency of a process. A run chart's creation involves plotting data points on a graph, with the time intervals on the horizontal axis and the specific measurements on the vertical axis. For...
80
01:13Mass Analyzers: Overview
723
The mass analyzer is a crucial component of the mass spectrometer. In the ionization chamber, the vaporized sample is bombarded with a high-energy electron beam to generate a radical cation and further fragment into neutral molecules, radicals, and cations. A series of negatively charged accelerator plates accelerate the cations into the mass analyzer. The mass analyzer separates ions according to their mass-to-charge (m/z) ratios and then directs them to the detector. The common types of mass...
723
02:07Improving Translational Accuracy
2.6K
2.6K