SPSC: Stream Processing Framework Atop Serverless Computing for Industrial Big Data

Area of Science:

• Computer Science
• Data Engineering
• Distributed Systems

Background:

• The increasing volume of data from smart manufacturing and information technologies necessitates efficient processing.
• Existing distributed stream processing systems (e.g., Apache Flink, Spark) face challenges with resource underutilization and high latency.
• Big data applications require scalable and performant solutions.

Purpose of the Study:

• To propose SPSC, a serverless-based stream computing framework designed to address the limitations of current big data processing systems.
• To achieve efficient task and data parallelism in stream processing.
• To evaluate the performance of the SPSC framework.

Main Methods:

• Developed SPSC, a serverless framework utilizing atomic streams and stateless Lambda functions as operators.
• Implemented a prototype on Amazon Web Services (AWS) using AWS Lambda, AWS Simple Queue Service (SQS), and AWS DynamoDB.
• Discretized events into atomic streams and employed context-irrelevant Lambda functions for parallel processing.

Main Results:

• The SPSC framework demonstrated superior performance compared to Alibaba's real-time computing Flink version.
• SPSC achieved a 10.12% performance improvement with comparable overhead.
• The serverless approach facilitated inherent data parallelism and task parallelism.

Conclusions:

• SPSC offers an efficient and performant alternative for big data stream processing in smart manufacturing and related fields.
• Serverless architectures can effectively address challenges of resource underutilization and latency in big data scenarios.
• The proposed framework shows significant potential for real-time data processing applications.