An efficient approach for low latency processing in stream data.

Area of Science:

• Big Data Analytics
• Real-Time Systems
• Stream Processing

Background:

• Continuous data generation from diverse sources requires efficient processing beyond traditional batch analytics.
• Real-time applications like stock market analysis, patient monitoring, and traffic management demand low-latency data processing.
• Dependent data streams from distributed environments introduce latency challenges in accurate analytics.

Purpose of the Study:

• To design and implement a stream processing model capable of handling data latency from distributed sources.
• To achieve end-to-end low latency in processing continuously arriving data.
• To enhance the accuracy of real-time analytics by incorporating dependent parameters.

Main Methods:

• Developed a stream processing model to manage varying data arrival latencies.
• Utilized a statistical approach to forecast potential data latency from distributed sources.
• Implemented preprocessing for at-least-once delivery and exactly-once processing semantics.
• Conducted stock market prediction using dependent parameters (USD, OIL, Gold prices) with equal arrival rates.

Main Results:

• The proposed model effectively handles latency, providing an end-to-end low-latency system.
• Incorporating affecting parameters significantly improved stock market prediction accuracy, as measured by Normalized Root Mean Square Error (NRMSE).
• System latency was reduced when the window size was synchronized with the data arrival rate.

Conclusions:

• The designed stream processing model offers a robust solution for real-time analytics with distributed data sources.
• Considering dependent parameters is crucial for accurate and meaningful stream data analytics.
• Optimizing window size relative to data arrival rate is key to minimizing system latency in stream processing.