Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B...

Area of Science:

• Computer Science
• Data Engineering
• Big Data Analytics

Background:

• Prior studies on Lambda Architecture (LA) primarily focused on hardware performance and scalability.
• The intricate design of LA has historically limited empirical examination of its data-processing accuracy.
• A significant literature gap exists regarding validated methodologies for assessing LA's accuracy.

Purpose of the Study:

• To introduce and validate a novel methodology for assessing data processing accuracy within the Lambda Architecture.
• To provide empirical evidence for theoretical assertions regarding LA's performance.
• To address the need for accuracy-focused evaluations beyond hardware and scalability.

Main Methods:

• Developed a methodology to evaluate prospective technologies across all LA layers.
• Examined layer-specific design limitations and implemented a uniform software development framework.
• Utilized unique metrics including data latency and processing accuracy under various conditions.

Main Results:

• Empirically demonstrated LA's "eventual consistency" across its Speed Layer (SL) and Batch Layer (BL).
• Confirmed that transient inconsistencies in the SL resolve, leading to precise results from the BL.
• Achieved a 100% accuracy rate under various severe data-ingestion scenarios, validating theoretical claims.

Conclusions:

• The novel methodology provides empirical validation for Lambda Architecture's data processing accuracy.
• LA exhibits eventual consistency, ensuring reliable results despite real-time processing fluctuations.
• The study sets a precedent for empirically supported big data architecture evaluations, applicable to other frameworks.