Evaluating the integration of Esper complex event processing engine and message brokers

Area of Science:

• Computer Science
• Data Engineering
• Software Architecture

Background:

• The proliferation of IoT and smart environments generates vast data, demanding low-latency processing systems.
• Existing Complex Event Processing (CEP) solutions often lack performance evaluations under varying loads, hindering new deployments.
• Scalability and fault tolerance in high-throughput data processing remain critical research areas.

Purpose of the Study:

• To evaluate the performance of Esper CEP engine versions integrated with RabbitMQ and Apache Kafka.
• To benchmark different software architectures for real-time data processing under increasing data rates.
• To provide developers with data-driven insights for selecting appropriate CEP and messaging broker solutions.

Main Methods:

• Defined a benchmark using Esper CEP event patterns and representative operators.
• Tested three architectures: Esper/RabbitMQ, Esper Enterprise/Kafka (single instance), and Esper Enterprise/Kafka (distributed).
• Measured CPU, RAM, latency, and throughput at increasing event input rates to identify system overload points.

Main Results:

• All architectures exhibited low CPU consumption across tested data rates.
• Memory usage was balanced and similar up to 10,000-15,000 events/second.
• Efficient response times were observed up to 10,000-15,000 events/second, varying by architecture and pattern.

Conclusions:

• Esper CEP offers efficient real-time data processing solutions with varying strengths in integration, scalability, and fault tolerance.
• The choice of CEP engine and messaging broker depends on specific system requirements.
• Performance benchmarks provide crucial data for informed system design decisions in IoT and smart environments.