Improvement of Kafka Streaming Using Partition and Multi-Threading in Big Data Environment

Area of Science:

• * Computer Science
• * Industrial Engineering

Background:

• * Increasing volumes of sensor data from Programmable Logic Controllers (PLCs) in manufacturing necessitate robust Big Data platforms.
• * Existing data management solutions struggle to efficiently handle large-scale, real-time manufacturing data.
• * The need for secure and scalable data infrastructure in smart manufacturing environments is critical.

Purpose of the Study:

• * To propose and evaluate a Hadoop-based ecosystem tailored for manufacturing Big Data challenges.
• * To integrate Apache Hadoop, HBase, Kafka, and Spark for comprehensive data management.
• * To implement public-key cryptography for secure data transmission within the ecosystem.

Main Methods:

• * Implementation of Apache Hadoop and HBase for large-scale Big Data storage.
• * Utilization of Apache Kafka as a data streaming pipeline with configurations for scalability (e.g., Kafka offset and partition).
• * Integration of Apache Spark for real-time data processing and analysis in conjunction with Kafka consumers.
• * Application of public-key cryptography for securing data transmission between Kafka producers and consumers.

Main Results:

• * The proposed Hadoop ecosystem effectively handles large-scale PLC sensing data from manufacturing environments.
• * Real-time data processing and analysis are achieved through the synergy of Kafka and Spark.
• * Public-key cryptography ensures secure data transmission, protecting sensitive manufacturing data.
• * The integrated system demonstrates enhanced performance in data storing, processing, and security.

Conclusions:

• * The developed Hadoop ecosystem provides a scalable and reliable solution for Big Data in manufacturing.
• * The combination of Big Data technologies and cryptographic methods significantly improves data management capabilities.
• * This approach enhances the overall efficiency, accuracy, and security of manufacturing data operations.