A comprehensive social media data processing and analytics architecture by using big data platforms: a case study of...

Area of Science:

• Computer Science
• Data Science
• Environmental Science

Background:

• Social media platforms generate vast amounts of real-time data.
• Analyzing social media streams presents significant big data challenges.
• Effective data processing architectures are crucial for extracting meaningful insights from social media.

Purpose of the Study:

• To propose an advanced architecture using Apache Hadoop and Apache Spark for social media data.
• To apply this architecture for collecting, storing, processing, and analyzing intensive Twitter data on flood topics.
• To evaluate the potential of Twitter messages for describing flood alert situations and extracting spatial information.

Main Methods:

• Development of a data workflow leveraging Apache Hadoop and Apache Spark.
• Implementation of a predictive machine learning model using Bayes Theorem for classifying flood-related tweets.
• Processing and analysis of approximately 100,000 Twitter messages collected over five days.

Main Results:

• The proposed architecture effectively handles the collection, storage, processing, and analysis of social media data streams.
• Results confirm the advantages of Spark and Hadoop for social media data processing, while noting existing challenges.
• Flood tweet analysis indicates that Twitter messages, with careful consideration, are informative for estimating regional flood alert situations.

Conclusions:

• Apache Hadoop and Spark provide a robust framework for analyzing social media data streams.
• Twitter messages contain valuable flood-related spatial information and can be used to assess flood alert situations.
• Further research is needed to address the challenges in real-time social media data analysis for disaster management.