E-ware: a big data system for the incremental discovery of spatio-temporal events from microblogs

Area of Science:

• Computer Science
• Data Science
• Artificial Intelligence

Background:

• Social media generates vast amounts of data, necessitating efficient methods for identifying significant events.
• Existing event detection methods often struggle with the dynamic and high-volume nature of big data streams.

Purpose of the Study:

• To introduce E-ware, a novel big data platform for incremental extraction and clustering of social events.
• To develop and apply unsupervised machine learning and NLP algorithms for pure incremental event discovery.
• To enable real-time tracking and dissemination of spatio-temporal events from social data.

Main Methods:

• Developed a scalable big data platform integrating data stream and geospatial processing.
• Implemented incremental clustering using temporal sliding windows to update event topic clusters.
• Utilized unsupervised machine learning and Natural Language Processing (NLP) algorithms for event discovery.
• Integrated an efficient spatio-temporal index for rapid retrieval and updates of evolving event clusters.

Main Results:

• E-ware demonstrated significant advantages in real-time incremental detection and tracking of events.
• The system effectively handles both spatial and temporal dimensions of event evolution.
• Experimental results on Twitter datasets validated the platform's effectiveness and efficiency.

Conclusions:

• E-ware provides a powerful solution for real-time, incremental event detection from big social data.
• The platform facilitates the development of advanced smart city applications, including proactive emergency management and event-enriched trip planning.
• This work advances the field of spatio-temporal event discovery in dynamic data streams.