Co-occurrence word model for news media hotspot mining-text mining method design

Area of Science:

• Natural Language Processing
• Data Mining
• Computational Linguistics

Background:

• The proliferation of online media necessitates advanced methods for identifying trending topics.
• Traditional hotspot mining algorithms lack the precision and speed required for contemporary news analysis.
• Existing methods often struggle with accuracy and timeliness in capturing emerging news trends.

Purpose of the Study:

• To address the limitations of current news media hotspot mining techniques.
• To propose an improved hotspot mining method for enhanced accuracy and timeliness.
• To leverage computational frameworks for efficient data processing.

Main Methods:

• Development of a novel co-occurrence word model incorporating word weights.
• Design of a hotspot mining algorithm using the co-occurrence model and improved smooth inverse frequency rank (SIFRANK).
• Integration of the Spark computing framework to optimize processing speed.

Main Results:

• The proposed algorithm identified a substantial number of new words across datasets (e.g., 16,871 in Weibo Short News).
• High heat weight values for keywords were achieved, indicating effective topic identification (e.g., up to 0.9991).
• Demonstrated strong performance metrics in accuracy, recall, and F1 score on benchmark datasets (e.g., Covid19 Tweets and President-elects Tweets).
• Significant improvements in running speed were observed after implementing the Spark framework.

Conclusions:

• The co-occurrence word model-based method enhances accuracy and efficiency in news media hotspot mining.
• The approach offers practical significance for real-time topic identification and analysis.
• The integration of Spark computing provides a scalable solution for large-scale text data processing.