An Earthquake Emergency Web Data Cleaning and Classification Method Based on Word Frequency and Position Weighting

Area of Science:

• Computer Science
• Information Science
• Emergency Management

Background:

• Effective emergency rescue decision-making relies on rapid processing of earthquake emergency web documents.
• Data classification is a critical bottleneck in the data cleaning process, directly impacting overall speed.

Purpose of the Study:

• To enhance the efficiency of earthquake emergency web document data cleaning.
• To develop an AI-based framework for robust data cleaning and fusion of emergency information.

Main Methods:

• Proposed a weighted frequency algorithm (P-TF-IDF) by enhancing TF-IDF with improved word frequency and location factors.
• Utilized N-gram feature word vectors to optimize the FastText model for efficient web document classification.
• Designed an AI-driven data cleaning framework incorporating recognition, classification, and repair rules for invalid data detection and conflict resolution.

Main Results:

• The P-TF-IDF algorithm and N-gram features significantly improved web document data classification efficiency.
• The AI framework successfully detected invalid data, resolved conflicts, and generated a complete, de-duplicated dataset.
• Achieved faster data cleaning, enabling quicker fusion of earthquake emergency network information.

Conclusions:

• The developed methods and framework substantially accelerate earthquake emergency web data cleaning.
• This enhanced data processing provides a solid foundation for effective emergency response and data visualization.