n-Gram-Based Text Compression

Area of Science:

• Natural Language Processing
• Data Compression
• Computational Linguistics

Background:

• Efficient text compression is crucial for managing large datasets in digital communication.
• Existing compression methods may not be optimized for the specific linguistic structures of Vietnamese.
• The development of specialized compression algorithms can significantly improve data handling efficiency.

Purpose of the Study:

• To propose and evaluate an efficient method for compressing Vietnamese text.
• To leverage n-gram dictionaries for enhanced compression ratios.
• To compare the proposed method against state-of-the-art compression techniques.

Main Methods:

• Text is segmented into n-grams (ranging from unigrams to five-grams).
• A sliding window approach (bigram to five-grams) is employed for optimal encoding stream generation.
• N-grams are encoded using byte representations derived from custom-built n-gram dictionaries.

Main Results:

• A 2.5 GB Vietnamese text corpus was used to construct n-gram dictionaries (totaling 12 GB).
• The proposed method achieved an average compression ratio of approximately 90%.
• Experimental results demonstrated superior performance compared to existing state-of-the-art methods.

Conclusions:

• The n-gram dictionary-based compression method is highly effective for Vietnamese text.
• The approach offers significant compression ratios, making it a valuable tool for data storage and transmission.
• Further research could explore adaptive dictionary generation for even greater efficiency.