A large-scale dataset for Chinese historical document recognition and analysis

Area of Science:

• Digital Humanities
• Computer Vision
• Historical Linguistics

Background:

• Chinese civilization boasts a rich history documented in numerous historical texts.
• Automated recognition and analysis of these documents are crucial for understanding ancient Chinese culture.
• Current deep-learning models are hindered by limited datasets lacking scale, character diversity, and book-level annotations.

Purpose of the Study:

• To introduce HisDoc1B, a novel, large-scale dataset designed for Chinese historical document recognition and analysis.
• To address the limitations of existing datasets in terms of scale, character categories, and annotation richness.
• To provide a valuable resource for advancing research in historical document analysis using deep learning.

Main Methods:

• Construction of HisDoc1B dataset comprising 40,281 books, over 3 million images, and over 1 billion characters.
• Inclusion of 30,615 character categories, book-level annotations, and punctuation annotations.
• Validation through extensive experiments demonstrating dataset quality and utility.

Main Results:

• HisDoc1B is the largest dataset for Chinese historical document analysis, exceeding existing datasets by over 200 times.
• It is the only dataset offering comprehensive book-level and punctuation annotations.
• Experimental results confirm the dataset's high quality and practical applicability for deep-learning models.

Conclusions:

• HisDoc1B significantly enhances the scale and scope of available resources for Chinese historical document analysis.
• The dataset's unique annotations facilitate more sophisticated research into ancient Chinese texts.
• HisDoc1B is poised to accelerate advancements in the automated recognition and analysis of historical Chinese documents.