A complex network approach to analyse pre-trained language models for ancient Chinese.

Area of Science:

• Computational Linguistics
• Digital Humanities
• Network Science

Background:

• Pre-trained language models are crucial for ancient Chinese compilation.
• Previous research has not holistically examined language model organization of ancient Chinese elements.
• Complex network analysis offers a novel perspective for this investigation.

Purpose of the Study:

• To explore how language models organize elements within the ancient Chinese system using complex networks.
• To analyze the structural properties of character and word co-occurrence and attention networks.
• To compare attention networks generated by SikuBERT with those from Chinese BERT.

Main Methods:

• Analysis of character and word co-occurrence networks in ancient Chinese.
• Static and dynamic network analysis of attention networks generated by SikuBERT.
• Comparison of attention network properties with co-occurrence networks and Chinese BERT.

Main Results:

• Attention networks exhibit small-world and scale-free properties.
• Over 80% of attention networks closely resemble co-occurrence networks.
• SikuBERT's attention networks are sparser than Chinese BERT's, with distinct character-word network gaps.
• Sentence segmentation minimally impacts network metrics, while POS tagging sparsifies networks.

Conclusions:

• Complex network analysis effectively reveals the organizational principles of language models for ancient Chinese.
• Attention mechanisms in SikuBERT capture significant linguistic relationships, mirroring co-occurrence patterns.
• Task-specific fine-tuning (e.g., POS tagging) influences the sparsity of attention networks.