A Large-Scale Analysis of Variance in Written Language

Area of Science:

• Computational linguistics
• Cognitive science
• Natural language processing

Background:

• Large text corpora have driven advances in distributional semantics and language learning models.
• Current models often treat language experience (corpora) agnostically, assuming uniform statistical properties.
• Knowledge representation is theorized as an interaction between cognitive processing and language structure.

Purpose of the Study:

• To investigate the variability of written language within fiction corpora.
• To assess whether linguistic patterns differ across genres, authors, and even within an author's works.
• To inform the selection and curation of corpora in computational linguistics.

Main Methods:

• Distributional semantic analysis of written language from a diverse collection of fiction books.
• Comparative analysis of linguistic features across different genres, authors, and individual works.
• Statistical examination of word usage and contextual redundancies.

Main Results:

• Significant variations in written language were found between different fiction genres.
• Linguistic differences were also observed between books of the same genre and even by the same author.
• The homogeneity assumption of corpora in distributional semantics is challenged.

Conclusions:

• The structure of language experience, specifically within corpora, is not uniform.
• Computational models of natural language processing must account for corpus-specific linguistic characteristics.
• More deliberate consideration and curation of corpora are essential for robust natural language processing research.