A Standardized Project Gutenberg Corpus for Statistical Analysis of Natural Language and Quantitative Linguistics

Area of Science:

• Corpus Linguistics
• Natural Language Processing
• Digital Humanities

Background:

• Project Gutenberg (PG) is a widely used text corpus for language analysis.
• Existing PG studies often use small, manually selected subsets or lack detailed preprocessing, hindering reproducibility.
• No standardized, full-text version of Project Gutenberg has been available.

Purpose of the Study:

• To present the Standardized Project Gutenberg Corpus (SPGC), a curated, full-size version of the complete PG data.
• To address the limitations of previous PG studies by providing a reproducible and comprehensive dataset.
• To facilitate research in language variability, corpus linguistics, NLP, and information retrieval.

Main Methods:

• Developed an open science approach to curate the complete Project Gutenberg dataset.
• Incorporated annotated metadata for content characterization.
• Published detailed methodology, processing code, and the corpus at multiple granularity levels.

Main Results:

• Created the Standardized Project Gutenberg Corpus (SPGC) with over 50,000 books and 3 billion word-tokens.
• Provided a broad characterization of PG content using metadata.
• Demonstrated the corpus's potential for analyzing language variability across time, subjects, and authors.

Conclusions:

• The SPGC offers a reproducible, pre-processed, full-size Project Gutenberg resource.
• This new scientific resource supports advancements in corpus linguistics, NLP, and information retrieval.
• The open science approach ensures transparency and facilitates future research using the PG corpus.