Benchmarking supervised classifiers within a design-of-experiments framework: Robust statistical inference in text...

Area of Science:

• Computational Linguistics
• Digital Humanities
• Statistical Learning

Background:

• Supervised classification with Bag-of-Words models is common in literary text mining.
• Existing benchmarking practices often suffer from methodological fragility, leading to unreliable results.
• Issues include improper feature selection, non-shared resampling splits, and over-reliance on average accuracy.

Purpose of the Study:

• To present a reproducible workflow for benchmarking supervised classifiers in text analysis.
• To apply a design-of-experiments framework for statistically coherent evaluation.
• To demonstrate the importance of rigorous methodology using Dante's Divina Commedia.

Main Methods:

• Implementation of a workflow combining leakage-free preprocessing and shared Monte Carlo train/test splits.
• Utilizing a design-of-experiments framework in R for text analysis.
• Employing paired statistical comparison, power assessment, and validation with mispaired designs.

Main Results:

• The proposed workflow ensures statistically coherent benchmarking of text classifiers.
• Leakage-free preprocessing is crucial, even for small biases, to prevent altered classifier rankings.
• Shared train/test splits are integral to the inferential design, not merely a technical detail.

Conclusions:

• Established text classification methods can yield reliable results under statistically sound conditions.
• Rigorous benchmarking is essential to avoid overstating performance differences between classifiers.
• Performance interpretation should consider transparency, stability, and relevance beyond accuracy alone.