Measuring Text Difficulty Using Parse-Tree Frequency.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Human-Computer Interaction

Background:

• Traditional text simplification methods rely on outdated readability formulas.
• Term familiarity is a successful metric, suggesting grammatical patterns may also be predictive of text difficulty.

Purpose of the Study:

• To introduce and evaluate grammar familiarity as a novel measure for text simplification.
• To compare grammar familiarity with existing readability formulas and user perception of text difficulty.

Main Methods:

• Created a database of 140K unique 3rd-level sentence parse structures from English Wikipedia.
• Calculated grammar frequencies and binned sentences into 11 frequency groups.
• Conducted a user study (N=6,600) measuring comprehension (Cloze test), perceived difficulty, and reading time.

Main Results:

• Sentences with more frequent grammatical structures were easier to understand, perceived as easier, and read faster.
• Grammar familiarity effectively predicted actual text difficulty, unlike traditional readability formulas.
• Readability formulas correlated with perceived difficulty but not actual comprehension.

Conclusions:

• Grammar familiarity offers a more effective approach to text simplification than traditional readability formulas.
• This metric can be applied to analyze grammar regularity across diverse text corpora.
• The findings suggest that sentence structure frequency is a key factor in text comprehension.