An exploration of automated narrative analysis via machine learning

Area of Science:

• Computational Linguistics
• Developmental Psychology
• Educational Technology

Background:

• Accurate assessment of narrative macrostructure is crucial for understanding child language development.
• Current methods often rely on subjective human scoring, which can be time-consuming and prone to variability.
• Automated scoring tools could enhance efficiency and consistency in evaluating narrative skills.

Purpose of the Study:

• To evaluate the accuracy of four machine learning (ML) methods in predicting narrative macrostructure scores.
• To compare ML model performance against scores assigned by human raters using a standardized rubric.
• To identify the most effective ML approach for automating narrative macrostructure assessment.

Main Methods:

• Trained predictive models on a corpus of 414 narratives from school-aged children (5-9 years).
• Compared ML methods utilizing hand-engineered features and those learning directly from raw text.
• Measured performance using Quadratic Weighted Kappa (QWK) for inter-rater reliability.

Main Results:

• The BERT model significantly outperformed other ML methods in scoring accuracy.
• BERT's performance was consistent with scores obtained by human raters using a valid and reliable rubric.
• The study identified BERT as a highly accurate automated scoring tool.

Conclusions:

• The BERT model shows significant promise for automating the scoring of narrative macrostructure.
• Automated scoring using BERT could be a valuable tool in clinical practice for assessing child language proficiency.
• Further research can explore the integration of BERT into clinical assessment workflows.