Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA)

Area of Science:

• Speech and Language Pathology
• Computational Linguistics
• Developmental Psychology

Background:

• Language sample analysis (LSA) is crucial for culturally sensitive language assessment in children.
• Literate language features in narratives are key indicators of language development.
• Existing automated scoring lacks progress-monitoring capabilities for literate language.

Purpose of the Study:

• To develop and validate an automated scoring system, the Literate Language Use in Narrative Assessment (LLUNA).
• To evaluate LLUNA's ability to score six specific literate language elements in narrative transcripts.
• To assess LLUNA's interrater reliability against expert and certified scorers.

Main Methods:

• Developed a hard-coded scoring system (LLUNA) for automated evaluation of literate language.
• LLUNA individually scores coordinating/subordinating conjunctions, meta-linguistic/meta-cognitive verbs, adverbs, and elaborated noun phrases.
• Interrater reliability was calculated using quadratic weighted kappa (κ).

Main Results:

• LLUNA demonstrated strong interrater reliability with an expert scorer across all six literate language elements.
• LLUNA's reliability surpassed certified, non-expert scorers on four elements and closely matched on two.
• The system effectively automates the scoring of specific literate language features.

Conclusions:

• LLUNA shows significant promise as a reliable tool for automating the assessment and progress-monitoring of literate language in narrative samples.
• This automated approach can enhance the efficiency and consistency of language sample analysis.
• LLUNA offers a viable solution for tracking developmental changes in children's narrative language skills.