Machine-Scored Syntax: Comparison of the CLAN Automatic Scoring Program to Manual Scoring

Area of Science:

• Linguistics
• Computational Linguistics
• Developmental Psychology

Background:

• The Index of Productive Syntax (IPSyn) is a measure of grammatical development.
• Automatic scoring tools like Computerized Language ANalysis (CLAN) aim to streamline IPSyn analysis.
• Accuracy of automated linguistic analysis tools is crucial for research and clinical applications.

Purpose of the Study:

• To evaluate the accuracy of machine-scored Index of Productive Syntax (IPSyn) using CLAN tools.
• To compare machine scoring results against manual scoring benchmarks.
• To introduce and utilize novel metrics for assessing automated syntactic analysis.

Main Methods:

• Comparison of machine-scored and manually-scored IPSyn data from 20 transcripts (10 children, 30 and 42 months).
• Analysis using traditional metrics (absolute point difference, point-to-point accuracy) and new metrics (Machine Item Accuracy - MIA, Cascade Failure Rate).
• Examination of differences in total scores, subscale scores (Noun Phrase, Verb Phrase, Question/Negation, Sentence Structures), and individual structures.

Main Results:

• Machine scoring showed a mean absolute point difference of 3.65 and 72.6% point-to-point agreement with manual scoring.
• Machine Item Accuracy (MIA) was 74.9%, with significantly more erroneous items than missed items.
• Noun Phrase and Verb Phrase subscales demonstrated higher accuracy than Question/Negation and Sentence Structures subscales.

Conclusions:

• The CLAN program's automatic scoring of IPSyn demonstrated notable inaccuracies compared to manual scoring.
• Recommendations for CLAN improvement include addressing second exemplar violations and implementing cascaded credit.
• Researchers and clinicians should routinely report detailed accuracy metrics, including MIA, to understand the limitations of machine-scored syntax.