Automated developmental sentence scoring using computerized profiling software

Area of Science:

• Linguistics
• Speech-Language Pathology
• Computational Linguistics

Background:

• Developmental Sentence Scoring (DSS) is a method for analyzing children's grammatical complexity.
• Automated analysis tools aim to improve efficiency and consistency in language assessment.
• The accuracy of automated DSS has not been extensively validated.

Purpose of the Study:

• To evaluate the accuracy of Computerized Profiling (CP) software in performing automated Developmental Sentence Scoring (DSS).
• To compare automated DSS results with traditional manual coding.
• To identify potential discrepancies in accuracy, particularly for children with language impairments.

Main Methods:

• Utilized 9,084 utterances from 48 school-age children (28 with language impairment).
• Performed both manual DSS coding and automated DSS coding using Computerized Profiling (CP) software.
• Calculated point-by-point agreement and correlation coefficients between manual and automated scores.

Main Results:

• Achieved 78% point-by-point agreement between CP's automated DSS and manual coding.
• Reported per-category agreement levels ranging from 0% to 98%.
• Observed a slight (approximately 2%) decrease in agreement for children with language impairments.
• Found a high correlation (r = .97) between CP-computed and manually computed scores, though CP scores were significantly higher.

Conclusions:

• Computerized Profiling (CP) software demonstrates substantial agreement with manual Developmental Sentence Scoring (DSS) but requires further refinement.
• Automated DSS analysis shows slightly reduced accuracy for children with language impairments.
• Continued research is necessary to enhance the precision of automated DSS tools for clinical language assessment.