Promises and pitfalls of machine scoring of the Index of Productive Syntax

The AC-IPSyn computerised system for scoring the Index of Productive Syntax (IPSyn) was evaluated. Twenty language samples, ten taken at 30 months and ten of the same children at 42 months, were each scored for the IPSyn by hand and by AC-IPSyn. Point differences and point-to-point reliability were examined at the levels of the total, subscale, and individual structure scores. Points missed and erroneously given at each level were also analysed. The difference in total scores between manual and AC-IPSyn scoring was relatively small; point-to-point agreement was lower than reported elsewhere. Age differences were also found. AC-IPSyn accuracy varied by subscale and structure, with results suggesting that AC-IPSyn be used at this point in conjunction with hand scoring of more error-prone and low frequency structures. The relatively small total point difference masked the lower reliability revealed by other measures, demonstrating the importance of detailed comparisons of manual and machine scoring.