Say What? Real-time Linguistic Guidance Supports Novices in Writing Utterances for Conversational Agent Training

Writing utterances to train conversational agents can be a challenging and time-consuming task, and usually requires substantial expertise, meaning that novices face a steep learning curve. We investigated whether novices could be guided to produce utterances that adhere to best practices via an intervention of real-time linguistic feedback. We conducted a user study in which participants were tasked with writing training utterances for a particular topic (intent) for a conversational agent. Participants received one of two types of linguistic guidance in real-time to shape their utterance-writing: (1) feedback on the lexical and syntactic properties and the variety of each utterance, or (2) sample utterances written by other users, to select or inspire the writing of new utterances. All participants also completed a control condition, in which they wrote utterances for a different intent without receiving any guidance. We investigated whether linguistic properties of the utterances differed as a function of whether the participant had received guidance, and if so, which type. Results showed that participants wrote longer and better quality utterances, with greater lexical and syntactic diversity, in both guidance conditions compared to when they received no guidance. These results demonstrate that giving novices explicit linguistic guidance can improve the quality of the training utterances they write, suggesting that this could be an effective way of getting new utterance writers started with much less training than most current practices require.