From automation to augmentation: a retrieval-augmented large language model framework for reshaping learning...

Generative artificial intelligence is now embedded in everyday university work, yet most evidence on its educational value still comes from surveys of student perception rather than from what students do. We reanalyzed the public log of Prof. Leodar, a course-specific retrieval-augmented chatbot deployed in an undergraduate data science module at Nanyang Technological University, Singapore. The original deployment study collected survey responses from 34 of 154 enrolled students (a 22% response rate). It released the complete query log of 12,330 anonymized learner queries over a 14-week pilot, in which the chatbot was powered by Anthropic Claude 3 (Sonnet) and a FAISS vector store. Because the survey sample is small and self-selected, and because the query log is a convenience sample clustered at the student level, this is a descriptive analysis: we report effect sizes and predictive accuracy as within-corpus summaries and do not generalize to other cohorts. We propose a query-level Automation-to-Augmentation framework that separates interactions in which the system simply delivers an answer from interactions that scaffold reasoning, debugging, or interpretation. Each query is assigned a topical category, one of five retrieval-use types, an augmentation label, and a Learning Augmentation Index (LAI) on a 0-100 scale; reading complexity is captured using the Flesch-Kincaid grade-level measure. We summarized the corpus with frequencies, Cramer's V (reported descriptively, with Bonferroni-corrected p-values), Kruskal-Wallis comparisons, and a 5-fold cross-validated logistic regression. Augmentation-oriented behavior accounts for 46.6% of all queries. The four technical categories (programming and debugging, data visualization, machine learning and modeling, and statistics and mathematics) together account for 57.3% of the corpus and carry the highest LAI (58.3 to 61.1). Retrieval-use type is more strongly associated with augmentation than topical category (Cramer's V = 0.226 versus 0.166), and the classifier recovers augmentation labels at AUC = 0.792 (accuracy 0.745, F1 0.666). Because the labels and the LAI are rule-based and the dataset contains no learning-outcome information, the contribution is best read as a transparent, reproducible behavioral analytics framework for course-specific RAG, rather than a measurement of learning itself.