Related Experiment Video
Updated: Jul 2, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
From automation to augmentation: a retrieval-augmented large language model framework for reshaping learning
Wang Lingling1, Sun Shijie1, Liu Xueyi2
1Jiangsu Medical College, Yancheng, Jiangsu, China.
Frontiers in Psychology
|July 1, 2026
Summary
Generative AI in university work shows potential for augmenting learning, not just automating tasks. Analysis of chatbot logs reveals nearly half of student queries aimed to enhance reasoning and problem-solving skills.
Area of Science:
- Educational Technology
- Artificial Intelligence in Education
- Human-Computer Interaction
Background:
- Generative artificial intelligence (AI) is increasingly integrated into university settings.
- Existing evidence on AI's educational impact often relies on student perceptions rather than actual usage data.
- A course-specific retrieval-augmented chatbot (Prof. Leodar) was deployed in an undergraduate data science module.
Purpose of the Study:
- To analyze student interactions with a course-specific retrieval-augmented chatbot (RAG) by re-examining its public query log.
- To develop and apply an "Automation-to-Augmentation" framework to categorize chatbot queries based on their learning support function.
- To quantitatively assess the extent to which chatbot use scaffolds student reasoning, debugging, and interpretation.
Main Methods:
- Reanalysis of a 14-week query log (12,330 anonymized queries) from a retrieval-augmented chatbot (Anthropic Claude 3, FAISS vector store).
- Development of a query-level framework classifying interactions into "automation" (direct answers) versus "augmentation" (scaffolding learning).
- Queries were categorized by topic, retrieval-use type, and assigned a Learning Augmentation Index (LAI); reading complexity was measured using Flesch-Kincaid grade level.
Main Results:
- Nearly half (46.6%) of all queries exhibited augmentation-oriented behavior, aiming to scaffold learning.
- Technical categories (programming, data visualization, machine learning, statistics) showed the highest LAI, indicating deeper learning engagement.
- Retrieval-use type was a stronger predictor of augmentation than topical category (Cramer's V=0.226 vs. 0.166), with a classifier achieving an AUC of 0.792 for augmentation labels.
Conclusions:
- Student interactions with course-specific RAG chatbots extend beyond simple information retrieval, with significant potential for augmenting learning processes.
- The proposed "Automation-to-Augmentation" framework provides a transparent and reproducible method for analyzing behavioral analytics in educational RAG systems.
- This behavioral analysis framework offers insights into AI's role in higher education, distinct from self-reported student perceptions.
