Artificial intelligence based personalized student feedback system -Sisu Athwala' to enhance exam performance of
Thilanka Seneviratne1, Supun Manathunga1, Wathsala Idirisingha1
1Department of Pharmacology, Faculty of Medicine, University of Peradeniya, Peradeniya, Sri Lanka.
Background:
In medical education, mentoring and feedback play crucial roles. Providing feedback on exam performance is a vital component as it allows students to improve. Feedback has to be tailor made and specific to the individual student. This needs lot of time and human resources, which are always not in abundance. Use of artificial intelligence (AI) is a promising proposition yet it comes with the integral problem of generating inaccurate responses by the Large language models (LLM). To alleviate and minimize this, we have developed our unique model 'Sisu Athwala' using retrieval augment generation (RAG) with custom LLM's.
Objective:
To design and implement an AI-based tool using RAG to provide customized feedback to medical students to enhance their exam performance, minimizing the risk of generating inaccurate responses by the LLM's. To evaluate the AI tool by expert student mentors and by the end users.
Methods:
The study was conducted at the Faculty of Medicine, University of Peradeniya, Sri Lanka. An AI based feedback tool was developed powered by Generative Pre-trained Transformers-4 (GPT-4) LLM using a RAG pipeline. Expert instruction sets were used to develop the data base through embedding model to minimize potential inaccuracies and biases. To generate user queries, students were provided with a self-evaluation form which was processed using Representative Vector Summarization (RVS). Hence most critical concerns of each student are distilled and captured accurately, minimizing noise or irrelevant details. The role of the AI tool was defined as a counsellor during Pre-processional alignment allowing professional manner throughout the interaction. User queries were processed using Open AI Application Programming Interface (API), utilizing GPT-4-turbo LLM. Students were invited to engage in conversations with the newly developed feedback tool. The AI tool was evaluated by the expert student mentors, as per its ability to give personalized feedback, use varied language expressions, and to introduce novel perspectives to students. End user perception on the use of AI tool was assessed using a questionnaire.
Results:
Post implementation end user survey of the Sisu Athwala AI tool was largely positive. 92% mentioned the advices given by the tool on stress management were helpful. 60% believed that the study techniques suggested were useful. While further 60% thought they are comfortable using the tool. 52% find the advices on exam performances were helpful. In their open comments some suggested to have the tool as a mobile APP. 15 expert student mentors took part in evaluating the tool. 100% agreed that it effectively addressed key points of student strengths and identifies areas for improvements going by the Pendleton model. 90% agreed that Sisu- Athwala gives clear actionable plans.
Conclusion:
Sisu Athwala AI tool provided comprehensive tailor made feedback and guidance to medical students which was well received by the end users. Expert student mentors evaluation of the material generated by the AI tool were quite positive. Though this is not a replacement for human mentors it supports mentoring to be delivered circumventing the human resource constraints.
