A novel framework for educational Q&A: Leveraging RAG and Code Interpreters for knowledge retrieval and logical...

Area of Science:

• Artificial Intelligence
• Educational Technology
• Natural Language Processing

Background:

• Traditional educational Q&A systems struggle with knowledge updates, reasoning accuracy, and complex computations.
• Limitations are pronounced in domains needing multi-step reasoning or real-time, specific knowledge.
• Large Language Models (LLMs) often face 'hallucination' and lack precise computational abilities.

Purpose of the Study:

• To develop an enhanced educational Q&A system by integrating Retrieval-Augmented Generation (RAG) with Large Language Model (LLM) Code Interpreters.
• To address challenges in knowledge currency, reasoning accuracy, and computational task handling.
• To improve the accuracy and reliability of educational Q&A systems.

Main Methods:

• Implemented a system combining RAG for dynamic knowledge retrieval with LLM Code Interpreters for logical reasoning and Python code execution.
• Evaluated the system on five diverse educational datasets: AI2_ARC, OpenBookQA, E-EVAL, TQA, and ScienceQA.
• Compared performance against vanilla LLMs, focusing on accuracy in mathematical problems and complex queries.

Main Results:

• The proposed RAG and Code Interpreter integration achieved an average accuracy improvement of 10-15 percentage points over vanilla LLMs.
• GPT-4o and Gemini-pro-1.5 demonstrated superior performance, particularly in scientific reasoning and multi-step computations.
• The system effectively mitigated LLM 'hallucination' by leveraging external, up-to-date knowledge sources.

Conclusions:

• Integrating RAG and Code Interpreters offers a promising pathway for more accurate, transparent, and personalized educational Q&A systems.
• The approach significantly enhances the learning experience by improving problem-solving capabilities in complex domains.
• Future research should address remaining challenges like retrieval failures, code execution errors, and multi-modal reasoning limitations.