An Explication and Classroom Field Study of the Virtual Human Interaction Lab's Expert (VHIL-E) LLM.

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Virtual and Augmented Reality

Background:

• Virtual Human Interaction Lab's Expert (VHIL-E) is a large language model (LLM) designed to represent research in virtual and augmented reality.
• Retrieval-Augmented Generation (RAG) is a key technique for enhancing LLM capabilities by integrating external knowledge sources.
• Implementing RAG involves a systematic process, termed the 'Seven Cs', for effective knowledge integration.

Purpose of the Study:

• To present best practices for implementing retrieval-augmented generation (RAG) systems.
• To evaluate the performance of RAG-based LLMs compared to traditional LLMs in an academic context.
• To analyze the impact of RAG constraints on the frequency of LLM hallucinations.

Main Methods:

• Collected and processed over 2.3 million words of research materials, including publications, talks, interviews, and course curricula.
• Implemented a RAG system (VHIL-E) following the 'Seven Cs' methodology: collecting, cleaning, classifying, chunking, creating embeddings, correlating embeddings, and connecting to an LLM.
• Conducted two studies: Study 1 compared RAG and Base GPT models on a multiple-choice test; Study 2 evaluated VHIL-E's hallucination rates in an open-ended task with student feedback.

Main Results:

• RAG implementations and Base GPT models achieved high performance (83-90%) on a multiple-choice test, comparable to student exam scores.
• In an open-ended task, students identified hallucinations in VHIL-E.
• Constraining VHIL-E to its RAG index significantly reduced hallucinations compared to allowing access to general GPT intelligence (Base/RAG Hybrid).

Conclusions:

• RAG systems can effectively represent specialized knowledge domains like virtual and augmented reality research.
• The implementation framework ('Seven Cs') provides a structured approach to building effective RAG systems.
• Constraining LLMs to specific knowledge bases is crucial for minimizing factual inaccuracies (hallucinations) in specialized applications.