Designing metaverse interaction systems for the Turkish language enhanced by fine-tuning and retrieval-augmented...

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Natural Language Processing

Background:

• The metaverse enables real-time interaction via immersive technologies, with Natural Language Processing (NLP) and Large Language Models (LLMs) enhancing human-computer engagement.
• Current LLM applications in virtual platforms often yield lengthy, irrelevant NPC responses, diminishing user immersion.
• Existing research primarily focuses on open-ended conversations, neglecting the specific constraints of metaverse NPC dialogue.

Purpose of the Study:

• To design NLP systems for AI-powered NPCs in the metaverse, generating concise, context-aware, and task-oriented outputs.
• To systematically investigate and compare fine-tuning and Retrieval-Augmented Generation (RAG) strategies for metaverse dialogue systems.
• To evaluate the performance of various decoder-only and encoder-decoder models using both standard and semantic metrics.

Main Methods:

• Comparative evaluation of fine-tuning versus RAG strategies across GPT-2, LLaMA, Qwen, mBART, and mT5 models.
• Utilized a combination of standard and semantic evaluation metrics, normalized with the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS).
• Developed and presented a Speech-to-Text (STT) → LLMs → Text-to-Speech (TTS) framework for real-time, personalized speech-based interaction.

Main Results:

• Retrieval-Augmented Generation (RAG) demonstrated more balanced performance, particularly with encoder-decoder models like mBART (~0.652) and mT5 (~0.555).
• RAG achieved effective results even when models were trained on smaller datasets.
• The proposed STT→LLMs→TTS framework enhances interaction quality through coherent and realistic speech communication.

Conclusions:

• RAG is a promising technique for improving NPC dialogue generation in the metaverse, offering concise and context-aware responses.
• Encoder-decoder architectures, when combined with RAG, show significant potential for metaverse applications.
• The integrated speech interaction framework facilitates more natural and immersive communication within metaverse environments.