Large language models for closed-library multi-document query, test generation, and evaluation

Area of Science:

• Artificial Intelligence
• Knowledge Management
• Information Science

Background:

• Technical professions face challenges in learning complex, evolving knowledge from large, frequently updated documents.
• Generating and revising knowledge tests requires tracking updates in extensive knowledge bases.
• Large Language Models (LLMs) offer a framework for AI-assisted knowledge acquisition and continuous learning.
• Retrieval-Augmented Generation (RAG) integrates pre-trained LLMs with domain-specific knowledge bases.

Purpose of the Study:

• To introduce methods (DaaDy, SQAD) for effective LLM-RAG question-answering on large documents.
• To present the AI for knowledge intensive tasks (AIKIT) solution for managing numerous documents for training and continuing education.
• To provide an open-source, containerized solution deployable on various systems.

Main Methods:

• Developed DaaDy (document as a dictionary) and SQAD (structured question answer dictionary) for LLM-RAG implementation.
• Created AIKIT, a containerized open-source solution integrating LLMs, RAG, vector stores, and a web interface.
• Employed document segmentation to improve question coverage for long source documents.

Main Results:

• Document segmentation enhances the coverage of LLM-RAG generated questions, especially for lengthy documents.
• AIKIT facilitates the use of multiple LLM models with multimodal RAG source documents.
• AIKIT retains LLM-RAG responses for queries across single or multiple LLM models.

Conclusions:

• AIKIT offers a user-friendly toolkit for leveraging LLM-RAG capabilities with complex information.
• The solution simplifies the integration and utilization of multiple LLM models.
• AIKIT supports continuous learning and knowledge management in technical fields by retaining query responses.