A scalable framework for evaluating multiple language models through cross-domain generation and hallucination...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Information Retrieval

Background:

• Large language models (LLMs) have advanced retrieval-augmented generation (RAG) systems.
• Challenges like semantic similarity, bias, and hallucinations persist in domain-specific LLM applications.

Purpose of the Study:

• Introduce MultiLLM-Chatbot, a scalable RAG-based benchmarking framework.
• Evaluate five popular LLMs (GPT-4-Turbo, CLAUDE-3.7-Sonnet, LLAMA-3.3-70B, DeepSeek-R1-Zero, Gemini-2.0-Flash).
• Assess LLM performance across five domains: Agriculture, Biology, Economics, Internet of Things (IoT), and Medical.

Main Methods:

• Generated 250 standardized queries from 50 peer-reviewed papers.
• Extracted and segmented PDF texts, embedded them, and indexed in Elasticsearch.
• Analyzed 1,250 model responses using cosine similarity, VADER sentiment analysis, TF-IDF, and Named Entity Recognition (NER).

Main Results:

• LLAMA-3.3-70B emerged as the top-performing model overall.
• LLAMA-3.3-70B led in all five evaluated domains.
• The framework provides a reproducible pipeline for domain-specific LLM benchmarking.

Conclusions:

• The MultiLLM-Chatbot framework offers a modular and adaptable solution for LLM benchmarking.
• Findings guide model selection for trustworthy LLM deployment in scientific and industrial sectors.
• The study addresses gaps in current LLM evaluation methodologies.