Consistency evaluation protocol: A reproducible framework for assessing large language model output repeatability

Area of Science:

• Artificial Intelligence
• Natural Language Processing

Background:

• Large Language Models (LLMs) exhibit stochastic behavior, leading to varied responses for identical inputs.
• Existing research on LLM behavior lacks a unified, reproducible methodology combining semantic and structural analysis.

Purpose of the Study:

• To propose and evaluate a novel consistency evaluation protocol for Large Language Models (LLMs).
• To address the research gap by integrating semantic and structural parameters for a comprehensive LLM consistency assessment.

Main Methods:

• Developed a consistency evaluation protocol involving repeated prompting and feature extraction.
• Assessed semantic consistency using embedding-based text representations.
• Analyzed structural variations through sentence length, word usage, and lexical diversity.

Main Results:

• Generated five independent responses for each of 20 input texts using Mistral-7B-Instruct-v0.2.
• Calculated semantic and structural stability, combining them into a Composite Consistency Score (CCS).
• Validated the framework through temperature sensitivity, prompt sensitivity, and metric validation tests.

Conclusions:

• The proposed protocol offers a reliable and reproducible method for evaluating LLM response consistency.
• The Composite Consistency Score (CCS) effectively quantifies both semantic and structural stability.
• This framework advances the understanding and control of LLM behavior in AI research.