S²AF: An action framework to self-check the Understanding Self-Consistency of Large Language Models.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Cognitive Science

Background:

• Large Language Models (LLMs) exhibit advanced language understanding but their grasp of implicit textual relationships is not fully understood.
• Assessing the depth of knowledge and logical consistency in LLM outputs is crucial for reliable AI development.

Purpose of the Study:

• Introduce 'Understanding Self-Consistency' as a metric for LLMs' ability to maintain logical and contextual coherence.
• Propose and evaluate the Self-Questioning and Answering Framework (S²AF) to measure this self-consistency.

Main Methods:

• Developed S²AF, a framework with a self-questioning and answering mechanism creating a closed loop for autonomous generation, questioning, answering, and evaluation.
• Tested S²AF on six LLMs across two datasets to objectively measure their self-consistency and knowledge relationship grasp.

Main Results:

• LLMs demonstrate measurable 'Understanding Self-Consistency' with differentiated abilities across reasoning paradigms.
• LLM performance can be enhanced using their own outputs via a 'self-enhanced Feedforward' mechanism.
• S²AF leverages factual logical relationships, independent of external data.

Conclusions:

• 'Understanding Self-Consistency' provides a novel perspective on LLM knowledge representation.
• The S²AF framework offers a robust method for evaluating and improving LLM logical coherence.
• Findings suggest potential for advancing Embodied Artificial Intelligence (EAI) through enhanced LLM self-consistency.