Multi-agent systems and credibility-based advanced scoring mechanism in fact-checking.

Area of Science:

• Artificial Intelligence
• Information Science
• Computational Social Science

Background:

• Misinformation and rumors on social networking services (SNS) pose significant challenges to online discourse.
• Large language models (LLMs) show promise for fact-checking but suffer from overconfidence and limitations in handling complex texts.
• Existing multi-source fact-checking approaches lack robust scoring mechanisms for credibility assessment.

Purpose of the Study:

• To propose and evaluate a novel multi-agent fact-checking (MAFC) framework.
• To address the limitations of LLM-based fact-checking, including overconfidence and binary classification constraints.
• To develop an effective scoring mechanism for integrating judgments from multiple information sources.

Main Methods:

• Developed a multi-agent fact-checking (MAFC) framework utilizing multiple agents with distinct information sources.
• Implemented a novel scoring mechanism to calculate text credibility based on agent judgments and confidence levels.
• Conducted comparative experiments to evaluate MAFC against baseline methods in fact-checking tasks.

Main Results:

• The proposed MAFC framework demonstrated superior performance compared to baseline methods.
• MAFC achieved better results in both binary and multi-label fact-checking tasks.
• Experimental validation confirmed the effectiveness of the novel scoring mechanism.

Conclusions:

• The MAFC framework offers an effective solution for improving fact-checking accuracy and reliability.
• The developed scoring mechanism enhances the integration of diverse information sources for credibility assessment.
• Further research is needed to address challenges in fact-checking, such as standardizing definitions and creating comprehensive datasets.