Weakly supervised veracity classification with LLM-predicted credibility signals.

Area of Science:

• Computational Linguistics
• Information Science
• Artificial Intelligence

Background:

• Assessing online content veracity is challenging due to the complexity of credibility signals.
• Automating credibility signal extraction requires high-accuracy models and large annotated datasets, which are often unavailable.
• Existing methods struggle with domain adaptation in misinformation detection.

Purpose of the Study:

• To introduce Pastel (Prompted weak Supervision with credibility signaLs), a weakly supervised method for extracting credibility signals.
• To leverage large language models (LLMs) for automated credibility signal extraction and content veracity prediction.
• To evaluate Pastel's performance against zero-shot and state-of-the-art supervised methods, particularly in cross-domain settings.

Main Methods:

• Utilizing prompted weak supervision with LLMs to extract various credibility signals from web content.
• Combining extracted credibility signals to predict the veracity of online content without human supervision.
• Validating the approach on four article-level misinformation detection datasets and performing cross-domain evaluations.

Main Results:

• Pastel outperforms zero-shot veracity detection by 38.3% and achieves 86.7% of supervised performance.
• In cross-domain settings, Pastel surpasses the state-of-the-art supervised model by 63%.
• 12 out of 19 proposed credibility signals show strong associations with veracity, with some domain-specific strengths.

Conclusions:

• Pastel offers an effective weakly supervised approach for automated credibility signal extraction and veracity prediction.
• The method demonstrates strong performance and significant improvements in cross-domain misinformation detection.
• Credibility signals are valuable indicators of content veracity, with potential for domain-specific applications.