Learning to detect AI texts and learning the limits

Area of Science:

• Cognitive Psychology
• Natural Language Processing
• Human-Computer Interaction

Background:

• The proliferation of AI-generated text necessitates methods for accurate detection.
• Misconceptions about AI writing characteristics, such as style and readability, are common.
• Overconfidence in identifying AI text can lead to errors, particularly in educational settings.

Purpose of the Study:

• To determine if individuals can learn to accurately discriminate between human and AI-generated texts with immediate feedback.
• To assess if feedback helps recalibrate self-perceived competence in AI text identification.
• To explore the criteria (textual style, readability) individuals use for discrimination.

Main Methods:

• GPT-4o generated AI texts; Koditex corpus provided human texts.
• 254 Czech native speakers participated in a text-pair identification task.
• Two conditions: immediate feedback vs. delayed feedback.

Main Results:

• Immediate feedback significantly improved accuracy and confidence calibration.
• Participants initially held incorrect assumptions about AI text features.
• Without feedback, errors peaked during high confidence; feedback mitigated this.
• Learners corrected misconceptions regarding AI stylistic rigidity and readability.

Conclusions:

• Targeted training with explicit feedback effectively teaches AI text differentiation.
• Feedback corrects misconceptions about AI stylistic features and readability.
• Accurate self-assessment is facilitated, reducing harmful overconfidence in educational contexts.