Rethinking AI-assisted writing instruction: feedback literacy scripts, calibration training, and student writing...

Area of Science:

• Educational Technology
• Writing Studies
• Artificial Intelligence in Education

Background:

• Generative AI is increasingly used in writing instruction, posing challenges in student feedback interpretation and critical use.
• Effective AI-assisted writing requires students to critically evaluate and utilize feedback for improvement.

Purpose of the Study:

• To investigate the impact of two metacognitive interventions on writing quality and self-assessment accuracy in AI-assisted writing.
• To compare the effects of a Feedback Literacy Script (FRAC) and an Assessment-Performance Calibration Activity (APCA) individually and in combination.

Main Methods:

• A 2x2 mixed factorial design with 120 undergraduate English majors across four conditions: regular AI use, FRAC only, APCA only, and FRAC+APCA.
• Data collected on writing quality gain, self-assessment accuracy, overconfidence, feedback uptake, and revision depth across four writing tasks.

Main Results:

• FRAC significantly improved writing quality, feedback uptake, and revision depth, enhancing students' processing of AI feedback.
• APCA demonstrated the strongest effect on self-assessment accuracy and overconfidence reduction, improving internal judgment.
• The combined FRAC+APCA intervention yielded the highest overall writing quality gains and retention but did not surpass APCA alone in self-assessment accuracy.

Conclusions:

• Writing quality and self-evaluation accuracy are related but distinct outcomes in AI-assisted writing.
• The educational value of AI-assisted writing hinges on learners' ability to evaluate feedback, calibrate self-judgment, and develop independent revision skills.
• Metacognitive interventions like FRAC and APCA are crucial for maximizing the benefits of AI in writing education.