Automated assignment grading with large language models: insights from a bioinformatics course

Area of Science:

• Educational Technology
• Artificial Intelligence in Education
• Bioinformatics Education

Background:

• Individualized feedback is crucial for student learning but challenging to provide at scale.
• Large language models (LLMs) offer a potential solution for efficient, personalized feedback.
• Evaluating LLM effectiveness in real educational settings is essential.

Purpose of the Study:

• To evaluate the practical effectiveness of LLM-based grading for written assignments.
• To compare the grading accuracy and feedback quality of LLMs against human teaching assistants (TAs).
• To assess the performance of both commercial and open-source LLMs.

Main Methods:

• A practical evaluation was conducted in an "Introduction to Bioinformatics" course with over 100 students.
• Students' text-based answers were graded using LLMs, with a subset receiving feedback from both LLMs and human TAs in a blind study.
• Six commercial and open-source LLMs were systematically evaluated and compared to human TA performance.

Main Results:

• LLMs, when prompted effectively, achieved grading accuracy and feedback quality comparable to human TAs.
• Open-source LLMs demonstrated performance on par with commercial LLMs.
• Student feedback quality ratings were similar for LLM-generated and human-generated feedback.

Conclusions:

• LLMs are a viable tool for grading written assignments, offering scalable and high-quality feedback.
• Open-source LLMs provide a cost-effective and privacy-preserving alternative for educational institutions.
• LLM-based grading can reduce instructor workload while enhancing student learning experiences.