The impacts of learning analytics and A/B testing research: a case study in differential scientometrics

Area of Science:

• Educational Technology
• Computer-Supported Learning
• STEM Education

Background:

• Online learning platforms generate vast data, fueling research through A/B testing and learning analytics (LA).
• The influence of these distinct research approaches on scientific discourse remains unclear.
• This study investigates how A/B testing and LA research on online STEM education platforms shape future scholarly conversations.

Purpose of the Study:

• To conduct the first scientometric study comparing the impact of A/B testing and LA research in online STEM education.
• To analyze how researchers build upon prior work from these two methodologies.
• To establish a novel method for evaluating research contributions in learning analytics.

Main Methods:

• Selected a pair of papers (one A/B testing, one LA) from the same research group, year, and conference on online STEM education.
• Analyzed citation networks by coding the reasons for citations from citing papers.
• Employed inter-rater reliability checks for accurate coding of citation purposes.

Main Results:

• The A/B testing paper received more self-citations and direct build-ons.
• The learning analytics (LA) paper was cited more frequently overall but often without direct engagement.
• Statistical comparison revealed differential citation patterns between the two research types.

Conclusions:

• The A/B testing paper demonstrated a more significant impact on subsequent research, contrary to citation counts alone.
• Learning analytics (LA) research, while highly cited, showed less direct influence on the discourse.
• The study provides a new framework for understanding research impact within learning analytics and online education.