Evaluating technology-enhanced learning: A comprehensive framework

The absence of a standard, comprehensive approach to evaluating technology-enhanced learning (TEL) limits the utility of individual evaluations, and impedes the integration and synthesis of results across studies.

To outline a comprehensive framework for approaching TEL evaluation in medical education, and to develop instruments for measuring the perceptions of TEL learners and instructors.

Using both theoretical constructs of inquiry in education and a synthesis of existing models and instruments, we outlined a general model for evaluation that links utility, principles, and practices. From this we derived a framework for TEL evaluation that identifies seven data collection activities: needs analysis; documentation of processes, decisions, and final product; usability testing; observation of implementation; assessment of participant experience; assessment of learning outcomes; and evaluation of cost, reusability, and sustainability. We then used existing quality standards and approaches to develop instruments for assessing the experiences of learners and instructors using TEL.

No single evaluation is likely to collect all of this information, nor would any single audience likely find all information elements equally useful. However, consistent use of a common evaluation framework across different courses and institutions would avoid duplication of effort and allow cross-course comparisons.