Development and validation of a digital learning resource quality evaluation framework: a mixed-methods study in health professions education

Digital learning resources (DLRs) are now core teaching tools in health professions education, yet quality evaluation remains constrained by subjective weighting bias and a persistent gap between educational importance and operational measurability. Because DLRs increasingly mediate what and how students learn, deficiencies in resource quality propagate directly into competency attainment, making quality evaluation a proximal determinant of student competency outcomes. This study aimed to develop and validate a practical DLR quality evaluation framework for medical and public health education.

A sequential mixed-methods design was employed. First, a modified Delphi procedure with 26 experts was used to construct the hierarchical indicator system. Second, an expert-based and data-derived integrated weighting model combining AHP and EWM was developed to balance expert judgment and data discriminatory power. Third, a measurability correction mechanism and a four-quadrant prioritization matrix were established. The framework was validated with 10 DLRs across 10 medical courses, involving 309 students.

The final system comprised 4 primary dimensions, 9 secondary indicators, and 14 tertiary indicators. Content Quality had the highest integrated weight (0.469), with Data Visualization (0.136) and Updated Content and Frontier Knowledge (0.103) as the top tertiary indicators. The four-quadrant matrix classified 7 indicators for direct implementation and 2 for targeted methodological development. Validation confirmed high internal consistency (Cronbach's α = 0.987) and satisfactory content validity (S-CVI = 0.951); absolute-agreement inter-rater ICCs were near zero, indicating substantial between-student variability in scoring levels.

This study provides a reliable, implementable DLR quality evaluation framework. By converting evaluation results into ranked, measurable improvement priorities, the framework directs resource revision toward the attributes that bear most directly on student competency development. The methodological workflow shows potential for transferability across nursing, pharmacy, public health and other health-related disciplines via contextual calibration, pending formal cross-context validation.