When assessment alignment masks generalizability: a methodological cautionary tale from an OBE pilot in nephrology

Research competency is increasingly recognized as essential for medical graduates, yet rigorous evaluation of educational interventions remains uncommon. Outcome-based education (OBE) aligns teaching activities with predefined competencies, but its application to research competency development in specialized clinical rotations is underexplored.

This methodological reflection draws on a feasibility pilot evaluating an OBE research-integrated teaching model in nephrology. A mixed-methods quasi-experimental design was employed, combining quantitative outcome data with qualitative feedback within a within-instructor counterbalanced framework. Eight instructors taught 126 fifth-year medical students across two consecutive 4-week blocks, with model crossover and no washout period. The primary outcome was a comprehensive examination (100-point scale); secondary outcomes included self-rated research competency, capstone proposal ratings, and platform usability.

Near-transfer assessment scores were higher in the OBE condition (mean difference 5.6 points, 95% CI 3.1-8.1, P = 0.001). Sensitivity analysis excluding the research design subscale (which directly overlapped with OBE micro-projects) attenuated the effect by 32% (mean difference 3.8 points, 95% BCa CI 1.2-6.4, P = 0.005). Self-rated competency gains and capstone scores favored the OBE group, but all assessments lacked far-transfer tasks.

The statistically significant differences reflect near-transfer performance on examination tasks structurally aligned with training activities, not generalizable competency acquisition. The design failed to deliver credible evidence due to four fatal flaws: (1) assessment-task alignment between the research design subscale and OBE micro-projects; (2) absence of graduated transfer assessment; (3) insufficient instructor sample (n = 8) to detect instructor-by-model interactions; and (4) lack of a washout period introducing cross-contamination. The control condition (passive didactic teaching) conflated OBE-specific effects with generic active learning benefits. This article provides no evidence that the OBE model improves research competency; its sole contribution is to identify and warn against assessment alignment bias as a common validity threat. We propose a graduated transfer assessment framework and methodological checklist for future studies.