Multimethods study to develop tools for competency-based assessments of implementation research training programmes...

Area of Science:

• Health Systems Research
• Implementation Science
• Educational Assessment

Background:

• Validated tools for evaluating competency-based implementation research (IR) training programs in low- and middle-income countries (LMICs) are scarce.
• Existing training evaluation methods may not adequately capture the nuances of IR competency development in diverse LMIC contexts.

Purpose of the Study:

• To develop and validate tools for assessing the performance of IR training programs based on a framework of core IR competencies specifically for LMICs.
• To evaluate the reliability and validity of these newly developed assessment tools.

Main Methods:

• Development of self-assessment (knowledge, self-efficacy) and objective assessment tools based on an IR competency framework.
• Utilized exploratory factor analyses and a one-parameter logistic model for validation using data from 166 trainees across five universities in LMICs.
• Conducted key informant interviews (KII) with trainees and trainers to assess the framework's and tools' utility.

Main Results:

• Developed two 16-item self-assessment tools (knowledge, self-efficacy) and one 40-item objective assessment tool.
• Self-assessment tools demonstrated high internal consistency (Cronbach's alpha > 0.97).
• Low correlation between self-assessed and objective IR knowledge pre-training (r=0.27) improved post-training (r=0.43), indicating potential over-reliance on self-assessment alone.

Conclusions:

• The developed IR competency framework and assessment tools are reliable and valid for evaluating IR training programs in LMICs.
• Findings suggest that self-assessment methods alone are insufficient for a comprehensive and valid evaluation of IR training performance in these settings.
• The tools provide a robust mechanism for guiding and improving IR training initiatives in resource-limited environments.