Measuring medical students' professional competencies in a problem-based curriculum: a reliability study.

Area of Science:

• Medical Education
• Competency-Based Assessment
• Problem-Based Learning

Background:

• Assessing professional competencies in medical students presents significant challenges.
• A novel instrument was developed by Problem-Based Learning (PBL) tutors to evaluate essential competencies.
• This study validates the reliability and validity of this instrument within PBL settings.

Purpose of the Study:

• To evaluate the reliability and validity of professional competency scores derived from a newly developed instrument.
• To assess the performance of medical students in interpersonal, cognitive, and professional behavior domains within PBL tutorials.

Main Methods:

• Forty-six Year 2 medical students in PBL tutorials were assessed by two faculty members across four occasions.
• An instrument with ten items rated on a five-point Likert scale measured three competency domains.
• Reliability was analyzed using G-theory, and criterion-related validity was assessed via correlation with written examination scores.

Main Results:

• The overall generalizability coefficient (G) for professional competency scores was 0.80, indicating acceptable reliability.
• The interpersonal domain showed the highest reliability (G=0.84), while professional behavior had the lowest (G=0.76).
• A positive correlation was found between written examination scores and cognitive competencies (r=0.46, P<0.01).

Conclusions:

• The assessment instrument demonstrates acceptable reliability for evaluating medical students' professional competencies in PBL tutorials.
• Further validation studies are recommended before its implementation for summative evaluations by PBL tutors.