Using cloud-based mobile technology for assessment of competencies among medical students.

Area of Science:

• Medical Education
• Clinical Assessment
• Health Professions Education

Background:

• Direct observation of medical student competency in clinical settings is difficult, leading to variable evaluations.
• Barriers include a lack of effective tools and transparent standards for assessing clinical skills.

Purpose of the Study:

• To develop and evaluate a web-based system, Just in Time Medicine (JIT), for direct observation and assessment of medical student clinical competency.
• To improve the reliability and validity of clinical skills assessments and provide timely feedback.

Main Methods:

• Developed a web-based content management system (JIT) for creating customizable assessment tools and capturing learner progress.
• Implemented a JIT-enabled clinical evaluation exercise (CEX) for third-year internal medicine students over two years.
• Assessed inter-rater reliability and validity using videotaped encounters and correlation with Objective Structured Clinical Examinations (OSCE).

Main Results:

• A total of 3567 CEXs were completed by 516 observers.
• Observer satisfaction with the JIT system was high (92%).
• Inter-rater reliability was 0.69, and CEX grades correlated with subsequent OSCE performance, indicating adequate validity.

Conclusions:

• The Just in Time Medicine (JIT) system is a feasible tool for capturing discrete clinical performance data.
• JIT enhances direct observation and assessment of medical student competency, with high user satisfaction.
• The system provides reliable and valid assessments, supporting performance-based student advancement.