Development and testing of a summative video-based e-examination in relation to an OSCE for measuring...

Objective: In the context of educating medical students, testing of competence in medical communication is carried out primarily with the Objective Structured Clinical Examination [1]. This makes it possible to assess practical performance, but it is resource-intensive and has a negative impact on test quality. The project "Digital test tool for measuring communication skills in medical studies" (digiRole) was funded by the BMBF (Federal Ministry of Education and Research) and its objective was to develop digital formats as electronic versions of an OSCE in order to test the communication competency of medical students. Such digital forms of examination should be cost-effective, be relevant to clinical practice and have high psychometric quality. In terms of content, the examination questions should incorporate factual and procedural knowledge as components of communication competency, although we assumed that procedural knowledge is more relevant than facutal knowledge to OSCE performance. This article describes the development and testing of a video-based, communication-related e-examination that is relevant to passing the test, which is the first milestone of the overall project. Methodology: We produced videos and related exam questions in the form of a situational judgement test [2] related to medical psychology and medical sociology, based on the educational content of a preclinical course on doctor-patient communication at the Mainz University Medical Center. In the summer semester of 2018, 226 students sat for this video-based single-choice e-examination (VSE). In the winter semester of 2018/2019, a different cohort of 192 students participated in the VSE as well as a tried-and-tested communication OSCE with five stations [3]. Results: The internal consistencies for the VSE in the summer semester of 2018 were α=.55, in the winter semester 2018/19 with α=.62 and for the OSCE with α=.60. There was a positive correlation between the performance of the students with the VSE and that with the OSCE (r=.21, p≤.01). Principal Axis Analyses did not reveal any dimensioning in terms of factual and procedural knowledge. In the evaluation, the majority of the students stated that the VSE was quite relevant to the practise of medical communication and were in favour of retaining this form of examination. Conclusion: The correlation between the VSE and the OSCE is relatively low, so that the VSE in this form is not a satisfactory predictor of an OSCE result. In terms of internal consistency, the VSE and the OSCE produced an almost identical result. It can also be assumed that the VSE can achieve a high degree of objectivity with the use of standardised video-based examinations as well as greater resource efficiency than OSCEs.