Efficacy of ChatGPT in solving attitude, ethics, and communication case scenario used for competency-based medical...

Competency-based medical education (CBME) is a method of medical training that focuses on developing learners' competencies rather than simply assessing their knowledge and skills. Attitude, ethics, and communication (AETCOM) are important components of CBME, and the use of artificial intelligence (AI) tools such as ChatGPT for CBME has not been studied. Hence, we aimed to assess the capability of ChatGPT in solving AETCOM case scenarios used for CBME in India.

A total of 11 case scenarios were developed based on the AETCOM competencies. The scenarios were presented to ChatGPT, and the responses generated by ChatGPT were evaluated by three independent experts by awarding score ranging from 0 to 5. The scores were compared with a predefined score of 2.5 (50% accuracy) and 4 (80% accuracy) of a one-sample median test. Scores among the three raters were compared by the Kruskal-Wallis H test. The inter-rater reliability of the evaluations was assessed using the intraclass correlation coefficient (ICC).

The mean score of solution provided by ChatGPT was 3.88 ± 0.47 (out of 5), indicating an accuracy of approximately 78%. The responses evaluated by three raters were similar (Kruskal-Wallis H P value 0.51), and the ICC value was 0.796, which indicates a relatively high level of agreement among the raters.

ChatGPT shows moderate capability in solving AETCOM case scenarios used for CBME in India. The inter-rater reliability of the evaluations suggests that ChatGPT's responses were consistent and reliable. Further studies are needed to explore the potential of ChatGPT and other AI tools in CBME and to determine the optimal use of these tools in medical education.