Evaluation of a Computer-Based Morphological Analysis Method for Free-Text Responses in the General Medicine...

Area of Science:

• Medical Education
• Health Informatics

Background:

• The General Medicine In-Training Examination (GM-ITE) in Japan assesses clinical knowledge for postgraduate residents.
• GM-ITE 2021 incorporated medical safety questions, including case diagnosis and presentation skills using video-based patient simulations.
• Scoring free-text responses poses a significant human resource challenge, potentially limiting examination implementation.

Purpose of the Study:

• To compare the efficacy of human versus machine scoring for free-text responses in the GM-ITE.
• To qualitatively analyze discrepancies between human and machine-generated scores to validate machine scoring.

Main Methods:

• Utilized voluntary free-text responses from residents answering GM-ITE video-based questions simulating a pulmonary embolism case.
• Human scores were derived from two independent scorers; machine scores used morphological analysis and word-matching against correct answers.
• Analyzed 39 valid responses from 104 collected cases.

Main Results:

• Discrepancies between human and machine scoring occurred in 7.2% of questions (14 out of 194).
• Identified specific areas for machine scoring improvement, such as maintaining comprehensive word lists and dictionaries.
• Acknowledged that some discrepancies were attributable to human scoring errors.

Conclusions:

• Machine scoring demonstrates comparable accuracy to human scoring for GM-ITE free-text responses.
• Implementation requires a straightforward program and calibration, offering a cost-effective solution for scoring.
• Automated scoring can enhance the efficiency and scalability of medical in-training examinations.