Challenges in assessing ethical training in contemporary medicine: Psychometric evidence from two retention exams

To evaluate the psychometric performance of two retention multiple-choice exams focused on practical ethics, deontology, and professional conduct administered to residents after interactive, in-person courses.

We analyzed item difficulty (p), corrected point-biserial correlation (r_pb*), and internal consistency (KR-20) under Classical Test Theory. Thematic categories were assigned to each item based on course blueprints and question stems. Two cohorts (Exam I on June 4, 2025; Exam II on September 8, 2025) completed 20-item tests each. Analyses were performed in Excel with verification in Python.

Exam I showed KR-20 = 0.61 with a high overall proportion correct (mean p ≈ 0.94), limiting score variance and discrimination (mean r_pb*≈0.28). Exam II exhibited improved reliability (KR-20 = 0.83) and higher mean discrimination (mean r_pb*≈0.52). Although most items in both exams are considered to be easy, in the context of a retention test this pattern is also compatible with efficient incorporation of practical content after teaching, particularly in thematic domains more readily retained by residents. Ceiling effects therefore indicate not only reduced psychometric spread, but also probable high short-term retention in several practical topics; however, items with very high p values still have limited utility for distinguishing between stronger and weaker performers and for identifying content areas that may require reinforcement. Items framed as practical clinical or professional scenarios ‒ especially those involving autonomy and communication, documentation, civil/criminal liability, and relationships with health insurers ‒ showed the strongest discrimination. By contrast, digital ethics and advertising items were generally easy and less discriminative, possibly because these rules were already familiar, more directly memoizable, or presented in especially explicit formats during teaching.

Psychometric monitoring of short retention exams in practical ethics and deontology supports continuous quality improvement and favors contextualized, case-based items that better differentiate residents in the domain of applicable professional rules and routine ethical-regulatory decisions. Such assessments should be a routine part of ethics curriculum development.