Determining Component Weights in a Communications Assessment Using Judgmental Policy Capturing

Area of Science:

• Medical Education
• Psychometrics
• Healthcare Communication Assessment

Background:

• Accurate weighting of performance assessment components is crucial in medical education.
• Judgmental policy capturing (JPC) offers a statistical method to understand expert decision-making strategies.
• The feasibility of JPC for defining weights in complex assessments needed investigation.

Purpose of the Study:

• To investigate the feasibility of using judgmental policy capturing (JPC) for determining appropriate weights for performance assessment components in medical education.
• To establish statistically derived weights for communication skills assessed in medical students.

Main Methods:

• Iterative judgmental policy capturing (JPC) was employed.
• A panel of four communication skill experts evaluated six core communication skill scores.
• Scores were derived from a communications objective structured clinical examination (OSCE) for medical students.

Main Results:

• Experts assigned varying importance to different communication skills.
• Information management received the lowest weight (8.5%).
• Rapport building (15.8%), agenda setting (15.4%), and addressing feelings (14.1%) received moderate, similar weights.
• Active listening (20.1%) and reaching common ground (25.5%) were weighted highest.

Conclusions:

• Judgmental policy capturing (JPC) is an effective method for determining weights in complex clinical assessment components.
• The derived weights can differ significantly from initial assumptions or traditional scoring methods.
• This approach provides a data-driven basis for refining assessment tools in medical education.