Improving reliability of a residency interview process.

Area of Science:

• Pharmacy Education
• Psychometrics
• Human Resources in Healthcare

Background:

• Pharmacy residency interviews are crucial for candidate selection.
• Existing evaluation forms may lack reliability and discrimination.
• Improving the interview process enhances the quality of resident selection.

Purpose of the Study:

• To enhance the reliability and discrimination of a pharmacy resident interview evaluation form.
• To improve the overall reliability of the pharmacy residency interview process.
• To minimize the impact of content specificity on interview outcomes.

Main Methods:

• Phase 1: Employed Many-Facet Rasch Measurement (MFRM) to refine an existing evaluation form.
• Phase 2: Utilized the modified form with interviewer pairs across four interview stations.
• Phase 3: Conducted individual evaluations with eight interviewers in one-on-one settings.

Main Results:

• The optimized form achieved a reliability of 0.98 and separated applicants into six distinct groups.
• Candidate variability was the primary source of variation, followed by content specificity.
• Phase 2 g-coefficient was 0.787, and confirmatory Phase 3 g-coefficient was 0.922.
• Increased interview stations improved process reliability and significantly reduced content specificity impact.

Conclusions:

• Developed a more reliable and discriminating evaluation form for pharmacy resident interviews.
• Established a process that effectively mitigates the influence of content specificity.
• The refined evaluation tool and process contribute to more objective candidate assessment.