Related Experiment Videos
Enhancing reliability in portfolio assessment: 'shaping' the portfolio
John Pitts1, Colin Coles, Peter Thomas
1NHSE (Wessex Deanery) and King Alfred's College, Winchester, UK.
Medical Teacher
|July 6, 2002
Summary
Assessing portfolios for general practice trainer courses showed moderate individual assessor reliability but poor inter-rater reliability. Portfolios are useful learning resources but problematic as summative assessment tools.
Area of Science:
- Medical Education
- Professional Development
- Assessment in Healthcare
Background:
- Portfolios are increasingly used in professional development.
- Assessing the reliability of portfolio-based assessments is crucial for their validity.
- General practice trainer education requires robust assessment methods.
Purpose of the Study:
- To evaluate the reliability of portfolio assessments for prospective general practice trainers.
- To determine if a reflective practice framework enhances assessment reliability.
- To assess both individual assessor consistency and inter-rater agreement.
Main Methods:
- A cohort of participants in a general practice trainer course assembled portfolios.
- A reflective practice model framework was provided to participants.
- A panel of assessors evaluated portfolio components and overall performance, with reliability analysis.
Main Results:
- Individual assessor reliability was moderate.
- Inter-rater reliability was insufficient for safe summative judgments.
- Agreement on the use of the reflective practice framework was poor among assessors.
- Reliability levels were comparable to previous subjective assessments.
Conclusions:
- Portfolios can serve as valuable learning resources.
- The use of portfolios as summative assessment tools is problematic due to reliability issues.
- Further research is needed to improve the reliability of portfolio assessment in professional development.