Related Experiment Video
Updated: Aug 12, 2026
10:26
Problem-Solving Before Instruction (PS-I): A Protocol for Assessment and Intervention in Students with Different Abilities
Published on: September 11, 2021
Enhancing reliability in portfolio assessment: 'shaping' the portfolio
John Pitts1, Colin Coles, Peter Thomas
1NHSE (Wessex Deanery) and King Alfred's College, Winchester, UK.
Medical Teacher
|July 6, 2002
Summary
Assessing portfolios for general practice trainer courses showed moderate individual assessor reliability but poor inter-rater reliability. Portfolios are useful learning resources but problematic as summative assessment tools.
Area of Science:
- Medical Education
- Professional Development
- Assessment in Healthcare
Background:
- Portfolios are increasingly used in professional development.
- Assessing the reliability of portfolio-based assessments is crucial for their validity.
- General practice trainer education requires robust assessment methods.
Purpose of the Study:
- To evaluate the reliability of portfolio assessments for prospective general practice trainers.
- To determine if a reflective practice framework enhances assessment reliability.
- To assess both individual assessor consistency and inter-rater agreement.
Main Methods:
- A cohort of participants in a general practice trainer course assembled portfolios.
- A reflective practice model framework was provided to participants.
- A panel of assessors evaluated portfolio components and overall performance, with reliability analysis.
Main Results:
- Individual assessor reliability was moderate.
- Inter-rater reliability was insufficient for safe summative judgments.
- Agreement on the use of the reflective practice framework was poor among assessors.
- Reliability levels were comparable to previous subjective assessments.
Conclusions:
- Portfolios can serve as valuable learning resources.
- The use of portfolios as summative assessment tools is problematic due to reliability issues.
- Further research is needed to improve the reliability of portfolio assessment in professional development.
Related Concept Videos
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
03:00Self-Evaluation: Self-Enhancement and Self-Verification
Social psychologists have documented that feeling good about ourselves and maintaining positive self-esteem is a powerful motivator of human behavior (Tavris & Aronson, 2008). In the United States, members of the predominant culture typically think very highly of themselves and view themselves as good people who are above average on many desirable traits (Ehrlinger, Gilovich, & Ross, 2005). Often, our behavior, attitudes, and beliefs are affected when we experience a threat to our...
02:13The Representativeness Heuristic
The representative heuristic describes a biased way of thinking, in which you unintentionally stereotype someone or something. For example, you may assume that your professors spend their free time reading books and engaging in intellectual conversation, because the idea of them spending their time playing volleyball or visiting an amusement park does not fit in with your stereotypes of professors.
01:29Self-Evaluation Maintenance Model
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
01:27Protecting Self-Esteem
Self-esteem, a central component of psychological well-being, is actively maintained through various cognitive and behavioral strategies. Individuals employ specific mechanisms to preserve a positive self-concept and mitigate threats to their self-worth, particularly in contexts involving social evaluation or personal feedback. Four primary techniques are commonly used to sustain self-esteem.Manipulating AppraisalsOne prominent strategy involves manipulating appraisals from others. Individuals...
01:27Halo Effect
The halo effect is a cognitive bias in which an individual's overall impression influences judgments about their specific traits. This psychological phenomenon leads people to associate positive characteristics with those they perceive as generally good and negative characteristics with those they view as bad. This effect is particularly influential in social perception, professional evaluations, and decision-making processes.The Psychological Basis of the Halo EffectThe halo effect is rooted...