Related Experiment Video
Updated: May 25, 2026
10:39
Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
Developing expert-derived rating standards for the peer assessment of lectures
Lori R Newman1, Dara D Brodsky, David H Roberts
1Shapiro Institute for Education and Research at Harvard Medical School, Boston, Massachusetts 02215, USA. lnewman@bidmc.harvard.edu
Summary
Developing reliable teaching peer review requires expert-defined standards. This study established precise lecture rating standards, identifying key behaviors and proficiency levels for effective peer rater training in academic settings.
Area of Science:
- Medical Education
- Academic Development
Background:
- Effective peer review of teaching necessitates accurate measurement of teaching behaviors by trained peer raters.
- Peer rater training must be grounded in expert-derived standards for teaching performance.
Purpose of the Study:
- To establish precise lecture rating standards for peer rater training.
- To enhance the credibility and reliability of teaching evaluations within an academic institution.
Main Methods:
- A panel of experts utilized a consensus-building model to rate 40 lectures from 2008 to 2010.
- Experts refined an existing instrument by adding precise behavioral descriptors for 11 criteria.
- Reliability of the developed standards was assessed through independent scoring of six lectures.
Main Results:
- High interrater reliability was achieved, with intraclass correlation coefficients ranging from 0.75 to 0.96.
- Strong positive associations (r = 0.752-0.886) were found between 10 of 11 criteria and overall performance.
- No significant differences in rater leniency or stringency were observed.
Conclusions:
- Two key themes, content and style, emerged from the analysis of lecturing variables.
- Recommendations for developing expert standards include interdisciplinary collaboration, consideration of diverse perspectives, and documentation of key behaviors.
- The established standards provide a foundation for future peer rater training programs.
Related Concept Videos
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
03:00Self-Evaluation: Self-Enhancement and Self-Verification
Social psychologists have documented that feeling good about ourselves and maintaining positive self-esteem is a powerful motivator of human behavior (Tavris & Aronson, 2008). In the United States, members of the predominant culture typically think very highly of themselves and view themselves as good people who are above average on many desirable traits (Ehrlinger, Gilovich, & Ross, 2005). Often, our behavior, attitudes, and beliefs are affected when we experience a threat to our...
01:10Review and Preview
In statistics, several tools are used to interpret the data. Measures of central tendency represent the characteristics of the data, such as mean, median, and mode. Additionally, measures of variance like standard deviation and range are used to find the spread of data from the mean. Relative standing measures the distance between data locations. Commonly used measures of relative standings are percentile, z score, and quartiles.
Percentiles are a type of fractile that partition data into...
Percentiles are a type of fractile that partition data into...
03:08Rate-Determining Steps
Relating Reaction Mechanisms
In a multistep reaction mechanism, one of the elementary steps progresses significantly slower than the others. This slowest step is called the rate-limiting step (or rate-determining step). A reaction cannot proceed faster than its slowest step, and hence, the rate-determining step limits the overall reaction rate.
The concept of rate-determining step can be understood from the analogy of a 4-lane freeway with a short-stretch of traffic-bottleneck caused due to...
In a multistep reaction mechanism, one of the elementary steps progresses significantly slower than the others. This slowest step is called the rate-limiting step (or rate-determining step). A reaction cannot proceed faster than its slowest step, and hence, the rate-determining step limits the overall reaction rate.
The concept of rate-determining step can be understood from the analogy of a 4-lane freeway with a short-stretch of traffic-bottleneck caused due to...
01:20Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
Evaluation of the teaching process enables the nurse to determine if the patient's learning needs were met and if training was effective. If the expected outcomes are not met, the care plan is revised, and additional education or reinforcement is provided. Nurses can ask questions after the session or obtain feedback to assess the patient's understanding of the topic.
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning, patient...
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning, patient...
01:37Social Loafing
Another way in which a group presence can affect performance is social loafing—the exertion of less effort by a person working together with a group. Social loafing occurs when our individual performance cannot be evaluated separately from the group. Thus, group performance declines on easy tasks (Karau & Williams, 1993). Essentially individual group members loaf and let other group members pick up the slack. Because each individual’s efforts cannot be evaluated, individuals become less...