Related Experiment Video
Updated: Sep 20, 2025
08:40
Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
Published on: June 12, 2019
7.6K
Assessing Interrater Reliability of a Faculty-Provided Feedback Rating Instrument
Daniel P Walsh1, Michael J Chen1, Lauren K Buhl1
1Department of Anesthesiology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA.
Summary
A new seven-item instrument reliably assesses faculty feedback quality in residency programs. This tool, when used by trained educators, can improve resident performance feedback and guide educational interventions.
Area of Science:
- Medical Education
- Healthcare Quality Improvement
Background:
- High-quality faculty feedback is crucial for resident development.
- Assessing faculty feedback reliability is necessary for targeted improvement.
Purpose of the Study:
- To pilot a feedback rating instrument for assessing interrater reliability.
- To evaluate the instrument's effectiveness after frame-of-reference training.
Main Methods:
- A seven-item feedback rating instrument was developed.
- Faculty educators underwent three frame-of-reference training workshops.
- Participants rated deidentified feedback comments, and interrater reliability was assessed.
Main Results:
- Moderate or greater interrater reliability was achieved on all seven rating categories.
- Gwet's agreement coefficient 1 and intraclass correlation were used for analysis.
- Reliable assessments of faculty feedback are possible with the trained instrument.
Conclusions:
- The piloted instrument demonstrates potential for reliable assessment of faculty feedback quality and utility.
- This tool can aid programs in evaluating feedback and the impact of interventions.
- Further validity evidence is needed to support its widespread use.
Related Concept Videos
02:16Surveys
15.4K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
15.4K
01:29Reliability and Validity
13.2K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.2K
01:21Friedman Two-way Analysis of Variance by Ranks
310
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
310
01:29Self-Evaluation Maintenance Model
9
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
9
01:22Self-Report Tests of Personality
472
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
472
00:54Ratio Level of Measurement
19.3K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
19.3K