Related Experiment Video
Updated: Aug 4, 2026
13:44
Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques
Published on: December 9, 2022
Interrater concordance for faculty grading of student performances in a problem-based learning course
N F Schor1, P Troen, S L Kanter
1Department of Pediatrics, University of Pittsburgh School of Medicine, Pennsylvania, USA. nfschor@vms.cis.pitt.edu
Summary
Faculty can achieve consistent grading in problem-based learning (PBL) courses when clear performance criteria are provided. This study demonstrates reliable interrater concordance in PBL, ensuring fair student assessment.
Area of Science:
- Medical Education
- Pedagogy
- Assessment
Background:
- Problem-based learning (PBL) is a student-centered educational approach.
- Assessing student performance in PBL can be challenging due to its collaborative and self-directed nature.
- Ensuring consistent and non-idiosyncratic grading is crucial for the validity of PBL.
Purpose of the Study:
- To evaluate the possibility of achieving consistent, non-idiosyncratic faculty grading in a problem-based learning (PBL) course.
- To assess interrater reliability among faculty facilitators in a PBL setting.
Main Methods:
- The study involved faculty facilitators grading 16 groups of students in the Integrated Case Studies and Medical Decision Making (ICS) course.
- Grades were assigned for overall performance and seven specific categories.
- Interclass correlation coefficients (ICC) were calculated to measure interrater concordance.
Main Results:
- Statistically significant concordance was achieved in aggregate scores for 100% of groups and overall scores for 90% of groups.
- Concordance was met in at least 75% of groups across six of seven performance categories.
- Practically significant concordance was observed in 83% of aggregate scores.
Conclusions:
- Consistent, non-idiosyncratic grading is achievable in PBL courses.
- Establishing specific criteria for performance evaluation is key to reliable grading.
- The findings support the validity of assessment in PBL environments.
Related Concept Videos
02:16Surveys
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
01:10Review and Preview
In statistics, several tools are used to interpret the data. Measures of central tendency represent the characteristics of the data, such as mean, median, and mode. Additionally, measures of variance like standard deviation and range are used to find the spread of data from the mean. Relative standing measures the distance between data locations. Commonly used measures of relative standings are percentile, z score, and quartiles.
Percentiles are a type of fractile that partition data into...
Percentiles are a type of fractile that partition data into...
01:10Bonferroni Test
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
01:09Comparing Experimental Results: Student's t-Test
The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
01:15Types of Aggregate Grading
Aggregate grading is crucial in economically obtaining a concrete mix with adequate strength, reasonable workability, and minimal segregation. There are four types of aggregate gradation: well-graded, uniformly (or one-sized) graded, gap-graded, and open-graded.
Well-graded aggregates include a complete range of necessary size fractions that fit together to create a dense matrix with minimal voids, represented by a smooth, continuous gradation curve. This type of grading ensures good...
Well-graded aggregates include a complete range of necessary size fractions that fit together to create a dense matrix with minimal voids, represented by a smooth, continuous gradation curve. This type of grading ensures good...
01:20Kendall's Coefficient of Concordance
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects or...