Related Experiment Video
Updated: Mar 7, 2026
Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties
Published on: September 27, 2020
Gestalt assessment of online educational resources may not be sufficiently reliable and consistent
Keeth Krishnan1, Brent Thoma2, N Seth Trueger3
1University of Toronto, Toronto, Ontario, Canada.
Purpose:
Online open educational resources are increasingly used in medical education, particularly blogs and podcasts. However, it is unclear whether these resources can be adequately appraised by end-users. Our goal was to determine whether gestalt-based recommendations are sufficient for emergency medicine trainees and attending physicians to reliably recommend online educational resources to others.
Methods:
Raters (33 trainees and 21 attendings in emergency medicine from North America) were asked to rate 40 blog posts according to whether, based on their gestalt, they would recommend the resource to (1) a trainee or (2) an attending physician. The ratings' reliability was assessed using intraclass correlation coefficients (ICC). Associations between groups' mean scores were assessed using Pearson's r. A repeated measures analysis of variance (RM-ANOVA) was completed to determine the effect of the level of training on gestalt recommendation scale (i. e. trainee vs. attending).
Results:
Trainees demonstrated poor reliability when recommending resources for other trainees (ICC = 0.21, 95% CI 0.13-0.39) and attendings (ICC = 0.16, 95% CI = 0.09-0.30). Similarly, attendings had poor reliability when recommending resources for trainees (ICC = 0.27, 95% CI 0.18-0.41) and other attendings (ICC = 0.22, 95% CI 0.14-0.35). There were moderate correlations between the mean scores for each blog post when either trainees or attendings considered the same target audience. The RM-ANOVA also corroborated that there is a main effect of the proposed target audience on the ratings by both trainees and attendings.
Conclusions:
A gestalt-based rating system is not sufficiently reliable when recommending online educational resources to trainees and attendings. Trainees' gestalt ratings for recommending resources for both groups were especially unreliable. Our findings suggest the need for structured rating systems to rate online educational resources.
Related Concept Videos
01:24Nursing Process for Patient and Caregiver Teaching I: Assessment and Diagnosis
It is critical to determine the patient's learning needs during the assessment. Determination of learning needs compounds data...
01:20Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
01:29Nursing Assessment
The nurse collects all aspects of the patient's health in the initial assessment, establishing priorities for ongoing focused assessments...
01:29Assessment of the Gastrointestinal System II: Health Perception Pattern
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
01:29Assessment of Ventilation II: Respiratory Depth and Rhythm
Respiratory depth measures the volume of air inhaled or exhaled during a breath. It can vary from shallow to deep and typically remains consistent when a person is at rest or asleep. Occasionally, individuals will automatically inhale deeply, known as sighing, which inflates the lungs with more air than normal breathing.
To assess respiratory depth, observe the degree of chest excursion or movement:
01:30Assessment of Airway, Skin Color, and Use of Accessory Muscles
Introduction
The initial evaluation of a patient's respiratory system...