Related Experiment Video
Updated: Jan 21, 2026
Hydra, a Computer-Based Platform for Aiding Clinicians in Cardiovascular Analysis and Diagnosis
Published on: September 26, 2018
Feasibility and preliminary validity evidence for remote video-based assessment of clinicians in a global health
Katherine A Smith1, Segolame Setlhare2, Allan DeCaen3
1Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States of America.
Insights
Serious childhood illnesses contribute to child mortality globally. A pilot study in Botswana found remote simulation assessments for healthcare providers showed reliability and feasibility for some skills, though not all scenarios performed well.
Area of Science:
- Global Health
- Medical Education
- Pediatric Emergency Medicine
Background:
- Serious childhood illnesses (SCI) like severe pneumonia, dehydration, sepsis, and malaria are leading causes of preventable child deaths worldwide.
- Inadequate recognition and treatment of SCI significantly impact child mortality rates in Botswana.
- Validated skills assessments for healthcare providers managing SCI are lacking in low and middle-income countries.
Purpose of the Study:
- To establish preliminary inter-rater reliability, validity evidence, and feasibility of an assessment tool for providers caring for SCI.
- To utilize simulated patients and remote video capture in community clinic settings in Botswana for skills assessment.
- To address the gap in validated assessment methods for SCI management in resource-limited settings.
Main Methods:
- A pilot study was conducted in primary care clinics in Kweneng, Botswana.
- Four SCI scenarios were developed using a modified Delphi technique and implemented with simulated patients.
- Sessions were video captured for independent review, with response process and internal structure analyzed using intra-class correlation (ICC) and Fleiss' Kappa; feasibility was assessed via a structured log for remote video capture.
Main Results:
- Eleven healthcare providers participated in the pilot study.
- Scenarios for Lower Airway Obstruction (ICC = 0.925) and Hypovolemic Shock from Severe Dehydration (ICC = 0.892) demonstrated excellent inter-rater reliability.
- Specific tasks like oxygen therapy (0.707), arranging transport (0.706), and fluid administration (0.701) showed substantial reliability, while other scenarios had poor reliability.
Conclusions:
- The initial development of the assessment tool shows promise, meeting many criteria for validity evidence.
- While some scenarios and tasks exhibited excellent reliability, others may require refinement, potentially due to manikin design or implementation challenges.
- Remote simulation assessment of certain clinical skills by healthcare providers in global health settings is both reliable and feasible.
Background:
Serious childhood illnesses (SCI), defined as severe pneumonia, severe dehydration, sepsis, and severe malaria, remain major contributors to amenable child mortality worldwide. Inadequate recognition and treatment of SCI are factors that impact child mortality in Botswana. Skills assessments of providers caring for SCI have not been validated in low and middle-income countries.
Objective:
To establish preliminary inter-rater reliability, validity evidence, and feasibility for an assessment of providers who care for SCI using simulated patients and remote video capture in community clinic settings in Botswana.
Methods:
This was a pilot study. Four scenarios were developed via a modified Delphi technique and implemented at primary care clinics in Kweneng, Botswana. Sessions were video captured and independently reviewed. Response process and internal structure analysis utilized intra-class correlation (ICC) and Fleiss' Kappa. A structured log was utilized for feasibility of remote video capture.
Results:
Eleven subjects participated. Scenarios of Lower Airway Obstruction (ICC = 0.925, 95%CI 0.695-0.998) and Hypovolemic Shock from Severe Dehydration (ICC = 0.892, 95%CI 0.596-0.997) produced excellent ICC among raters while Lower Respiratory Tract Infection (LRTI, ICC = 0, 95%CI -0.034-0.97) and LRTI + Distributive Shock from Sepsis (0.365, 95%CI -0.025-0.967) were poor. Oxygen therapy (0.707), arranging transport (0.706), and fluid administration (0.701) demonstrated substantial task reliability.
Conclusions:
Initial development of an assessment tool demonstrates many, but not all, criteria for validity evidence. Some scenarios and tasks demonstrate excellent reliability among raters, but others may be limited by manikin design and study implementation. Remote simulation assessment of some skills by clinic-based providers in global health settings is reliable and feasible.
Related Concept Videos
02:55The Evidence for Evolution
01:29Assessment of the Gastrointestinal System II: Health Perception Pattern
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
01:21Nursing Assessment of the Genitourinary System I: Health History
01:50Global Climate Change
01:29Reliability and Validity
01:28Physical Assessment of the Respiratory Tract I: Health History
Subjective Data
Subjective data provides vital information about the patient's health history and symptoms. This data is typically collected through interviews in which patients describe their experiences, symptoms, and concerns.
Health history and...