Related Experiment Video
Updated: Jan 3, 2026
Use of a Video Scoring Anchor for Rapid Serial Assessment of Social Communication in Toddlers
Published on: March 14, 2018
Inter-rater reliability in a bespoke scoring tool: the Paediatric Observation Priority Score
Adam Bonfield1, Damian Roland2,3
1College of Medicine Biological Sciences and Psychology, University of Leicester, Leicester, UK ab798@le.ac.uk.
Insights
Inter-rater reliability (IRR) for the Paediatric Observation Priority Score (POPS) was assessed. The study found moderate to excellent IRR among healthcare professionals using POPS for paediatric patients.
Area of Science:
- Pediatric Emergency Medicine
- Clinical Assessment Tools
- Healthcare Quality Improvement
Background:
- Inter-rater reliability (IRR) is crucial for the consistent application of clinical scoring systems.
- The Paediatric Observation Priority Score (POPS) is a bespoke tool for assessing paediatric deterioration in emergency departments (EDs).
- Limited data exists on the IRR of POPS, hindering its widespread adoption and validation.
Purpose of the Study:
- To determine the inter-rater reliability (IRR) of the overall Paediatric Observation Priority Score (POPS).
- To investigate the IRR of individual parameters within the POPS.
- To assess the consistency of POPS scoring among diverse healthcare professionals in a pediatric ED setting.
Main Methods:
- An experimental, single-centre study conducted in a UK children's ED.
- 45 healthcare professionals (nurses, healthcare assistants, doctors) evaluated 11 prerecorded paediatric video assessments.
- Intraclass correlation coefficient (ICC) was used for overall POPS reliability, and Fleiss kappa for individual parameters.
Main Results:
- The overall ICC for POPS was 0.84 (95% CI 0.71 to 0.95), indicating excellent reliability.
- Fleiss kappa values for individual parameters ranged from 0.44 to 1, demonstrating moderate to excellent reliability across most parameters.
- Parameters like oxygen saturation, response, medical history, heart rate, and temperature showed excellent IRR (kappa ≥ 0.76).
Conclusions:
- The POPS demonstrates statistically moderate to excellent inter-rater reliability.
- The scoring system is reliable when used by healthcare professionals with varying experience levels.
- POPS is a dependable tool for assessing clinical presentations and identifying deterioration in children within the ED setting.
Objective:
Inter-rater reliability (IRR) is rarely determined for scoring systems used to recognise deterioration in children. Thus, the primary objective of this study was to determine the IRR of the Paediatric Observation Priority Score (POPS), a bespoke paediatric scoring system for ED use. The IRR of both the overall POPS and its individual parameters are to be investigated.
Methods:
This is an experimental, single-centre study based in the Leicester children's ED, England. A purposive sample of nursing, healthcare assistants and doctors were recruited from October 2017 to December 2017. Eleven prerecorded video assessments were shown to 45 participants asked to generate a POPS for each child. The participants were blinded to each other's scores and the triage POPS. IRR among the participants POPS was measured using the intraclass correlation coefficient (ICC). Fleiss kappa was used to determine the IRR of individual parameters.
Results:
Overall, the ICC across all assessments by staff was 0.84 (95% CI 0.71 to 0.95). Fleiss kappa for parameters ranged from 0.87 to 1 for oxygen saturation, 0.48 to 0.91 for work of breathing, 0.55 to 1 for response, 0.46 to 0.87 for gut feeling, 0.53 to 1 for medical history, 0.76 to 1 for heart rate, 0.44 to 0.96 for respiratory rate and 0.51 to 1 for temperature.
Conclusions:
This study suggests there is statistically moderate to excellent IRR of the POPS when assessing a variety of clinical presentations between different healthcare professionals with a range of experience.
More Related Videos
Related Concept Videos
01:20Assessment of Ventilation I: Respiratory Rate
A Ventilation assessment is critical for monitoring a patient's health status. Respiration, one of the most accessible vital signs, provides insights into the function of numerous body systems and can indicate serious health issues, such as brainstem injuries from head trauma.
Critical Guidelines for Assessing Ventilation:
01:21Methods of Documentation III: PIE