Related Experiment Video
Updated: Mar 5, 2026
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
Using a stakeholder-engaged approach to develop and validate electronic clinical quality measures
Jill Boylston Herndon1, Krishna Aravamudhan2, Ronald L Stephenson3
1Key Analytics and Consulting, LLC, Gainesville, Florida, USA (Dr Herndon was with the Department of Health Outcomes and Policy, University of Florida College of Medicine, Gainesville, Florida, USA, when this study was conducted).
Objective:
To describe the stakeholder-engaged processes used to develop, specify, and validate 2 oral health care electronic clinical quality measures.
Materials And Methods:
A broad range of stakeholders were engaged from conception through testing to develop measures and test feasibility, reliability, and validity following National Quality Forum guidance. We assessed data element feasibility through semistructured interviews with key stakeholders using a National Quality Forum-recommended scorecard. We created test datasets of synthetic patients to test measure implementation feasibility and reliability within and across electronic health record (EHR) systems. We validated implementation with automated reporting of EHR clinical data against manual record reviews, using the kappa statistic.
Results:
A stakeholder workgroup was formed and guided all development and testing processes. All critical data elements passed feasibility testing. Four test datasets, representing 577 synthetic patients, were developed and implemented within EHR vendors' software, demonstrating measure implementation feasibility. Measure reliability and validity were established through implementation at clinical practice sites, with kappa statistic values in the "almost perfect" agreement range of 0.80-0.99 for all but 1 measure component, which demonstrated "substantial" agreement. The 2 validated measures were published in the United States Health Information Knowledgebase.
Conclusion:
The stakeholder-engaged processes used in this study facilitated a successful measure development and testing cycle. Engaging stakeholders early and throughout development and testing promotes early identification of and attention to potential threats to feasibility, reliability, and validity, thereby averting significant resource investments that are unlikely to be fruitful.
More Related Videos
10:39Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
06:28E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy
Published on: August 1, 2019
Related Concept Videos
01:23Methods of Documentation V: CBE
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
01:30Methods of Documentation VII: EMR
01:30Health Information Technology and Healthcare Information System
Health Information Technology, commonly called HIT, integrates advanced information systems and technology in healthcare settings. Its primary functions include:
01:29Clinically Relevant Drug Product Specifications: Methods of Establishment
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
01:19Quality Assurance