Related Experiment Video
Updated: Dec 6, 2025

Development of a Virtual Reality Assessment of Everyday Living Skills
Published on: April 23, 2014
Development, content validation, and reliability of the Assessment of Real-World Observational Studies (ArRoWS)
Briana Coles1, Freya Tyrer1, Humaira Hussein2
1Leicester Real World Evidence Unit, University of Leicester, Leicester, UK.
Purpose:
The objective was to develop and test a pragmatic critical appraisal tool, the Assessment of Real-World Observational Studies (ArRoWS), to quickly and easily assess the quality of real-world evidence studies using electronic health records.
Methods:
The initial ArRoWS tool was developed by identifying items frequently found in existing validated assessment instruments and adapting these items to specifically assess real-world evidence studies. The tool was revised based on recommendations from an expert panel of 14 senior academic individuals specializing in epidemiology and content validity was measured. During March 2018-January 2019, 47 large, observational studies related to cardiometabolic medicine were identified through a search algorithm and assessed by three pairs of raters using the ArRoWS tool.
Results:
The final version of the ArRoWS had 16 items including nine core items and seven study design-specific items with item-specific content validity indexes ranging from 0.64 to 1.00. The scale-level content validity index of the ArRoWS appraisal tool was 0.91. When the ArRoWS tool was pilot tested, the observed agreement between assessor pairs on whether the study provided high-quality real-world evidence was 85.7%, 68.8%, and 58.8%. The prevalence adjusted bias-adjusted kappa for the assessor pairs was 0.71, 0.38, and 0.18.
Conclusion:
The ArRoWS is a simple tool to standardize the assessment of real-world evidence studies.
Related Concept Videos
Observational Studies
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
Reliability and Validity
Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...
Data Validation
Key parameters for method validation include:
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Naturalistic Observations

