Establishing Trust in Pharmaceutical Data with an Independent Verification and Validation Methodology

Our aim is to demonstrate a general-purpose data and knowledge validation approach that enables reproducible metrics for data and knowledge quality and safety. We researched widely accepted statistical process control methods from high-quality, high-safety industries and applied them to pharmacy prescription data being migrated between EHRs. Natural language medication instructions from prescriptions were independently categorized by two terminologists as a first step toward encoding those medication instructions using standardized terminology. Overall, the weighted average of medication instructions that were matched by reviewers was 43%, with strong agreement between reviewers for short instructions (K=0.82) and long instructions (K=0.85), and moderate agreement for medium instructions (K=0.61). Category definitions will be refined in future work to mitigate discrepancies. We recommend incorporating appropriate statistical tests, such as evaluating inter-rater and intra-rater reliability and bivariate comparison of reviewer agreement over an adequate statistical sample, when developing benchmarks for health data and knowledge quality and safety.