Ensuring the Quality of Evidence: Using the Best Design to Answer Health IT Questions

The quality of logic in a research design determines the value of the results and our confidence regarding the validity of the findings. The purpose of this contribution is to review the principles of research design as they apply to research and evaluation in health IT. We review the architecture of research design, the definitions of cause, sources of bias and confounds, and the importance of measurement as related to the various types of health IT questions. The goal is to provide practitioners a roadmap for making decisions for their own specific study. The contribution is organized around the Threats to Validity taxonomy and explains how different design models address these threats through the use of blocking, factorial design, control groups and time series analysis. The contribution discusses randomized experiments, and includes regression discontinuity designs and various quasi-experimental designs with a special emphasis on how to improve pre/post designs. At the end, general recommendations are provided for improving weaker designs and general research procedures.