Validation of an AI-Assisted Framework for Systematic Bias Assessment in Observational Studies

The rapid expansion of medical literature has led to variability and contradictions in study findings, making it increasingly difficult to distinguish meaningful signals from noise. Much of this variability arises from methodological limitations, including confounding, selection bias, and reverse causation. Although artificial intelligence (AI)-assisted tools exist for risk-of-bias assessment, most are designed for systematic reviews and are not tailored to identifying epidemiologic biases in observational studies. Structured, scalable approaches are needed to evaluate validity in real-world evidence research.

To develop and validate EpiVise, an AI-assisted, expert-informed, rule-based framework for identifying major sources of bias in pharmacoepidemiologic studies and to assess its agreement with expert epidemiologist evaluations.

Recently published pharmacoepidemiologic studies from high-impact journals (post- July 2025) were independently evaluated by EpiVise and two expert epidemiologists across predefined bias domains, including measured confounding, confounding by indication, selection bias, immortal time bias, and disease latency bias. Agreement was assessed using weighted kappa statistics. In addition, synthetic study scenarios with predefined embedded biases were constructed to evaluate framework performance under controlled conditions.

Among published studies (10 studies; 60 ratings), agreement between EpiVise and expert assessments was substantial (weighted κ = 0.75; 95% confidence interval [CI], 0.63-0.87). Twelve ratings (20.0%) were discordant, all limited to adjacent categories. In synthetic scenarios (10 studies; 50 ratings), agreement was also substantial, with 40 of 50 ratings concordant (80.0%) and a weighted κ of 0.72 (95% CI, 0.61-0.83).

EpiVise demonstrated substantial agreement with expert epidemiologist assessments in both published and synthetic study evaluations. As a scalable and reproducible framework for identifying common epidemiologic biases, EpiVise may enhance evidence appraisal, peer review, and clinical or regulatory decision-making. Further validation across broader study designs and therapeutic areas is warranted.