Developing and Refining the Pregnancy Algorithm CErtaInty Tool (ACE-IT) for Validating Pregnancy Outcomes

Algorithm validation studies of health outcomes using real-world data are used to generate real-world evidence. We developed the Algorithm CErtaInty Tool (ACE-IT) to enable users to appraise whether a given real-world data-based algorithm is fit for purpose for a specific decision context, using the outcome of ischemic stroke as an exemplar. Our objective was to refine, develop, and tailor the ACE-IT to address whether validated algorithms for pregnancy and birth outcomes are fit for purpose for a specific decision context. We also tested the feasibility, acceptability, appropriateness, and content validity of the Pregnancy ACE-IT for the evaluation of an algorithm on pregnancy/birth outcomes. We engaged an expert panel of pharmacoepidemiologists and regulators to guide the development of the Pregnancy ACE-IT. The panel contributed to crafting a vignette on the selected pregnancy outcome of neural tube defects. Twenty-seven external raters used the Pregnancy ACE-IT to appraise whether a selected peer-reviewed algorithm to identify neural tube defects was fit for purpose and generalized as a target data source. Most raters found the Pregnancy ACE-IT acceptable and appropriate, although some did not find it easy to use. The Pregnancy ACE-IT represents a suitable, flexible, acceptable, and appropriate tool with relevant content for determining whether algorithms for pregnancy outcomes are fit for purpose for a specific decision context. Future studies to encourage broader acceptance and improve ease of use will require familiarizing users with the tool's operational characteristics and evaluating its suitability in the context of additional pregnancy/birth outcomes in different therapeutic areas.