Workflow for defining reference chemicals for assessing performance of in vitro assays

Instilling confidence in use of in vitro assays for predictive toxicology requires evaluation of assay performance. Performance is typically assessed using reference chemicals--compounds with defined activity against the test system target. However, developing reference chemical lists has historically been very resource-intensive. We developed a semi-automated process for selecting and annotating reference chemicals across many targets in a standardized format and demonstrate the workflow here. A series of required fields defines the potential reference chemical: the in vitro molecular target, pathway, or phenotype affected; and the chemical's mode (e.g. agonist, antagonist, inhibitor). Activity information was computationally extracted into a database from multiple public sources including non-curated scientific literature and curated chemical-biological databases, resulting in the identification of chemical activity in 2995 biological targets. Sample data from literature sources covering 54 molecular targets ranging from data-poor to data-rich was manually checked for accuracy. Precision rates were 82.7% from curated data sources and 39.5% from automated literature extraction. We applied the final reference chemical lists to evaluating performance of EPA's ToxCast program in vitro bioassays. The level of support, i.e. the number of independent reports in the database linking a chemical to a target, was found to strongly correlate with likelihood of positive results in the ToxCast assays, although individual assay performance had considerable variation. This overall approach allows rapid development of candidate reference chemical lists for a wide variety of targets that can facilitate performance evaluation of in vitro assays as a critical step in imparting confidence in alternative approaches.