AI-assisted, human-in-the-loop pipeline for converting historical documents into structured database: a feasibility...

Most states have environmental guidelines or legislation that require any known site of subsurface contamination to undergo a baseline environmental assessment to document contaminant types, concentrations, and subsurface characterization (soil types, groundwater levels, etc.). The resulting publicly available documents are essential in understanding past contamination sources and the potential for future transport or exposure from these sites. Most states have an overwhelming archive of such documents that remain in only paper format, while newer documents are often available digitally. The older documents are very hard to evaluate and use in an efficient manner. Chemical data embedded in the documents were generated by multiple laboratories in varying formats, making them difficult to integrate and use for data analysis and reporting. There is an unmet need to systematically and efficiently extract the chemical data and ensure that the data are both human-readable and machine-readable and meet the principles of FAIR (Findable, Accessible, Interoperable, and Reusable). This study aimed to evaluate the feasibility of an AI-assisted, human-in-the-loop pipeline for extracting, reconstructing, harmonizing, and standardizing tabular chemical data from legacy environmental documents under real-world conditions. This study developed CLEAR-Extract, a scalable, flexible, and modular pipeline integrating optical character recognition, AI-assisted table reconstruction, and structured human quality assurance. Two pipeline configurations-a single-AI configuration emphasizing flexibility and a dual-AI configuration emphasizing efficiency and stability-were applied sequentially to historical brownfield documents from Michigan. Feasibility was assessed at the address level based on successful reconstruction of qualifying chemical tables, correct exclusion of non-qualifying documents, identification of true processing failures, and the degree of required human intervention. Across 99 candidate addresses, the pipeline successfully processed 98% of sites by either reconstructing qualifying chemical concentration tables or correctly excluding documents lacking relevant data. Only two addresses were classified as true failures due to severe document quality limitations. The resulting harmonized database captured volatile organic compound measurements spanning multiple decades and heterogeneous reporting formats, standardized into a common schema suitable for downstream environmental health analysis. This paper demonstrated that CLEAR-Extract, an AI-assisted, human-in-the-loop pipeline, could achieve high reconstruction success from heterogeneous legacy environmental documents. The approach also demonstrated a scalable and transferable framework for converting unstructured historical documents into a FAIR-aligned, analyzable database for environmental health research and regulatory decision-making.