Development and Implementation of an AI System for Generating Clinical Urine Drug Test Sign-Outs

Modern natural language processing tools have potential to improve clinical workflows, but few have been successfully deployed in practice.

To describe the development, deployment, and evaluation of an artificial intelligence (AI) language tool for generating preliminary sign-outs to support a urine drug testing service.

In this prognostic study, large language models (LLMs) were used to extract substance use patterns from clinical urine drug test interpretations at a single medical center between January 1, 2014, and February 29, 2024. An AI model using these data was trained to predict substance use from qualitative and quantitative urine testing results. Predicted substance use patterns were used to create preliminary clinical sign-out statements, which were then integrated into an existing clinical workflow.

Predeployment and postdeployment user studies were performed to evaluate model performance and user experience within the workflow. Statistical differences between event rates were calculated using χ2 tests, and between means using t tests. Differences between human and LLM labelers were calculated using the McNemar test.

A total of 83 553 urine tests from 26 459 patients (12 413 male [46.9%]; mean [SD] age, 47.5 [16.7] years) were analyzed. LLM-based extraction of substance-use patterns was 99.9% accurate (13 509 of 13 520 tests), outperforming human labeling. Substance use prediction was similarly accurate, with area under the receiver operating curve greater than 0.99 for 23 of 26 substances. Workflow integration of the AI tool reduced clinical sign-out times by 28.5 seconds per case (23% efficiency gain), and by 65 seconds per case (51% efficiency gain) when integrated alongside a second, non-AI workflow improvement.

In this prognostic study, AI-based interpretation of urine drug testing was fast and accurate, providing notable efficiency gains to the clinical service. These findings suggest that natural language processing tool integration can provide substantial clinical benefit, without compromising quality of care.