Evaluating large language models for automated REDCap support ticket triage and response

To evaluate the utility of large language models (LLMs) without and with retrieval-augmented generation (RAG) resources for automating Research Electronic Data Capture (REDCap) support at a large academic medical center, focusing on ticket complexity, categorization, and response drafting.

We randomly selected 90 REDCap support tickets from 6316 received in 2023, stratified by estimated complexity (low, moderate, high) using ticket metadata. Four LLM configurations (GPT-4o Default; GPT-4o + Vanderbilt REDCap documentation; GPT-4o + University of Michigan [UM] REDCap documentation; GPT-4o + combined documentation) were evaluated on 3 tasks per ticket: complexity assessment, category assignment, and email response drafting. Human REDCap experts provided gold-standard ratings and assessed email drafts.

Compared to the gold standard, the GPT-4o Default environment most closely matched human complexity ratings, while specialized RAG environments tended to overestimate complexity. For ticket support categorization, all LLM configurations achieved high agreement with human judgments (Cohen's kappa ≈0.85), correctly categorizing 81%-89% of tickets. In generating ticket responses, the GPT-4o + REDCap + UM environment provided the most correct answers, particularly for low and moderate complexity tickets, where about two-thirds of responses were substantially or fully correct.

Large language models have potential to automate ticket triage and response drafting for routine REDCap support, with context augmentation improving response accuracy for lower complexity tickets.

Large language model-based support tools hold promise for reducing human workload for REDCap support tickets. Continued human oversight, iterative prompt engineering, and expanded local documentation are needed for reliable, effective deployment.