Evaluating large language models for automated REDCap support ticket triage and response.

Area of Science:

• Artificial Intelligence
• Health Informatics
• Clinical Research Informatics

Background:

• Research Electronic Data Capture (REDCap) is vital for clinical research data management.
• Automating REDCap support can improve efficiency and reduce human workload.
• Large Language Models (LLMs) show potential for automating technical support tasks.

Purpose of the Study:

• To evaluate LLMs with and without retrieval-augmented generation (RAG) for automating REDCap support.
• To assess LLM performance in ticket complexity assessment, categorization, and response drafting.
• To compare different RAG configurations using institutional documentation.

Main Methods:

• 90 REDCap support tickets were randomly selected and stratified by complexity.
• Four LLM configurations (GPT-4o Default, GPT-4o + Vanderbilt docs, GPT-4o + UM docs, GPT-4o + combined docs) were tested.
• LLM outputs were compared against human expert gold standards for accuracy.

Main Results:

• GPT-4o Default best matched human complexity ratings; RAG models tended to overestimate.
• All LLM configurations showed high agreement with human categorization (Cohen's kappa ≈0.85).
• The GPT-4o + combined documentation model yielded the most accurate responses, especially for low/moderate complexity tickets.

Conclusions:

• LLMs can automate REDCap support ticket triage and response drafting.
• Context augmentation via RAG enhances response accuracy for less complex issues.
• Effective deployment requires human oversight, prompt engineering, and comprehensive local documentation.