Performance of a large language model in the informed consent process for participation in a clinical trial

The performance of large language models (LLMs) in the informed consent process for a randomized clinical trial (RCT) remains unknown. The objective of this proof-of-concept study was to evaluate whether a large language model constrained to trial-specific documentation can deliver technically accurate, reproducible responses to informed consent questions with minimal hallucination, as a foundational validation step prior to participant-facing evaluation; and to assess the feasibility of using a second large language model to reproduce document-grounded accuracy assessments in alignment with human evaluators, as a proof of concept for scalable evaluation, rather than to establish independent ground truth correctness. Mean accuracy scores across all responses were consistently rated high by humans (4.8, 95% CI: 4.7-4.9) and the moderator LLM (4.7, 4.6-4.8). Readability was appropriate (human: grade 7.5, 7.0-8.0; LLM: grade 6.4, 6.0-6.9) and there was near identical semantic consistency (human: 0.91, 0.89-0.92; LLM: 0.91, 0.89-0.92). Inter-rater reliability of accuracy ratings demonstrated significant differences between human assessors (κ = 0.1, 0.0-0.2) and the moderator LLM (κ = 0.8, 0.8-0.8). Human raters gave different accuracy ratings to the same answers generated by the consent LLM due to perceptions that responses were technically accurate but inadequate because they lacked conversational tone, awareness of context, and empathy. We developed a set of novel LLM-driven tools that improve the efficiency of the informed consent process by enabling people designing RCTs to answer potential subjects' questions and validate those models efficiently with machines, rather than people.