Evaluating the Automated Extraction of Radiation Oncology Trial Data at Scale from the ClinicalTrials.gov Database...

Background The rapid growth of biomedical publishing in radiation oncology has created a need for automated methods to extract structured information from unstructured text. Large language models (LLMs) show promise in biomedical natural language processing, but their use for clinical trial data curation remains to be systematically evaluated. Methods We evaluated an LLM-based extraction system for radiation oncology trials on ClinicalTrials.gov. Forty-five classification questions spanning variables such as trial design, tumor type, stage, radiotherapy modality, systemic therapy, and outcomes were implemented using the data-element-extractor Python library with three open-source LLMs (GPT-OSS-20B, GPT-OSS-120B, and Qwen-3.6-27B). A manually curated ground-truth of 94 trials was used for performance evaluation based on accuracy, precision, recall, and F1-score. Results Qwen-3.6-27B achieved the highest overall accuracy of 92.0% across all questions. For the 39 Boolean questions, it demonstrated 93.0% accuracy, 81.6% precision, 83.7% recall, and an F1-score of 82.6%, with 23 Boolean questions achieving an F1-score ≥80.0%. Three numerical questions reached accuracy ≥80.0%. Applied to the full dataset, the system characterized the clinical landscape of radiotherapy trials, detailing the prevalence of advanced techniques (e.g., 13.7% SBRT) and the integration of systemic agents, including chemotherapy (57.3%) and immunotherapy (26.1%). Conclusion LLM-based extraction can enable accurate, scalable curation of radiotherapy trial data. While challenges remain with complex categorical or numerical variables, open-source LLMs can serve as effective components of transparent, auditable clinical research pipelines.