Related Experiment Video
Updated: May 6, 2026
A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
Artificial Intelligence Applications for Automated Data Extraction and Secondary Use of Clinical Information in
Julian Greß1,2, Gordon Otto2, Sebastian Sommer2,3
1Technical University of Munich, Munich, Germany.
Background And Objective:
Manual data extraction is a major bottleneck in uro-oncology, limiting research and quality assurance. Although artificial intelligence (AI) offers a scalable solution, the quality and generalizability of current evaluations remain unclear. This review aims to assess the performance, validation strategies, and real-world implementation of AI for automated data extraction in uro-oncology, encompassing a methodological spectrum from rule-based natural language processing to large language models, and to provide recommendations for rigorous evaluation standards.
Methods:
A systematic search of PubMed, Web of Science, and Embase was conducted through May 2025 following the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines. The search was restricted to studies published from 2020 onward to focus on modern AI capabilities. Two reviewers independently screened records, extracted data, and assessed risk of bias using the Prediction model Risk Of Bias Assessment Tool (PROBAST).
Key Findings And Limitations:
Fourteen studies, encompassing between 100 and 66 532 patient records and approximately 120 000 individual clinical documents across genitourinary cancers, were included. AI models demonstrated high technical performance on structured data extraction, with reported F1 scores frequently exceeding 0.90. However, 86% (12/14) relied solely on internal validation; only two studies reported external validation. Nine studies (64%) described workflow benefits such as improved efficiency and reduced manual abstraction time. Most studies were retrospective and single center, with heterogeneous reporting that precluded a meta-analysis. Evidence for clinical application, cost effectiveness, calibration, and long-term sustainability was limited. These limitations highlight the need for robust external validation, human-in-the-loop verification, improved calibration reporting, equity assessments, and an implementation-science approach.
Conclusions And Clinical Implications:
AI shows strong potential for automating data extraction in uro-oncology, but clinical translation is limited by insufficient external validation and methodological heterogeneity. A shift from isolated performance metrics toward demonstrated robustness and trustworthy clinical application is needed to support reliable clinical use.
Patient Summary:
In this study, we reviewed how artificial intelligence (AI) is being used to extract information automatically from medical reports on urological cancers. We found that most AI systems can identify important clinical details very accurately, but these are usually tested in only one hospital and not yet shown to work reliably in other settings. This means that while AI has great potential to save time and improve data quality, more testing in everyday clinical practice is needed before it can be used safely and routinely.
