Advanced Approaches to Generating High-validity Real-world Evidence in Asthma

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Real-World Evidence Generation

Background:

• Asthma is a complex disease requiring high-quality data for robust real-world evidence (RWE).
• Quantitative data quality measures are crucial for key research variables.
• This study compares traditional RWE methods with AI-driven approaches using electronic health record (EHR) data.

Purpose of the Study:

• To compare the accuracy of asthma feature extraction between traditional structured EHR data analysis and advanced AI techniques applied to unstructured clinical data.
• To evaluate the impact of AI on the quality and validity of RWE for asthma research.

Main Methods:

• Extracted 18 asthma-related features from 6037 healthcare encounters across 3481 patients.
• Established a manual reference standard via chart abstraction with dual annotator review.
• Assessed inter-rater reliability using Cohen's kappa and accuracy using F1-score.

Main Results:

• Traditional methods achieved an average F1-score of 52.2%.
• Advanced AI approaches yielded an average F1-score of 94.7%, an 81.4% relative increase.
• Inter-rater reliability was 0.80 (Cohen's kappa), indicating a credible reference standard.

Conclusions:

• Advanced AI approaches significantly enhance the quality of real-world data for asthma research.
• High-quality RWE, including granular clinical features, can be generated using AI on routinely collected healthcare data.
• Data quality measurement is key to supporting high-validity RWE generation.