Evaluating generative artificial intelligence's limitations in health policy identification and interpretation

Area of Science:

• Policy epidemiology
• Health policy analysis
• Artificial intelligence in public health

Background:

• Policy epidemiology relies on human subject-matter experts (SMEs) for systematic policy analysis.
• Emerging infectious diseases necessitate comprehensive health policy assessment across nations.
• Generative artificial intelligence (GAI) tools offer potential for resource reduction in policy research.

Purpose of the Study:

• To evaluate the accuracy and precision of GAI tools compared to SMEs in identifying and interpreting health policies.
• To assess GAI's utility in large-scale health policy data collection and analysis projects.

Main Methods:

• A comparative study assessing GAI tool responses against SME analysis.
• Utilized two validated policy datasets: emergency/childhood vaccination and quarantine/isolation policies from UN Member States.
• Analyzed concordance rates, data collection speed, and systematic inaccuracies across WHO regions.

Main Results:

• SME and GAI concordance was 78.09% for vaccination and 67.01% for quarantine/isolation policies.
• GAI significantly accelerated data collection but exhibited systematic inaccuracies, particularly in the African and Eastern Mediterranean regions.
• GAI showed lowest concordance with SMEs during policy interpretation tasks.

Conclusions:

• GAI tools require significant development for accurate global health policy identification and nuanced interpretation.
• GAI demonstrates utility as a quality assurance tool for health policy identification.
• Further research is needed to refine GAI for complex public health policy analysis.