Mining Social Media Data for Influenza Vaccine Effectiveness Using a Large Language Model and Chain-of-Thought...

Area of Science:

• Public Health
• Computational Epidemiology
• Artificial Intelligence in Medicine

Background:

• Influenza vaccine effectiveness (VE) estimation is crucial for public health policy.
• Current VE estimation methods face limitations like selection bias and delayed reporting.

Purpose of the Study:

• To explore the use of large language models (LLMs) with few-shot chain-of-thought (CoT) prompting for real-time influenza VE estimation.
• To mine social media data for improved influenza surveillance.

Main Methods:

• Annotated over 4,000 tweets from the 2020-2021 flu season for vaccination status and test outcomes.
• Developed and tested LLM-based CoT prompting strategies for data extraction.
• Compared LLM performance against traditional supervised fine-tuning methods.

Main Results:

• Achieved high inter-annotator agreement during tweet annotation.
• The best LLM prompting strategy reached an F1 score above 87% for identifying vaccination status and test outcomes.
• LLM approaches significantly outperformed traditional supervised fine-tuning methods.

Conclusions:

• LLM-based prompting is effective for extracting relevant influenza information from social media.
• This method provides a valuable real-time surveillance tool complementing traditional epidemiological studies.
• LLMs offer a promising avenue for enhancing public health decision-making through rapid VE estimation.