Generative artificial intelligence and machine learning methods to screen social media content.

Area of Science:

• Computational Social Science
• Digital Health
• Artificial Intelligence in Research

Background:

• Social media research faces challenges with vast, evolving data and irrelevant search results.
• Effective methods are needed to screen social media content for specific research questions.
• Generative AI and machine learning offer potential solutions for content screening.

Purpose of the Study:

• To present generative artificial intelligence (AI) and machine learning methods for screening social media content.
• To apply these methods to identify TikTok content related to e-cigarette use during pregnancy.
• To evaluate the effectiveness of ChatGPT-4 in screening this specific content.

Main Methods:

• Searched TikTok for pregnancy and vaping content using 70 hashtag pairs, obtaining 11,673 posts.
• Extracted video, description, and metadata; transcribed audio using Whisper; and performed object/text detection.
• Utilized ChatGPT-4 for content analysis, refined by human coder cross-checking and review.

Main Results:

• ChatGPT-4 classified 44.86% of videos as pregnancy-related, 36.91% as vaping-related, and 8.91% as both.
• Human reviewers confirmed vaping and pregnancy content in 45.38% of posts identified by ChatGPT.
• Human review of excluded posts showed a 99.06% agreement rate with ChatGPT's screening.

Conclusions:

• ChatGPT demonstrates mixed capacity for screening social media content converted to text via machine learning.
• ChatGPT's sensitivity was lower than human coders but effective for initial screening and excluding irrelevant content.
• Future research should focus on enhancing ChatGPT's sensitivity for social media content analysis.