Assessing Variability in ChatGPT Responses: A Case Study on Simulating Online User Inputs

Area of Science:

• Public Health
• Artificial Intelligence
• Health Equity

Background:

• Generative artificial intelligence (AI) offers potential for public health information access.
• Biases in AI training data can lead to unfair or inequitable application outcomes.
• Investigating AI bias is crucial for ensuring trustworthy and effective public health tools.

Purpose of the Study:

• To assess how social variables (race, gender, sexual orientation) influence responses from generative AI tool ChatGPT.
• To evaluate potential biases in ChatGPT's public health advice, specifically regarding HIV.
• To compare response disparities between ChatGPT versions 3.5 and 4.0.

Main Methods:

• Structured question format used to query ChatGPT versions 3.5 and 4.0.
• Simulated first-time interactions with questions focused on HIV advice.
• Responses analyzed for comprehensiveness and inclusion of social determinants and culturally sensitive resources across different demographic inputs.

Main Results:

• Certain social variables were associated with less comprehensive HIV advice from ChatGPT.
• Both AI versions rarely mentioned social determinants of health.
• Culturally sensitive resources were sporadically referenced in AI responses.

Conclusions:

• Disparities in AI-generated public health advice linked to social variables were identified.
• The findings underscore the need for AI systems to integrate diverse data sources to mitigate bias.
• Further research is required to ensure AI tools promote health equity.