Comparing Emotional Valence from Human Quantitative Ratings and Qualitative Narrative Data on Using Artificial...

Area of Science:

• Gerontology
• Artificial Intelligence Ethics
• Health Informatics

Background:

• African American family caregivers of persons with Alzheimer's disease and related dementias (ADRD) face unique challenges.
• Understanding caregiver attitudes towards Artificial Intelligence (AI) is crucial for developing supportive technologies.
• Existing survey methods may not fully capture the nuances of caregiver sentiment.

Purpose of the Study:

• To compare emotional valence scores derived from machine learning (ML) natural language processing (NLP) with human ratings.
• To assess the perceived attitudes of African American family caregivers regarding AI use in ADRD care.
• To evaluate the reliability of NLP in sentiment analysis of qualitative survey data.

Main Methods:

• A survey was administered to 627 African American family caregivers of persons with ADRD from April to May 2024.
• Participants responded to open-ended questions on ten AI use cases, followed by ratings.
• Three ML algorithms (AFINN, Bing, Syuzhet) were applied to detect emotional valence from text data.

Main Results:

• Mean emotional valence scores from NLP analysis were consistently negative across all algorithms (AFINN: -1.61, Bing: -1.40, Syuzhet: -0.67).
• Human ratings of emotional valence were positive (2.30).
• A significant difference was found between machine and human ratings (p=0.0001).

Conclusions:

• Machine-based sentiment analysis using NLP may misinterpret or inaccurately reflect the emotional valence of caregiver attitudes towards AI.
• Human ratings provide a more positive and potentially accurate assessment of caregiver sentiment.
• Findings underscore the need for careful consideration of survey design and interpretation methods in the era of AI and NLP.