Related Experiment Video
Updated: Jul 4, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Can generative AI infer thinking style from language? Evaluating the utility of AI as a psychological text analysis
1Department of Communication, Michigan State University, East Lansing, MI, USA. dmm@msu.edu.
Abstract:
Generative AI, short for Generative Artificial Intelligence, a class of artificial intelligence systems, is not currently the choice technology for text analysis, but prior work suggests it may have some utility to assess dynamics like emotion. The current work builds upon this empirical foundation to consider how analytic thinking scores from a large language model chatbot, ChatGPT, were linked to analytic thinking scores from dictionary-based tools like Linguistic Inquiry and Word Count (LIWC). Using over 16,000 texts from four samples and tested against three prompts and two large language models (GPT-3.5, GPT-4), the evidence suggests there were small associations between ChatGPT and LIWC analytic thinking scores (meta-analytic effect sizes: .058 < rs < .304; ps < .001). When given the formula to calculate the LIWC analytic thinking index, ChatGPT performed incorrect mathematical operations in 22% of the cases, suggesting basic word and number processing may be unreliable with large language models. Researchers should be cautious when using AI for text analysis.
Related Concept Videos
Language and Cognition
Introduction to Cognitive Psychology
This field emerged in the mid-20th century, following a period dominated by behaviorism, which...
Non-equilibrium in the Cell
Cognitivism
Previously dominated by behaviorism, which prioritized observable behaviors and largely ignored mental processes, psychology transformed in the 1950s. Cognitive psychologists argue that understanding how we think and process...
Reason and Intuition
Higher Mental Functions of the Brain: Language
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...

