Implicit Perception of Differences between NLP-Produced and Human-Produced Language in the Mentalizing Network

Area of Science:

• Cognitive Neuroscience
• Artificial Intelligence
• Natural Language Processing (NLP)

Background:

• Natural Language Processing (NLP) aims to create human-quality language for machine communication.
• Objective criteria are needed to measure the quality of NLP-generated language.
• Previous research has not fully explored neural responses to distinguish AI-generated text from human text.

Purpose of the Study:

• To investigate if neural activity in the mentalizing network can differentiate AI-produced language from human-produced language.
• To establish implicit perception as a criterion for evaluating NLP language quality.
• To assess the stability of perceived personality in chatbot interactions.

Main Methods:

• Behavioral tests using social chatbots (Google Meena, Microsoft XiaoIce) to assess perceived personality variance.
• Functional magnetic resonance imaging (fMRI) during an identity rating task.
• Analysis of neural activity in the mentalizing network (DMPFC, rTPJ) in response to chatbot vs. human chats.

Main Results:

• Behavioral tests showed greater perceived personality variance in chatbot conversations compared to human conversations.
• fMRI revealed distinct patterns of brain activity in the mentalizing network for chatbot-generated text versus human-generated text.
• These neural differences were detectable even when human judges could not subjectively distinguish the language source.

Conclusions:

• Mentalizing network activity provides an empirical basis for distinguishing AI-produced language from human language.
• Implicit neural perception can serve as a novel criterion for measuring NLP language quality.
• This approach offers a promising method for developing more sophisticated NLP evaluation tools.