Graph Representations for Reading Comprehension Analysis using Large Language Model and Eye-Tracking biomarker.

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Reading comprehension is crucial for cognitive development.
• Comparing human and Large Language Model (LLM) language understanding is increasingly important.
• Previous studies analyzed word-level comprehension but lacked depth.

Purpose of the Study:

• To compare human and LLM reading comprehension using a graph-based text representation.
• To investigate LLM consistency in understanding semantic relationships within text.
• To inform human-AI co-learning strategies.

Main Methods:

• Utilized an LLM-based AI agent to create graph structures from text, representing words as nodes and semantic relationships as edges.
• Employed question-oriented prompts to guide graph construction.
• Compared the distribution of human eye fixations on graph elements (nodes and edges).

Main Results:

• LLMs demonstrate high consistency in language understanding at the graph topological structure level.
• Eye fixation patterns on graph structures provide insights into semantic relevance.
• The graph-based approach offers a deeper understanding than word-level analysis.

Conclusions:

• LLMs exhibit robust and consistent semantic understanding when text is represented as a graph.
• Graph-based text representation enhances the analysis of reading comprehension.
• Findings support the development of effective human-AI collaboration in learning.