Developing a Conversational Agent's Capability to Identify Structural Wrongness in Arguments Based on Toulmin's Model...

Area of Science:

• Argumentation theory
• Computational linguistics
• Artificial intelligence

Background:

• Toulmin's model provides a framework for analyzing argument structure.
• Conversational agents can support user learning and argument development.
• Identifying structural components of arguments is crucial for assessing their quality.

Purpose of the Study:

• To evaluate the utility of Toulmin's model for assessing argument components within a conversational agent.
• To develop and test classifiers for identifying claims, warrants, and evidence in user arguments.
• To explore the application of these classifiers in facilitating coherent learning conversations.

Main Methods:

• Utilized Toulmin's model to define argument components: claim, warrant, and evidence.
• Developed machine learning classifiers to detect the presence and direction of claims, warrants, and evidence.
• Trained and evaluated classifiers on a dataset of user arguments concerning entity intelligence.
• Proposed a conditional dialogue structure based on Bloom's taxonomy for learning conversations.

Main Results:

• Claim detection achieved a weighted average F1 score of 0.91.
• Warrant detection achieved a weighted F1 score of 0.88.
• Evidence detection achieved a weighted average F1 score of 0.80.
• These scores indicate high accuracy for identifying structural argument components.

Conclusions:

• Toulmin's model is effective for structuring argument assessment in conversational agents.
• Accurate identification of argument components supports coherent learning dialogues.
• Future work should focus on agents that detect complex argumentation errors and support argumentation learning.