Sherlock: A Semi-automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

Area of Science:

• Computer Science
• Artificial Intelligence
• Knowledge Representation

Background:

• Automated quiz generation systems often lack domain independence and fine-grained difficulty control.
• Existing methods struggle to accurately assess the cognitive difficulty of educational content.

Purpose of the Study:

• To develop a semi-automatic system (Sherlock) for generating domain-independent quizzes with adjustable difficulty levels.
• To propose a novel hybrid semantic similarity measure for assessing knowledge difficulty using linked data.

Main Methods:

• Developed Sherlock, a system leveraging linked data and RDF resource descriptions for quiz generation.
• Implemented a novel hybrid semantic similarity measure to quantify knowledge difficulty.
• Evaluated the system's performance using extensive experiments and human quiz testing.

Main Results:

• The proposed semantic similarity measure significantly outperformed four baseline methods, achieving over 47% gain in clustering accuracy.
• Sherlock demonstrated domain-independent quiz generation capabilities.
• Human quiz tests confirmed a strong correlation between the model's accuracy and pairwise quiz similarity.

Conclusions:

• The novel semantic similarity measure is effective for controlling quiz difficulty and improving automated quiz generation.
• Sherlock offers a flexible and accurate framework for creating educational assessments.
• The approach bridges cognitive science principles with linked data for enhanced learning tools.