Exploration of designing an automatic classifier for questions containing code snippets-A case study of Oracle SQL...

Area of Science:

• Computer Science
• Educational Technology

Background:

• Automatic classification of exam questions aids in test assembly.
• Classifying SQL questions with code snippets presents unique challenges compared to text-only questions.

Purpose of the Study:

• To design and evaluate automatic classifiers for Oracle SQL certification exam questions containing code snippets.
• To identify the most effective feature representation schemes and machine learning methods for this classification task.

Main Methods:

• Utilized factorial experiments to analyze the impact of feature representation (e.g., TF-IDF) and machine learning models (e.g., Logistics Regression, Support Vector Machine).
• Evaluated classifier performance using weighted macro-average AUC, F1, and Precision metrics.

Main Results:

• The TF-IDF feature representation scheme combined with the Logistics Regression model yielded the best results for AUC and F1 scores.
• TF-IDF with Support Vector Machine excelled in weighted macro-average Precision.
• Feature representation scheme was identified as the primary factor influencing classifier performance.

Conclusions:

• The TF-IDF scheme is highly effective for classifying Oracle SQL exam questions with code.
• Logistics Regression and Support Vector Machine are viable machine learning models for this application.
• Feature representation significantly impacts the success of automatic question classification systems.