Interactive Visual Exploration of Rule-Based Model Logic

Rule-based machine learning models, including those derived from decision trees or forests, are often considered inherently interpretable. However, human understanding is hindered by model size, rule complexity, and interdependencies between features. Moreover, rule sets extracted from ensemble models can contain contradictory, incomplete, or counterintuitive logic, even when the overall model achieves high predictive accuracy. This paper introduces a visual analytics methodology designed to support systematic exploration of rule-based model logic and its alignment with domain knowledge. Our approach integrates overview visualizations, interactive filtering, contradiction analysis, and topic modeling. This enables analysts to detect illogical or implausible rules, assess their potential impact, and refine the model to improve its interpretability and trustworthiness. A key distinction of our method is its ability to support reasoning about model behavior both with and without access to labeled data. We demonstrate the approach through two real-world case studies: evaluating logical consistency in a vessel movement classifier and analyzing feature relationships in a COVID-19 prediction model. These studies show how visual analytics can facilitate logic-focused model critique beyond traditional performance metrics and enable valuable domain-relevant insights.