Trust, but Verify-Post-Hoc Analysis of Industrial Machine Learning via Interpretability Metric Embedding and...

In industrial machine learning, predictive performance alone is insufficient to ensure reliable deployment, as model behaviour may vary across different regions of the input space under limited data and evolving process conditions. This work investigates whether such variation can be systematically analysed through post-hoc methods. A model-agnostic framework is proposed in which interpretability metrics, including residuals and feature attributions, are embedded into a low-dimensional space and approximated using a continuous surrogate model. This representation enables the analysis of model behaviour as a structured landscape, rather than as isolated pointwise explanations. The approach is applied to ceramic heating element production, where two distinct regimes are identified. One corresponds to a stable region with consistent and accurate predictions, while the other reflects a transitional regime associated with increased ambiguity and sensitivity to feature interactions. These regimes are shown to align with known process conditions and temporal variation. The results demonstrate that model behaviour can be organised into coherent regions that are not observable through aggregate performance metrics alone. This provides a structured basis for post-hoc analysis, supporting targeted interpretation and further investigation of model reliability in industrial settings.