Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts

Area of Science:

• Data Science
• Human-Computer Interaction
• Scientific Communication

Background:

• Predictive model performance presentation is a communication bottleneck hindering data scientist and subject matter expert (SME) collaboration.
• Standard metrics (accuracy, error) inadequately convey model risks, strengths, and limitations, leading to SME distrust or underutilization.
• Existing communication gaps stem from unfamiliar terminology, metrics, and visualizations, discouraging SME engagement and knowledge transfer.

Purpose of the Study:

• To investigate communication gaps between data scientists and SMEs regarding predictive model performance.
• To develop and evaluate a set of communication guidelines using visualization to bridge these gaps.
• To enhance SME confidence and understanding of model capabilities and limitations.

Main Methods:

• An iterative study involving data scientists and SMEs to identify communication challenges.
• Derivation of communication guidelines centered on visual representations of model performance.
• Demonstration of guidelines in a regression modeling context with subsequent SME feedback collection.

Main Results:

• SMEs reported increased comfort in discussing model performance after guideline implementation.
• SMEs demonstrated greater awareness of model trade-offs and risks.
• Visualizations facilitated a more contextualized understanding of model utility beyond raw numbers.

Conclusions:

• Visualization-based communication guidelines improve SME comprehension and trust in predictive models.
• Bridging the communication gap enhances the effective adoption and application of data science models.
• Contextualizing model performance visually empowers SMEs for informed decision-making.