On Evaluating Runtime Performance of Interactive Visualizations.

Area of Science:

• Scientific Visualization
• Computer Graphics
• Human-Computer Interaction

Background:

• Evaluation of visualization techniques has shifted from runtime performance to user behavior studies.
• Existing methodologies for user studies are well-established, but a similar foundation for runtime performance evaluation is lacking.
• Current performance evaluations in visualization papers are often limited in scope, hindering comparability and generalization.

Purpose of the Study:

• To develop a comprehensive framework for evaluating the runtime performance of volume and particle visualization techniques.
• To gain a deeper understanding of qualitative runtime behavior and quantitative parameter dependencies.
• To establish best practices for runtime performance evaluation in scientific visualization.

Main Methods:

• Analysis of 50 recent visualization papers to identify current evaluation practices.
• Development of a framework for exhaustive performance evaluation.
• Conducting millions of measurements across ten different Graphics Processing Units (GPUs) for volume and particle rendering techniques.

Main Results:

• Identified limited parameter investigation in existing studies, questioning comparability and generalizability.
• Statistical analysis revealed insights into independent and linear parameter behavior, as well as non-obvious performance effects.
• The framework facilitated an extensive evaluation of visualization techniques.

Conclusions:

• There is a critical need for standardized and thorough methodologies in evaluating scientific visualization runtime performance.
• The developed framework provides a foundation for more rigorous performance quantification.
• Recommendations are provided for best practices to improve the reliability and comparability of future performance evaluations.