Evaluating a Novel Summary Visualization for Clinical Trial Reports: A Usability Study

Contributions of clinical trials are captured in published reports that are unstructured and often require extensive manual review to gain a deeper understanding of the study itself. Our goal is to increase comprehension and decrease the time necessary to understand these reports through the use of visualization tools. In this paper, we specify and evaluate the visualization of a previously developed representation as well as gain insight from user input for further development. The usability experiment consisted of a two-arm study with users either having or not having access to the visualization. A user questionnaire was used to measure time spent and accuracy in comprehension; intuitiveness and reproducibility of the visualization; and preferences. We found that having the visualization required on average 28.1% less time (25.8 min vs. 35.8 min, p=0.01) while maintaining similar accuracy (73.7% vs. 67.0%). Users were then asked to create their own visualizations, with their visualizations averaging 86.1% similar to the gold standard. All participants either preferred the visualization over the status quo or preferred both equally. These results demonstrate that novel visualizations for trial reports could provide time savings and achieve similar accuracy as reviewing the paper itself. Understanding the strength and quality of clinical trials can be alleviated with a visualization that makes content explicit.