A scoping review of approaches to evaluating workflow management systems for bioinformatics users

Workflow Management Systems (WMSs) make it easier to run bioinformatics analyses by combining tools into reusable workflows. However, selecting the right WMS can be challenging, particularly for users from noncomputational backgrounds. Many reviews of WMSs focus on technical details or only consider usability from the perspective of developers. The first objective of this scoping review was to identify the characteristics of bioinformatics WMSs that are most relevant for life scientists. The second objective was to explore how these characteristics have previously been evaluated. The review included 21 papers published since 2018 that evaluated bioinformatics WMSs based on criteria relating to the user experience. Published papers and websites describing 55 currently available WMSs were also included to identify characteristics highlighted by their developers. Twelve themes emerged from these evaluation criteria and WMS characteristics: Basic Computing, Functions, Security, Scalability, Cost/Efficiency, Sustainability, Usability, Learnability, Reproducibility, FAIRness (Findability, Accessibility, Interoperability, Reusability), Flexibility, and Support. Evaluations focusing on the needs of noncomputational users preferred Graphical User Interfaces and platforms that provided plenty of guidance for users. Papers prioritizing the needs of developers instead favoured text-based interfaces and flexible platforms that gave users greater control. In addition to these contrasting views on what was considered a positive characteristic, differences in how criteria were defined and scored meant that evaluations could not be compared between papers and would be impossible for users to repeat on new or updated WMSs. Users do not currently have a clear approach to follow when selecting a WMS.