A qualitative assessment of using ChatGPT as large language model for scientific workflow development

Area of Science:

• Computational Science
• Bioinformatics
• Data Science

Background:

• Scientific workflow systems are crucial for reproducible and scalable data analysis.
• Implementing these workflows is challenging due to complex tools and infrastructure.
• Limited user support and examples hinder workflow adoption.

Purpose of the Study:

• To evaluate the effectiveness of large language models (LLMs), specifically ChatGPT, in supporting users with scientific workflows.
• To assess LLM performance in comprehending, adapting, and extending scientific workflows across different domains.

Main Methods:

• Conducted three user studies in two distinct scientific domains.
• Evaluated ChatGPT's capabilities in understanding, modifying, and extending scientific workflows.
• Analyzed user interactions and workflow outcomes to identify LLM performance and limitations.

Main Results:

• LLMs demonstrated high accuracy in comprehending and explaining scientific workflows.
• Performance was reduced when attempting to exchange components or extend workflows purposefully.
• Limitations were identified in scenarios requiring complex modifications and extensions.

Conclusions:

• LLMs show significant potential for assisting with scientific workflow comprehension.
• Further research is essential to enhance LLM performance in workflow adaptation and extension.
• Addressing identified limitations will improve the utility of LLMs in scientific data analysis.