Workflow analysis of data science code in public GitHub repositories.

Area of Science:

• Computer Science
• Software Engineering
• Data Science

Background:

• Understanding data science coding practices is crucial for improving developer tools and workflows.
• Existing literature suggests data science coding is iterative but lacks detailed empirical evidence.

Purpose of the Study:

• To empirically investigate the iterative and explorative nature of data science coding.
• To identify common transitions and patterns in data science workflows.
• To inform the development of better tooling for data scientists.

Main Methods:

• Analysis of 470 publicly available Jupyter notebooks from GitHub repositories.
• Manual annotation of code cells into distinct data science steps by five domain experts.
• Application of first-order Markov chain models to analyze transitions between steps.

Main Results:

• Empirical evidence confirms the iterative nature of data science workflows.
• Identified specific transition patterns and probabilities between different data science activities.
• Machine learning models achieved an F1-score of approximately 71% in predicting data science steps.

Conclusions:

• Data science coding workflows are characterized by frequent, iterative transitions between diverse tasks.
• The findings provide valuable insights into the practical implementation of data science.
• The annotated dataset and developed models show promise for automating workflow analysis and supporting data scientists.