AI support for data scientists: An empirical study on workflow and alternative code recommendations

Area of Science:

• Computer Science
• Data Science
• Human-Computer Interaction

Background:

• AI coding assistants are popular but their utility in data science tasks is under-researched.
• Exploring alternative analytical paths is crucial for robust data science conclusions.
• The role of AI assistants in facilitating path exploration in data science is unknown.

Purpose of the Study:

• To investigate how AI coding assistants impact data scientists' workflow.
• To determine if AI assistants can support exploration of alternative data science paths.
• To evaluate the acceptance and helpfulness of AI code recommendations, including alternatives.

Main Methods:

• A mixed-methods study was conducted with data scientists using an AI coding assistant.
• Quantitative analysis assessed acceptance and helpfulness of AI recommendations (including alternatives).
• Qualitative insights were gathered on user interactions and challenges.

Main Results:

• Including the data science step in prompts significantly improved recommendation acceptance.
• The presence of alternative recommendations did not significantly impact acceptance or helpfulness.
• Significant differences were observed in recommendation acceptance and usefulness between descriptive and predictive tasks.

Conclusions:

• AI assistants can support data science tasks, with prompt engineering being key.
• Current AI assistants may not effectively facilitate the exploration of diverse analytical paths.
• User sentiment towards AI assistance in data science is generally positive.