Evaluation of large language model-driven AutoML in data and model management from human-centered perspective

Area of Science:

• Artificial Intelligence
• Machine Learning Operations (MLOps)

Background:

• Organizations face challenges adopting machine learning (ML) due to technical complexity.
• Implementing ML solutions often impacts operational efficiency and creates adoption barriers.

Purpose of the Study:

• To investigate how Large Language Models (LLMs) can improve ML accessibility via a human-centered Automated Machine Learning (AutoML) framework.
• To evaluate the organizational impact of an LLM-based AutoML approach compared to traditional ML implementation methods.

Main Methods:

• A user study was conducted with 15 professionals from diverse roles and technical backgrounds.
• The study compared an LLM-based AutoML framework against traditional ML implementation techniques.
• Performance metrics included ML implementation success rates, accuracy, development time, and error resolution time.

Main Results:

• LLM-based interfaces significantly improved ML implementation success rates (93.34% of users).
• Accuracy improved by 10%-25% for 46.67% of users and >25% for another 46.67%.
• Development time was reduced by 50%, error resolution time by 73%, and learning curves were accelerated.

Conclusions:

• LLM-based AutoML democratizes ML capabilities by bridging technical skills gaps through natural language interfaces.
• This approach enhances ML implementation success, reduces development time, and improves human-AI collaboration.
• Empirical evidence supports natural language as an effective interface for complex technical systems like ML.