Knowledge sharing in manufacturing using LLM-powered tools: user study and model benchmarking

Area of Science:

• Manufacturing Technology
• Artificial Intelligence
• Knowledge Management

Background:

• Manufacturing operations are increasingly knowledge-intensive, straining operator training and support.
• Effective knowledge sharing is crucial for factory efficiency and operator development.

Purpose of the Study:

• To develop and evaluate a Large Language Model (LLM)-based system for knowledge retrieval and sharing in factories.
• To assess the system's impact on operator efficiency and information access.
• To benchmark various LLMs for their suitability in manufacturing knowledge management.

Main Methods:

• Development of an LLM-based system to access factory documentation and expert knowledge.
• User study conducted in a factory setting to evaluate system adoption and perceived benefits.
• Benchmarking of commercial and open-source Large Language Models (LLMs).

Main Results:

• The LLM system demonstrated benefits in quicker information retrieval and issue resolution.
• Operators expressed a preference for learning from human experts when available.
• GPT-4 showed superior performance among tested LLMs, with open-source models offering competitive alternatives.

Conclusions:

• LLM-based systems offer a promising approach to enhance knowledge management in manufacturing.
• Balancing AI support with human expertise is key for effective operator training.
• Further research into optimizing LLM applications in industrial settings is warranted.