How good are large language models at product risk assessment?

Area of Science:

• Product Safety Engineering
• Artificial Intelligence in Risk Management

Background:

• Product safety professionals are responsible for assessing consumer risks from product use and misuse.
• Generative artificial intelligence (AI), particularly large language models (LLMs), presents a potential tool for enhancing product risk assessment processes.

Purpose of the Study:

• To investigate the utility of LLMs, such as ChatGPT, in various product risk assessment tasks.
• To evaluate the performance of LLMs in identifying failure modes, conducting Failure Mode and Effects Analysis (FMEA), and suggesting risk mitigations.

Main Methods:

• Developed prompts for six consumer products covering failure mode identification, FMEA table creation, and risk mitigation strategies.
• Input prompts into ChatGPT and other LLMs, recording the generated outputs.
• Administered a survey to product safety professionals to assess the quality and utility of the AI-generated outputs.

Main Results:

• LLMs demonstrated strengths in divergent thinking tasks, such as brainstorming potential failure modes and risk mitigations.
• Identified errors and inconsistencies in AI-generated results, with guidance often perceived as generic or lacking expert depth.
• Similar performance patterns were observed across different LLMs tested.

Conclusions:

• LLMs show potential as assistive tools for ideation in product risk assessment, particularly for brainstorming.
• Human expertise remains crucial for critical review and validation of AI-generated content in product safety.
• The role of AI may evolve to support experts, shifting their focus towards higher-level analysis and validation.