From LLMs to AI agents: a systematic benchmark for SAO structure extraction in patent analytics

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Robotics

Background:

• The exponential growth in patent literature necessitates advanced automated methods for information extraction.
• Extracting Subject-Action-Object (SAO) relationships is crucial for understanding technological innovation within patents.
• Existing methods struggle with the complexity and volume of patent data, especially in specialized domains like collaborative robotics.

Purpose of the Study:

• To develop and evaluate an information extraction benchmark framework for the collaborative robotics domain.
• To compare the performance of large language models (LLMs) including GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, with and without agent architectures.
• To assess the impact of agent frameworks on the accuracy and efficiency of SAO relationship extraction from patent literature.

Main Methods:

• Construction of a specialized information extraction benchmark for collaborative robotics patents.
• Implementation of a closed-loop "reflection and refinement" model incorporating thought chain and ReAct strategies.
• Comparative analysis of three LLMs (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) and their agent-based counterparts, with parameter tuning for factual accuracy.
• Statistical significance testing using a t-test on 501 independent samples.

Main Results:

• Agent architectures positively impacted the performance of all tested LLMs.
• The Gemini 1.5 Pro agent achieved the highest performance with an F1 score of 89.58% and a BERTScore of 87.73%.
• Agent-based models significantly outperformed baseline LLMs, especially in parsing complex nested clauses and filtering legal jargon, despite increased computational cost (40-60x).

Conclusions:

• Agent architectures provide a significant performance boost for SAO relationship extraction in patent analysis.
• Gemini 1.5 Pro, when utilized with an agent framework, demonstrates superior capability in handling complex patent data within the collaborative robotics domain.
• The study offers a methodological framework and empirical evidence for deploying agent-based LLMs in specialized technical domains, justifying the computational overhead for enhanced recall and risk mitigation.