How Does a Generative Large Language Model Perform on Domain-Specific Information Extraction?─A Comparison between...

Area of Science:

• Materials Science
• Natural Language Processing
• Computational Chemistry

Background:

• Generative Large Language Models (LLMs) are transforming Natural Language Processing.
• The efficacy of LLMs for domain-specific information extraction remains under investigation.
• Materials science literature lacks sufficient training data for specialized extraction tasks.

Purpose of the Study:

• To compare the performance of GPT-4 and ChemDataExtractor for extracting band gap information from materials science literature.
• To evaluate the accuracy and identify the strengths and weaknesses of each method without requiring training data.
• To assess the potential of LLMs for domain-specific information extraction in data-scarce fields.

Main Methods:

• A comparative analysis of GPT-4 and a rule-based method (ChemDataExtractor) for band gap extraction.
• Manual evaluation of extraction accuracy on 415 randomly selected scientific articles.
• Error analysis to understand the performance differences and limitations of each model.

Main Results:

• GPT-4 achieved significantly higher correctness (87.95%) compared to ChemDataExtractor (51.08%).
• GPT-4 demonstrated strengths in resolving interdependencies and recognizing complex material names.
• GPT-4 exhibited weaknesses in hallucination, identifying band gap values, and types, which were mitigated by prompt revision.

Conclusions:

• Generative LLMs, specifically GPT-4, offer a more accurate approach for band gap information extraction in materials science.
• The study validates the utility of LLMs for domain-specific extraction tasks, particularly where training data is limited.
• Prompt engineering can further enhance the performance of LLMs in specialized scientific information extraction.