Synthesized annotation guidelines are knowledge-lite boosters for clinical information extraction

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Biomedical Informatics

Background:

• Information extraction relies on annotation guidelines, which are labor-intensive and knowledge-intensive to create.
• Large language models (LLMs) are increasingly used for generative information extraction via prompting and few-shot learning.
• Existing prompts with examples resemble manual annotation guidelines, but their construction is a bottleneck.

Purpose of the Study:

• To leverage LLMs' capabilities to automatically synthesize high-quality annotation guidelines.
• To reduce the human effort and expertise required for creating annotation guidelines.
• To improve the performance of downstream LLMs in information extraction tasks.

Main Methods:

• A zero-shot hierarchical prompt engineering method was developed.
• LLMs' knowledge summarization and text generation capacities were utilized.
• The method requires minimal human input for guideline synthesis.

Main Results:

• LLM-synthesized guidelines improved zero-shot clinical named entity recognition performance on multiple benchmarks (e.g., i2b2, n2c2).
• Performance gains ranged from 0.2% to 25.86% for different LLMs compared to a no-guideline baseline.
• Synthesized guidelines performed equivalently or better than human-written guidelines in most tasks.

Conclusions:

• LLMs can generate consistent, high-quality annotation guidelines automatically.
• The pre-trained knowledge within LLMs enables guideline synthesis without explicit human instruction.
• Researchers may still need to refine definition nuances for specific project requirements.