The COLIEE 2025 Competition on Legal Information Extraction and Entailment: Overview, Discussion, and Dataset...

Area of Science:

• Legal Informatics
• Artificial Intelligence in Law
• Natural Language Processing for Legal Text

Background:

• The 12th Competition on Legal Information Extraction and Entailment (COLIEE) aimed to advance AI capabilities in legal text analysis.
• This edition expanded beyond traditional case and statute law to include a pilot task on Tort law prediction and rationale extraction.

Purpose of the Study:

• To evaluate diverse AI approaches for legal information retrieval and entailment.
• To identify effective methods for tasks such as case law retrieval, statute law question answering, and Tort law prediction.

Main Methods:

• Utilized a combination of traditional Information Retrieval (IR) techniques (e.g., BM25) and advanced neural re-ranking methods.
• Employed Large Language Models (LLMs) with prompt engineering for complex legal reasoning and question-answering tasks.
• Developed multi-stage retrieval pipelines integrating initial candidate selection with LLM-based re-ranking.

Main Results:

• Dominant approaches in case law retrieval (Task 1) and statute law retrieval (Task 3) featured multi-stage pipelines combining IR and neural re-ranking.
• The statute law entailment/QA task (Task 4) saw top performance using LLMs with prompt engineering, achieving 0.9041 accuracy.
• The pilot Tort law tasks showed strong performance with LLMs, with specific models excelling in prediction (acc.=0.765) and rationale extraction (F1=0.712).

Conclusions:

• Multi-stage retrieval pipelines and LLMs are highly effective for legal information extraction and entailment tasks.
• The success in statute law tasks (3 and 4) suggests a strong potential for automated legal reasoning in civil codes.
• The competition's results support the introduction of a new statute retrieval task in future COLIEE editions.