MirathQA: A dataset for evaluating large language models on Hanbali Islamic inheritance reasoning tasks

Area of Science:

• Computational Linguistics
• Artificial Intelligence
• Islamic Jurisprudence

Background:

• Arabic large language models (LLMs) face challenges in exact arithmetic and multi-step reasoning, particularly in specialized domains like Islamic inheritance law ('ilm al-farā'iḍ/علم الفرائض).
• Existing Arabic datasets are insufficient for systematically evaluating LLM performance in complex legal rule application, heir identification, and calculation adjustments like proportional reduction ('awl/عول) and return (radd/رد).

Purpose of the Study:

• To introduce MirathQA, a novel, comprehensive dataset designed to benchmark Arabic LLM capabilities in Islamic inheritance law, specifically focusing on the Hanbali school of jurisprudence (المذهب الحنبلي).
• To provide a resource for advancing AI applications in legal reasoning, education, and interdisciplinary research within Islamic legal contexts.

Main Methods:

• Curated a dataset of 1394 questions from 242 real-life Islamic inheritance cases sourced from Arabic materials.
• Each case includes a textual description, verified solutions, heir shares, and annotations for blocked heirs (ḥijb/حجب) and other complexities.
• Developed additional multiple-choice and true/false questions to assess specific reasoning skills, and structured the data into training (70%), validation (15%), and test (15%) splits at the case level.

Main Results:

• The MirathQA dataset offers a robust benchmark for evaluating Arabic LLMs on tasks requiring precise legal reasoning, sequential rule application, and handling of specific inheritance scenarios.
• The dataset includes detailed annotations for heirs, blocked heirs, and calculations, facilitating in-depth analysis of LLM performance in Islamic inheritance law.
• Data is provided in both CSV and Excel formats, ensuring accessibility for diverse research and development needs.

Conclusions:

• MirathQA addresses the critical need for structured Arabic datasets in evaluating AI for Islamic inheritance law, promoting systematic assessment and development.
• This resource is valuable for advancing Arabic LLM reasoning capabilities in domain-specific legal applications and supporting educational initiatives.
• The dataset supports interdisciplinary research at the intersection of AI, law, and computational linguistics.