LLM ethics benchmark: a three-dimensional assessment system for evaluating moral reasoning in large language models

Area of Science:

• Artificial Intelligence
• AI Ethics
• Computational Linguistics

Background:

• Large language models (LLMs) are increasingly used in critical societal roles.
• Existing methods for evaluating AI ethical decision-making are insufficient, leading to accountability gaps.
• There is a need for precise evaluation of AI moral reasoning to ensure alignment with human values.

Purpose of the Study:

• To establish a novel framework for systematically evaluating the moral reasoning capabilities of LLMs.
• To address the limitations of current assessment methodologies in evaluating nuanced ethical decision-making.
• To enable precise identification of ethical strengths and weaknesses in LLMs.

Main Methods:

• Developed a framework to quantify LLM alignment with human ethical standards.
• Evaluated LLMs across three dimensions: foundational moral principles, reasoning robustness, and value consistency.
• Utilized benchmark datasets and an evaluation codebase for systematic assessment.

Main Results:

• The framework enables precise identification of ethical strengths and weaknesses in LLMs.
• Quantified alignment with human ethical standards across diverse scenarios.
• Facilitated targeted improvements for stronger LLM alignment with societal values.

Conclusions:

• The novel framework provides a systematic approach to evaluating LLM moral reasoning.
• Public release of datasets and codebase promotes transparency and collaborative advancement in ethical AI.
• The study contributes to bridging accountability gaps in AI development and deployment.