Uncovering Large Language Model Weaknesses in Character and Word Understanding and Manipulating.

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence (AI)

Background:

• Large language models (LLMs) demonstrate advanced capabilities in various NLP tasks, often matching or exceeding human performance.
• Despite their strengths, LLMs exhibit significant limitations in fundamental character and word manipulation, hindering practical applications like content creation and text editing.

Purpose of the Study:

• To introduce the Character and Word Understanding and Manipulation (CWUM) benchmark for evaluating LLMs in Chinese and English.
• To comprehensively assess the performance of nine advanced LLMs on CWUM tasks, identifying specific areas of deficiency.

Main Methods:

• Development of the CWUM benchmark, comprising 23 diverse text editing tasks (counting, identification, insertion, reversal).
• Evaluation of nine state-of-the-art LLMs on the CWUM benchmark, analyzing performance through quality and quantity metrics.
• Investigation of various methods to enhance LLM performance on CWUM, including supervised fine-tuning (SFT).

Main Results:

• LLMs demonstrated significant failures on CWUM tasks, performing substantially worse than humans on tasks requiring basic text manipulation.
• Analysis revealed specific deficiencies in LLMs' fundamental language understanding and manipulation capabilities.
• Supervised fine-tuning (SFT) proved effective in improving LLM performance on CWUM tasks while preserving generalization to new tasks.

Conclusions:

• Current LLMs struggle with essential character and word editing, despite their advanced NLP abilities.
• The CWUM benchmark effectively highlights these limitations, providing a critical tool for future LLM development.
• Targeted training methods like SFT can enhance LLM performance in text manipulation without compromising broader language understanding.