Redefining text-to-SQL metrics by incorporating semantic and structural similarity.

Area of Science:

• Artificial Intelligence
• Database Management
• Natural Language Processing

Background:

• Text-to-SQL systems are rapidly advancing, necessitating sophisticated benchmarks.
• Current evaluation metrics for text-to-SQL lack granularity, failing to capture nuances in SQL query equivalence.
• Existing metrics overlook partial correctness, structural variations, and semantic equivalence.

Purpose of the Study:

• To propose a novel metric for SQL query comparison that addresses limitations of current evaluation methods.
• To provide a more precise assessment of SQL query similarity at both semantic and execution result levels.
• To enable more accurate ranking and development of text-to-SQL tools and models.

Main Methods:

• Developed a new metric for comparing SQL queries based on semantic and execution result similarity.
• Designed the metric for granular evaluation of SQL query differences.
• Utilized distribution analysis to compare model performance.

Main Results:

• The proposed metric offers a more precise assessment of SQL query similarity than existing methods.
• Experimental results demonstrate the metric's effectiveness in evaluating text-to-SQL models.
• The metric can identify specific query differences, such as missing operators or variations in ordering.

Conclusions:

• The novel metric significantly improves the evaluation of text-to-SQL systems.
• This approach facilitates the distinction between models handling simple versus complex queries.
• The metric provides valuable training signals for developing more accurate language models for SQL generation.