Democratizing cloud data lake analytics: natural language access to Apache Iceberg via LLM agents.

Area of Science:

• Data Engineering
• Artificial Intelligence
• Database Systems

Background:

• Enterprise data lakes are crucial for business insights but inaccessible to non-technical users due to SQL requirements.
• Existing text-to-SQL models suffer from security vulnerabilities (SQL injection) and limitations in handling data lake specifics.
• Inconsistent metric definitions across organizations hinder reliable data analysis.

Purpose of the Study:

• To develop a natural language interface for querying Apache Iceberg data lakes.
• To address security vulnerabilities and improve the accuracy of text-to-SQL systems.
• To enable business analysts to access data insights without SQL expertise or schema knowledge.

Main Methods:

• Implemented the LangChain Iceberg Toolkit utilizing large language models (LLMs) for natural language understanding.
• Employed a hybrid query execution architecture combining PyIceberg's type-safe API and DuckDB's SQL engine.
• Integrated a YAML-based semantic layer for mapping business terms to data structures and ensuring consistent metric definitions.

Main Results:

• Achieved 100% query success rate across 100 systematically designed queries.
• Demonstrated a 33 percentage-point accuracy improvement over a schema-aware text-to-SQL baseline (100% vs. 67%).
• Reduced SQL injection attack success rate to 0%, a significant improvement from prior research.

Conclusions:

• The LangChain Iceberg Toolkit democratizes enterprise data access by enabling natural language querying of Apache Iceberg data lakes.
• The system offers a secure, accurate, and efficient solution for data insights, overcoming limitations of previous text-to-SQL approaches.
• The production-ready, open-source implementation validates the practical viability of conversational data access.