Mynodbcsv: lightweight zero-config database solution for handling very large CSV files

Area of Science:

• Computer Science
• Data Science
• Database Systems

Background:

• Rapid growth in scientific and industrial data volumes presents analysis challenges.
• Data format heterogeneity and lack of standardization hinder efficient exploration.
• Traditional databases struggle with datasets having a large number of columns (wide data).

Purpose of the Study:

• To present a system for accessing very large Comma-Separated Values (CSV) datasets using Structured Query Language (SQL).
• To overcome the limitations of traditional database systems in handling wide-format datasets.
• To provide a user-friendly, efficient, and low-overhead solution for CSV data analysis.

Main Methods:

• Developed a C++ based system with a "no copy" approach, keeping data in original CSV files.
• Implemented "zero configuration" for schema-less data access.
• Utilized query rewriting, dynamic index creation, and direct data retrieval for efficient query execution.
• Supported millions of columns and per-value typing for mixed data.

Main Results:

• The system efficiently accesses large CSV datasets via SQL, even with millions of columns.
• Demonstrated "no copy" and "zero configuration" for simplified data handling.
• Achieved high performance through query optimization and direct CSV access.
• Showcased compatibility with MATLAB via a simple network protocol.

Conclusions:

• The presented system offers a powerful and flexible solution for analyzing large, wide-format CSV datasets.
• It significantly reduces the effort required for data exploration compared to traditional methods.
• The freeware system requires no installation and minimal prerequisites, making it accessible for researchers.