Real and synthetic data sets for benchmarking key-value stores focusing on various data types and sizes

Area of Science:

• Computer Science
• Data Management
• Database Systems

Background:

• Key-value stores are crucial for big data due to their flexibility with diverse data types.
• Benchmarking these stores requires representative datasets that capture variety and volume.

Purpose of the Study:

• To present real and synthetic datasets for benchmarking key-value stores.
• To address the need for datasets with varied data types and sizes in big data applications.

Main Methods:

• Crawled four types of real-world datasets from Twitter (text, hashtags, geo-location, follower counts).
• Generated four types of synthetic datasets by varying size (volume).
• Developed algorithms for data crawling via REST and streaming APIs and for synthetic data generation.

Main Results:

• Successfully created diverse real datasets and scalable synthetic datasets.
• Demonstrated the utility of these datasets in benchmarking popular key-value stores like LevelDB, RocksDB, and Berkeley DB.
• Presented an example algorithm for basic operations on LevelDB.

Conclusions:

• The presented datasets are valuable resources for evaluating key-value store performance.
• The methods allow for the creation of tailored datasets for specific benchmarking needs.
• These resources facilitate comparisons and performance analysis in big data contexts.