Field of genes: using Apache Kafka as a bioinformatic data repository

Area of Science:

• Bioinformatics
• Data Science
• Computer Science

Background:

• Bioinformatic research relies on large-scale datasets from repositories like NCBI's RefSeq.
• Current data formats (unstructured vs. highly structured) present trade-offs in usability and scalability.
• A need exists for a flexible data approach balancing structure, performance, and scale.

Purpose of the Study:

• To introduce lightly structured data storage using Apache Kafka for bioinformatic datasets.
• To demonstrate a proof-of-concept for NCBI's RefSeq database using this approach.
• To evaluate the performance and scalability compared to traditional flat files.

Main Methods:

• Implementing a proof-of-concept RefSeq database using Apache Kafka.
• Storing lightly structured data for parallel access and streamed processing.
• Measuring performance and scalability against flat file storage.

Main Results:

• The Apache Kafka-based system demonstrates near-linear scalability with added compute nodes.
• This approach significantly outperforms traditional flat file methods.

Conclusions:

• Apache Kafka provides a fast, scalable, and general-purpose solution for bioinformatic data storage.
• This technology is suitable for public reference datasets (e.g., RefSeq) and private clinical/experimental data.
• It bridges the gap between low structure and high performance/scale in bioinformatics.