A fully scalable online pre-processing algorithm for short oligonucleotide microarray atlases

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Large-scale microarray data collections are rapidly accumulating, offering potential for holistic genome function characterization.
• Current preprocessing techniques lack scalability, hindering the full utilization of these valuable genome-wide profiling data resources.
• Existing scalable probe-level techniques are limited to specific platforms and rely on restricted training sets.

Purpose of the Study:

• To introduce a fully scalable online-learning algorithm for probe-level analysis and preprocessing of large microarray datasets.
• To overcome the scalability limitations of current microarray data preprocessing methods.
• To enable the analysis of tens of thousands of arrays for comprehensive genome function characterization.

Main Methods:

• Development of a fully scalable online-learning algorithm for probe-level analysis.
• Application of the algorithm to large microarray atlases containing tens of thousands of arrays.
• Sequential hyperparameter updates using small, consecutive batches of data for efficient memory usage.

Main Results:

• The algorithm scales linearly with sample size, making it applicable to all short oligonucleotide platforms.
• The model can identify individual probes affected by noise and biases using comprehensive data collections.
• This approach circumvents extensive memory requirements of standard methods.

Conclusions:

• The developed algorithm provides a scalable solution for probe-level analysis of large microarray collections.
• It offers tools to guide array design and quality control by pinpointing noisy or biased probes.
• This novel approach unlocks new opportunities for leveraging contemporary microarray data for genome research.