Efficient Compression and Indexing for Highly Repetitive DNA Sequence Collections

In this paper, we focus upon the important problem of indexing and searching highly repetitive DNA sequence collections. Given a collection G of t sequences S_i of length n each, we can represent G succinctly in 2nH_k(T) + O(n^' loglogn) + o(q n^') + o(tn) bits using O(t n² + q n^') time, where H_k(T) is the kth-order empirical entropy of the sequence T ∈ G that is used as the reference sequence, n^' is the total number of variations between T and the sequences in G, and q is a small fixed constant. We can restore any length len substring S[ sp, ..., sp + len-1] of S ∈ G in O(n_s^' + len(logn)² / loglogn) time and report all positions where P occurs in G in O(m ·t + occ ·t ·(logn)²/loglogn ) time. In addition, we propose a dynamic programming method to find the variations between T and the sequences in G in a space-efficient way, with which we can build succinct structures to enable efficient search. For highly repetitive sequences, experimental results on the tested data demonstrate that the proposed method has significant advantages in space usage and retrieval time over the current state-of-the-art methods. The source code is available online.