A General Framework for Fast Co-clustering on Large Datasets Using Matrix Decomposition.

Area of Science:

• Data Mining
• Machine Learning
• Computational Statistics

Background:

• Co-clustering, simultaneously grouping rows and columns of data matrices, is vital for applications like document analysis and recommendation systems.
• Existing co-clustering algorithms often exhibit high time complexity (O(m x n)) and significant memory requirements, limiting scalability for large datasets.
• The need for efficient co-clustering methods that can handle large-scale data without requiring full in-memory storage is critical.

Purpose of the Study:

• To propose a general framework, CRD, for scalable co-clustering of large data matrices.
• To develop a co-clustering approach with linear time complexity (O(m+n)) and reduced memory footprint.
• To demonstrate the effectiveness and efficiency of CRD compared to existing methods on diverse datasets.

Main Methods:

• CRD framework leverages sampling-based matrix decomposition techniques for efficient processing.
• The proposed method achieves linear time complexity with respect to the dimensions of the data matrix (m rows, n columns).
• CRD operates without necessitating the entire data matrix to be loaded into main memory, enabling scalability.

Main Results:

• Extensive experiments on synthetic and real-world datasets validate the performance of CRD.
• CRD achieves competitive accuracy compared to established co-clustering algorithms.
• The computational cost of CRD is significantly lower than traditional methods, demonstrating its efficiency.

Conclusions:

• CRD provides a scalable and computationally efficient solution for co-clustering large data matrices.
• The framework's ability to handle large datasets with reduced memory requirements opens new possibilities for various data mining applications.
• CRD represents a significant advancement in co-clustering methodology, offering a practical alternative for big data analysis.