Self organization of a massive document collection.

Area of Science:

• Computer Science
• Information Retrieval
• Machine Learning

Background:

• Organizing large document collections is challenging.
• Textual similarity is key for document organization.
• Scaling machine learning algorithms is crucial for big data.

Purpose of the Study:

• To implement a scalable system for organizing vast document collections.
• To adapt the self-organizing map (SOM) algorithm for high-dimensional data.
• To demonstrate the system's capability with a large-scale experiment.

Main Methods:

• Utilized the self-organizing map (SOM) algorithm.
• Employed statistical representations of document vocabularies as feature vectors.
• Developed a method to scale SOM for large, high-dimensional datasets using 500-dimensional stochastic feature vectors from weighted word histograms.

Main Results:

• Successfully mapped 6,840,568 patent abstracts onto a 1,002,240-node SOM.
• Demonstrated the scalability of the SOM algorithm for massive document collections.
• The system effectively organized documents based on textual similarities.

Conclusions:

• The implemented system provides an effective and scalable solution for organizing large document collections.
• The adapted SOM algorithm successfully handles high-dimensional data for document clustering.
• This approach has significant implications for information retrieval and knowledge management.