Distributed Fast Self-Organized Maps for Massive Spectrophotometric Data Analysis ^†

Area of Science:

• Data Science
• Astronomy
• Distributed Computing

Background:

• Massive datasets (hundreds of Gigabytes) necessitate distributed computing for knowledge extraction.
• Classical algorithms require adaptation for parallel processing to handle Big Data.

Purpose of the Study:

• To propose a parallel, scalable, and optimized Self-Organized Map (SOM) design.
• To analyze massive data from the European Space Agency's Gaia spacecraft.
• To enable extrapolation of the methodology to other domains.

Main Methods:

• Development of a distributed Self-Organized Map (SOM) implementation.
• Performance comparison of sequential vs. distributed SOMs using Apache Hadoop and Apache Spark.
• Analysis of proposed optimizations for distributed SOMs.

Main Results:

• The distributed SOM design demonstrates enhanced performance for Big Data analysis.
• Apache Hadoop and Apache Spark implementations show significant scalability.
• Optimizations lead to efficient processing of large-scale datasets.

Conclusions:

• The proposed distributed SOM is effective for analyzing massive astronomical data.
• The methodology is adaptable to various Big Data domains.
• A domain-specific visualization tool aids in exploring astronomical SOMs.