SecDedoop: Secure Deduplication with Access Control of Big Data in the HDFS/Hadoop Environment.

Area of Science:

• Computer Science
• Data Storage
• Cybersecurity

Background:

• Data deduplication is crucial for minimizing storage costs but faces security and space challenges.
• Big data environments like Hadoop Distributed File System (HDFS) require efficient and secure storage solutions.
• Existing deduplication methods often lack robust security and optimized resource allocation.

Purpose of the Study:

• To propose a secure data deduplication system (SecDedoop) with access control for big data on HDFS.
• To enhance data confidentiality and storage efficiency in distributed file systems.
• To address security vulnerabilities and space limitations in current data deduplication techniques.

Main Methods:

• Implemented elliptic curve cryptography for secure data confidentiality and key management.
• Utilized TF-IDF for word weighting and SHA-3 hashing for duplicate data detection within file chunks.
• Employed a modified Particle Swarm Optimization (PSO)-based MapReduce model for optimal data node selection.
• Integrated MongoDB for fast file indexing and modified Fuzzy C-Means (FCM) clustering for file organization.

Main Results:

• SecDedoop successfully achieves secure data deduplication with enhanced confidentiality.
• The system demonstrates improved storage efficiency by storing only unique data copies.
• Optimized data node selection using PSO and efficient indexing/clustering with MongoDB and FCM contribute to performance gains.
• Performance evaluation using various metrics indicates SecDedoop outperforms previous approaches.

Conclusions:

• SecDedoop provides a secure and efficient solution for big data deduplication on HDFS.
• The integration of advanced cryptographic and optimization techniques addresses key challenges in data storage.
• The proposed system offers a robust framework for managing large-scale, sensitive datasets securely and cost-effectively.