Artificial Intelligence-Based Anomaly Detection Technology over Encrypted Traffic: A Systematic Literature Review.

Area of Science:

• Cybersecurity
• Network Security
• Artificial Intelligence

Background:

• The rise of cyber-attacks in unencrypted environments necessitates secure communication channels like Transport Layer Security (TLS).
• Attackers exploit protected channels, making traditional Deep Packet Inspection (DPI) based anomaly detection ineffective against encrypted traffic.
• Artificial Intelligence (AI) and statistical traffic analysis offer promising alternatives for detecting threats within encrypted communications.

Purpose of the Study:

• To systematically review existing Artificial Intelligence (AI)-based anomaly detection techniques specifically designed for encrypted network traffic.
• To identify and analyze the methodologies, datasets, feature engineering, and algorithms employed in current research.
• To provide a comprehensive overview of the state-of-the-art in AI-driven encrypted traffic anomaly detection.

Main Methods:

• Conducted a systematic literature review based on predefined research questions and eligibility criteria.
• Screened and assessed the quality of collected research articles, selecting 30 relevant studies.
• Analyzed selected studies based on dataset characteristics, feature extraction/selection, preprocessing steps, AI algorithms, and performance metrics.

Main Results:

• Identified a diverse range of AI techniques applied to anomaly detection in encrypted traffic.
• Observed that some AI techniques are adaptations from unencrypted traffic analysis, while others are novel developments for encrypted environments.
• Confirmed the feasibility and variety of AI-based approaches for discerning malicious activities within protected communication channels.

Conclusions:

• AI-based anomaly detection is a viable and evolving field for securing encrypted network traffic.
• Further research is needed to explore unique AI techniques tailored for the complexities of encrypted data.
• The findings provide a valuable resource for researchers and practitioners aiming to enhance cybersecurity in encrypted communication environments.