Anomaly Detection in Large-Scale Networks With Latent Space Models.

Area of Science:

• Computer Science
• Network Security
• Data Mining

Background:

• Large, sparse networks present challenges for real-time anomaly detection.
• Existing methods struggle with dynamic network activity and computational complexity.

Purpose of the Study:

• To develop an efficient real-time anomaly detection method for directed activity on large, sparse networks.
• To improve the detection of network security threats by modeling latent network dynamics.

Main Methods:

• A dynamic logistic model incorporating sender/receiver latent factors and popularity scores was developed.
• Latent nodal attributes were estimated using a variational Bayesian approach with time-varying capabilities.
• A case-control approximation was employed to reduce computational complexity from O(N^2) to O(E).

Main Results:

• The algorithm was tested on enterprise network event records from over 25,000 computers.
• The method successfully identified a red team attack.
• Detection rates were significantly improved compared to models lacking latent interaction terms.

Conclusions:

• The developed anomaly detection method is effective for large, sparse networks.
• The inclusion of latent factors and a case-control approximation enhances detection efficiency and accuracy.
• This approach offers a promising solution for real-time network security monitoring.