Data-Driven Packet Loss Estimation for Node Healthy Sensing in Decentralized Cluster

Area of Science:

• Computer Science
• Distributed Systems
• Network Reliability

Background:

• Decentralized clustering is prevalent in modern IT for high availability and fault tolerance.
• Toolkits like Akka facilitate cluster construction, commonly employing Gossip protocols for membership management.
• Existing link failure detection mechanisms are insufficient when nodes stochastically drop packets, corrupting cluster member status.

Purpose of the Study:

• To address the limitations of Gossip-based decentralized clustering when nodes exhibit packet loss.
• To formulate the problem of maintaining cluster integrity as evaluating link quality and finding a maximum clique in the connectivity graph.
• To propose a novel algorithm capable of handling stochastic packet drops in decentralized clusters.

Main Methods:

• Formulating the problem into two sub-problems: link quality evaluation and maximum clique finding (an NP-Complete problem).
• Developing a two-model algorithm driven by application-layer data to solve these sub-problems.
• Utilizing simulations with statistical data and a real-world product for validation.

Main Results:

• The proposed algorithm effectively addresses the issue of node packet loss in Gossip-based clusters.
• Demonstrated good performance in simulations and with a real-world product.
• Successfully evaluated link quality and identified maximum cliques in the connectivity graph.

Conclusions:

• The novel algorithm enhances the robustness of decentralized clustering systems against stochastic packet drops.
• This approach improves the reliability and accuracy of membership management in distributed systems.
• The findings have practical implications for building more resilient and dependable IT infrastructure.