Lifeguard : SWIM-ing with Situational Awareness.

SWIM is a peer-to-peer group membership protocol that uses randomized probing and gossip to obtain attractive scaling and robustness properties. Sensitivity to slow message processing, due to factors such as CPU exhaustion, network delay or loss, can lead SWIM to declare healthy members faulty. To counter this, SWIM adds a Suspicion mechanism, that trades increased failure detection latency for a lower false positive failure detection rate. However, relatively short lived periods of slow message processing commonly experienced in data centers can still lead to healthy members being marked as failed. observe that the Suspicion mechanism still assumes timely processing of some messages. In particular, refutation of a suspicion can only succeed if it is processed by the suspecting member in a timely manner. However, missing expected responses could indicate a member is experiencing slow message processing, and an episode of slow message processing at a given group member is likely to impact multiple of its interactions with other members in a short period of time. Based on these insights, we define a set of extensions to SWIM that allow a member to dynamically adjust its timeouts to mitigate timeliness issues. We call these extensions Lifeguard. analyze the effect of Lifeguard using synthetic benchmarks that vary message processing delays in a controlled manner. Across the wide range of cases tested, Lifeguard is able to reduce the false positive rate by a factor of more than 50x, while modestly increasing failure detection latency and message load. Furthermore, by modifying tuning parameters, Life- guard allows users to reduce median detection latency by 45% while still reducing false positives at healthy members by 3x compared to without Lifeguard. The tuning parameters allow users to choose a suitable trade-off between lower false positives and lower detection latency.