Deter: Deterministic Tcp Replay For Performance Diagnosis

TCP performance problems are notoriously difficult to diagnose because subtle differences in TCP parameters and features may lead to completely different performance. The gold standard for diagnosis is to collect packet traces and trace TCP executions. However, it is not easy to use these tools in large-scale data centers where many TCP connections interact with each other. In this paper, we introduce DETER, a deterministic TCP replay tool, which runs lightweight recording all the time at all the hosts and then replays selected collections where operators can collect packet traces and trace TCP executions for diagnosis. The key challenge for deterministic TCP replay is the butterfly effect-a small timing variation causes a chain reaction between TCP and the network that drives the system to a completely different state in the replay. To eliminate the butterfly effect, we propose to replay individual TCP connection separately and capture all the interactions between a connection with the applications and the network. We show that DETER has low recording overhead and can help diagnose many TCP performance problems such as long latency related to receive buffer shrinking, zero windows, late fast retransmission, frequent retransmission timeout, and problems related to the switch shared buffer.