Co-Designing Systems for Microsecond-Scale Consistent Tail Latency in Datacenters
COM1 Level 3
MR1, COM1-03-19
Abstract:
Modern datacenter applications increasingly demand consistent microsecond-scale tail latency to meet stringent performance requirements. While advances in I/O devices and operating systems have made such performance levels attainable, critical challenges remain – including coordination overheads across distributed servers, unpredictable workload imbalance, and static OS configurations that fail to adapt to dynamic workloads.
Our research introduces three systems that address these challenges through two co-design principles: (1) co-designing distributed systems with network hardware to accelerate distributed consensus and enable dynamic load balancing, and (2) co-designing dataplane operating systems with machine learning for automatic performance tuning. First, Hydra reduces coordination overheads in distributed consensus by leveraging programmable switches as a distributed set of sequencers to order request messages directly in the network, improving median latency by up to 13× and throughput by 47%. Second, Capybara addresses unpredictable workload imbalance by co-designing library operating systems with programmable switches to enable live migration of layer-4 protocol states (e.g., TCP) across servers, reducing tail latency by over 99% under realistic workloads. Finally, we introduce our ongoing work on designing an ML-native dataplane OS architecture for automatic parameter tuning, presenting case studies that demonstrate substantial performance gains through dynamic parameter optimization, such as a 2× improvement in packet transmission rate.
Overall, this thesis provides practical solutions to critical datacenter performance challenges, demonstrating how cross-layer systems co-design can effectively achieve consistent microsecond-scale tail latency for modern datacenter applications.