A Robust Approximation for Multiclass Multiserver Queues with Applications to Microservices Systems.

Model-based management of software applications in the cloud is based on predicted delays at scaled out services. These services are modeled as FIFO (first-in first-out) multiservers, with many servers, users and types of operation (classes of service). Efficient approximations for these multiservers either scale badly for large systems, or have convergence and accuracy problems. This work investigates three scalable approximations in depth. The best (called AB) combines class aggregation and a binomial approximation to the queue state (which assumes that users behave independently). Over the parameters of greatest relevance, two-thirds of the errors are less than 5%. The largest errors, up to about 30%, occur near the onset of saturation.