Sub-Linear Scalability of MQTT Clusters in Topic-Based Publish-Subscribe Applications

Message Queuing Telemetry Transport (MQTT) is a widespread protocol for topic-based publish-subscribe architectures supporting IoT and social networks applications. MQTT brokers are logical entities that couple publishers and subscribers and play a critical role in such architectures. MQTT brokers can be implemented either as standalone servers or in a cluster configuration. Clusters of brokers increase reliability, availability and overall performance, since operations can be highly parallelized among the brokers that form the cluster. The load-balancing strategy in a cluster usually consists in connecting an incoming client to a randomly selected broker. This random-attach strategy, it is very simple, but generates a significant amount of inter-broker traffic, as we demonstrate through theoretical and experimental evaluations. Inter-broker traffic is an overhead for the system and it increases the CPU load of the brokers, compromising the scaling behavior of the whole cluster. Indeed, we found that a linear increase of the number of brokers forming a cluster does not necessarily provide an equivalent linear gain in performance, and such a scaling penalty can be surprisingly significant, in the order of 40%. To solve this issue and improve performance, we propose a novel load-balancing strategy that envisages the use of multiple MQTT sessions per client to reduce inter-broker traffic and that can be implemented by means of a greedy algorithm. We show feasibility and effectiveness of our strategy for IoT and social-network applications by means of simulations and real measurements. The resulting scaling penalty is reduced to 10%.