Scalable optimization methods for Distributed Machine Learning

As data grows, there is a need to develop scalable, efficient and most importantly distributed machine learning algorithms. Distributed algorithms fall into two main categories (a) horizontal partitioning, where the data is distributed across multiple slaves. The main drawback of this strategy is that the model parameters need to be replicated on every machine. This is problematic when the number of classes, and consequently the number of parameters is very large, and hence cannot fit in a single machine. The other strategy is (b) vertical partitioning, where the model parameters are partitioned. However, here the data needs to be replicated on each machine, thus failing to scale to massive datasets.