Merging Models Pre-Trained on Different Features with Consensus Graph

Learning global models effectively on private and decentralized datasets has become an increasingly important challenge of machine learning when applied in practice. Federated Learning (FL) has recently emerged as a solution paradigm to address this challenge. In particular, the FL clients agree to a common model parameterization in advance, which can then be updated collaboratively via synchronous aggregation of their local model updates. However, such strong requirement of modeling homogeneity and synchronicity across clients makes FL inapplicable to many practical learning scenarios that cannot afford such requirements. For example, in distributed sensing, a network of heterogeneous sensors sample from different data modalities of the same phenomenon. Each sensor thus requires its own specialized model. Local learning therefore needs to happen in isolation but inference still requires merging the local models for better performance. To enable this, we investigate a feature fusion approach that extracts local feature representations from local models and incorporates them into a global representation to train a more holistic predictive model. We study two key aspects of this feature incorporation. First, we develop an alignment algorithm that draws accurate correspondence between feature components which are arbitrarily arranged across clients. Next, we propose learning a consensus graph that captures the high-order interactions between these feature components, which reveals how data with heterogeneous features can be stitched together coherently to train a better model. The proposed framework is demonstrated on four real-life data sets including monitoring and predicting power grids and traffic networks.