Characterization and Implications for Federated Learning Systems

Abstract:

Data is the new oil of the 21st century. However, its full potential is often hindered by isolation into data islands due to privacy concerns or communication bandwidth, limiting its ability to serve as a shared resource. Applying machine learning techniques to distributed datasets presents significant challenges. Federated learning seeks to bridge this gap by enabling collaborative data usage across multiple parties without direct data sharing.

Despite the promise, federated learning systems face several critical challenges affecting their efficiency. First, the distributed nature of the approach results in heterogeneous data distribution among different data owners. Second, the gradient aggregation process in federated learning incurs substantial communication overhead, especially when dealing with large models or numerous clients. Third, practical federated platforms require a robust and efficient aggregation algorithm to integrate gradients from diverse sources. Designing such platforms is a non-trivial task due to privacy, data heterogeneity, communication cost constraints, and their interplay inherent to the FL systems.

This thesis addresses these efficiency challenges in federated learning systems. First, we propose a benchmark system highlighting two key issues: Heterogeneous data distribution and communication overhead. Then, to tackle these challenges, we develop two solutions: One is a generalized neuron-matching algorithm that performs layer-wise matching on machine learning models to address data distribution disparities. The other is a hybrid gradient compression method to reduce communication costs during federated training. Finally, we propose a new federated learning framework that integrates these solutions with optimization techniques, providing an open and convenient platform for federated learning algorithm development.