Addressing Label Skews in Federated Learning

Abstract:

Federated learning (FL) collaboratively trains machine learning model across different parties without exposing private data, addressing privacy concerns and adhering to regulations like GDPR. In practice, FL faces label skews, where some clients have few or no samples for some classes. Label skews lead to biased updates, lower accuracy, and slow or unstable training, especially with few rounds or in one-shot settings.

This thesis studies both multi-round and one-shot FL. To address label skews in multi-round FL where bandwidth is not a bottleneck, We introduce FedConcat, a method that concatenates local models' feature extractors to form the global model, which effectively keeps local knowledge. By using clustering techniques to group clients based on label distributions, we reduce the global model's size. FedConcat achieves higher accuracy and lower communication costs than existing methods.

One-shot FL means FL with only one communication round, which suits scenarios where bandwidth is a bottleneck. To address label skews in one-shot FL, we propose FedOV (Federated Learning by Open-set Voting). This approach allows models to admit uncertainty during inference. We propose data destruction and adversarial outlier enhancement techniques to further improve accuracy.

We also give theory for the learnability of one-shot FL via open-set learning (OSL). Guided by this, we benchmark 12 OSL methods and run comprehensive experiments. We find that synthetic outlier generation and self-supervised prediction reliably raise ensemble accuracy under label skews. Based on this, we propose FedAdav, which adaptively fuses complementary OSL signals. It substantially improves one-shot FL on label skews.