How does Knowledge Distillation help Federated Learning?

fixit_fiona

As per my understanding, typically in FL, there is a global server that interacts with various client devices. The global server and the client both possess a ML models. The client(s) update their models locally and then send the weights across to the server where it is averaged. The paper , Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data, has the following paragraph - "To...

Originally asked by desert_ranger on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

The logit vectors in the aforementioned paper are actually the outputs of the models and not the reference values in the dataset: Each model output is a set of logit values normalized via a softmax function, hereafter denoted as a logit vector The average here serves as a proxy for communication efficient comparison of "teacher" and "student" outputs.

Full answer by user1360448: https://ai.stackexchange.com/a/42303 (CC BY-SA 4.0).

Need it done for your business? UphillTech.