https://github.com/sarodyatawatta/federated-pytorch-test
Federated learning with PyTorch (federated averaging and consensus optimization): with 'reduced' bandwidth
https://github.com/sarodyatawatta/federated-pytorch-test
clustering consensus-model consensus-optimization deep-learning federated-learning lbfgs pytorch sagecal variational-autoencoder
Last synced: 25 days ago
JSON representation
Federated learning with PyTorch (federated averaging and consensus optimization): with 'reduced' bandwidth
- Host: GitHub
- URL: https://github.com/sarodyatawatta/federated-pytorch-test
- Owner: SarodYatawatta
- License: apache-2.0
- Created: 2019-08-11T10:40:21.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2024-04-30T11:30:43.000Z (over 1 year ago)
- Last Synced: 2024-04-30T13:05:08.359Z (over 1 year ago)
- Topics: clustering, consensus-model, consensus-optimization, deep-learning, federated-learning, lbfgs, pytorch, sagecal, variational-autoencoder
- Language: Python
- Homepage:
- Size: 301 KB
- Stars: 40
- Watchers: 2
- Forks: 7
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# federated-pytorch-test
We train CNN models __without__ having access to the full dataset. The CIFAR10 dataset is used in all examples. The CNN models can be chosen from simpler models similar to PyTorch or Tensorflow demos and also ResNet18. In all cases, we use only 1/K (where K is user defined) of the data for training each CNN model. We also compare the performance of federated averaging and consensus optimization in training the K models, without sharing the training data between models. Note that we only pass __a subset of parameters__ between the models, unlike in normal federated averaging or consensus. This __reduces the bandwidth__ required enormously!
The stochastic LBFGS optimizer is provided with the code. Further details are given [in this paper](https://ieeexplore.ieee.org/document/8755567). Also see [this introduction](http://sagecal.sourceforge.net/pytorch/index.html).
GPU acceleration is enabled when available, set ```use_cuda=True```.
Files included are:
``` lbfgsnew.py ```: New LBFGS optimizer
``` simple_models.py ```: Relatively simple CNN models for CIFAR10, derived from PyTorch/Tensorflow demos, also ResNet18
``` no_consensus_multi.py ```: Train K models using 1/K of the training data for each model
``` federated_multi.py ```: Train K models using 1/K of the data, with federated averaging, K can be varied
``` fedprox_multi.py ```: Train K models using 1/K of the data, with federated proximal algorithm, K can be varied, based on [this paper](https://arxiv.org/abs/1812.06127)
``` consensus_multi.py ```: Train K models using 1/K of the data, with consensus optimization (adaptive) ADMM, K can be varied
``` federated_vae.py ```: Train K variational autoencoders, using federated averaging
``` federated_vae_cl.py ```: Train K variational autoencoders for clustering, using federated averaging, based on [this paper](https://arxiv.org/abs/2005.04613)
``` federated_cpc.py ```: Train K models using contrastive predictive coding, using LOFAR data, based on [this paper](https://arxiv.org/abs/1905.09272)
This images compares training K=1 and K=10 models, stad alone training using ```no_consensus_multi.py```, with consensus optimization ```consensus_multi.py``` and with federated averaging ```federated_multi.py```. The upper bound is using the full dataset for training ( K=1 ) while using 1/K of the data gives the lower bound.