{"id":19066258,"url":"https://github.com/epfml/error-feedback-sgd","last_synced_at":"2025-04-28T12:27:37.688Z","repository":{"id":40977614,"uuid":"168591140","full_name":"epfml/error-feedback-SGD","owner":"epfml","description":"SGD with compressed gradients and error-feedback:  https://arxiv.org/abs/1901.09847","archived":false,"fork":false,"pushed_at":"2022-06-28T17:11:10.000Z","size":1196,"stargazers_count":28,"open_issues_count":2,"forks_count":9,"subscribers_count":7,"default_branch":"master","last_synced_at":"2023-10-20T09:29:05.325Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epfml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-31T20:25:11.000Z","updated_at":"2023-09-30T06:45:07.000Z","dependencies_parsed_at":"2022-08-18T21:10:59.394Z","dependency_job_id":null,"html_url":"https://github.com/epfml/error-feedback-SGD","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Ferror-feedback-SGD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Ferror-feedback-SGD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Ferror-feedback-SGD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Ferror-feedback-SGD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epfml","download_url":"https://codeload.github.com/epfml/error-feedback-SGD/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223772191,"owners_count":17199977,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T00:55:39.401Z","updated_at":"2024-11-09T00:55:40.326Z","avatar_url":"https://github.com/epfml.png","language":"Jupyter Notebook","readme":"# Error-feedback SGD\n\nWe present here the code of the experimental parts of the paper [Error Feedback Fixes SignSGD and other Gradient Compression Schemes](https://arxiv.org/abs/1901.09847).\n\nThe implementation is based on [this repository](https://github.com/kuangliu/pytorch-cifar)'s code and uses PyTorch.\n\n\n\n## Requirements\n\nThe following packages were used for the experiments. Newer versions are also likely to work.\n\n- torchvision==0.2.1\n- numpy==1.15.4\n- torch==0.4.1\n- pandas==0.23.4\n- scikit_learn==0.20.3\n\nTo install them automatically: `pip install -r requirements.txt`\n\n## Organization\n\n- `notebooks/` contains jupyter notebook files with plotted results and experiments.\n- `optimizers/` contains the custom optimizer, namely ErrorFeedbackSGD.\n- `models/` contains the deep net architectures. Only VGG and Resnet were experimented.\n- `results/` contains the results of the experiments in pickle files.\n- `utils/` contains utility functions for saving/loading objects, convex optimization, progress bar...\n- `checkpoints/` contains the saved models' checkpoints with all the nets parameters. The folder is empty here as those files are very large.\n\n## Notations\n\nA few notations in the code don't match the notations from the paper. In particular,\n\n- What is called signSGD in the paper is the scaled sign SGD in the code, as the gradients are rescaled by their norm.\n- What is called ef-signSGD in the paper is scaled sign SGD with memory in the code. The `memory` parameter can also be used with other compressions besides the sign.\n\n## Usage\n\n- `main.py` can be called from the command line to run a single network training and testing. It can take a variety of optional arguments. Type `python main.py --help` for further details.\n- `utils.hyperparameters.py` facilitate the definition of all the hyper-parameters of the experiments.\n- `tune_lr.py` allows to tune the learning rate for a network architecture/data set/optimizer configuration.\n- `main_experiments.py` contains the experiments presented in the paper, section 6.\n\n# Reference\nIf you use this code, please cite the following [paper](http://proceedings.mlr.press/v97/karimireddy19a/karimireddy19a-supp.pdf)\n\n    @inproceedings{karimireddy19a,\n      title = \t {Error Feedback Fixes {SignSGD} and other Gradient Compression Schemes},\n      author = \t {Karimireddy, Sai Praneeth and Rebjock, Quentin and Stich, Sebastian U. and Jaggi, Martin},\n      booktitle = \t {ICML - Proceedings of the 36th International Conference on Machine Learning},\n      pages = \t {3252--3261},\n      year = \t {2019}\n    }\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Ferror-feedback-sgd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepfml%2Ferror-feedback-sgd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Ferror-feedback-sgd/lists"}