https://github.com/feifeibear/communication-efficient-dnn
https://github.com/feifeibear/communication-efficient-dnn
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/feifeibear/communication-efficient-dnn
- Owner: feifeibear
- License: mit
- Created: 2018-04-12T20:40:43.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-05-24T22:08:35.000Z (about 7 years ago)
- Last Synced: 2025-01-23T00:41:16.822Z (4 months ago)
- Language: Python
- Size: 83 KB
- Stars: 3
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: Change.txt
- License: LICENSE
Awesome Lists containing this project
README
# Communication Efficient Deep Learning Training Method
This is a code repository used to reproduce the results in paper [DEEP GRADIENT COMPRESSION: REDUCING THE COMMUNICATION BANDWIDTH FOR DISTRIBUTED TRAINING](https://arxiv.org/pdf/1712.01887.pdf)
It is based off [convNet.pytorch](https://github.com/eladhoffer/convNet.pytorch) with some helpful options such as:
- Simulate data parallel senario on multiple machines on one GPU
- Gradient pruning
- Training on several datasets
- Complete logging of trained experiment
- Graph visualization of the training/validation loss and accuracy
- Definition of preprocessing and optimization regime for each model
## Dependencies
- [pytorch]()
- [torchvision]() to load the datasets, perform image transforms
- [pandas]() for logging to csv
## Data
- Configure your dataset path at **data.py**.
- To get the ILSVRC data, you should register on their site for access:
## Experiment examples
```bash
sh ./submit_pruning.sh
```
- See *run_experiments.sh* for more examples
## Model configuration
Network model is defined by writing a .py file in models folder, and selecting it using the model flag. Model function must be registered in models/\_\_init\_\_.py
The model function must return a trainable network. It can also specify additional training options such optimization regime (either a dictionary or a function), and input transform modifications.
e.g for a model definition:
```python
class Model(nn.Module):
def __init__(self, num_classes=1000):
super(Model, self).__init__()
self.model = nn.Sequential(...)
self.regime = {
0: {'optimizer': 'SGD', 'lr': 1e-2,
'weight_decay': 5e-4, 'momentum': 0.9},
15: {'lr': 1e-3, 'weight_decay': 0}
}
self.input_transform = {
'train': transforms.Compose([...]),
'eval': transforms.Compose([...])
}
def forward(self, inputs):
return self.model(inputs)
def model(**kwargs):
return Model()
```
## Author
Jiarui Fang
[email protected]