Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/tmulc18/distributeddeeplearningreads

Papers and blogs related to distributed deep learning
https://github.com/tmulc18/distributeddeeplearningreads

Last synced: about 2 months ago
JSON representation

Papers and blogs related to distributed deep learning

Awesome Lists containing this project

README 

        
# Distributed Deep Learning Reads



Compilation of literature related to distributed deep learning.  Pull requests welcome :)



* [100-epoch ImageNet Training with AlexNet in 24 Minutes](https://arxiv.org/abs/1709.05011)

* [Accumulated Gradient Normalization](https://arxiv.org/abs/1710.02368)

* [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://arxiv.org/pdf/1706.02677.pdf)

* [Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization](http://papers.nips.cc/paper/5751-asynchronous-parallel-stochastic-gradient-for-nonconvex-optimization.pdf)

* [Asynchrony begets Momentum, with an Application to Deep Learning](https://arxiv.org/abs/1605.09774)

* [Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations](http://www.cs.fsu.edu/~xyuan/paper/09jpdc.pdf)

* [Bringing HPC Techniques to Deep Learning](http://research.baidu.com/bringing-hpc-techniques-deep-learning/)

* [Deep learning with Elastic Averaging SGD](https://arxiv.org/abs/1412.6651)

* [Distributed Delayed Stochastic Optimization](https://arxiv.org/abs/1104.5525)

* [Don't Decay the Learning Rate, Increase the Batch Size](https://arxiv.org/abs/1711.00489)

* [FireCaffe: near-linear acceleration of deep neural network training on compute clusters](https://arxiv.org/abs/1511.00175)

* [Heterogeneity-aware Distributed Parameter Servers](https://ds3lab.org/wp-content/uploads/2017/07/sigmod2017_jiang.pdf)

* [Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent](https://people.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf)

* [How to scale distributed deep learning?](https://arxiv.org/abs/1611.04581)

* [ImageNet Training in Minutes](https://arxiv.org/abs/1709.05011)

* [Joeri Hermans ADAG Blog](http://joerihermans.com/ramblings/distributed-deep-learning-part-1-an-introduction/)

* [Large Scale Distributed Deep Networks](https://static.googleusercontent.com/media/research.google.com/en//archive/large_deep_networks_nips2012.pdf)

* [Meet Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow](https://eng.uber.com/horovod/)

* [More Effective Distributed ML via a Stale

Synchronous Parallel Parameter Server](http://repository.cmu.edu/cgi/viewcontent.cgi?article=1163&context=machine_learning)

* [Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs](https://arxiv.org/abs/1606.04487)

* [On Parallelizability of Stochastic Gradient Descent for Speech DNNs](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ParallelSGD-ICASSP2014-published.pdf)

* [On Scalable Deep Learning and Parallelizing Gradient Descent](https://github.com/JoeriHermans/master-thesis/tree/master/thesis)

* [One weird trick for parallelizing convolutional neural networks](https://arxiv.org/abs/1404.5997)

* [Parallel training of DNNs with Natural Gradient and Parameter Averaging](https://arxiv.org/abs/1410.7455)

* [Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines](https://arxiv.org/abs/1512.06216)

* [PowerAI DDL](https://arxiv.org/abs/1708.02188)

* [Revisiting Distributed Synchronous SGD](https://arxiv.org/pdf/1604.00981.pdf)

* [Scalable Distributed DNN Training Using Commodity GPU Cloud Computing](https://s3-us-west-2.amazonaws.com/amazon.jobs-public-documents/strom_interspeech2015.pdf)

* [SparkNet: Training Deep Networks in Spark](https://arxiv.org/abs/1511.06051)

* [Staleness-aware Async-SGD for Distributed Deep Learning](https://arxiv.org/abs/1511.05950)

* [TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning](https://arxiv.org/abs/1705.07878)