https://github.com/thomashirtz/noisy-networks

Minimal implementation of the network layers of the paper "Noisy Networks for Exploration" using Pytorch.
https://github.com/thomashirtz/noisy-networks

deep-learning deep-neural-networks machine-learning noisy-networks python pytorch reinforcement-learning

Last synced: about 2 hours ago
JSON representation

Minimal implementation of the network layers of the paper "Noisy Networks for Exploration" using Pytorch.

• Host: GitHub
• URL: https://github.com/thomashirtz/noisy-networks
• Owner: thomashirtz
• Created: 2020-10-26T15:43:17.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2024-03-11T21:55:15.000Z (8 months ago)
• Last Synced: 2024-03-11T22:50:50.253Z (8 months ago)
• Topics: deep-learning, deep-neural-networks, machine-learning, noisy-networks, python, pytorch, reinforcement-learning
• Language: Python
• Homepage:
• Size: 30.3 KB
• Stars: 7
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# noisy-networks

This repository provide a minimal implementation of the paper [Noisy Networks for Exploration](https://arxiv.org/pdf/1706.10295.pdf) using Pytorch. Integration examples of this network on reinforcement learning algorithms/tasks are avalaible on my [reinforcement-learning](https://github.com/thomashirtz/reinforcement-learning) github repository.

### Principle 

Noisy layers are similar linear layers, except that a noise that can be tuned during the training (sigma) is added.



Become:  



### DQN Implemetation example

```python

import torch.nn as nn

class DQN(nn.Module):

    def __init__(self, input_features, output_features, hidden_units):

        super().__init__()

        self.fc_1 = nn.Linear(input_features, hidden_units)

        self.fc_2 = nn.Linear(hidden_units, output_features)

    def forward(self, x):

        x = F.relu(self.fc_1(x))

        return self.fc_2(x)

```

Become:  

```python

import torch.nn as nn

from noisynetworks import FactorisedNoisyLayer

class DQN(nn.Module):

    def __init__(self, input_features, output_features, hidden_units):

        super().__init__()

        self.noisy_layer_1 = FactorisedNoisyLayer(input_features, hidden_units)

        self.noisy_layer_2 = FactorisedNoisyLayer(hidden_units, output_features)

    def forward(self, x):

        x = F.relu(self.noisy_layer_1(x))

        return self.noisy_layer_2(x)

```

For replay function that works with batch, the rest of the code is almost unchanged, since by default, everytime the forward loop is called the noise will change. The other exploration techniques such as epsilon-greedy can be removed.

### Implementation details

`nn.Parameter()` is indispensable when using tensor as custom parameter, otherwise, the optimizer will not know that they exist.  

`self.register_buffer()` in the initialization allows to link those parameters to the layer, without setting them as trainable.  

`F.linear(x, weight=self.mu_weight, bias=self.mu_bias)` allows to elegantly do the linear computation.  

`torch.ger(a, b)` allows to elegantly create a 2D tensor with two 1D tensors.  

`tensor.uniform_(a, b)` allows to fill a tensor with an uniform distribution bounded by a and b.  

`tensor.fill_(x)` allows to fill a tensor with x.  

Adding the `data` in the lines such as `self.sigma_bias.data.fill_(self.sigma)` allows to not have in place modification errors.  

In the forward loop, there is the possibility to either manually sample the noise by setting `sample_noise` to `False`.  

```

if sample_noise:

    self.sample_noise()

```

When the `self.training` is set to `False`, it is possible to do a forward pass without the noise:  

```

if not self.training:

    return F.linear(x, weight=self.mu_weight, bias=self.mu_bias)

```

In the case of the Independent version, be careful to not input a tuple into the `torch.FloatTensor(features)`, otherwise it will create a tensor with those values. Instead, it is possible to unpack them `torch.FloatTensor(*features)`.  

## Installation

Direct Installation from github using pip by running this command:

```shell

pip install git+https://github.com/thomashirtz/noisy-networks#egg=noisynetworks

```

## Original Paper

```BibTeX

@paper{fortunato2019noisy,

      title={Noisy Networks for Exploration}, 

      author={Meire Fortunato and Mohammad Gheshlaghi Azar and Bilal Piot and Jacob Menick and Ian Osband and Alex Graves 

              and Vlad Mnih and Remi Munos and Demis Hassabis and Olivier Pietquin and Charles Blundell and Shane Legg},

      year={2019},

      eprint={1706.10295},

      archivePrefix={arXiv}

}

```