https://github.com/i-m-iron-man/growing_neural_nets_for_rl
Exploring constructive or growing neural networks for topology optimization in a reinforcement learning task
https://github.com/i-m-iron-man/growing_neural_nets_for_rl
Last synced: 7 months ago
JSON representation
Exploring constructive or growing neural networks for topology optimization in a reinforcement learning task
- Host: GitHub
- URL: https://github.com/i-m-iron-man/growing_neural_nets_for_rl
- Owner: i-m-iron-man
- Created: 2022-04-22T21:23:05.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-04-24T22:03:26.000Z (over 3 years ago)
- Last Synced: 2025-01-22T16:15:31.872Z (9 months ago)
- Language: C++
- Homepage:
- Size: 252 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Growing_Neural_Nets_for_RL
The project is about exploring constructive or growing neural networks for topology optimization in a reinforcement learning task.
## Step 1) Implementation of Neural Networks (NN) in C++
A simple and intiutive implementation of NN is needed so that their topology can be augmented later while learning and thier complete functionality is exposed.
Net.cpp & Net.h contain implementation of a NN, where one can create an NN as a object having member functions such as adding neurons, connecting neurons forward pass etc. It relies on:
1. Class Neuron (defined in `Neuron.h` & `Neuron.cpp`) to represent a single neuron and it's functionality like activating it's input to output.
2. Class Bond (defined in `Bond.h` & `Bond.cpp`) to represent a single bond or connection, connecticting 2 neurons and it's functionality like scaling the input signal by a scalar.
### status: Done
## Step 2) Implementation of an environment
An environment class in C++, similar to an OpenAI Gym environment is needed so that our agent can operate in it.
The opensource and fast physics simulation library [MuJoCo](https://mujoco.org/) is used for the task.
`Env.h` and `Env.cpp` contain the implementation of the ["Half-Cheetah"](https://gym.openai.com/envs/HalfCheetah-v2/) environment.
The XML file for the environment is imported from the `model` sub-folder.
### status: Done
## Step 3) Implementation of a Hebbian-Learning Algorithm.
Hebbian learning is needed here for two main reasons:
1. If the architecture of network changes mid-execution during a back-prop algorithm then the learning may be adversly-effected due to catastrophic forgetting.
2. In future, growth may occur such that it leads to formation of cyclic connections making the NN a general RNN. Again, Since the toplology of the RNN will keep augmenting standard RNN-backprop algos like n-BPPT, RTRL may not be effective.
A recent awesome method introduces in the work [Meta-Learning through Hebbian Plasticity in Random Networks](https://github.com/enajx/HebbianMetaLearning.git) is used for this purpose. `Growing_Machines.cpp` & `Growing_Machines.h` contain a C++ implementation for the approach. Also, multi-threading is used to speed up the training process. Class Net, Class Neuron and Class Bond are updated accordingly.
### status: Tuning of hyper-parameters remaining
## Step 4) Implementation of emperical Growing Rules.
### status: In Progress
## Inspiration
1. [Weight Agnostic Neural Networks](https://arxiv.org/abs/1906.04358)
2. [A Constructive Neural Network for Evolving a Machine Controller in Real-Time](https://link.springer.com/chapter/10.1007/978-3-642-04512-7_12)