https://github.com/anjum48/rl-examples
Examples of published reinforcement learning algorithms in recent literature implemented in TensorFlow
https://github.com/anjum48/rl-examples
artificial-intelligence openai-gym python reinforcement-learning tensorflow
Last synced: 9 months ago
JSON representation
Examples of published reinforcement learning algorithms in recent literature implemented in TensorFlow
- Host: GitHub
- URL: https://github.com/anjum48/rl-examples
- Owner: Anjum48
- License: mit
- Created: 2017-11-14T16:47:40.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2020-08-03T09:42:11.000Z (almost 6 years ago)
- Last Synced: 2025-10-06T20:01:53.123Z (9 months ago)
- Topics: artificial-intelligence, openai-gym, python, reinforcement-learning, tensorflow
- Language: Python
- Homepage:
- Size: 17 MB
- Stars: 103
- Watchers: 7
- Forks: 26
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# rl-examples
Examples of published reinforcement learning algorithms in recent
literature implemented in TensorFlow.
Most of my research is in the continuous domain, and I haven't spent much
time testing these in discrete domains such as Atari etc.
*BipedalWalker-v2 solved using DPPO with a LSTM layer. CarRacing-v0 solved using PPO with a joined actor-critic network*
## Algorithms Implemented
Thanks to DeepMind and OpenAI for making their research openly available.
Big thanks also to the TensorFlow community.
| Algorithm | Paper |
| --------- | ------------------------------------------------------- |
| DPPG | [Continuous control with deep reinforcement learning](https://arxiv.org/abs/1509.02971) |
| A3C | [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783) |
| PPO | [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) |
| DPPO | [Emergence of Locomotion Behaviours in Rich Environments](https://arxiv.org/abs/1707.02286) |
| GAE | [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438) |
- GAE was used in all algorithms except for DPPG
- Where possible, I've added an LSTM layer to the policy and value functions.
This sometimes achieved higher scores in some environments, but can have stability issues
- In some environments, having a joint network for the actor & critic performs better (i.e. where CNNs are used).
These scripts are suffixed, e.g. `ppo_joined.py`
## Training
All the Python scripts are written as standalone scripts (but share some common functions in `utils.py`).
Just run them directly in your IDE. Or in a terminal using the `-m` flag:
```
rl-examples$ python3 -m ppo.ppo_joined
```
The models and TensorBoard summaries are saved in the same directory as the script.
DPPO has a helper script to set off the worker threads:
```
rl-examples$ sh dppo/start_dppo.sh
```
## Requirements
- Python 3.6+
- OpenAI Gym 0.10.3+
- TensorFlow 1.11
- Numpy 1.13+
DPPO was tested on a 16 core machine using CPU only, so the helper
script will need to be updated for your particular setup.
For my setup, there was usually no speed advantage training BipedalWalker on the
CPU vs GPU (GTX 1080), but CarRacing did get a performance boost due to the usage of CNN layers
## Issues/Todo's
- Work needed to find the correct parameters for PPO in discrete action spaces for Atari
- The LSTM batching in A3C is incorrect. Need to fix this (see `ppo_lstm.py` for the correct implementation)
- Distributed Proximal Policy Optimisation with the LSTM (`dppo_lstm.py`) is sometimes a bit unstable,
but does work at low learning rates