{"id":15065055,"url":"https://github.com/garethjns/reinforcement-learning-keras","last_synced_at":"2025-04-10T13:09:59.102Z","repository":{"id":39727738,"uuid":"264685776","full_name":"garethjns/reinforcement-learning-keras","owner":"garethjns","description":"Reinforcement learning algorithms implemented in Keras (tensorflow==2.3) and sklearn","archived":false,"fork":false,"pushed_at":"2023-03-25T00:57:07.000Z","size":36695,"stargazers_count":6,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T11:56:49.380Z","etag":null,"topics":["dqn","keras","policy-gradient","q-learning","reinforcement-learning","tensorflow2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/garethjns.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-05-17T14:29:46.000Z","updated_at":"2022-03-25T19:00:59.000Z","dependencies_parsed_at":"2023-12-15T22:00:34.401Z","dependency_job_id":"ffe9d50a-78da-40d6-ba99-cfb1a553f31d","html_url":"https://github.com/garethjns/reinforcement-learning-keras","commit_stats":{"total_commits":89,"total_committers":3,"mean_commits":"29.666666666666668","dds":0.4831460674157303,"last_synced_commit":"8fd4b334400ff6061d9c67c6df441cf0812c9748"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2Freinforcement-learning-keras","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2Freinforcement-learning-keras/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2Freinforcement-learning-keras/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2Freinforcement-learning-keras/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/garethjns","download_url":"https://codeload.github.com/garethjns/reinforcement-learning-keras/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247846835,"owners_count":21006090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dqn","keras","policy-gradient","q-learning","reinforcement-learning","tensorflow2"],"created_at":"2024-09-25T00:30:12.275Z","updated_at":"2025-04-10T13:09:59.082Z","avatar_url":"https://github.com/garethjns.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reinforcement learning in Keras\n![Tests](https://github.com/garethjns/reinforcement-learning-keras/workflows/Tests/badge.svg?branch=master) [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=garethjns_reinforcement-learning-keras\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=garethjns_reinforcement-learning-keras)\n\nThis repo aims to implement various reinforcement learning agents using Keras (tf==2.2.0) and sklearn, for use with OpenAI Gym environments.\n \n![Episode play example](images/DDQN_doom_corridor_example_at_8k.gif)\n \n# Planned agents\n- Methods\n  - Off-policy\n      - [Linear Q learning](#linearq)\n        - [x] [Mountain car](#lq_mountaincar)\n        - [x] [CartPole](#lq_cart-pole)\n      - [Deep Q learning](#dqn)\n        - [x] [Mountain car](#dqn_mountaincar)\n        - [x] [CartPole](##dqn_cart-pole)\n        - [x] [Pong](#pong)\n        - [x] [Vizdoom](#vizdoom)\n        - [x] [GFootball](#gfootball)\n        - [Model extensions](#extensions)\n          - [x] Replay buffer\n          - [ ] Unrolled Bellman\n          - [x] Dueling architecture\n          - [ ] Multiple environments\n          - [x] Double DQN\n          - [ ] Noisy network\n    - [ ] Policy gradient methods    \n        - [x] [REINFORCE](#reinforce)\n            - [ ] Mountain car\n            - [x] CartPole\n            - [ ] Pong\n        - [ ] Actor-critic\n            - [ ] Mountain car\n            - [x] CartPole\n            - [ ] Pong\n            \n\n# General references\n - [Deep reinforcement learning hands-on, 2nd edition](https://www.amazon.co.uk/Deep-Reinforcement-Learning-Hands-optimization/dp/1838826998/), Maxim Lapan\n - [The Lazy Programmers'](https://lazyprogrammer.me/) courses: \n   - [Artificial Intelligence: Reinforcement learning in Python](https://www.udemy.com/course/artificial-intelligence-reinforcement-learning-in-python/learn/)\n   - [Advanced AI: Deep reinforcement learning in Python](https://www.udemy.com/course/deep-reinforcement-learning-in-python/learn/)\n   - [Cutting-edge AI: Deep reinforcement learning in Python](https://www.udemy.com/course/cutting-edge-artificial-intelligence/learn/)\n - Lilian Weng's overviews of reinforcement learning. I try and use the same terminology as used in these posts.\n   - [A (Long) Peek into Reinforcement Learning](https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html)\n   - [Policy Gradient Algorithms](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html)\n - Multiple Github repos and Medium posts on individual techniques - these are cited in context.\n   \n# Set-up\n````bash\ngit clone \ncd reinforcement-learning-keras\npip install -r requirements.txt\n````\n\n# Implemented algorithms and environment examples\n\n## Deep Q learner \u003ca name=\"dqn\"\u003e\u003c/a\u003e\n### Pong \u003ca name=\"pong\"\u003e\u003c/a\u003e\nPong-NoFrameSkip-v4 with various wrappers.\n\n![Episode play example](images/DQNAgentPong.gif) ![Convergence](images/DQNAgentPong.png)  \n\nModel:  \nState -\u003e action model -\u003e [value for action 1, value for action 2] \n\nA [deep Q learning](https://www.nature.com/articles/nature14236) agent that uses small neural network to approximate Q(s, a). It includes a replay buffer that allows for batched training updates, this is important for 2 reasons:\n - As this method is off-policy (the action is selected as argmax(action values)), it can train on data collected during previous episodes. This reduces correlation in the training data.\n - This is important for performance, especially when using a GPU. Calling multiple predict/train operations on single rows inside a loop is very inefficient. \n\nThis agent uses two copies of its model:\n - One to predict the value of the next action, which us updated every episode step (with a batch sampled from the replay buffer)\n - One to predict value of the actions in the current and next state for calculating the discounted reward. This model is updated with the weights from the first model at the end of each episode.\n\n#### Run example\n````python\nfrom tf2_vgpu import VirtualGPU\nfrom rlk.agents.q_learning.deep_q_agent import DeepQAgent\nfrom rlk.environments.atari.pong.pong_config import PongConfig\n\nVirtualGPU(4096) \nagent = DeepQAgent(**PongConfig('dqn').build())\nagent.train(verbose=True, render=True, max_episode_steps=10000)\n````\n\n\n### Cart-pole \u003ca name=\"dqn_cart-pole\"\u003e\u003c/a\u003e\nUsing cart-pole-v0 with step limit increased from 200 to 500.\n\n![Episode play example]images/DQNAgent.gif) ![Convergence]images/DQNAgent.png)  \n\n#### Run example\n````python\nfrom tf2_vgpu import VirtualGPU\nfrom rlk.agents.q_learning.deep_q_agent import DeepQAgent\nfrom rlk.environments.cart_pole.cart_pole_config import CartPoleConfig\n\nVirtualGPU(256) \nagent = DeepQAgent(**CartPoleConfig('dqn').build())\nagent.train(verbose=True, render=True)\n````\n\n### MountainCar (not well tuned) \u003ca name=\"dqn_mountaincar\"\u003e\u003c/a\u003e\n![Episode play example](images/DeepQAgentMC.gif) ![Convergence](images/DeepQAgentMC.png)  \n\n#### Run example\n````python\nfrom tf2_vgpu import VirtualGPU\nfrom rlk.agents.q_learning.deep_q_agent import DeepQAgent\nfrom rlk.environments.mountain_car.mountain_car_config import MountainCarConfig\n\nVirtualGPU(256)\nagent = DeepQAgent(**MountainCarConfig('dqn').build())\nagent.train(verbose=True, render=True, max_episode_steps=1500)\n````\n\n### Extensions \u003ca name=\"extensions\"\u003e\u003c/a\u003e\n#### Dueling DQN\n![Episode play example]images/DuelingDQNAgent.gif) ![Convergence]images/DuelingDQNAgent.png)  \n\nThe [dueling](https://arxiv.org/abs/1511.06581) version is exactly the same as the DQN, expect with slightly different model architecture. The second to last layer is split into two layers with the units=1 and units=n_actions. The idea is that the model might learn V(s) and action advantages (A(s)) separately, which can speed up convergence.  \n\nThe output of the network is still action values, however preceding layers are not fully connected; the values are now V(s) + A(s) and a subsequent Keras lambda layer is used to calculate the action advantages.\n \n#### Run example\n````python\nfrom tf2_vgpu import VirtualGPU\nfrom rlk.agents.q_learning.deep_q_agent import DeepQAgent\nfrom rlk.environments.cart_pole.cart_pole_config import CartPoleConfig\n\nVirtualGPU(256) \nagent = DeepQAgent(**CartPoleConfig('dueling_dqn').build())\nagent.train(verbose=True, render=True)\n````\n\n## Linear Q learner \u003ca name=\"linearq\"\u003e\u003c/a\u003e\n### Mountain car \u003ca name=\"lq_mountaincar\"\u003e\u003c/a\u003e\n![Episode play example](images/LinearQAgentMC.gif) ![Convergence](images/LinearQAgentMC.png)  \n\nModel:  \nState -\u003e model for action 1 -\u003e value for action 1    \nState -\u003e model for action 2 -\u003e value for action 2\n\nThis agent is based on [The Lazy Programmers](https://lazyprogrammer.me/) 2nd reinforcement learning course [implementation](https://github.com/lazyprogrammer/machine_learning_examples/tree/master/rl2/cartpole). It uses a separate SGDRegressor models for each action to estimate Q(a|s). Each step, the model for the selected action is updated using .partial_fit.  Action selection is off-policy and uses epsilon greedy; the selected either the argmax of action values, or a random action, depending on the current value of epsilon.\n\nEnvironment observations are preprocessed in an sklearn pipeline that clips, scales, and creates features using RBFSampler.\n\n\n````python\nfrom rlk.agents.q_learning.linear_q_agent import LinearQAgent\nfrom rlk.environments.mountain_car.mountain_car_config import MountainCarConfig\n\nagent = LinearQAgent(**MountainCarConfig('linear_q').build())\nagent.train(verbose=True, render=True, max_episode_steps=1500)\n````\n\n### CartPole \u003ca name=\"lq_cart-pole\"\u003e\u003c/a\u003e\n![Episode play example](images/LinearQAgent.gif) ![Convergence](images/LinearQAgent.png)  \n\n#### Run example\n````python\nfrom rlk.agents.q_learning.linear_q_agent import LinearQAgent\nfrom rlk.environments.cart_pole.cart_pole_config import CartPoleConfig \n\nagent = LinearQAgent(**CartPoleConfig('linear_q').build())\nagent.train(verbose=True, render=True)\n````\n\n \n## REINFORCE (policy gradient) \u003ca name=\"reinforce\"\u003e\u003c/a\u003e\n### CartPole\n\n![Episode play example]images/REINFORCEAgent.gif) ![Convergence](images/REINFORCEAgent.png)  \n\nModel:  \nState -\u003e model -\u003e [probability of action 1, probability of action 2]  \nRefs:  \nhttps://github.com/Alexander-H-Liu/Policy-Gradient-and-Actor-Critic-Keras\n\n[Policy gradient](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf) models move the action selection policy into the model, rather than using argmax(action values). Model outputs are action probabilities rather than values (π(a|s), where π is the policy), making these methods inherently stochastic and removing the need for epsilon greedy action selection. \n\nThis agent uses a small neural network to predict action probabilities given a state. Updates are done in a Monte-Carlo fashion - ie. using all steps from a single episode. This removes the need for a complex replay buffer (list.append() does the job). However as the method is on-policy it requires data from the current policy for training. This means training data can't be collected across episodes (assuming policy is updated at the end of each). This means the training data in each batch (episode) is highly correlated, which slows convergence.\n\nThis model doesn't use any scaling or clipping for environment pre-processing. For some reason, using the same pre-processing as with the DQN models prevents it from converging. The cart-pole environment can potentially return really huge values when sampling from the observation space, but these are rarely seen during training. It seems to be fine to pretend they don't exist, rather than scaling inputs based environment samples, as done with in the other methods.\n\n````python\nfrom rlk.agents.policy_gradient.reinforce_agent import ReinforceAgent\nfrom tf2_vgpu import VirtualGPU\nfrom rlk.environments.cart_pole.cart_pole_config import CartPoleConfig\n\nVirtualGPU(256)\nagent = ReinforceAgent(**CartPoleConfig('reinforce').build())\nagent.train(verbose=True, render=True)\n````\n\n## Doom \u003ca name=\"vizdoom\"\u003e\u003c/a\u003e\n\n### Set up\nInstall these two packages:\n  - [ViZDoom](https://github.com/mwydmuch/ViZDoom)\n  - [ViZDoomGym](https://github.com/shakenes/vizdoomgym) \n\nAdditionally, to save monitor wrapper output, install the following packages:\n````BASH\nsudo apt install libcanberra-gtk-module libcanberra-gtk3-module\n````\n\n### VizdoomBasic-v0  \u003ca name=\"vizdoom\"\u003e\u003c/a\u003e\n#### DQN\n![Episode play example](images/DQNAgentDoom.gif) ![Convergence](images/dqn_VizdoomBasic-v0.png)  \n\n\n````python\nfrom tf2_vgpu import VirtualGPU\nfrom rlk.agents.q_learning.deep_q_agent import DeepQAgent\nfrom rlk.environments.doom.vizdoom_basic_config import VizDoomBasicConfig\n\nVirtualGPU(256)\nagent = DeepQAgent(**VizDoomBasicConfig(agent_type='dqn', mode='stack').build())\nagent.train(n_episodes=1000, max_episode_steps=10000, verbose=True, render=True)\n````\n\n### VizDoomCorridor-v0\n#### Double dueling DQN\n![Episode play example](images/DDQN_doom_corridor_example_at_8k.gif) ![Convergence](images/DDDQN_doom_corridor_at_10k.png)  \n\nThe DQNs struggle to solve this environment on their own. See scripts and [readme](scripts/doom/README.md) in [scripts/doom/](scripts/doom) for an example training with additional experience collection with (scripted) bots.\n\n\n### GFootball \u003ca name=\"gfootball\"\u003e\u003c/a\u003e\n\nWork in progress. Involves pre-training the agent on historical data, and sampling experience from (policy) bots. \n\nSee notes in [scripts/gfootball/readme.md](scripts/gfootball/README.md) \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarethjns%2Freinforcement-learning-keras","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgarethjns%2Freinforcement-learning-keras","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarethjns%2Freinforcement-learning-keras/lists"}