{"id":15930215,"url":"https://github.com/ianstenbit/art-cog","last_synced_at":"2025-04-03T14:18:05.184Z","repository":{"id":150821722,"uuid":"179603512","full_name":"ianstenbit/art-cog","owner":"ianstenbit","description":null,"archived":false,"fork":false,"pushed_at":"2019-04-05T01:42:38.000Z","size":79,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-09T03:14:02.396Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ianstenbit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-05T01:42:13.000Z","updated_at":"2019-04-06T18:25:39.000Z","dependencies_parsed_at":"2023-04-10T11:13:53.429Z","dependency_job_id":null,"html_url":"https://github.com/ianstenbit/art-cog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ianstenbit%2Fart-cog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ianstenbit%2Fart-cog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ianstenbit%2Fart-cog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ianstenbit%2Fart-cog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ianstenbit","download_url":"https://codeload.github.com/ianstenbit/art-cog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247014520,"owners_count":20869376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-07T00:40:28.968Z","updated_at":"2025-04-03T14:18:05.140Z","avatar_url":"https://github.com/ianstenbit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep Q-learning for Atari Games\nThis is an implementation in Keras and OpenAI Gym of the Deep Q-Learning\nalgorithm (often referred to as Deep Q-Network, or DQN) by Mnih et al.\non the well known Atari games.\n  \nRather than a pre-packaged tool to simply see the agent playing the game,\nthis is a model that needs to be trained and fine tuned by hand and has\nmore of an educational value.\nThis code tries to replicate the experimental setup described in\n[the original DeepMind paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf).\n\nA similar project on Deep Q-Learning applied to videogames can be found\non [this repo of mine](https://github.com/danielegrattarola/deep-q-snake).\n\n## Acknowledgments\nMake sure to cite the paper by Mnih et al. if you use this code for\nyour research:\n```\n@article{mnih2015human,\n    title={Human-level control through deep reinforcement learning},\n    author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others},\n    journal={Nature},\n    volume={518},\n    number={7540},\n    pages={529--533},\n    year={2015},\n    publisher={Nature Research}\n}\n```\n\nand remember to give me a shoutout by linking to my\n[Github profile](https://github.com/danielegrattarola),\n[blog](https://danielegrattarola.github.io), or\n[Twitter](https://twitter.com/riceasphait).\n\nA big thank you to Carlo D'Eramo\n([@carloderamo](https://github.com/carloderamo)), who helped me\ndevelop this code and spent a good amount of time debugging in the\nbeginning.\n\n## Setup\nTo run the script you'll need the following dependencies:\n- [Keras](http://keras.io/#installation)\n- [OpenAI Gym](https://gym.openai.com/)  \n- [PIL](http://www.pythonware.com/products/pil/)\n- [h5py](http://packages.ubuntu.com/trusty/python-h5py)\n\nwhich should all be available through Pip.\n  \nNo additional setup is needed, so simply clone the repo:\n```sh\ngit clone https://gitlab.com/danielegrattarola/deep-q-atari.git\ncd deep-q-atari\n```  \n  \n## Usage\nA default training session can be run by typing:\n```sh\npython atari.py -t\n```  \nwhich will train the model with the same parameters as described in \n[this Nature article](http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html), \non the `MsPacmanDeterministic-v4` environment.\n  \nBy running:\n```sh\npython atari.py -h\n```  \nyou'll see the options list. The possible options are:\n`-t, train`: train the agent;  \n`-l, --load`: load the neural network weights from the given path;  \n`-v, --video`: show video output;  \n`-d, --debug`: run in debug mode (no output files);\n`--eval`: evaluate the agent;\n`-e, --environment`: name of the OpenAI Gym environment to use \n(default: MsPacman-v0);\n`--minibatch-size`: number of sample to train the DQN at each update;  \n`--replay-memory-size`: number of samples stored in the replay memory;  \n`--target-network-update-freq`: frequency (number of frames) with which\nthe target DQN is updated;\n`--avg-val-computation-freq`: frequency (number of DQN updates) with which the \naverage reward and Q value are computed;  \n`--discount-factor`: discount factor for the environment;  \n`--update-freq`: frequency (number of steps) with which to train the DQN;  \n`--learning-rate`: learning rate for the DQN;\n`--epsilon`: initial exploration rate for the agent;  \n`--min-epsilon`: final exploration rate for the agent;  \n`--epsilon-decrease`: rate at which to linearly decrease epsilon;  \n`--replay-start-size`: minimum number of transitions (with fully random policy) \nto store in the replay memory before starting training;  \n`--initial-random-actions`: number of random actions to be performed by the \nagent at the beginning of each episode;  \n`--dropout`: dropout rate for the DQN;  \n`--max-episodes`: maximum number of episodes that the agent can experience \nbefore quitting;  \n`--max-episode-length`: maximum number of steps in an episode;  \n`--max-frames-number`: maximum number of frames for a run;  \n`--test-freq`: frequency (number of episodes) with which to test the agent's \nperformance;  \n`--validation-frames`: number of frames to test the model like in table 3 of the\n paper  \n`--test-states`: number of states on which to compute the average Q value;  \n  \nThe possible environments on which the agent can be trained are all the \nenvironments in the Atari gym package.\nA typical usage of this script on an headless server (e.g. EC2 instance) would \nlook like this:\n```sh\npython atari.py -t -e BreakoutDeterministic-v4\n```\nIf you want to see the actual game being played by the agent, simply add the `-v`\nflag to the above command (note that this will obviously slow the collection of\nsamples.\n  \n## Output\nYou'll find some csv files in the output folder of the run (`output/runYYYMMDD-hhmmss`)\n which will contain raw data for the analysis of the agent's performance.\n  \nMore specifically, the following files will be produced as output:  \n1. **training_info.csv**: will contain the episode length and cumulative \n(non-clipped) reward of each training episode;    \n2. **evaluation_info.csv**: will contain the episode length and cumulative \n(non-clipped) reward of each evaluation episode;  \n3. **training_history.csv**: will contain the average loss and accuracy for each\n training step, as returned by the `fit` method of Keras;  \n4. **test_score_mean_q_info.csv**: will contain the average score and Q-value \n(computed over a number of held out random states defined by the `--test-states`\n flag) calculated at intervals of N DQN updates (where N is set by the\n `--avg-val-computation-freq` flag);  \n5. **log.txt**: a text file with various information about the parameters of the \nrun and the progress of the model;  \n6. **model_DQN.h5, model_DQN_target.h5**: files containing the weights of the \nDQN and target DQN (both files will be saved when the script quits or is killed \nwith `ctrl+c`). You can pass any of these files as argument with the `--load` \nflag to initialize a new DQN with these weights (Note: the DQN architecture must\nbe unchanged for this to work);","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fianstenbit%2Fart-cog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fianstenbit%2Fart-cog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fianstenbit%2Fart-cog/lists"}