{"id":16841797,"url":"https://github.com/redtachyon/rl-pong","last_synced_at":"2025-07-02T13:36:44.246Z","repository":{"id":52300167,"uuid":"225735130","full_name":"RedTachyon/rl-pong","owner":"RedTachyon","description":"Implementation of PPO for playing the game of Pong, originally used in the Reinforcement Learning class at Aalto University","archived":false,"fork":false,"pushed_at":"2021-04-30T21:45:35.000Z","size":642,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-24T12:13:20.104Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedTachyon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-03T23:12:09.000Z","updated_at":"2020-04-25T18:14:32.000Z","dependencies_parsed_at":"2022-09-12T17:42:01.364Z","dependency_job_id":null,"html_url":"https://github.com/RedTachyon/rl-pong","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Frl-pong","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Frl-pong/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Frl-pong/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTachyon%2Frl-pong/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedTachyon","download_url":"https://codeload.github.com/RedTachyon/rl-pong/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244161415,"owners_count":20408311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:43:01.965Z","updated_at":"2025-03-18T05:23:56.791Z","avatar_url":"https://github.com/RedTachyon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pong AI via multi-agent training\n\nThe environment code was written by the course instructors/TAs at Aalto University. The training procedure (Collector, PPOTrainer), Tensorboard integration and most of the general classes (BaseModel, Agent) have been written by Ariel (@RedTachyon), and some of the models and the evaluation API integration have been written by Elisabeth (@LizTheElephant).\n\nNote from Ariel: this is put up primarily as a showcase repo right now. I'm currently developing a research project in a similar framework, albeit this code has been significantly simplified for the purposes of the course. Nevertheless, since most of the code I'm the most proud of is sitting on some company servers, I wanted to show this as an example of my coding style.\n\n## Quickstart for understanding the code\n\n### MultiAgentEnv\n\nThis is the general structure template for a multi-agent environment. Everything is handled\nwith dictionaries. For example, with agents named [\"Agent0\", \"Agent1\"], an action where \neach of them takes the action `0` is ```{\"Agent0\": 0, \"Agent1\": 0}```. Observations, rewards etc are\ntreated analogously\n\n### BaseModel (`models.py`)\n\nThe basic abstraction used in this code. It's just a PyTorch model equipped with certain utilities \nfor handling recurrent policies (via the hidden `state`). Subclasses should overwrite the \n`.forward()` method (standard PyTorch module method) and `.get_initial_state()` - in case of \nnonrecurrent policies, just return an empty tuple. For recurrent policies, return a tuple with the state vectors\nof the appropriate size (e.g. for LSTMs it's h and c)\n\n### Agent (`agents.py`)\n\nA wrapper around the BaseModel that handles its interactions with the environment, and is able to \ncompute some things that are necessary for PPO. Should work with any model, but evaluate_actions might\nneed some more work for recurrent policies for speed. Also, might need some attention for convolutional models,\nbut might also not if the model is appropriately implemented.\n\n### Collector (`rollout.py`)\n\nThis class is responsible for gathering data and nothing else. It holds an environment and\na number of agents that will act in that environment. It also holds a `Memory` object that is basically\na fancy dictionary for storing and basic processing of the data. In `Memory`, note the `.get_torch_data()`\nmethod - it gathers all the stored experience and converts it into a dictionary of tensors, ready for usage\n\n### Trainer (`trainers.py`)\n\nThis class performs the actual training. Currently only PPO is included so the \ngeneral format is not specified, but in this case, the main methods are `.train_on_data()` which\nperforms a single PPO update using the passed batch of data, and `.train()` which executes a full training\nloop, including data collection using a `Collector` and a PPO update with each batch.\n\nCould at some point be refactored to separate out the training logic from the PPO logic.\n\n\n### General remarks\n\nAll hyperparameters etc. are handled by configs. These should all be easily serializable so that \nthey can be saved (via pickle) along with the TensorBoard logs\n\n### Tensorboard\n\nThe training is automatically logged into TensorBoard as long as `tensorboard_name` is passed in the config.\nTo view the training graphs, run `tensorboard --logdir ~/tb_logs --bind_all` and see them at `localhost:6006`\nin your browser. (alternatively, skip --bind_all and go to the url displayed in the terminal)\n\n### Compatibility\n\nThe Pong codes represents the actions and observations as either raw values (single-agent) of tuples of values (multi-agent).\nTo ensure compatibility, there's two convert methods in `utils.py` converting between representations.\nRight now, if there aren't any bugs, it's enough to include `\"tuple_mode\": True` in the trainer config\nand everything should work. The Pong representation is only used to directly interact \nwith the environment - otherwise, everything is kept in the dictionary form as described at the beginning of this document.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredtachyon%2Frl-pong","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredtachyon%2Frl-pong","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredtachyon%2Frl-pong/lists"}