{"id":16403903,"url":"https://github.com/marcometer/episodic-transformer-memory-ppo","last_synced_at":"2025-08-09T09:20:49.427Z","repository":{"id":39083484,"uuid":"488545449","full_name":"MarcoMeter/episodic-transformer-memory-ppo","owner":"MarcoMeter","description":"Clean baseline implementation of PPO using an episodic TransformerXL memory","archived":false,"fork":false,"pushed_at":"2024-06-18T13:35:45.000Z","size":25017,"stargazers_count":172,"open_issues_count":1,"forks_count":22,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-11T02:11:32.774Z","etag":null,"topics":["actor-critic","deep-reinforcement-learning","episodic-memory","gated-transformer-xl","gtrxl","memory-gym","on-policy","policy-gradient","pomdp","ppo","proximal-policy-optimization","pytorch","transformer","transformer-xl","trxl"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarcoMeter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-04T10:32:14.000Z","updated_at":"2025-04-08T21:48:24.000Z","dependencies_parsed_at":"2024-10-11T05:50:49.266Z","dependency_job_id":"c700a1da-60c5-4917-ba18-67fb6ee20928","html_url":"https://github.com/MarcoMeter/episodic-transformer-memory-ppo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarcoMeter%2Fepisodic-transformer-memory-ppo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarcoMeter%2Fepisodic-transformer-memory-ppo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarcoMeter%2Fepisodic-transformer-memory-ppo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarcoMeter%2Fepisodic-transformer-memory-ppo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarcoMeter","download_url":"https://codeload.github.com/MarcoMeter/episodic-transformer-memory-ppo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248328161,"owners_count":21085261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actor-critic","deep-reinforcement-learning","episodic-memory","gated-transformer-xl","gtrxl","memory-gym","on-policy","policy-gradient","pomdp","ppo","proximal-policy-optimization","pytorch","transformer","transformer-xl","trxl"],"created_at":"2024-10-11T05:50:42.864Z","updated_at":"2025-04-11T02:11:39.000Z","avatar_url":"https://github.com/MarcoMeter.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TransformerXL as Episodic Memory in Proximal Policy Optimization\n\nThis repository features a PyTorch based implementation of PPO using TransformerXL (TrXL). Its intention is to provide a clean baseline/reference implementation on how to successfully employ memory-based agents using Transformers and PPO.\n\n# Features\n\n- Episodic Transformer Memory\n  - TransformerXL (TrXL)\n  - Gated TransformerXL (GTrXL)\n- Environments\n  - Proof-of-concept Memory Task (PocMemoryEnv)\n  - CartPole\n    - Masked velocity\n  - Minigrid Memory\n    - Visual Observation Space 3x84x84\n    - Egocentric Agent View Size 3x3 (default 7x7)\n    - Action Space: forward, rotate left, rotate right\n  - [MemoryGym](https://github.com/MarcoMeter/drl-memory-gym)\n    - Mortar Mayhem\n    - Mystery Path\n    - Searing Spotlights (WIP)\n- Tensorboard\n- Enjoy (watch a trained agent play)\n\n# Citing this work\n\n```bibtex\n@article{pleines2023trxlppo,\n  title = {TransformerXL as Episodic Memory in Proximal Policy Optimization},\n  author = {Pleines, Marco and Pallasch, Matthias and Zimmer, Frank and Preuss, Mike},\n  journal= {Github Repository},\n  year = {2023},\n  url = {https://github.com/MarcoMeter/episodic-transformer-memory-ppo}\n}\n```\n\n# Contents\n\n- [Installation](#installation)\n- [Train a model](#train-a-model)\n- [Enjoy a model](#enjoy-a-model)\n- [Episodic Transformer Memory Concept](#episodic-transformer-memory-concept)\n- [Hyperparameters](#hyperparameters)\n      - [Episodic Transformer Memory](#episodic-transformer-memory)\n      - [General](#general)\n      - [Schedules](#schedules)\n- [Add Environment](#add-environment)\n- [Tensorboard](#tensorboard)\n- [Results](#results)\n\n# Installation\n\nInstall [PyTorch](https://pytorch.org/get-started/locally/) 1.12.1 depending on your platform. We recommend the usage of [Anaconda](https://www.anaconda.com/).\n\nCreate Anaconda environment:\n```bash\nconda create -n transformer-ppo python=3.7 --yes\nconda activate transformer-ppo\n```\n\nCPU:\n```bash\nconda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cpuonly -c pytorch\n```\n\nCUDA:\n```bash\nconda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge\n```\n\nInstall the remaining requirements and you are good to go:\n\n```bash\npip install -r requirements.txt\n```\n\n# Train a model\n\nThe training is launched via `train.py`. `--config` specifies the path to the yaml config file featuring hyperparameters. The `--run-id` is used to distinguish training runs. After training, the trained model will be saved to `./models/$run-id$.nn`.\n\n```bash\npython train.py --config configs/minigrid.yaml --run-id=my-trxl-training\n```\n\n# Enjoy a model\n\nTo watch an agent exploit its trained model, execute `enjoy.py`. Some pre-trained models can be found in: `./models/`. The to-be-enjoyed model is specified using the `--model` flag.\n\n```bash\npython enjoy.py --model=models/mortar_mayhem_grid_trxl.nn\n```\n\n# Episodic Transformer Memory Concept\n\n![transformer-xl-model](./docs/assets/trxl_architecture.png)\n\n# Hyperparameters\n\n#### Episodic Transformer Memory\n\n\u003ctable\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth\u003eHyperparameter\u003c/th\u003e\n      \u003cth\u003eDescription\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd\u003enum_blocks\u003c/td\u003e\n      \u003ctd\u003eNumber of transformer blocks\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eembed_dim\u003c/td\u003e\n      \u003ctd\u003eEmbedding size of every layer inside a transformer block\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003enum_heads\u003c/td\u003e\n      \u003ctd\u003eNumber of heads used in the transformer's multi-head attention mechanism\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003ememory_length\u003c/td\u003e\n      \u003ctd\u003eLength of the sliding episodic memory window\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003epositional_encoding\u003c/td\u003e\n      \u003ctd\u003eRelative and learned positional encodings can be used\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003elayer_norm\u003c/td\u003e\n      \u003ctd\u003eWhether to apply layer normalization before or after every transformer component. Pre layer normalization refers to the identity map re-ordering.\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003egtrxl\u003c/td\u003e\n      \u003ctd\u003eWhether to use Gated TransformerXL\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003egtrxl_bias\u003c/td\u003e\n      \u003ctd\u003eInitial value for GTrXL's bias weight\u003c/td\u003e\n    \u003c/tr\u003e    \n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n#### General\n\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd\u003egamma\u003c/td\u003e\n      \u003ctd\u003eDiscount factor\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003elamda\u003c/td\u003e\n      \u003ctd\u003eRegularization parameter used when calculating the Generalized Advantage Estimation (GAE)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eupdates\u003c/td\u003e\n      \u003ctd\u003eNumber of cycles that the entire PPO algorithm is being executed\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003en_workers\u003c/td\u003e\n      \u003ctd\u003eNumber of environments that are used to sample training data\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eworker_steps\u003c/td\u003e\n      \u003ctd\u003eNumber of steps an agent samples data in each environment (batch_size = n_workers * worker_steps)\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eepochs\u003c/td\u003e\n      \u003ctd\u003eNumber of times that the whole batch of data is used for optimization using PPO\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003en_mini_batch\u003c/td\u003e\n      \u003ctd\u003eNumber of mini batches that are trained throughout one epoch\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003evalue_loss_coefficient\u003c/td\u003e\n      \u003ctd\u003eMultiplier of the value function loss to constrain it\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003ehidden_layer_size\u003c/td\u003e\n      \u003ctd\u003eNumber of hidden units in each linear hidden layer\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003emax_grad_norm\u003c/td\u003e\n      \u003ctd\u003eGradients are clipped by the specified max norm\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n#### Schedules\n\nThese schedules can be used to polynomially decay the learning rate, the entropy bonus coefficient and the clip range.\n\n\u003ctable\u003e\n    \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd\u003elearning_rate_schedule\u003c/td\u003e\n      \u003ctd\u003eThe learning rate used by the AdamW optimizer\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003ebeta_schedule\u003c/td\u003e\n      \u003ctd\u003eBeta is the entropy bonus coefficient that is used to encourage exploration\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eclip_range_schedule\u003c/td\u003e\n      \u003ctd\u003eStrength of clipping losses done by the PPO algorithm\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n# Add Environment\n\nFollow these steps to train another environment:\n\n1. Implement a wrapper of your desired environment. It needs the properties `observation_space`, `action_space` and `max_episode_steps`. The needed functions are `render()`, `reset()` and `step`.\n2. Extend the `create_env()` function in `utils.py` by adding another if-statement that queries the environment's \"type\"\n3. Adjust the \"type\" and \"name\" key inside the environment's yaml config\n\nNote that only environments with visual or vector observations are supported. Concerning the environment's action space, it can be either discrte or multi-discrete.\n\n# Tensorboard\n\nDuring training, tensorboard summaries are saved to `summaries/run-id/timestamp`.\n\nRun `tensorboad --logdir=summaries` to watch the training statistics in your browser using the URL [http://localhost:6006/](http://localhost:6006/).\n\n# Results\n\nEvery experiment is repeated on 5 random seeds. Each model checkpoint is evaluated on 50 unknown environment seeds, which are repeated 5 times. Hence, one data point aggregates 1250 (5x5x50) episodes. Rliable is used to retrieve the interquartile mean and the bootstrapped confidence interval. The training is conducted using the more sophisticated DRL framework [neroRL](https://github.com/MarcoMeter/neroRL). The clean GRU-PPO baseline can be found [here](https://github.com/MarcoMeter/recurrent-ppo-truncated-bptt).\n\n## Mystery Path Grid (Goal \u0026 Origin Hidden)\n\n![mpg_off_results](./docs/assets/mpg_off.png)\n\nTrXL and GTrXL have identical performance. See [Issue #7](https://github.com/MarcoMeter/episodic-transformer-memory-ppo/issues/7).\n\n## Mortar Mayhem Grid (10 commands)\n\n![mmg_10_results](./docs/assets/mmg_10.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcometer%2Fepisodic-transformer-memory-ppo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarcometer%2Fepisodic-transformer-memory-ppo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcometer%2Fepisodic-transformer-memory-ppo/lists"}